Why trace and delay conditioning are sometimes (but not always) hippocampal dependent: A computational model

Abstract

A recurrent-network model provides a unified account of the hippocampal region in mediating the representation of temporal information in classical eyeblink conditioning. Much empirical research is consistent with a general conclusion that delay conditioning (in which the conditioned stimulus CS and unconditioned stimulus US overlap and co-terminate) is independent of the hippocampal system, while trace conditioning (in which the CS terminates before US onset) depends on the hippocampus. However, recent studies show that, under some circumstances, delay conditioning can be hippocampal-dependent and trace conditioning can be spared following hippocampal lesion. Here, we present an extension of our prior trial-level models of hippocampal function and stimulus representation that can explain these findings within a unified framework. Specifically, the current model includes adaptive recurrent collateral connections that aid in the representation of intra-trial temporal information. With this model, as in our prior models, we argue that the hippocampus is not specialized for conditioned response timing, but rather is a general-purpose system that learns to predict the next state of all stimuli given the current state of variables encoded by activity in recurrent collaterals. As such, the model correctly predicts that hippocampal involvement in classical conditioning should be critical not only when there is an intervening trace interval, but also when there is a long delay between CS onset and US onset. Our model simulates empirical data from many variants of classical conditioning, including delay and trace paradigms in which the length of the CS, the inter-stimulus interval, or the trace interval is varied. Finally, we discuss model limitations, future directions, and several novel empirical predictions of this temporal processing model of hippocampal function and learning.

1- Introduction

Classical conditioning, in which a cue (the conditioned stimulus or CS) is paired with a reflex-evoking unconditioned stimulus (US) until the CS comes to produce an anticipatory response (the conditioned response or CR) has proven a useful testbed for examining the psychological principles and neurobiological substrates of learning. Under many circumstances, delay conditioning, in which the CS and US overlap and co-terminate, is spared or even mildly facilitated following hippocampal damage (e.g., Berger, Rinaldi, Weisz, & Thompson, 1983; Gabrieli et al., 1995; Ito, Everitt, & Robbins, 2005; Ito, Robbins, McNaughton, & Everitt, 2006; Schmaltz & Theios, 1972); conversely, under many conditions, hippocampal lesion disrupts trace conditioning, in which CS offset occurs before US onset, producing a temporal gap known as the trace interval (e.g., Beylin et al., 1999; McGlinchey-Berroth, Carrillo, Gabrieli, Brawn, & Disterhoft, 1997; Solomon, Vander Schaaf, Thompson, & Weisz, 1986; Weisz, Solomon, & Thompson, 1980). This has led some researchers to assume that the hippocampus plays a more important role in trace than in delay conditioning (Beylin, et al., 1999; McGlinchey-Berroth, et al., 1997; Solomon, et al., 1986)—as we discuss below, this assumption ignores empirical findings on the role of the hippocampus in delay conditioning. Along the same lines, some studies of humans and non-human animals use trace conditioning as a canonical task to demonstrate evidence of hippocampal dysfunction in transgenic models, healthy aging, and pharmacological models (Brown, Comalli, De Biasi, & Woodruff-Pak, 2010; Disterhoft et al., 1999).

However, there are strong reasons to challenge this delay/trace dichotomy. First, it has long been known that the hippocampus shows learning-related changes during acquisition of the delay CR in intact animals and humans. These changes include the development of responses by some hippocampal pyramidal neurons that precede the behavioral eyeblink CR and mirror its form (Berger, Alger, & Thompson, 1976; Berger, et al., 1983; Berger & Thompson, 1978; Green & Arenos, 2007; Thompson et al., 1980). Initially, these responses occur in the US period, but increases in the CS period occur at about the time that behavioral CRs appear, and decline with continued training (for review, see Christian & Thompson, 2003). Similar hippocampal activity occurs in rabbits given trace conditioning (Weiss, Kronforst-Collins, & Disterhoft, 1996) but not in rabbits given unpaired CS/US trials (Solomon, et al., 1986).

Functional imaging studies in humans show a similar pattern of learning-related activity in the hippocampus during delay eyeblink conditioning to those observed in animals (Blaxton et al., 1996; Cheng, Disterhoft, Power, Ellis, & Desmond, 2008; Knight, Cheng, Smith, Stein, & Helmstetter, 2004 Stein & Helmsetter, 2004; Logan & Grafton, 1995; Schreurs & Alkon, 2001). Thus, these data suggest that – even if the hippocampus is not necessary for acquisition of a delay CR – it nevertheless normally plays a role. This challenges the simple view of delay conditioning as hippocampal-independent, and begs a more nuanced view of the difference between brain substrates that are sufficient to mediate a learned response, versus those that are normally involved.

Second, under many conditions, delay conditioning is spared or slightly enhanced following hippocampal lesion. Specifically, the ability of hippocampal lesioned animals to acquire a delay eyeblink CR depends on the length of the CS interval. Specifically, while hippocampal-lesioned rats can acquire an eyeblink CR when the delay between CS onset and US onset is short, they are impaired when the delay is lengthened (Beylin et al., 2001). Thus, short-delay conditioning is spared by hippocampal lesion, but long-delay conditioning is not. Further, disruption of the hippocampus, via electrical stimuli or pharmacological intervention, can retard acquisition of even a short-delay CR (Kaneko & Thompson, 1997; Sakamoto et al., 2005; Salafia, Chiaia, & Ramirez, 1979; Salafia, Romano, Tynan, & Host, 1977; Solomon & Gottfried, 1981; Solomon, Solomon, Schaaf, & Perry, 1983). Together, these results document that delay conditioning is not always spared following hippocampal lesion or disruption.

Third, although trace conditioning is often disrupted by hippocampal lesion, this is not always the case. For example, Thompson and colleagues (1980) speculated that the hippocampus might be involved in trace conditioning, to bridge the temporal gap between CS and US. Solomon and colleagues (Solomon, et al., 1986) presented an early study showing that dorsal hippocampal lesions disrupted trace eyeblink conditioning in rabbits, by decreasing the number of CRs. However, other studies followed that reported no trace conditioning impairment in hippocampal-lesioned animals (James, Hardiman, & Yeo, 1987; Port, Romano, Steinmetz, Mikhail, & Patterson, 1986).

Another factor affecting the hippocampal-dependence of trace conditioning may be differences in the trace interval used. In studies where the trace interval has been explicitly varied, a deficit in trace conditioning appears only for long trace intervals. Thus, for example, hippocampal-lesioned rabbits are impaired on eyeblink CR acquisition with a long (500 ms) but not a short (100 ms CS or 300 ms) trace interval (Moyer, Deyo, & Disterhoft, 1990). There may also be interactions between CS duration, and trace interval: Steinmetz and colleagues (Walker & Steinmetz, 2008) found that hippocampal-lesioned rats were impaired relative to controls on acquisition of an eyeblink CR when the CS duration was 50 ms and the trace interval was 500 ms, but not when the CS duration was 500 ms and the trace interval was 50 ms. In addition, Shors and colleagues (Beylin, et al., 2001) showed that – although hippocampal-lesioned rats were impaired at both trace and long-delay eyeblink conditioning – once the lesioned animals had acquired a long-delay CR, they could then learn and perform the trace CR. Together, these results document that, at least under some circumstances, subjects with hippocampal lesion can acquire a trace CR as well as matched controls.

In summary, while the idea that trace conditioning is hippocampal-dependent whereas delay conditioning is hippocampal-independent provides a useful rule of thumb, it is not sufficient to adequately address the full range of existing data. An additional complication involves the inter-stimulus interval (ISI). When the ISI is short, response systems (such as the brainstem and cerebellum for eyeblink conditioning) can successfully learn CS-US associations and produce a well-timed CR. When the ISI is longer, the hippocampus helps to bridge the temporal gap between CS and US, facilitating production of a well-timed CR. Thus, in the case of eyeblink conditioning, both short-delay and short-trace paradigms can be acquired without impairment by hippocampal-lesioned animals; however, both long-delay and long-trace paradigms are disrupted following hippocampal lesion.

Consistent with this view, although infant rats (with immature hippocampus) can acquire short-delay conditioning, they are impaired at both long-delay and trace conditioning, which emerge in parallel during later development (Barnet & Hunt, 2005; Ivkovich, Paczkowski, & Stanton, 2000). Thus, these data all suggest that in addition to the presence of a trace interval, the duration between CS onset and US arrival determines whether hippocampal mediation is required, as is the case in short- and long-delay conditioning

Interestingly, when Hoehler and Thompson (1980) first speculated that trace conditioning might be hippocampal dependent, they did so based on their studies of ISI manipulations in eyeblink conditioning, and on their findings that the hippocampus appeared to be involved in forming a temporal map of the learned behavioral response to be made, allowing for the CR to be accurately timed even when the ISI is beyond the timing parameters that are “optimal” for the basic associative substrate. Thus, for example, the “optimal” parameters for eyeblink conditioning (operationalized in terms of acquisition speed) may be a few hundred milliseconds and may reflect temporal processing mechanisms in other structures such as the cerebellum.

As discussed by Christian and Thompson (2003), optimal temporal parameters for learning in the cerebellum are 50–200 ms between CS and US presentation. Most trace conditioning studies on animals use a trace interval of 500 ms to induce hippocampal involvement in task learning. Optimal parameters for fear conditioning tend to be an order of magnitude longer (usually > 1 sec, see for example, Bevins & Ayres, 1995), and may reflect temporal processing mechanisms in the amygdala. But in fear conditioning, just as in eyeblink conditioning, hippocampal lesions affect trace conditioning as a function of the trace interval, so that hippocampal lesions impair expression of contextual fear conditioning with long but not short trace intervals (Chowdhury, Quinn, & Fanselow, 2005 2005; Pang et al., 2010). In other words, we argue that the hippocampus plays a similar role in both eyeblink and fear conditioning, but temporal differences in optimal ISI length for eyeblink and fear conditioning acquisitions are, respectively related to processing in the cerebellum and amygdala.

Extending this principle beyond classical conditioning, other brain systems mediate other behavioral responses, and each may have an operating window of temporal delays that can be spanned; in each case, these brain systems alone may be capable of mediating learning with sufficiently short delays, but as the delay is lengthened, hippocampal mediation becomes critical. This basic idea is consistent with a large number of theories of hippocampal-region function (e.g. Hoehler & Thompson, 1980; Rawlins, 1985; Wallenstein, Eichenbaum, & Hasselmo, 1998) and finds broad support from a range of preparations. For example, in delayed non-matching-to-place in an eight-arm radial maze, rats with hippocampal lesion can learn under short delays, but are impaired when the delay period is extended (Lee & Kesner, 2003). Similarly, in delayed non-match to sample (DNMS), primates with lesions limited to the hippocampus (sparing nearby medial temporal areas) can learn the non-matching task as well as controls, but show increasing impairments as the delay between sample and response is lengthened (Zola-Morgan & Squire, 1986). In each case, the important factor determining hippocampal dependence is not presence or absence of a stimulus-free gap, but rather the length of the interval across which information must be maintained before responding.

Again, similar findings have also been reported in human studies. For example, humans with bilateral hippocampal damage are impaired relative to healthy controls on temporal and spatial estimation at long, but not short, delays (Kesner & Hopkins, 2001). Patients with medial temporal lobe and hippocampal lesions also perform much better on relational memory tasks when the interval between learning and memory test is short than long. Specifically, Squire and colleagues (Jeneson, Mauldin, & Squire, 2010) found that patients with medial temporal lobe damage do not show impairment in performing relational learning tasks when the delay between learning and test is short (1s). Similarly, Ryan and Cohen (2004) have tested amnesic patients on a relational memory task using both short and long delay periods. They have found that patients are impaired only for the long-interval condition.

Thus, a conceptualization of eyeblink conditioning in which the length of the ISI, in addition to the presence of a trace interval, determines hippocampal dependence would appear to help integrate our understanding of the eyeblink conditioning literature with that of other preparations.

In this paper, we present an extension of our prior trial-level computational models of hippocampal function and stimulus representation. Our models assumed that the hippocampal region interacted with other brain systems, such as cortex and cerebellum, during associative learning, specifically by forming new stimulus representations that provided information about stimulus-stimulus and contextual regularities (Gluck & Myers, 1993; Myers & Gluck, 1994; Gluck & Myers, 2001; Moustafa et al., 2009). These models were correctly able to account for the effects of hippocampal lesion and disruption on various trial-level phenomena such as acquisition, discrimination, latent inhibition, and contextual shift effects. A later extension which modeled the effects of cholinergic manipulations by altering the hippocampal region learning rate was correctly able to address the effects of cholinergic agonists and antagonists on classical conditioning (Myers et al., 1996; Myers et al., 1998; Moustafa et al., 2010). However, these earlier models simulated trial-level information only, meaning that they could simulate whether a CR is given on a particular trial, but could not address within-trial events, such as the relative timing of CS and US onset. As such, these earlier trial-level models could not address the differences between delay and trace conditioning, nor the effects of manipulating the length of the ISI or trace interval. The need to address these aspects of the empirical data partially motivates the current work.

Our new model simulates performance in various delay and trace eyeblink conditioning data within a unified framework. Specifically, the current model includes adaptive recurrent collateral connections that aid in the representation of intra-trial temporal information. With this model, as in our prior models, we argue that the hippocampus is a general-purpose system that learns to predict the next state of all stimuli given the current state of variables encoded by activity in recurrent collaterals. As such, the model correctly predicts that hippocampal involvement in associative learning, including classical conditioning, should be most critical not only when there is an intervening trace interval, but also when there is a long delay between CS onset and US onset, as in short-delay vs. long-delay conditioning.

Modeling

Figure 1 shows a schematic diagram of the current model, which builds off our prior models of hippocampal-region processes in classical conditioning. Like our earlier models (Gluck & Myers, 1993, 2001; Moustafa, Myers, & Gluck, 2009), the present model conceives of the hippocampal region (Figure 1, green) as a predictive autoencoder, which learns to predict the next state of the world given current inputs. In the process, the hippocampal-region network forms new stimulus representations in its internal layer that compress (or make more similar) the representations of co-occurring inputs while differentiating (making less similar) the representations of inputs that make different predictions about future events such as US arrival. In other words, the essential function of the hippocampus in our model is monitoring environmental regularities and using prior experience and current inputs to predict what (out of all possible events) is likely to happen next. Classical conditioning is a good example of prediction processes because the most salient event -- the US arrival -- can be predicted with high accuracy by learning the CS and the ISI. In our model, the hippocampal network learns not only whether a particular CS will be followed by a US, but when this will occur.

Figure 1.

The hippocampal model which includes recurrent connections within the motor and hippocampal networks, as well as processing of within-trial events. In the motor network, CS inputs project through modifiable weights to the output node, which in turn project to motor areas that drive the behavioral response (eyeblink CR). The prediction error module receives excitatory US projections and inhibitory CR projections, and provides the response error (US-CR) as a “teaching signal” to the motor network. The hippocampal-region network receives inputs detailing the current state of all inputs at time t, including presence or absence of CSs, contextual cues, US, and CR. The hippocampal-reigon network learns to produce outputs that predict the state of all inputs at the next timestep t+1; in the process it forms new stimulus representations in its internal node layer that are sensitive to stimulus co-occurrence and association with the US. In the intact model, these new representations also provided as input to the motor network, which can then map from them to new behavioral responses. Arrows represent weighted connections; filled circles = inhibitory connections.

Also as in our prior models, the hippocampal-region network communicates with a second motor network (Figure 1, blue), which is assumed to represent some of aspects cortical and cerebellar substrates of motor learning. The motor output network is modeled as a single adaptive node that learns to map from weighted inputs specifying the presence of CSs, as well as contextual or background stimuli and an efferent copy of the CR. The activities of the hippocampal-region network hidden layer units are also provided as inputs to the motor output network, allowing the motor output network to incorporate the adaptive representations formed in the hippocampal-region network into its own ongoing learning. The output from the motor response network represents the behavioral CR. The difference between this output (CR) and the US constitutes an error signal that can be used to train the connection weights in the motor response network, using an error correction rule such as the least-mean squares or LMS rule (Widrow & Hoff, 1960); full details of the learning rule and other model details are provided in the Methods section below.

The major differences between this model and the prior models are (1) the consideration of each trial not as a discrete event, but as a series of timepoints, (2) the addition of recurrent pathways within the hippocampal-region and motor output networks. We discuss each of these points below; full simulation details are provided in the Methods section.

First, to simulate within-trial events, each trial is divided into a number of timesteps, which represent small time intervals within a conditioning trial (e.g. 50 ms). Typically, contextual inputs are present during the entire trial, and are present alone during the first several timesteps of a trial; then one or more CSs may be presented for a specified number of timesteps; the US, when present, appears for a single timestep. The US may overlap with the CS (as in delay conditioning) or may occur after CS cessation (as in trace conditioning). A further series of context-alone presentations ends the trial (simulating the intertrial interval or ITI). Figure 2 provides schematic illustrations of some example paradigms that we simulate in our model.

Figure 2.

Schematic illustration of stimulus events during a single trial of conditioning. (A) Delay conditioning (ISI=4): On each trial several context-alone presentations are given (not shown), followed by CS onset (left dashed line); the CS remains present for 5 timesteps. The US appears for a single timestep (US onset marked by right dashed line) and co-terminates with the CS. Additional context-alone presentations complete the trial (not shown). (B) Trace conditioning (ISI=4) is similar, except that the CS is present for only two timesteps, producing a two-timestep trace interval (TI=2) before US arrival. (C) Long-delay conditioning (ISI=8) is similar to short-delay conditioning except that the CS is present for 8 timesteps before the US appears; CS and US co-terminate. (D) Short-trace conditioning in which the US appears on the next timestep after CS offset. Abbreviation, ISI, interstimulus interval; TI, trace interval.

A second difference between the prior and current models is that Figure 1 includes recurrent connections within the hippocampal and motor output models. Specifically, the hippocampal region network includes recurrent connections within the internal layer, while the motor output network contains a feedback CR pathway, carrying information regarding the current state of the CR. Provision of feedback within the hippocampal network allowed the activation of internal layer nodes at any time step to be a function of external (CS and contextual) input and also of the adaptive representation of input from a previous timestep. Because the weights on these recurrent connections are adaptive, it is possible for a sequence of activation patterns to be stored in the network that “buffers” input information over several timesteps. Importantly, this buffering function is not pre-wired into the network, but emerges dynamically as a result of training and learning. Anatomical studies support the existence of such recurrent loops in the hippocampal region, particularly hippocampal subfield CA3 (Amaral, Ishizuka, & Claiborne, 1990; Amaral & Witter, 1989) as well as the dentate gyrus (Amaral, Scharfman, & Lavenex, 2007). Prior models of the hippocampus have also included recurrent connections to simulate conditioning tasks (see for example, Rodriguez & Levy, 2001).

In this model as in our prior models, hippocampal lesion is simulated by disabling learning in the hippocampal-region network, in which case the motor output network can still learn new responses by modifying weights from the CS and contextual inputs, but no new adaptive stimulus representations are formed in the hippocampal region (Gluck & Myers, 1993; Myers & Gluck, 1994). In addition, and also as in our prior models, the effects of cholinergic agonists and antagonists are simulated by raising (agonist) or lowering (antagonist) learning rates in the hippocampal-region network (Myers et al., 1996; Myers et al., 1998; Moustafa et al., 2010).

2- Results

The recurrent model of Figure 1 successfully simulates the basic findings usually interpreted as evidence that trace conditioning is hippocampal-dependent but delay conditioning is hippocampal-independent. Figure 3A shows that, for a short ISI (ISI=4), delay conditioning (Figure 2A) is acquired by the intact system more quickly than trace conditioning; this is consistent with empirical data (Beylin, et al., 2001). As mentioned above, the role of the hippocampus in our model is predicting next state of stimuli. Accordingly, the more unpredictable the CS (i.e., if CS is not always present at some timesteps including the trace interval as in trace conditioning), the more difficult the prediction problem becomes, and the longer it takes to learn to predict the right US at the right time. This explains why trace conditioning generally takes longer to acquire than delay conditioning.

Figure 3.

Simulation results of delay and trace conditioning depicted in Figure 2A,B. (A) For a given short ISI (here, ISI=4), the intact model acquires a delay CR more quickly than a trace CR. (B) Under the same parameters, the lesioned model can learn a delay CR but is severely impaired at acquiring the trace CR. In this figure, and in subsequent figures depicting learning curves, the mean model output is defined as the output of the motor network at timestep t-1, where t is the time of US arrival on that trial; results shown are averaged over 5 simulation runs; bars represent standard error of the mean. (Results – not shown – are similar if CRs are scored as the peak output during the time interval between CS onset and US onset.) In all subsequent figures, HL refers to hippocampal-lesioned model.

Interestingly, mathematical models and empirical data also show the existence of an “Aha!” moment during learning by human subjects, defined as an abrupt increase in performance (Bower, 1961; Trabasso & Bower, 1968). Rats and rabbits also show evidence of an “aha” effect, manifest as abrupt increases in conditioned eyeblink responses during both delay and trace conditioning (Gallistel, Fairhurst, & Balsam, 2004). We argue that the abrupt jump in performance in our model occurs because early in training the internal layer of the hippocampal module learns to represent the CS during all timesteps of the ISI. Once this process is complete, there is a fairly quick process of mapping from these representations to behavioral output, resulting in an abrupt increase in performance.

Figure 3B shows that, given ISI=4, delay conditioning in the lesioned model is comparable to that in the intact model, but that trace conditioning is severely impaired; again, this is consistent with empirical data (Berger, et al., 1983; Beylin, et al., 1999; Gabrieli, et al., 1995; Ito, et al., 2005; Ito, et al., 2006; McGlinchey-Berroth, et al., 1997; Schmaltz & Theios, 1972; Solomon, et al., 1986; Weisz, et al., 1980). Without the hippocampal region network, the model output network alone cannot perform trace conditioning since it cannot form internal-layer node representations as described above to span CS-free intervals in trace conditioning. We obtained similar results when the hippocampal network was left in place but hippocampal plasticity was blocked, which is in agreement with empirical data from Sakamoto et al., who found that the administration of NMDA blockers to hippocampus in mice abolished trace but not delay conditioning.

However, the simple learning curves of Figure 3 mask several additional points: first, both delay and trace conditioning in the intact model are affected by ISI; second, the pattern of impaired and spared learning in the lesioned model reflects not only the presence or absence of a trace interval, but also the ISI. In each case, the effects of ISI manipulation in the model parallel those observed empirically. Below, we present simulation results, along with discussion of the relevant empirical data, for (1) delay conditioning at a variety of inter-stimulus intervals (ISIs) and (2) trace conditioning with short and without trace intervals.

2.1. Delay Conditioning

We first present simulation results for delay conditioning with short and long ISI, in the intact and lesioned models.

2.1.1. Delay conditioning: Short ISI

In delay conditioning with a short ISI (ISI=4), both the intact and lesioned models can learn a CR within a few hundred trials, as shown in Figure 3 above. Figure 4A shows the same data, with intact and hippocampal lesioned model data presented together for comparison, and also shows the absence of model output during the pre-CS period of each trial (one timestep before CS onset). This is consistent with the large body of evidence documenting that hippocampal lesion does not impair acquisition of a delay eyeblink CR in rabbits (Schmaltz & Theios, 1972), rats (Christiansen & Schmajuk, 1992), or humans (Gabrieli, et al., 1995).

Figure 4.

Short-delay (ISI=4) conditioning in the model. (A) Both intact and lesioned model can learn the eyeblink (EB) CR (CR, solid lines) while maintaining low background responding measured at the timestep before CS onset (pre-CS, dashed lines). (B) Activity of the hippocampal-region output node learning to predict the next state of the US in the intact model. (C) Individual responses of two representative hippocampal network hidden units during the last conditioning trial, one (light blue) which responds at CS onset and continues to respond throughout the CS period, and one (dark blue) which responds during the CS period, close to the time of expected US arrival. The x-axis indicates within-trial events: .=context only (no CS or US present on that timestep); c=CS present; u=US present; *=both CS and US present; HL, hippocampal lesion; Hipp; hippocampus.

However, the recurrent model allows analysis not only of learning across trials, as in Figure 4A, but also consideration of within-trial events, including the shape of the CR. Empirical data have also documented learning-related changes in hippocampal neuronal activity during delay eyeblink conditioning. Specifically, as discussed above, hippocampal neurons of the hippocampus show activity during conditioning trials that is similar in form to the learned response. This “hippocampal model” of the behavioral response tends to peak slightly earlier than the behavioral CR (e.g., Berger & Thompson, 1978). Figure 4B shows activity pattern of the output node in the hippocampal-region network that is trained to predict the next state of the US. A similar pattern to that shown in Figure 4B develops in the hippocampal-region output node that learns to predict the CR (not shown); other hippocampal-region output nodes respond to CS onset, while still others respond to neither the CS nor the US. This is consistent with empirical data showing that, although some hippocampal CA1 pyramidal neurons show the “hippocampal model” of the CR, other neurons respond to the CS or do not show CS-evoked changes in activity (e.g., Berger & Thompson, 1978).

To better explain how the hippocampus participates in learning delay conditioning, Figure 4C shows the activity of two hippocampal hidden nodes which increase their activity during the ISI, but at different time steps after CS onset. Other hippocampal hidden units (not shown) have other response profiles, some for example stay on or off during the entire experiment, and these do not contribute to model performance (for similar results, see Rodriguez & Levy, 2001). This variation in responses among hippocampal network hidden nodes is similar to the neural activity of the hippocampus, where different neurons respond at different points within a trial; together, the set of hippocampal network hidden units is sufficient to represent events and timesteps spanning the trial duration.

2.1.2 Delay Conditioning: Long ISI

Figure 5A shows learning curves for the intact and lesioned model in a long-delay paradigm, in which CS duration is 9 timesteps (ISI=8). Under these parameters, the hippocampal lesion model does not reach the same level of responding as the intact model. Empirical data likewise suggest that, although hippocampal-lesioned animals can learn short-delay eyeblink CRs, they are impaired at long-delay conditioning (Beylin, et al., 2001; Port, Mikhail, & Patterson, 1985). Here also, the hippocampal network’s prediction of the US is very similar to, but precedes, the behavioral response (Figure 5B). As mentioned above, the role of the hippocampus in US prediction also explains behavioral differences in short and long-delay conditioning. In our model, the more unpredictable the CS (i.e., the more distal in time the CS is from the US as in long-delay conditioning), the more difficult the prediction problem becomes, and the longer it takes to learn to predict the right US at the right time, as in long-delay conditioning.

Figure 5.

Long-delay (ISI=8) conditioning in the model. (A) as in Figure 4, both the intact and HL model can learn long-delay condition but at a lower rate. (B) Hippocampal prediction of the US in the intact model that produced the responses in (A). (C) Individual responses of three representative hippocampal network hidden units during the last conditioning trial, showing that different units respond to CS onset (purple), later in the CS period (blue), or to predicted US arrival (green).

We also found that the hippocampal-lesioned model is more likely to produce CRs at the wrong time during the ISI interval. To better understand the hippocampus’s role in long-delay conditioning, Figure 5C shows the activity of several representative hippocampal hidden nodes. As in the simulation of delay conditioning (Figure 4C), some hippocampal units increase their activity after CS onset (e.g. Figure 5C, purple line), while others show activity later during the ISI. The hippocampal-lesioned model does not form such traces, and thus shows impairment performing long-delay conditioning tasks.

Figure 6 shows mean trials to a criterion of 10 consecutive trials with CR>0.8 for the intact model, under a range of ISI from 1 to 12 timesteps. The finding that learning in the intact model is faster for short ISI than long ISI is consistent with empirical data (Hoehler & Thompson, 1980; Millenson, Kehoe, & Gormezano, 1977; c.f. Salafia, Terry, & Daston, 1975). Our simulation results are in agreement with empirical studies. For example, Beylin et al. (2001) contrast two ISIs (short vs. long-delay conditioning), and other studies have also shown that even longer ISIs are more difficult to learn (Servatius, Brennan, Beck, Beldowicz, & Coyle-DiNorcia, 2001; Solomon & Groccia-Ellison, 1996).

Figure 6.

Effect of varying ISI. (A) Trials to criterion on delay conditioning in the intact model varies as a function of ISI.

2.2. Trace Conditioning in Intact and Lesioned Brains

In this section, we first present simulation results of trace conditioning with short trace interval, and then in a modified trace conditioning paradigm where CS and US do not overlap, but US appears on the timestep immediately following CS cessation).

2.2.1. Trace conditioning

Figure 7A presents the trace conditioning data from Figure 3, with intact and lesioned models plotted together for ease of comparison. Figure 7A shows that the intact model begins to give a trace response after about 500–600 trials, and a strong response with little responding in the pre-CS period after about 1,000 trials. Note that by the end of 1,000 trials, the hippocampal-lesioned model has not begun to give any trace responses (Figure 7B). This is consistent with the severe impairment in trace conditioning observed in rabbits with hippocampal lesion (Moyer, et al., 1990; Solomon, et al., 1986; Weisz, et al., 1980) and in amnesic humans with bilateral hippocampal damage (McGlinchey-Berroth, et al., 1997).

Figure 7.

Trace conditioning paradigm with ISI=4 and TI=2. (A, B) Across-trials responding in the intact and HL model given 100 training trials (left) and in the HL model given 10,000 training trials, right). (C) Activity of the hippocampal-region output node learning to predict the next state of the US in the intact model. (D) Individual responses of three representative hippocampal network hidden units during the last conditioning trial. As in long-delay conditioning, some nodes respond to CS onset (blue), others later in the CS period (purple), and some peak at the time of expected US arrival (brown).

Note that the development of a well-trained trace responses takes about as many trials as it takes the intact model to reach criterion in the long-delay (ISI=8) conditioning task. Similarly, Beylin et al. (2001) found that, while trace was more “difficult” (in terms of trials to criterion) than delay conditioning when ISI was equated, long-ISI delay conditioning could be about as “difficult” as short-ISI trace conditioning for intact rabbits.

Figure 7C shows the hippocampal prediction of the US; after 1,000 training trials, the hippocampal network has developed a well-timed prediction of the US. As in earlier figures, the hippocampal activity slightly precedes the arrival of the US, because the hippocampal network is trained to predict upcoming events. But earlier in training, around trial 500, the hippocampal network is already giving a well-timed response – arguably before the behavioral CR has emerged. This is consistent with empirical data showing that a small percentage of hippocampal pyramidal cells show a “model” of the behavioral CR in trace conditioning (Weiss, et al., 1996), and that this hippocampal neuronal activity precedes development of a behavioral trace CR (McEchron & Disterhoft, 1997; Weible, O'Reilly, Weiss, & Disterhoft, 2006).

Figure 7D shows representative hippocampal hidden units’ activity during the ISI and trace interval. As in the simulation of long-delay conditioning (Figure 5C), some hippocampal units increase their activity at different timesteps during the ISI and trace interval. This is agreement with earlier computational and empirical results that different hippocampal cells code for CS representation during the trace interval (for similar results, see Rodriguez & Levy, 2001). Importantly, below, we show how hippocampal hidden units’ activity develop over learning until they become stable after the “AHA” moment (see Appendix).

2.2.2. Trace conditioning: Short Interval

In a much easier version of the trace conditioning task, CS duration remains at 2 timesteps but the trace interval is shortened so that the US appears immediately after CS cessation (TI=0), although (unlike delay conditioning) the CS and US do not overlap. The intact model learns this task very quickly – reaching criterion within about 100 trials (Figure 8A). As mentioned above, the role of the hippocampus in US prediction also explains behavioral differences in long- and short-trace conditioning. In our model, the CS is closer in time to the US presentation in the short-trace conditioning, and thus US prediction is easier in this case, and it takes fewer trials to predict the right US at the right time.

Figure 8.

The “simplest” version of trace conditioning in the model, in which US onset occurs just after CS cessation (TI=0); see Figure 6D for task description. (A) The intact model learns this task quickly; the HL model is impaired, but learns to produce some CRs after extended training.

The hippocampal lesion model is still impaired, although some hippocampal lesion simulations can acquire a weak trace CR with extended training. Along similar lines, Walker & Steinmetz (2008) and Moyer et al. (1990) found that, although hippocampal lesion animals were impaired at trace conditioning, their performance was improved if the TI was reduced – although the hippocampal lesion animals were still impaired relative to control animals. From these data, Walker and Steinmetz (2008) concluded that, while the hippocampus is important for trace conditioning, the relative lengths of CS and trace interval are also important.

2.3. Pharmacological Manipulations

Finally, we have simulated the effects of cholinergic agents on delay and trace conditioning. First, we have also found that reducing hippocampal plasticity in the model (by decreasing learning rate values) slightly impairs delay conditioning but severely impairs trace conditioning (Figure 9A), similar to the results of cholinergic antagonists on eyeblink conditioning discussed above.

Figure 9.

Simulation results show that (A) cholinergic antagonists mildly affect delay conditioning but severely impair trace conditioning. In contrast, (B) mild doses of cholinergic agonists slightly enhance delay conditioning and significantly enhance performance in trace eyeblink conditioning, while (C) large doses of cholinergic agonists mildly impair delay conditioning, but severely impair trace conditioning.

In addition, we simulated the effects of low-dose and high-dose of cholinergic agonists on delay and trace conditioning by increasing hippocampal plasticity in the model (through manipulation of learning rate parameter in the hippocampal module). In empirical studies, mild doses of cholinergic agonists slightly enhance delay conditioning in aged rabbits (Woodruff-Pak & Santos, 2000; Woodruff-Pak, Vogel, & Wenk, 2001), and significantly enhance performance in trace eyeblink conditioning (Simon, Knuckley, & Powell, 2004); the model shows the same effects (Figure 9B). Our simulation results also suggest that enhancing learning abilities in healthy animals using cholinergic agonists may improve more difficult tasks, such as trace conditioning (Simon, et al., 2004). Finally, administration of a large dose of cholinergic agonists mildly impairs delay conditioning, but severely impairs trace conditioning in the model (Figure 9C). This is a new prediction of the model, and it is plausible since other experimental studies have shown that the administration of large doses of cholinergic agonists to animals interferes with the performance of hippocampal-based tasks (Dumery, Derer, & Blozovski, 1988; Ennaceur & Meliani, 1992).

3- Discussion

Here, we have presented a computational model of the hippocampal region and its role in stimulus representation, that includes the ability to simulate within-trial events, and thus to address not only trial-level data regarding whether a behavioral response is emitted, but within-trial data regarding the timing of that response. Applied to classical eyeblink conditioning, the model is correctly able to account for the findings that, in intact animals, learning is slower as the ISI increases (also see Servatius, et al., 2001), that for a given ISI delay CRs are acquired faster than trace CRs, and that some (but not all) neurons in the hippocampus show activity patterns that predict and precede the form of the behavioral CR. Our model is also in agreement with studies showing that the hippocampus participates in appetitive (Flesher, Butt, & Kinney-Hurd, 2011; Seager, Asaka, & Berry, 1999) and fear (Esclassan, Coutureau, Di Scala, & Marchand, 2009) trace conditioning.

The model also addresses data from hippocampal-lesioned animals including impairments of trace and delay conditioning that vary as a function of ISI; thus, both trace and long-delay conditioning are impaired in the lesioned model, but both delay and trace conditioning are spared if the ISI is sufficiently short. Finally, our current computational model shows how the hippocampus role in stimulus prediction explains the effects of hippocampal lesion, and cholinergic agonists and antagonists on delay and trace conditioning. To our knowledge, this is the first model to simulate these various delay and trace conditioning results, from both intact and lesioned animals, using one single framework.

In sum, our model not only simulates behavioral differences between delay and trace conditioning, but also addresses a large body of empirical data on the effects of manipulating ISI on conditioning. Along the same lines, our model shows that the same computational theory of hippocampal function—that is, prediction of future state of the environment—explains performance in various conditioning studies. Third, unlike prior models of delay and trace conditioning, our model also shows how hippocampal lesion (and thus inability to correctly predict US timing) interferes with some conditioning tasks (e.g., trace conditioning), but not others (e.g., delay conditioning). Our model also in agreement with data showing that long-delay conditioning is acquired more slowly than short-delay conditioning using a fear conditioning paradigm (Barnet & Hunt, 2005).

3.1. Computational role of the hippocampus in US prediction learning and pattern separation

The hippocampus in our model has two related functions: one is predicting upcoming events (in the output layer) but the other is developing new stimulus representations in the hidden layer that allow this prediction. These functions are interrelated. In other words, if the model is presented with two sets of stimuli that predict the same outcome, through learning, they will have largely similar representations in the hidden layer of the hippocampus. On the other hand, if the model is presented with two sets of stimuli that predict different outcomes, they will have different representation in the hidden layer of the hippocampus. This latter process is also known as pattern separation or differentiation. Since the model is presented with relevant stimuli (including CSs, and USs) and irrelevant stimuli (including contextual information such as smell/shape of the cage or testing room, which are not related to current study being that our focus is on the difference between delay and trace conditioning), the model learns to form separate representations of context and other stimuli. Not only that, but the model also learns to form representation of the CS in the hidden layer of the hippocampus during the trace interval of trace conditioning tasks. Weight modification (i.e., learning) occurs in connections in the hippocampus that help form separate representation of CS and context. Learning takes place in all weights in the hippocampal module, as we explain in Methods and Material (section 4.2). This mechanism explains how the hippocampus participates in US prediction, pattern separation, and learning to represent stimuli that are not perceptually present as in trace conditioning.

Importantly, the role of the hippocampus in US prediction learning also explains behavioral differences between long- and short-trace conditioning. In our model, the CS is closer in time to the US presentation in the short-trace conditioning, and thus US prediction is easier in this case, and it takes fewer trials to learn to predict the US at the right time than in long-trace conditioning. The same analysis also applies to short- and long-delay conditioning. To summarize, the main function of the hippocampus in our model is US prediction, which then explain its role in learning to represent input information in the hippocampus, which in turn explains the hippocampal role in conditioning tasks.

3.2. Comparison to Other Hippocampal Models of Conditioning

The conception of the hippocampal region as a general-purpose prediction device, learning to map from current inputs to expectation of future events, and helping to span temporal gaps between stimulus and desired response, has gained a great deal of currency in the last decade, as empirical data have emerged suggesting that hippocampal “place cells” encode not only spatial location but also “location” in temporal sequences (e.g., Dragoi & Buzsaki, 2006; Johnson & Redish, 2007; Lehn et al., 2009; Pastalkova, Itskov, Amarasingham, & Buzsaki, 2008). This is not the first computational model to instantiate such a function; other computational models have also focused on putative hippocampal-region roles in temporal sequence learning or short-term buffering of information, at various levels of biological and empirical detail (Hasselmo, Fransen, Dickson, & Alonso, 2000; Hazy, Frank, & O'Reilly R, 2007; Hazy, Frank, & O'Reilly, 2006; Howe & Levy, 2007; Levy, 1996; Wiebe, Staubli, & Ambros-Ingerson, 1997), and it is entirely possible that some or all of these earlier models could also address the delay, trace, and ISI results which have been simulated here. To date, only a subset of such existing models of hippocampal function has been explicitly applied to address classical conditioning. Of this small set, most existing models have assumed some degree of “hard-wired” connections that allow the hippocampal region to perform temporal processing.

For example, some models have assumed that temporal information is provided to the hippocampus from external sources. For example, Schmajuk and colleagues proposed a model of hippocampal-region function based on the attentional models of Pearce and Hall (Pearce & Hall, 1980) in which the hippocampus computes the aggregate prediction of reinforcement – i.e. the expectation of US arrival based on all available cues (e.g. CSs) – and the difference between this aggregate prediction and the actual US is then used to compute the associability of cues contributing to this prediction (also see Schmajuk, Larrauri, & Labar, 2007; Schmajuk & Moore, 1988). These models assumed that each CS had a memory trace, which was maximal when the CS was present, and then decayed back to baseline. This trace could be associated with a subsequent US, even if the CS and US did not overlap, and thus the model could perform trace conditioning. Similar assumptions of an explicit CS trace that occurs outside the hippocampus are made in other models (Buhusi & Schmajuk, 1996; Schmajuk & DiCarlo, 1992; Schmajuk & Moore, 1989).

Other models, often known as “tapped delay line” models, have suggested that the CS trace could arise in the hippocampus, with different dentate gyrus cells responding at different delays to a CS (Grossberg & Merrill, 1996; Ludvig, Sutton, Verbeek, & Kehoe, 2009; Zipser, 1986), generating a spectral representation of the CS; a CS could be associated with a US at a specific ISI (with or without a trace interval) by adjusting the weights from the cell that responded to the CS with the correct delay. Such models generally require one cell to represent each possible CS at each possible delay, which leads to a problem of combinatorial explosion. Further, the biological plausibility of models invoking tapped delay lines is weakened by the fact that cellular recordings do not show any obvious CS storage within the hippocampus (Rodriguez & Levy, 2001) and that, although hippocampal units display CS-related activity during the trace interval in eyeblink conditioning, this activity shifts across training so that, in a well-trained animal, neuronal activity tends to model the time-course of the behavioral CR (e.g.,Solomon, et al., 1986).

In contrast to models that assume tapped delay lines or other explicit representations of a CS trace, the current model follows in a different tradition of prior models suggesting that the hippocampus’s ability to maintain stimulus traces across short delays is not hardwired but adaptive. These models assume that the high degree of internal recurrency in the hippocampus, particularly within field CA3, could allow the hippocampus to store sequences of neural activity by forming a reverbatory memory (Levy & Sederberg, 1997; Rodriguez & Levy, 2001; Wallenstein & Hasselmo, 1997a, 1997b; Wiebe, Staubli, & Ambros-Ingerson, 1997; Yamazaki & Tanaka, 2005). Network models with recurrent connections can adaptively learn to buffer information across a stimulus-free interval without requiring a multitude of hardwired delay lines (Levy, 1989). A few prior recurrent network models have been explicitly applied to classical conditioning and to trace conditioning in particular (e.g., Howe & Levy, 2007; Rodriguez & Levy, 2001; Yamazaki & Tanaka, 2005).

For example, Rodriguez and Levy (2001) have considered a biologically-based model of CA3 in which a CS input excites a subset of cells, which in turn excite other cells at a short delay, and so on until a final group of cells representing the US is excited at the correct temporal distance from CS onset (see also Levy & Sederberg). This model has the virtue that its ability to span a CS-free interval is learned, rather than hardwired into the network via tapped delay lines. Howe and Levy (2007) subsequently showed that such a model could correctly predict data showing that various subpopulations of hippocampal neurons are activated by the CS, by the US, or during the trace interval, as well as data showing that the emergence of neuronal activity that accurately predicts US onset occurs suddenly after a period of training in rabbits. However, this model does not consider representational changes in the hippocampus, nor does it interact with a motor output module, and so it does not directly generate CRs, nor can it simulate hippocampal lesion data.

Similar to several of these prior models, the current model includes a recurrent network as a model of the hippocampal region, in which different subsets of hippocampal cells maintain representation of CS information during the ISI in delay and trace eyeblink conditioning. This is supported by an empirical study by McEchron and Disterhoft (1997), who recorded from the CA1 of the hippocampus in rabbits during trace eyeblink conditioning; results showed that different hippocampal neurons maintains representation of CS at different timesteps during the trace interval. As similar to trace conditioning, other neurophysiological studies have shown that different hippocampal neurons are activated at different timesteps during task performance, including spatial navigation (Pastalkova, et al., 2008) and sequence learning (MacDonald, Lepage, Eden, & Eichenbaum, 2011).

3.3. Model Limitations

In turn, the current model also suffers from some limitations. First, it is a simple connectionist model, with abstract nodes that do not simulate the biophysical properties of neurons. As similar to the Rodriguez and Levy model, Itskov et al. (2011) show how a recurrent hippocampal network and hippocampal cells can maintain events over time. While the Rodriguez and Levy model is applied to trace conditioning data, the Itskov model is applied to spatial navigation tasks. In contrast to Levy’s and Itskov’s models, the abstraction of our model allows us to simulate a large number of behavioral data that were not simulated by Levy’s or other models of the hippocampus. Nevertheless, future models should address how physiologically detailed models of the hippocampus (as in the model of Rodriguez & Levy, 2001) can simulate performance in a large number of behavioral studies, including conditioning data (as in our model).

Furthermore, our model considers the hippocampal region as a single functional system, without considering the anatomical and functional differences of different subregions. However, empirical data strongly suggest that some putatively hippocampal-dependent representational processes depend more on entorhinal cortex than on the hippocampus proper (Allen et al., 2002; Coutureau et al., 2002; Jarrard, 1993; Shohamy, Allen, & Gluck, 2000). Other experiments have shown that lesion to CA1, but not to CA3, interferes with performing paired associates tasks that include temporal delay (Kesner, Hunsaker, & Gilbert, 2005). Consistent with this, prior modeling work has shown that some aspects ascribed by our model to the hippocampal region as a whole could emerge naturally from the anatomy and physiology, including redundancy compression in the entorhinal cortex (C. E. Myers, Gluck, & Granger, 1995) and pattern separation in the dentate gyrus (C. E. Myers & Scharfman, 2009, 2010). The pattern separation function of the hippocampus in our model is also much in line with the conjunctive encoding function proposed in O’Reilly’s models (O'Reilly & Norman, 2002; O'Reilly & Rudy, 2001).

Future empirical work should determine whether trace conditioning similarly depends primarily on one or more of the hippocampal subregions. In particular, Czerniawski et al. (2009) have suggested that ventral, but not dorsal, hippocampal lesions impair trace conditioning, which might mean either that the ventral hippocampus is more important for trace conditioning than the dorsal hippocampus, or might merely reflect the relative importance for trace conditioning of inputs that preferentially target the ventral hippocampus. This view is, however, challenged by other physiological experimental studies, which argue that dorsal CA1 neurons are more active than ventral CA1 neurons during trace conditioning (Weible, et al., 2006). Future modeling work could explore these possibilities.

Another limitation of model is not addressing the differential roles of ventral vs. dorsal hippocampus in conditioning (nor simulating subregions of the hippocampus including septum, CA1, CA3, and dentate gyrus). Importantly, there is no consensus in the literature in the role of dorsal vs. ventral hippocampus in conditioning, and it is not clear how both interact during acquisition and performance. For example, Burman et al. (2006) argue that dorsal (septal) hippocampus is important for acquisition, while the ventral hippocampus is important for expression of fear responses (for similar results, also see Kjelstrup et al., 2002). Interestingly, González-Pardo et al. (2012) found opposite results: dorsal hippocampus being important for expression of fear responses, and ventral hippocampus for acquisition! Unlike the Burman et al. results, Czerniawsk et al. (2012) found that both ventral and dorsal hippocampi are required for the acquisition of trace conditioning. On the other hand, Wang et al. (2012) found that dorsal hippocampus is important for contextual fear conditioning, and that dorsal or ventral hippocampus is sufficient for subsequent conditioning in a different context, while Kenney et al. (2012) suggest that there is a competition between dorsal and ventral hippocampus on control over behavior in contextual conditioning tasks. Future modeling work, which takes into account differences in connectivity patterns and cell types in the dorsal vs. ventral hippocampus, might be able to help reconcile some of these conflicting results.

Another limitation of our model is the finding that trace conditioning recruits additional brain areas, such as the supplementary motor area (Knight, et al., 2004) and anterior cingulate cortex (Han et al., 2003), which are not simulated in our model. The function of these brain areas in trace conditioning is perhaps related to the short-term encoding of conditioned stimuli during the trace interval. It is possible that the recurrent connection in our model perhaps corresponds to hippocampal interactions with other cortical areas responsible for maintaining information during trace intervals, though future computational modeling work should address this point more explicitly.

The medial prefrontal cortex (mPFC) is also important for trace conditioning. For example, lesion of mPFC impairs acquisition of long-interval trace but not short-interval trace or delay eyeblink CRs in the rabbit (McLaughlin, Skaggs, Churchwell, & Powell, 2002), and also impairs extinction of previously-learned trace eyeblink CRs (Weible, Weiss, & Disterhoft, 2007). It has been suggested that the mPFC plays a role in contextual-dependent suppression of learned responses (Milad et al., 2007; Morgan & LeDoux, 1995; Resstel, Joca, Correa, & Guimaraes, 2008), and medial prefrontal cortex may be important in suppressing a response to the CS during a trace interval and/or helping to suppress responses to contextual stimuli that are present at the time of US arrival. An important goal for future research will be considering the interaction between the hippocampus and mPFC in classical conditioning. Similarly, the model does not simulate the functionality of the rubro-trigeminal pathway which has been demonstrated to inhibit activity in the inferior olive following stimulation of the magnocellular red nucleus which presumably plays an important role in eyeblink conditioning (Weiss, Houk, & Gibson, 1990).

The current model focuses on the hippocampal region’s role in associative learning; other hippocampal-dependent processes, such as declarative (consciously-mediated) memory, are beyond the scope of the model, but may also play a role in trace conditioning. Several studies have now shown that, in human trace eyeblink conditioning, participants who self-report becoming aware of the stimulus contingencies early in the conditioning session emit more CRs later in the session than participants who report becoming aware later in the session or not at all (Clark, Manns, & Squire, 2002; Clark & Squire, 1998). There was no such interaction between awareness and conditioning under delay contingencies (Manns, Clark, & Squire, 2001), suggesting that awareness in delay conditioning could perhaps be epiphenomenal. Similarly, in a two-cue discrimination task, awareness of stimulus contingencies was associated with emergence of differential eyeblink CRs under trace but not delay conditioning (Clark & Squire, 1998). A possible conclusion to be drawn from these data is that trace but not delay conditioning requires conscious awareness, probably mediated by hippocampal declarative memory systems (Clark, et al., 2002), at least in humans.

While the data correlating awareness and trace conditioning are robust, there are at least three reasons to be cautious in assuming a causative link. First, although eyeblink conditioning appears to share very similar substrates across species from rodents to rabbits to primates (including humans), it is unclear whether conscious awareness is required in non-human animals, or how such awareness might be assessed. Second, as noted by LaBar and Disterhoft (LaBar & Disterhoft, 1998), patients with amygdala lesions show disrupted conditioning with spared declarative knowledge of the stimulus contingencies (Bechara et al., 1995), while partial medial temporal lobe damage that spares declarative memory for stimulus contingencies can be insufficient to support development of conditioning (Daum, Channon, Polkey, & Gray, 1991). These data suggest that awareness per se is not sufficient for conditioning to occur. Third, as mentioned above, although humans and other animals with bilateral hippocampal damage can acquire delay CRs as quickly as controls, this learning is not necessarily “normal” – for example, there may be ill-timed short-latency CRs (e.g., Christiansen & Schmajuk, 1992; Clark, et al., 2002) and impairment at long-delay conditioning (Beylin, et al., 2001).

A final important limitation of the current model is that it ignores consolidation processes (McGaugh, 2000). Although the hippocampus is important for acquisition of trace eyeblink CRs, it apparently is not the final site of memory storage, because trace CRs are abolished in rabbits given bilateral hippocampal lesion one day, but not one month, after trace conditioning (Kim, Clark, & Thompson, 1995). Apparently, the hippocampus either functions as a temporary memory store, or else supports the gradual formation of trace eyeblink associations elsewhere, so that eventually the memories are stored outside the hippocampal region and can survive hippocampal lesion. This final storage site is a matter of debate, but may involve association cortex; in the case of eyeblink conditioning, the cerebellum is also a possibility. The current model could be expanded to explore these possibilities as further empirical data emerge to constrain the model. It is also worth noting that memories can be consolidated, and even strengthened, during sleep (Walker & Stickgold, 2006). In this context, it is interesting to note that sleep deprivation may have a particularly detrimental effect on learning that involves temporal information (such as recency judgments) and motor sequence learning, and that sleep deprivation is associated with decreases in hippocampal nerve growth factor (Walker & Stickgold, 2006). Since any plasticity or consolidation that occurs during sleep happens (by definition) in the absence of external sensory stimuli (and thus, during a stimulus-free trace interval), the relationship between trace conditioning and hippocampal activity during sleep may prove profitable for further explorations using both empirical techniques and computational models.

3.4 Future directions

One important question is, how does the role of the hippocampus in prediction explains its role in long-term memory? In our earlier work, we have addressed the relationship between the hippocampus role in both episodic long-term memory and classical conditioning (see for example, Gluck, Meeter, & Myers, 2003; Meeter, Myers, & Gluck, 2005). We conceptualize long-term memory as binding of information in one single unit. For example, one's memory of visiting a certain place with some friends at certain time is the binding of the where, who, and when together. The relationship between the hippocampus role in US prediction and long-term memory is rather indirect. First, we have previously shown that the role of the hippocampus in US prediction can support associative binding of information, as in contextual conditioning, in which subjects learn to associate contextual and cue information (see for example, Gluck & Myers, 1993; Myers & Gluck, 1994) and associative learning in humans (Moustafa, Keri, Herzallah, Myers, & Gluck, 2010): while learning to predict the US, the hippocampal module’s internal layer learns associative properties of input stimuli. Accordingly, long-term memory (which is fast binding of information, see for example Meeter, et al., 2005) is a special case of associative learning. Similar modeling studies have also addressed relationships between episodic memory and associative learning (see for example the computational model of Li, Naveh-Benjamin, & Lindenberger, 2005).

Experimental studies have shown that the hippocampus is involved in the reactivation of recently acquired memories, particularly during slow-wave sleep. This reactivation consequently leads to the stabilization of memories (Marshall & Born, 2007). It was also shown that hippocampus-mediated declarative-memory improves after a night's sleep (Ellenbogen, Hulbert, Stickgold, Dinges, & Thompson-Schill, 2006). Moreover, additional experimental findings suggest that the hippocampus sends vast amounts of information to the cortex only during slow-wave sleep (Montgomery, Sirota, & Buzsaki, 2008) and that following sleep deprivation, activity in the medial temporal lobes is decreased and its memory capacity reduced (Drummond et al., 2000; Gais et al., 2007; Takashima et al., 2006). Such findings were theorized as supporting the role of the hippocampus as a fast episodic learning system which, during short-wave sleep, gradually transfers its content to the more robust memory storage located in the cortex (McClelland, McNaughton, & O'Reilly, 1995). In sum, it was found that sleep affects memory consolidation by modulating various learning mechanisms and that these learning mechanisms are preferentially affected by different stages of sleep. Following earlier modeling studies on sleep (Hinton, Dayan, Frey, & Neal, 1995; Norman, Newman, & Perotte, 2005), we also argue that sleep enhances offline learning with input stimuli occurs. According to our modeling framework, offline learning during sleep should enhance binding of information in the hippocampus, and thus improved episodic memory. Although explicit simulation of sleep is beyond the scope of the current paper, future modeling work could address data showing how sleep deprivation impairs conditioning tasks (Hagewoud, Bultsma, Barf, Koolhaas, & Meerlo).

In addition, we are currently designing a computational model of the interactions of the hippocampal region, mPFC, and the amygdala in fear conditioning, in which the hippocampal-region network plays a similar function to its role in eyeblink conditioning (as shown here). In other words, we argue that the hippocampal-region is essential for learning stimulus prediction, a process essential to learning various forms of conditioning paradigms.

In sum, our model provides a unified account on the role of the hippocampus in delay and trace conditioning suggesting that the hippocampal region is a general-purpose prediction device, with the ability to help other brain substrates span temporal gaps between stimulus arrival and desired time of behavioral response; this role is critical when the temporal gap is longer than can be bridged by the brain substrates mediating the behavioral response, whether or not there is an accompanying trace interval.

4- Methods and Material

The recurrent hippocampal model was implemented in objective-C++ using the Xcode 3.0 applications development suite for Macintosh OS 10.5.

4.1. External Inputs and Timing Consideration

Each trial is divided into a number of timesteps; for most of the simulations presented here, there are 30 timesteps per trial. At each timestep t, external inputs consist of a 18-element vector[x₁(t)…x₁₈(t)] detailing the presence (1.0) or absence (0.0) of 18 possible cues. For the experiments reported here, the first three elements of the vector are considered as CSs (CS₁, CS₂, CS₃) and the remaining 15 elements are considered as contextual elements. As in prior models (Moustafa, et al., 2009; Myers & Gluck, 1994), CSs are usually presented phasically, with discrete onset and cessation during each trial, while contextual cues are presented tonically throughout the trial and across trials, but otherwise there is no special treatment given to CSs vs. contextual cues in the model.

For the first several timesteps of each trial, only contextual cues are present. One or more CSs may then be presented for a fixed number of timesteps, and the US may be presented for a single timestep to co-terminate with the CS (simulating delay conditioning) or a few timesteps after CS cessation (to simulate trace conditioning). The remaining timesteps in the trial are additional context-alone presentations. As in empirical studies, the time of CS onset is varied pseudorandomly across trials so that CS onset cannot be predicted simply by passage of time from last CS presentation. Figure 2 shows schematic examples within-trial events during a single trial of short-delay, trace, and long-delay paradigms.

At the start of a simulation run, the model is given 100 trials (each consisting of 30 timesteps) of exposure to the context(s) alone. All results shown are averaged across 5 simulation runs, except figures depicting within-trial data, which each show data from one individual, representative, simulation run.

4.2. Hippocampal-Region Network

As in prior models (Gluck & Myers, 1993; Moustafa et al., 2009), the hippocampal region is implemented as an predictive autoencoder network. At each timestep t, the network receives 20 inputs detailing the state of the world, including the 18 CS and context inputs [x₁(t)…x₁₈(t)] (via entorhinal cortex), an efferent copy of the CR from the previous timestep (x₁₉(t) = CR(t–1)), and the US at the present timestep x₂₀(t = US(t).

The hidden layer includes 10 nodes j, each fully connected to the inputs, and computing activation as:

$$y_j(t)=f[\sum _i{x}_j(t)w_{ij}(t)+\sum _{j\prime }{y}_{j\prime }(t-1)w_{j\prime j}(t)+{\theta }_j]$$ [1]

for all input layer nodes i and hidden layer nodes j’. θ_j is node j’s bias, which is treated as a weight from an input that is always active.

The hidden layer units also project to the motor cortex (possibly via waystations in cortex, thalamus, or brainstem), providing the hippocampal-mediated adaptive representations as a secondary set of inputs that can be used to drive CRs. This projection is not assumed to be instantaneous; rather the hippocampal hidden layer activations at time t-1 are provided as input to the motor network at time t.

Finally, the output layer includes 20 nodes k that are fully connected to the hidden layer nodes j and compute activation as:

$$y_k(t)=f[\sum _j{y}_j(t)w_{jk}(t)+{\theta }_k]$$ [2]

The resulting array y_k (t) is a prediction of the next state of the external inputs x_i(t+1). The difference between the predicted and actual inputs is used as an error signal to drive plasticity in the hippocampal network, with each hippocampal output-layer node k updating its weights as:

$$w_{jk}(t+1)=w_{jk}(t)+\mathrm{\Delta }{w}_{jk}(t)\mathrm{\Delta }{w}_{jk}(t)={\beta }^{*}{\partial }_k{(t)}^{*}{y}_i(t)+{\alpha }^{*}\mathrm{\Delta }{w}_{jk}(t-1){\partial }_k(t)={(x_k(t)-y_k(t))}^{*}f\prime (y_k(t))$$ [3]

for all hidden-layer nodes j, where f’(x)=x(1-x), α is a momentum term set to 0.9 and β is a learning rate parameter set to 0.5 if US(t)=1, and to 0.05 otherwise. Each hidden-layer node j then updates its weights as:

$$w_{ij}(t+1)=w_{ij}(t)+\mathrm{\Delta }{w}_{ij}(t)\mathrm{\Delta }{w}_{ij}(t)={\beta }^{*}{\partial }_j{(t)}^{*}{x}_i(t)+{\alpha }^{*}\mathrm{\Delta }{w}_{ij}(t-1){\partial }_j(t)={(\sum _k{\partial }_k{(t)}^{*}{w}_{jk}(t))}^{*}f\prime (y_j(t))$$ [4]

for all input-layer nodes i. All weights in the cortical and hippocampal networks (including biases) are initialized from the uniform distribution U(−0.3,+0.3) at the start of each simulation run.

4.3. Motor Output Network

The motor output network is responsible for the generation of motor responses during conditioning. It is implemented as a single adaptive node, receiving inputs representing CSs, contextual stimuli, an efferent copy of the previous CR, and the activations of the hippocampal-region network hidden layer nodes from the previous timestep. The output of this node at time t, which is assumed to drive the behavioral CR, is computed as:

$$CR(t)=f(\sum _i{x}_i(t)w_{ik}(t)+\sum _j{y}_j(t-1)w_{jk}(t)+CR(t-1)w_{CR}(t))$$ [5]

for all inputs i and all hippocampal hidden layer nodes j. Note that, because both the hippocampal inputs y_j and the efferent copy of the CR are from the previous timestep (t-1), the input layer is not getting any information as input about whether a US has arrived on the current timestep t. The w are modifiable weights on each of the inputs, and f(x)=1/(1+e^−x)

Weights in the motor response network are updated proportional to output error δ(t)=US(t)-CR(t):

$$w_i(t+1)=w_i(t)+{\beta }^{*}\partial {(t)}^{*}C{S}_i(t)w_j(t+1)=w_j(t)+{\beta }^{*}\partial {(t)}^{*}{y}_j(t)w_{CR}(t+1)=w_{CR}(t)+{\beta }^{*}\partial {(t)}^{*}CR(t)$$ [6]

for each input i and hippocampal hidden node j. β is a learning rate parameter; if US(t)=1, β=0.05; else β=0.005. All weights w are initialized to 0 at the start of a simulation run.

In the model, hippocampal-region damage is simulated by deleting the hippocampal-region network. In this case, the motor output network still receives (and can modify weights from) inputs representing CS and context, as well as the efferent copy of the CR. Lastly, as in prior work (Myers et al., 1996; Myers et al., 1998; Moustafa et al., 2010), we simulate the effects of cholinergic antagonists by decreasing the learning rate parameter in the hippocampal-region network, and cholinergic agonists by increasing hippocampal-region learning rate.

Appendix: the development of stable activation in the hippocampus during trace conditioning

In this section, we explain how the activation of hidden layer nodes o of the hippocampal module develops during trace conditioning. Figure 10 shows several plots showing the activity of the 10 hidden nodes of the hippocampus along with the motor output node during trace conditioning. Each of these plots shows the activity of each node at time step t-1 only, where t is defined as the time step where US presentation occurs. The plots show that there are clear correlations between the activities of nodes in the hidden layer and the motor output node. Importantly, in some of the hippocampal hidden nodes, activity stabilizes after the “AHA” moment. Overtraining the model did not alter the activation of hidden units, suggesting they become stable after the AHA moment.

Figure 10.

Activity of hippocampal hidden units at time step before US presentation during trace conditioning, showing prediction of US during learning. The first plot shows activity of the output of the Motor network. The other 10 figures show activity of the hippocampal hidden units during trace conditioning. The “AHA” moment here is roughly at trial 500 (first plot, Top left). Some hidden unit also show increase of activity around the “AHA” moment (see in particular, hidden units # 2,3,4,6,8,9); they activities become stable after the AHA moment, and are not altered by overtraining the network. Abbreviation: hh = hippocampal hidden unit.