Retrospective and prospective responses arising in a modeled hippocampus during maze navigation by a brain-based device

Abstract

Recent recordings of place field activity in rodent hippocampus have revealed correlates of current, recent past, and imminent future events in spatial memory tasks. To analyze these properties, we used a brain-based device, Darwin XI, that incorporated a detailed model of medial temporal structures shaped by experience-dependent synaptic activity. Darwin XI was tested on a plus maze in which it approached a goal arm from different start arms. In the task, a journey corresponded to the route from a particular starting point to a particular goal. During maze navigation, the device developed place-dependent responses in its simulated hippocampus. Journey-dependent place fields, whose activity differed in different journeys through the same maze arm, were found in the recordings of simulated CA1 neuronal units. We also found an approximately equal number of journey-independent place fields. The journey-dependent responses were either retrospective, where activity was present in the goal arm, or prospective, where activity was present in the start arm. Detailed analysis of network dynamics of the neural simulation during behavior revealed that many different neural pathways could stimulate any single CA1 unit. That analysis also revealed that place activity was driven more by hippocampal and entorhinal cortical influences than by sensory cortical input. Moreover, journey-dependent activity was driven more strongly by hippocampal influence than journey-independent activity.

Place field activity reflecting past, present, and future events has been found in hippocampal responses of rodents performing spatial memory tasks (1–3). In studies using a plus maze, a journey consisted of a trajectory from one start arm to one goal arm, with a forced choice at the intersection. Place fields, for which activity varies in different journeys through the maze, are called journey-dependent, and place fields for which activity is present in multiple journeys through the same maze arm are called journey-independent. Journey-dependent responses are either retrospective, where neural activity is present in the goal arm after choice, or prospective, where neural activity is present in the start arm before choice.

Place cell activity depends on the anatomy and detailed circuit connectivity of the hippocampal region and its surrounding areas. The input to the hippocampus consists of highly processed neocortical information from multiple sensory modalities, which converges onto the medial temporal lobe. After processing through the hippocampus and the entorhinal cortex, the output diverges in broad projections back to the neocortex (4, 5). Within the hippocampus itself, there are several levels of looping over different time scales (6–9). The looping of information within the hippocampus allows it to integrate sensory input over time, providing an essential basis for episodic memory (10).

Although current recording techniques are of critical value in exploring hippocampal functions, it is presently not feasible to trace all neuronal circuits contributing to these functions. To provide a tractable theoretical means to interpret data derived from such complex neural systems, we have constructed brain-based devices (BBDs) that allow us to fully examine a simulated nervous system at all levels while the device carries out a behavioral task (11, 12). A BBD consists of a robotic platform, which is controlled by a large-scale neuronal simulation based on features of vertebrate neuroanatomy and neurophysiology. By studying a BBD behaving in a real environment, we are able to investigate all of the detailed dynamic interactions of its nervous system with its body and the environment. Our approach contrasts with other robotic models of hippocampal function (13–15) in that we have focused on large-scale neuroanatomy, examining its effects on neural dynamics.

We previously described the performance of Darwin X (10, 16), a BBD model incorporating a hippocampus and surrounding cortical regions. The device was challenged by a task similar to the Morris water maze (17). By examining the responses of the BBD's simulated brain, we could follow the formation of place cell activity and the detailed dynamics of the pathways that gave rise to such activity. Analysis of the data revealed that CA1 units were driven mainly by the trisynaptic loop in early learning trials but increasingly by perforant path connections in late trials. Because Darwin X was tested in an open-field environment where it rarely took the same route to a particular location, it was difficult to quantify journey-dependent neural responses.

In the present paper, we describe the performance of Darwin XI, a BBD designed to investigate the emergence of journey-dependent activity. To test for journey-dependent responses, Darwin XI explored a standard plus maze, which constrained its paths. The sequential nature of the plus maze task is more suitable for the investigation of prospective and retrospective hippocampal responses; this maze has been used in rodent studies of episodic memory (1). During maze navigation, journey-dependent responses emerged in Darwin XI's hippocampal neuronal units. Because the computational design of a BBD allows us to determine the detailed microcircuitry and synaptic activity of all neuronal units, we are able to trace structural and functional connections in complete detail by a procedure called backtrace analysis (16). In the present study, such an analysis of neural population activity revealed the differential contributions of the BBD's cortical sensory areas and hippocampus to the firing of journey-independent and -dependent neuronal units.

Results

Task and Behavior.

The task was purposely designed to be comparable to the one described by Ferbinteanu and Shapiro (1). Darwin XI had to navigate a plus maze in which it began on a start arm, moved to an intersection, and there chose one of two goal arms (see Fig. 1A). At the end of one of the goal arms was a reflective piece of construction paper that Darwin XI could not detect visually from a distance but could detect when nearby with a downward-facing infrared sensor. The infrared sensor triggered a neural reward response (T+, colored red in Fig. 2A), which in turn triggered a value system response (see S, colored red in Fig. 2A).

Fig. 1.

Darwin XI's environment and experimental protocol. (A) Darwin XI at the choice point of its plus-maze environment. Darwin XI began a trial alternately at the east or west start arm and used its artificial whiskers to follow the maze arm until it reached the choice point. As it followed the maze wall, its whiskers sensed patterns of pegs, its camera sensed color cue cards on the perimeter, its compass provided heading, and its laser provided range information. (B) In the beginning of training, Darwin XI was given a rewarding stimulus when it chose the south goal arm (B1). After it successfully learned that task, the rewarding stimulus was switched to the north goal arm (B2).

Fig. 2.

Schematic of the neural simulation of Darwin XI. (A) The schematic shows the organization of the simulation. Sensory inputs are shown in blue, motor output in green, and value system in red. Simulated neural areas: SI, SII, somatosensory; SMAP, population-coded laser localization; V1, early visual; IT, inferotemporal; Pr, parietal; ATN, anterior thalamic nucleus; M_hdg, motor; BF, basal forebrain; S, value system; T+, rewarding stimulus; and MTL, medial temporal lobe, including hippocampus. (B) The schematic shows the details of the MTL. Projections from cortical sensory areas converge on the entorhinal cortex input layer, EC_in, and EC_in projects to the DG, CA3, and CA1 fields of the hippocampus. CA1 projects to EC_out, which in turn projects to cortical sensory areas. Learning occurs at synapses that have experience-dependent plasticity (dotted lines). Value-dependent synapses from CA1 influence behavioral choices (dotted lines with yellow markers). Inhibitory neuronal units and connections are omitted for clarity. See SI Text for details.

Darwin XI had a camera for vision, artificial whiskers for texture discrimination (18), an internal compass for determining head direction, and a laser range finder for estimating position [see supporting information (SI) Text]. Darwin XI used its whiskers for following the wall of a maze arm and infrared sensors that detected the absence of walls at the intersection of the plus maze. The cues on the maze were ambiguous, and location determination required integration of input from at least two different sensory modalities. The maze environment was constructed to take full advantage of Darwin XI's multiple sensory streams by having different-colored objects on the perimeter of the maze and different textures along the maze arms.

Four individual “subjects” were created, each composed of the same robotic device and neural architecture but differing in the distribution of initial synaptic weights and microscopic connectivity (see SI Text). Each subject began successive trials alternately in each start arm (west, east, west, …) and was trained until it found the reward containing goal arm in 8 of a block of 10 trials. Initially, the reward signal was given at the end of the south goal arm, and the four Darwin XI subjects took 40, 50, 50, and 30 trials, respectively, to reach 80% criterion. After this performance, the reward was switched to the north arm to reverse training. The four subjects then required an additional 120, 140, 110, and 70 trials, respectively, to again reach 80% criterion (see Fig. 1B).

Neural Simulation.

Darwin XI's simulated nervous system incorporated a detailed model of the entorhinal cortex, the hippocampus, and sensory cortical regions (Fig. 2). The architecture was based on known neuroanatomical and neurophysiological parameters obtained from the literature (4–9). The simulation contained 57 neural areas, 80,000 neuronal units, and 1.2 million synapses. Simulated areas are denoted by italics (e.g., CA1). Each neuronal unit was described by a mean firing rate model in which the activity of each unit corresponded to the firing rate of 100 neurons over 200 ms. See SI Text and SI Tables 2 and 3 for further details.

The model included the three sensory cortical regions used in Darwin X (10, 16), visual pathways from camera input (V1-Color → IT and V1-Edge → PR in Fig. 2A), and HD input from an internal compass (HD → ATN in Fig. 2A). Darwin XI had two additional sensory regions: an artificial whisker texture area (SII in Fig. 2A) and a pseudocortical area containing an estimate of location in the environment obtained through a laser rangefinder (SMAP in Fig. 2A). Each sensory input created organized activity in a neuronal area analogous to a neocortical area in the vertebrate brain. Inputs from all these sensory areas converged sparsely on the input layers of entorhinal cortex, which in turn projected by the perforant path to the dentate gyrus and the CA1 and CA3 subfields of hippocampus (EC_in → DG, EC_in → CA3, and EC_in → CA1 in Fig. 2B). The model also contained the trisynaptic loop through the hippocampus and back to the output layers of the entorhinal cortex (EC_in → DG → CA3 → CA1 → EC_out in Fig. 2B).

Strengths of synapses within the hippocampus were activity-dependent (19, 20), and synapses from CA1 to a motor cortical area, M_hdg, were modified by a temporal difference reinforcement learning rule (10, 21) based on the activity of a simulated dopaminergic value system, S. Differences occurred in the activity of M_hdg as Darwin XI viewed each goal arm by panning its camera at the maze intersection. A softmax probability distribution, based on M_hdg activity, was used to choose the direction in which Darwin XI turned. Further details can be found in SI Text.

Place Activity.

Place unit activity was calculated from maps of mean firing rates accumulated after multiple trials of journeys of the same type. A journey is defined by its start and end points, e.g., an east (E) start and north (N) goal is an EN journey. There are four possible journeys: EN, WN (W, west), ES (S, south), and WS. A place field is defined as any contiguous location in the mean firing rate map of a particular journey for which mean unit activity is >10% of the maximal firing rate. Mean firing rate maps were composed of the 20 trials carried out before reaching criterion, excluding error trials where the BBD entered the unrewarded goal arm. A journey-dependent place field is defined as having activity >10% of the maximal firing rate during one journey through an arm and <5% of the maximal firing rate on the other journey through the same arm. Journey-dependent fields on goal arms are said to be retrospective, and journey-dependent fields on start arms are said to be prospective. Representative examples of journey-independent, retrospective, and prospective neuronal units from the CA1 region of Darwin XI are shown in Fig. 3.

Fig. 3.

Representative rate maps. Arrows indicate the directions of the four possible journeys. A rate map is shown for each maze arm on each journey; color represents the mean firing rate of a neuronal unit at the indicated location. (A) Journey-independent place units are active in multiple journeys through the same maze arm. (B) Retrospective place units are more active on a goal arm for a particular journey. (C) Prospective place units are more active on a start arm for a particular journey.

Neuronal units in Darwin XI's CA1 area developed place fields through experience-dependent plasticity while traversing the plus maze. Table 1 shows metrics that quantify place field activity. The place units across all four subjects had, on average, more than one bit of spatial information. Place units also had continuous fields (high coherence) and higher firing rates in those locations than in the rest of the maze (high selectivity). The values in Table 1 are comparable to values recorded in rodent hippocampus (22, 23).

Table 1.

Place unit metrics

Metrics	Mean	SD	Maximum	Minimum
Spatial information	1.01	0.66	4.56	0.13
Coherence	0.83	0.15	0.98	0.06
Selectivity	8.79	7.36	57.09	2.07

Values are calculated by combining the results of the 687 CA1 units across four subjects that had either journey-dependent or -independent place fields. Spatial information, given in bits, is derived by considering a cell as a communication channel whose input is the device's location. Coherence is a nearest-neighbor 2D autocorrelation that measures the smoothness of firing. Selectivity is an arbitrary measure equal to the spatial maximum firing rate divided by the mean firing rate of the neuronal unit. The place units from Darwin XI's CA1 possess fields that have metrics comparable to rodent hippocampus (22, 23).

Darwin XI's position in the maze could be determined accurately, with a median error of 0.4 m (approximately the diameter of the robot), from the population vector of activity in CA1 (see Methods and SI Text for reconstruction details). Additionally, when Darwin XI was in the start arm, the population vector of activity in CA1 could predict the direction it would take with 94% accuracy. When Darwin XI was in the goal arm, the direction it came from could be predicted from CA1 activity with 83% accuracy.

Of the 2,304 CA1 neuronal units (576 CA1 neuronal units per subject, four subjects), 384 had journey-dependent and 303 journey-independent fields (see Fig. 4). This approximately equal distribution of journey-dependent and -independent fields in hippocampal place units is similar to findings in rodent hippocampus (1).

Fig. 4.

The proportion of place units that are prospective (gray), retrospective (white), or journey-independent (black). The proportions of journey-dependent and -independent are roughly of the same magnitude, as found in rodent hippocampus (1).

Backtrace Analysis.

Analysis of the BBD model allowed tracing of all neuronal activity and synaptic efficacy changes during behavior. To examine neural responses that led to a behavioral event, we introduced a methodology called backtrace analysis (16). This method traces functional anatomical pathways by choosing a particular neural reference unit and recursively examining the dynamics of all anatomically connected neuronal units that contributed to the activity of that reference unit.

Neural responses that led to journey-independent and -dependent place activity were subjected to backtrace analysis. The backtrace started with the onset of place activity in a CA1 reference unit at time t during a journey. The analysis then examined every unit that made a synaptic connection onto the reference unit, took those that had nonzero activity at time t−1 (one simulation cycle back, 200 ms), and added them to the backtrace. The process continued recursively on those units added at time t−1 and so on until t−6. This resulted in a graph with vertices consisting of neuronal units and edges consisting of the synaptic influences that those units had on those in the next time step. The influence a presynaptic unit had on a postsynaptic unit was computed to be the presynaptic activity times the synaptic weight.

We randomly chose a number of reference neuronal units in the CA1 region, 43 of which had journey-dependent and 19 had journey-independent place activity. We ran 307 backtraces from these 62 CA1 reference neuronal units; for each reference unit, backtraces were made for multiple trials at the onset of place unit activity. From three to six time steps before the onset of reference unit activity, there were very few shared neuronal units in different backtraces (see SI Fig. 7). However, within two cycles before the onset of reference unit activity, there was a higher percentage of the same neuronal units in the different backtraces. These findings indicate that the potential pathways leading to the activity of a reference unit form a highly degenerate set (i.e., many different circuit structures can lead to the same unit output). As found in the analysis of Darwin X, the backtraces were comprised mostly of entorhinal cortical units (see SI Table 2) and reflected a prevalent activity of the perforant path (80%) over the trisynaptic loop (20%).

A detailed comparison of backtraces from retrospective, prospective, and journey-independent units is shown in Fig. 5. Each backtrace revealed a multiplicity of neural pathways through the network leading to place activity. The two journey-dependent backtraces revealed stronger hippocampal influence than the journey-independent backtrace. The backtrace for the prospective neuronal unit showed a strong contribution of the trisynaptic pathway influence leading to activity (see Fig. 5C). In all backtraces, there was a shift in activity from time step to time step between the simulated entorhinal cortex and hippocampal areas. This interaction reflects the looping anatomical projections within and between these areas.

Fig. 5.

Each diagram depicts a backtrace that starts from a single reference unit (yellow marker). The x axis depicts time ranging from six cycles, t−6, before the reference unit firing to the firing event at time t. Each colored dot represents a neuronal unit: the reference CA1 unit is yellow, hippocampal units are red, enthorhinal cortical units are magenta, and sensory cortical units are blue. Lines between neuronal units denote the synaptic influence (presynaptic activity times synaptic weight). Thicker darker lines correspond to higher synaptic influence. See Fig. 2 for regions corresponding to abbreviations on the vertical axis. (A) Backtrace from the representative journey-independent CA1 unit shown in Fig. 3A. (B) Backtrace from the representative retrospective CA1 unit shown in Fig. 3B. (C) Backtrace from representative prospective CA1 reference unit shown in Fig. 3C. Note that the backtraces from the retrospective and prospective units (B and C) had greater hippocampal influence and lower sensory cortical influence than the backtrace from the journey-independent unit (A).

Hippocampal influence consisting of activation coming from DG, CA3, and CA1 was greater in journey-dependent than -independent backtraces (Fig. 6). In both journey-dependent and -independent backtraces, hippocampal and entorhinal influence was greater than that of sensory input.

Fig. 6.

Proportions of synaptic influence among neuronal areas during 307 backtraces of journey-dependent and -independent units. The open bars refer to the journey-dependent place units, which include retrospective and prospective place units. The filled bars refer to journey-independent place units. Error bars reflect standard error. Note that the hippocampal influence is stronger in journey-dependent than -independent units (Wilcoxon; P < 0.01). Sensory influence was less in the journey-dependent than -independent units (Wilcoxon; P < 0.01).

Discussion

BBDs such as Darwin X and XI allow the modeler to examine all aspects of the simulated brain during behavior in the environment. This method offers the opportunity to examine circuit details of the simulated nervous system in ways not possible with real nervous systems. In previous work, we showed that a BBD could learn a task involving episodic memory, similar to that posed by the Morris water maze. Upon exploration in the environment, Darwin X developed place fields in its hippocampus (10, 16). Backtrace analysis permitted us to examine the patterns of neural activity that determined its behavior in an open field environment. This analysis highlighted the perforant path from the entorhinal cortex to CA1 as an effective functional projection during goal-directed behavior.

The present work used Darwin XI, which incorporated additional sensory modalities and corresponding sensory cortical areas. In contrast to the many possible paths traversed by Darwin X in an open field environment, Darwin XI was tested in a plus maze task that forced it to choose between a limited set of sequential paths. Journey-independent and -dependent place activity emerged in Darwin XI's CA1 during the exploration of this environment (see Figs. 3 and 4). These place units had metrics comparable to those found in the rat (see Table 1). The place fields arose from experience-dependent changes in synaptic strength during training in the plus maze. Because the value system in Darwin XI's brain does not directly affect its simulated hippocampus, retrospective and prospective units arise from network dynamics within simulated sensory, entorhinal, and hippocampal areas.

To better understand these network dynamics, we carried out backtrace analyses on journey-independent and -dependent place units in CA1. From this analysis, we observed relatively strong entorhinal and hippocampal influences leading to place activity (see Figs. 5 and 6). The hippocampal influence was significantly stronger for journey-dependent than for -independent backtraces (see Fig. 6), suggesting that the hippocampus played a more important role for those units that required memory of the past to predict the future.

The dynamics of backtraces over time suggest a possible mechanism for how these journey-dependent responses arise from anatomical loops between the hippocampus and the entorhinal cortex (see Fig. 5). Strong synaptic influences and activity shifts occurred between the entorhinal cortex and areas within the hippocampus. During these shifts, the entorhinal cortex and hippocampal areas process both current and previous sensory input. For example, at one time step, many CA1 neuronal units are driven by entorhinal and other hippocampal neuronal units, whereas at the next time step, CA1 neuronal units drive entorhinal units. At this stage, we can only speculate whether this is the actual mechanism underlying the acquisition and recall of episodic memories that put together both place and time. It would be of interest to trace activity further back in time than a few seconds, but this procedure is presently not feasible, because backtraces may grow exponentially with time. In any case, the backtrace data (Fig. 5) clearly show the presence of many degenerate pathways among populations of neuronal units (24). That approximately half the units in the present experiments are journey-dependent suggests they must also function in many other tasks.

The observation that performance of a simple task, such as that imposed by the plus maze, results in the appearance of retrospective and prospective activity raises questions about the causal chains leading to this activity. The answer to these questions will depend on the mechanisms by which this activity arises, such as those described above. Previous studies have used time-series analyses to explore these mechanisms (25–27). One of these studies, which applied Granger causality to the Darwin X network, revealed a small core of neuronal units that were apparently causal to the firing of a CA1 reference unit (27). Although the use of such theoretical work based on BBDs is of great potential in highlighting important issues, satisfactory answers will ultimately depend on relating such work to further studies of living animals.

Methods

Neuronal Simulation.

A previous model of the hippocampus in Darwin X has been described elsewhere in complete algorithmic detail (10). Exceptions are noted in the SI Text.

Device and Environment.

The robotic device was 0.35 m in both diameter and height and moved at ≈5 cm/s. A CCD color camera mounted on a pan-tilt unit provided visual input to the neural simulation. Deflection-sensing artificial whiskers were arranged on each side of the device in two groups (18). Whiskers arranged in a row parallel to the ground were used for wall-following, and whiskers stacked vertically were used to detect texture patterns of pegs set at different heights along the wall. A magnetic compass provided heading input to the neural simulation. A rear-facing laser rangefinder provided distance-to-object information used to the device's position. A set of three IR transceivers, mounted front left, front middle, and front right, were used to detect the absence of walls at the maze intersection.

The maze (see Fig. 1) was 5 m on the east–west axis and 4 m on the north–south axis. Textures formed from pegs arranged in vertical and horizontal patterns were present on the walls of each arm of the maze. There were four textures paired on the walls of opposite maze arms. All maze walls of the maze were black. Visible cue cards hanging on the perimeter of the maze were colored red, yellow, green, and blue. Each different-colored cue card had a different stripe width. At the end of the goal arm was a platform, which could not be detected by the camera but had a distinctive reflectivity that could be detected by a downward pointing IR transceiver; this signal was used to activate area T+ of the neuronal simulation.

Device Behavior.

The device used its whiskers to follow the walls of the maze, as described in ref. 18. When the three front-mounted IRs detected the maze intersection, the device stopped and panned its camera 90° to the left and then 180° to the right. After each pan, which correspondingly changed the activity in the HD area, the device kept the camera stationary for 13 simulation cycles. During this time, the neuronal activity in the motor area M_hdg was summed for each direction, and a goal arm was chosen with a softmax probability, β = 40, proportional to the M_hdg activities. When the device detected the reflective floor plate at the end of the goal arm, T+ was activated, which in turn activated the value system S. The device again repeated the pan left and then right behavior to potentiate value-dependent synapses between the currently active CA1 units, the motor area M_hdg, and the value area S.

Analysis.

Mean firing-rate maps of neuronal activity were obtained by partitioning the maze arms into a place grid consisting of 10-cm-wide slices across the maze arms. The location of the device at each time step of the simulation was recorded by two overhead cameras.

Two types of reconstruction analyses were used to see how well place and journey could be determined from population activity. The first reconstruction, which estimated the device's position, was calculated based on the vector of CA1 population activity. A rate map of CA1 population activity vectors at each position in the maze was computed from the 20 trials before behavioral criterion for each subject. For each simulation cycle, the place in the rate map that most closely matched the CA1 activity at that simulation cycle was used to estimate position. The second reconstruction analysis predicted the arm the device was going to or coming from. There are eight possible combinations of journey and arm; a mean population activity vector was calculated for each. The population activity vector that most closely matched CA1 activity during a maze arm traversal was used to estimate the journey type. Both reconstruction methods used the Euclidean distance between population activity vectors as a metric for comparison.

Abbreviation

BBD

brain-based device.