The hippocampus uses information just encountered to guide efficient ongoing behavior

Abstract

Adaptive ongoing behavior requires using immediate sensory input to guide upcoming actions. Using a novel paradigm with volitional exploration of visuo-spatial scenes, we revealed novel deficits among hippocampal amnesic patients in effective spatial exploration of scenes, indicated by less systematic exploration patterns than those of healthy comparison subjects. The disorganized exploration by amnesic patients occurred despite successful retention of individual object locations across the entire exploration period, indicating that exploration impairments were not secondary to rapid decay of scene information. These exploration deficits suggest that amnesic patients are impaired in integrating memory for recent actions, which may include information such as locations just visited and scene content, to plan immediately forthcoming actions. By utilizing a novel task that measured the on-line links between sensory input and behavior, we observed the critical role of the hippocampus in modulating ongoing behavior.

Introduction

The mammalian hippocampus is critical for creating long-lasting representations of scenes, events, and their constituent elements (Cohen & Eichenbaum, 1993; Eichenbaum & Cohen, 2001). Damage to the human hippocampus results in severe impairment in processing the arbitrary relationships among individual items that, bound together, form scenes and events. Impaired memory for relationships following hippocampal damage include co-occurrence relations between arbitrary words (Giovanello, Verfaellie, & Keane, 2003), spatial relations between objects and their locations in scenes (Crane & Milner, 2005; Hannula, Tranel, & Cohen, 2006), and temporal relations describing the order of events (Konkel, Warren, Duff, Tranel, & Cohen, 2008). Likewise, brain-imaging studies show activity in the hippocampus during the encoding (Davachi, Mitchell, & Wagner, 2003; Ranganath et al., 2004) and retrieval (Giovanello, Schnyer, & Verfaellie, 2004) of relations. However, one fundamental type of processing has traditionally been considered outside the purview of hippocampal function. During ongoing, goal-directed behavior, optimal action often relies on memory for recently encountered information. For example, during visual search in a scene, representations for regions already searched must be formed to direct upcoming search efficiently into unexplored regions. Does the hippocampus play a role in forming these representations over the brief timescales involved in ongoing exploration?

This issue is of considerable interest because accumulating evidence shows that the fundamental relational binding function of the hippocampus identified within long-term memory paradigms is also critically relevant for short-delay and in-the-moment processing (see Olsen, Moses, Riggs, & Ryan, 2012 for review). In short-delay relational memory tasks, there are now consistent observations that patients with hippocampal damage are impaired in remembering relations between objects and their locations in scenes (Hannula, Tranel, & Cohen, 2006), and between configurations of features within scenes (Hartley et al., 2007), across delays of a few seconds. Also, such impairment in processing relations in scenes has been observed online without any imposed delays. Specifically, amnesics were found to exhibit qualitatively different eye movements during scene viewing (Ryan & Cohen, 2004). The deficits were observed during the initial visual exploration of a scene, implying that amnesia impairs relational processing that occurs from moment-to-moment. While the gist of a scene can be acquired within a quick glance (Biederman, Rabinowitz, Glass, & Stacy, 1974), the relations among the many items that make up a rich scene can only be appreciated via repeated fixations to discrete locations. The reported impairments may thus arise from the inability to bridge these gaps in order to bind serially acquired percepts into a coherent scene. Indeed, it has long been theorized that the hippocampus might bind pieces of information presented at discrete moments into coherent representations. An early study found that the extent of hippocampal damage correlated with impairments in the recall of complex figures that were studied piecemeal—that is, for which information fragments needed to be bound across space and/or time during study (Jones-Gotman, 1986). Likewise, hippocampal activity during encoding predicts subsequent relational memory for information presented piecemeal (Staresina & Davachi, 2009).

One ramification of hippocampal involvement in moment-to-moment processing is that the hippocampus may contribute to strategy implementation during ongoing behavior, when adaptive behavior is based on evolving representations of recently acquired information. For example, during visual search, strategic behavior entails searching in not-yet-explored regions and avoiding previously searched regions. This requires that memory for visited and unvisited areas be updated continuously. One possibility is that the hippocampus continuously binds discrete pieces of information into coherent representations, which then serve as the basis for this sort of strategic sampling behavior.

To investigate this issue, we developed a restricted viewing paradigm that captures a crucial aspect of scene viewing – sampling of different regions through sequential glimpses – while constraining exploration behavior such that the effectiveness of sampling strategies can be reliably measured. In naturalistic scene viewing, multiple factors drive eye movements, for example, the meaning of the scene, the salience of a region, etc. Therefore, it is difficult to isolate the sole impact of previous fixations on scene exploration. Here, we used simple, novel scenes and a restricted-viewing design for which the most effective exploration strategy is unambiguous and therefore can be quantified. Scenes containing only two objects were viewed through a “restricted viewing” window that moved based on input provided by subjects via a joystick. That is, each scene was covered almost entirely by a black mask (Fig. 1A), and moment-to-moment control of the viewing window provided piecemeal viewing of the scene, one small region at a time. Each trial began with the viewing window positioned on one of the two objects (the “start”), and subjects attempted to locate the second object (the “goal”). Upon finding the goal, participants returned the viewing window to the start object using the most direct route possible (Fig. 1B). This paradigm bears some resemblance to real-world visual search — in both cases, only a restricted region of space is under scrutiny at any moment (corresponding to the foveated portion of the visual field during real-world exploration). Efficient exploration of the scene required memory for previously visited locations and/or the scene content acquired over multiple successive glimpses. In addition, the use of a joystick constrained upcoming search to immediately adjacent regions, and constrained the optimal strategy to the creation of systematic search paths (scanning up and down, or left to right), such that deviation from this optimal strategy could be quantified.

Figure 1. Example trial.

(a) A representative background image is shown (left) along with the same background image covered by the mask that was used to restrict viewing (right). (b) Each trial began with the viewing window centered on the start object. Participants moved the viewing window with a joystick to search for the goal. When they reached the goal, they returned to the start object to terminate the trial.

Each scene was presented for exploration in this way on six discrete trials, with each trial separated by two intervening search-and-return trials for other scenes containing different objects. This design allowed both search and return performance for each scene to be assessed both across trials and within trials, thus, with and without interposed trials. Repeated presentations of the same scenes provided an opportunity to assess the consequences of different exploration behaviors in amnesics and comparisons (if any) on subsequent learning, and allowed for replication of short-delay impairment with this novel restricted viewing paradigm. We assessed two aspects of within-trial performance: (1) search efficiency during the first search attempt, and (2) return performance from the goal object to the start object, which indicated the retention of the start location from the beginning of the trial to the end of the trial. Our focus was on the search efficiency measure, which intended to capture the role of the hippocampus in remembering and updating scene information acquired over successive glimpses, given that efficient search requires using an updated representation of past movements and/or the scene to plan the immediately forthcoming exploratory movement. In contrast, return performance provided information on retention capabilities for individual scene elements across the delay imposed by the search, and was used to test whether search efficiency deficits in amnesia could be attributed to rapid forgetting of scene features.

Materials & Methods

Participants

Four amnesic patients with hippocampal damage (three male and one female) and six neurologically intact comparison participants each matched to one of the patients in terms of age, sex, handedness, and education, participated. The amnesic patients were selected from the Patient Registry of the Division of Cognitive Neuroscience at the University of Iowa. The comparison participants were recruited from the Champaign-Urbana community. All procedures were approved by the institutional review boards at the University of Illinois and the University of Iowa. Informed consent was obtained from each participant before testing began. For all of the patients, amnesia was secondary to an anoxic/hypoxic episode, due either to an episode of seizures leading to status epilepticus (patient 1846; see Warren et al., 2012), or to cardiac or cardiopulmonary arrest (in three patients). Structural MRI scans, performed on three of the patients, confirmed bilateral damage that was restricted in large part to the hippocampus compared to a gender- and age-matched comparison group (Allen, Tranel, Bruss, & Damasio, 2006). One of the patients wears a pacemaker, and was therefore not eligible to undergo MRI scanning, but based on etiology of anoxia it is assumed that damage is limited to the hippocampus (Hopkins, Gale, Johnson, & Anderson, 1995; Rempel-Clower, Zola, Squire, & Amaral, 1996; Zola-Morgan, Squire, & Amaral, 1986). The Studentized residual differences in hippocampal and parahippocampal volume with reference to a matched comparison group are presented in Table 1 (for more detailed information about the imaging protocol and volumetric data see (Allen et al., 2006)). The Studentized hippocampal volume was more than two standard deviations below the comparison group for each patient. Patient 2363 had normatively small volumes of overall cerebrum gray matter and parietal gray matter, and his parahippocampal gray matter was normatively small. However, inspection of his MRI scan by an anatomist suggested that his parahippocampal region was intact and not significantly atrophied (Allen, personal communication). Patient 1606 had additional volume reduction in temporal gray matter, and he is the only patient for whom perirhinal damage could not be ruled out confidently. None of the patients had significant volume reduction in the frontal lobe.

Table 1. Amnesic Subject Characteristics.

Patient	Age	Sex	Handedness	HC Residual	PHC Residual		WMS-III	CFT	WAIS-III			Working Memory
					gray	white	GMI		VIQ	PIQ	FSIQ	WMI	DS	Arith	Sent Rept
1606	60	M	R	−3.99	−2.46	−2.36	66	11	94	89	91	76	7	9	8
1846	44	F	R	−4.23	−1.28	−2.19	57	6	89	79	84	85	10	7	11
2363	51	M	R	−2.64	−2.26	−0.37	73	5	112	83	98	88	8	11	10
2563	52	M	L	NA	NA	NA	75	7	98	105	102	99	14	6	13

Age: at time of testing. HC Residual: Hippocampal volume (bilateral) relative to a group of 87 healthy comparison participants (see Allen et al., 2006) presented as a Studentized residual value (i.e., corrected for normative, regression-derived volumetric trends in age and sex, scaled as a normally-distributed variable). PHC Residual: Parahippocampal region (gray matter and white matter) volume relative to a group of 87 comparison participants (Studentized residual, see above). WMS-III, Wechsler Memory Scale-III (GMI, General Memory Index); CFT, Complex Figure Test (delayed recall raw score); WAIS-III, Wechsler Adult Intelligence Scale-III (VIQ, Verbal IQ; PIQ, Performance IQ; FSIQ, Full Scale IQ); WMI, WMS-III Working Memory Index; DS, WAIS-III Digit Span subtest; Arith, WAIS-III Arithmetic subtest; Sent Rept, Multilingual Aphasia Examination Sentence Repetition subtest. WMS-III and WAIS-III yield mean scores in the normal population of 100 with a standard deviation of 15.

All of the patients had severe memory impairments that interfered with daily life and prevented them from returning to their former employment since the onset of their amnesia. Thorough neuropsychological testing of each patient confirmed that their memory impairment was disproportionate to any impairment in general cognitive function. The General Memory Index score, obtained from the Wechsler Memory Scale-III, was two standard deviations lower than the mean Full Scale IQ score obtained from each patient on the Wechsler Adult Intelligence Scale-III. Each patient was also severely impaired in delayed recall tests such as the Complex Figure Task, with mean score of 6.4 out of 36. In contrast, their performance on several standardized working memory tests that assessed working memory for items was normal, a result that is consistent with well-established findings that performance under short-delay conditions is intact when memory for simple items is tested (Cave & Squire, 1992). Neuropsychological test scores are provided in Table 1.

Procedure

Instructions and practice were given before testing began. The stimuli were 48 computer-rendered scenes with a resolution of 1600 × 1200 pixels created using Bryce 5.0 software. Each scene had a colored background with distinct top and bottom parts (Fig. 1A). Scene-unique novel objects were positioned according to a grid of 24 (6 × 4) cells of equal size, which were not apparent in the scene image (Fig. 1A). Two cells were occupied by the “start” and “goal” objects, sized 4.8° of visual angle. Throughout a trial, the scene was masked in black except for a small viewing window that was smaller than the objects. The viewing window occupied 2.4° of visual angle, and expanded to 4.8° of visual angle when it was enlarged to show the start and goal objects fully. The position of the viewing window was controlled by the participants, who manipulated a joystick to move it to the area they wished to inspect. Each trial consisted of two phases, search and return (Fig. 1B). The trial began with the viewing window centered on the start object. The viewing window was then enlarged for 2s to reveal the start object fully. When the viewing window returned to its original size, the search phase began. Participants then explored the scene in search of the goal. Upon locating it, the viewing window was again enlarged for 2s to reveal the goal fully, and when the window returned to its original size the return phase began. Upon returning to the start object, the viewing window was once again enlarged for 2s to signal the end of the trial.

Participants were given unlimited time to search for and return from the goal; however, for repetitions of the same scene (Presentations 2-6), a beep was sounded if the length of the search path was more than 80% of that in the previous presentation, unless it was within 130% of the optimal search path length. The purpose of the beep was to encourage participants to use knowledge obtained from previous presentation(s), if any, to guide their search. During return, no beep would sound. Participants were told to search and return as quickly possible. Instructions and practice were given before testing began.

There were two conditions in the experiment: in the same-start condition, all three start objects in a block occupied the same location; in the different-start condition, they occupied different locations. All three goals occupied different locations in both conditions. Statistical tests revealed no significant difference in performance between the two conditions. Therefore, analyses are reported collapsed across the two conditions.

The 48 scenes were grouped into 16 blocks of three. Each block was shown six times consecutively, and the presentation order of scenes within a block remained the same across repetitions. Across participants, each scene was seen equally often in each condition. For both conditions, each of the 24 cells was used equally often as the start and goal locations across participants. Within each block, each scene was seen equally often as the first, second, and third trial across participants.

Three of the patients (2363, 2563, and 1846) completed a second session of the experiment on different counterbalancing orders in order to achieve full counterbalancing. The two sessions were separated in time by several months. For patients who were tested twice, a different comparison participant was recruited to complete that particular counterbalancing order, so that comparison participants were not exposed to the experiment more than once (except for 2363, who was tested twice but only one comparison was available, in his case, there were half as many comparison trials for analysis as amnesic trials).

Data Analysis

Percent Coverage

Exploration efficiency was measured using percent coverage, calculated for each initial search attempt (repeat search attempts were not considered in this analysis, as they would reflect a combination of exploration efficiency as well as across-trial scene learning, and therefore could differ between patients and comparisons based on either factor). Percent-coverage scores were obtained by creating a graphical depiction of the portion of the background image that was uncovered by the viewing window for the trial (i.e., a static image of the total the area viewed via the entire search path for the trial). A tight bounding rectangle was drawn around this viewed area, and percent-coverage was calculated as the fraction of the total area of the bounding rectangle that was viewed via the search path (Fig. 2A). All calculations were performed via a computer script, and accuracy was confirmed via visual inspection.

Figure 2. Real-time exploration deficit in amnesia shown by analysis of search paths.

(a) Search paths are illustrated as black lines superimposed over representative background images. Coverage was calculated as the percentage of the total space traversed in a given time (bounded by a red square for illustration). A less efficient search path (top) revealed less of the total space traversed, resulting in a lower percent-coverage value than the bottom path. (b) Coverage of searched area in amnesics and comparison participants. (c) Coverage of searched area subdivided into short (10-20 s), medium (21-50 s), and long (50+ s) search duration in amnesics and comparison participants. Error bars indicate SE.

Behaviors unhelpful for finding the target (e.g., revisiting previously searched areas, “crossing” the path that has already been traversed, etc.) generally detract from percent-coverage scores, whereas behaviors helpful for finding the target (e.g., exploring each area only once, using an orderly pattern so that no back-tracking is necessary to uncover all areas, etc.) increase percent-coverage scores. Furthermore, the use of the bounding rectangle to constrain the overall search space partially accounts for differences among trials in overall search duration, and hence the amount of the background image that could have been covered in the given time, given that the viewing window moved with fixed velocity. In other words, the bounding rectangle approximately normalized the size of actual search space across trials of different durations. However, the bounding-box correction alone does not fully account for overall search duration differences, especially for searches that include multiple revisits to the same location, because it does not directly penalize for revisiting areas of space that were already explored. In other words, a search with multiple revisits can lead to fuller coverage of a (smaller) bounding rectangle compared to searches of the same duration that do not involve revisiting the same locations, depending on the geometry of the search path in each case. In order to more fully account for the influence of variability in search duration, we also explicitly matched trials of similar search duration between amnesic and comparison subjects. This matching, coupled with the bounding-box correction, was used to directly control for the overall longer searches in amnesic subjects versus comparisons (see below) when comparing between these groups. For all analyses of search efficiency, trials lasting less than 10 s were excluded because these generally comprised straight-line paths or very simple paths, indicating that the subject uncovered the target serendipitously without effortful search requiring strategic control. These trials therefore did not capture the search behavior of interest. The proportion of trials excluded for each patient-comparison pair was: 0.3 vs. 0.21 for 1606, 0.33 vs. 0.33 for 1846, 0.11 vs. 0.25 for 2363, and 0.22 vs. 0.19 for 2563. There was no significant difference in the proportion of trials excluded from this analysis for patients compared to comparison (t(3)=0.18, P>0.1).

Initial Heading Error

Another performance measure, initial heading error, was used to assess retention of start and goal locations based on viewing window movements. Initial heading error was calculated as the deviation from the optimal heading between the start object and the goal, i.e., the angle between the actual and the optimal heading. To calculate the initial heading, instantaneous movement vectors were obtained from the first 60 samples of each trial (i.e., first second of navigation). Movement vectors were calculated from successive sampling points (e.g., x₁-x₀, y₁-y₀), and averaged, thus providing the mean direction the joystick moved during the first second of search. Because the central tendency of its distribution was significantly different from zero, the absolute value of each initial heading was taken and the result was then log transformed to produce a distribution that conformed to the assumptions of parametric analyses.

Cumulative Error

Originally developed by Gallagher, Burwell, & Burchinal (1993) to measure rats’ performance in the Morris Water Maze task, cumulative error was used in this study to measure deviation from the optimal path between the start object and the goal. Cumulative error was calculated by sampling the path taken to navigate between the start object and the goal at a rate of 60 Hz, then summing the distance from the goal at each sampling point. This measure takes proximity to the goal location into account, hence offering an advantage over measures of performance that do not (e.g., total path length; see Gallagher et al. (1993) for comparisons between different kinds of measurements). The distribution of cumulative error deviated significantly from normality, and therefore a log transformation was applied to the raw data.

Statistical Tests

To examine across-trial learning, a mixed-model repeated measures ANOVA with between-subject factor group (amnesics, comparison participants), within-subject factors phase (search, return) and trial (trial 1, 2, …, 6), was performed. After finding a significant group × phase × trial interaction (cumulative error: F(5,40)=26.8, P<0.001, ε = 0.30; initial heading error: F(5,40)=11.8, P=0.001, ε=0.36), separate 2-way mixed model repeated measures ANOVAs were performed for each phase, with between-subject factor group and within-subject factor trial. T-tests were performed for post-hoc comparisons. They were Bonferroni corrected for multiple comparisons, with overall alpha held at 0.05 and individual alpha at 0.01.

Results

Within-trial search efficiency

We first sought to assess the role of the hippocampus in using just-acquired information to guide upcoming action within a single trial; that is, in real-time processing that occurs with no interposed study-test delay. We therefore examined the efficiency of search attempts, given that efficient search requires ongoing integration of the scene and/or movement information obtained throughout the search in order to plan the forthcoming exploration. Search efficiency was quantified by percent coverage, where higher percent coverage indicates higher search efficiency.

Analysis was first performed on trials where the search duration for amnesics and comparisons was matched. Such matching was essential in order to achieve a controlled contrast of amnesic and comparison search efficiency: given that the viewing window was of fixed area and moved with fixed velocity, by matching trial duration, both amnesic and comparison trials were equivalent in the maximum percent-coverage value that could have been obtained. Each amnesic trial was paired with a randomly selected trial from his/her comparison that matched for the overall duration of search. The path traversed by each amnesic subject is shown as Supplementary Material, side-by-side with the corresponding matched comparison trial. For the set of search-duration-matched trials, percent-coverage was significantly lower for each patient relative to the matched comparison (each P<0.05; Supplementary Material). The mean duration-matched percent-coverage value was 46.7% for patients and 59.4% for comparisons (t(3)=4.30, P=0.02).

We also performed the same analysis with all trials ≥10 s included, without matching for trial duration. Again, percent-coverage values were lower for each amnesic patient versus his/her comparison subject(s) (44% vs. 63% for patient 1606 vs. comparison; 38% vs. 44% for patient 1846 vs. comparison; 50% vs. 54% for patient 2363 vs. comparison; and 47% vs. 51% for patient 2563 vs. comparison; mean values across subjects provided in Fig. 2B). When trials were grouped based on search duration into short (10-20 s), medium (21-50 s), and long (50+ s) searches (Fig. 2C), a marginally significant main effect of group (patient vs. comparison, F(1,3)=7.9, P=0.06) was found, showing that our results are robust even when a loose comparison is made based on all trials. A non-significant group-by-duration interaction (F(2,6)=2.5, P=0.2) indicated that searches were less effective for amnesics vs. comparisons for all search durations, without significant variation across durations.

Next, we sought to determine whether amnesics showed efficient search early on in a trial and then deteriorated, or they never searched efficiently from the beginning. To accomplish this, we examined the distribution of percent coverage among searches of similar durations. The rationale is that, if amnesics started efficiently but deteriorated, their searches would have similar coverage as those of controls when searches are short but not long, because amnesics would have deteriorated less in short searches. All trials ≥ 10 s were included. Searches were grouped into five duration categories: 10-20s, 21-30s, 31-40s, 41-50s, and >50s. For each duration category, the distribution of percent coverage of all searches in that category is shown as a histogram (Supplementary Figure 1). Visual inspection of the histograms suggests that amnesics searched less efficiently for all search durations, as they had a higher proportion of searches with low coverage in all the histograms. Confirming this visual impression, and consistent with the analysis above where searches were classified into short, medium, and long, searches had lower percent coverage in amnesics than in comparisons for all search durations (main effect of group: F(1,3)=16.41, p=0.03). There was no group-by-duration interaction (F(4,12)=1.02, p>0.4), suggesting amnesics did not search efficiently even in the beginning.

One of our patients (1606) had additional damage in parahippocampal cortex and also the lowest coverage score of all amnesic participants. It is therefore possible that parahippocampal cortex also contributed to disorganized search behavior. However, all other amnesics with damage limited to hippocampus each had a significantly lower coverage score compared to their matched comparisons, suggesting that hippocampal damage alone was sufficient to elicit the impairment. When patient 1606 was excluded from the analysis where we included all trials ≥10 s, the patient versus comparison difference in search efficiency remained significant (P=0.029).

Within-trial return performance

We then assessed how well amnesics retained the start location within a trial across the duration of search. Similar results were obtained for the initial heading error and the cumulative error measures: amnesics’ error was larger for the six trials overall than for comparisons (heading error: F(1,8)=11.9, P<0.01, Fig.3B, 3D; cumulative error: F(1,8)=15.6, P<0.01, Fig. 4B), indicating that they had poorer retention of the start location than comparisons. However, two additional analyses revealed that amnesics’ poorer retention was not due to poor memory for the start location per se, but rather as a result of their longer search. First, we performed an ANCOVA on overall return performance (collapsed across Presentations 1 to 6) with search duration as a covariate. The group effect was no longer significant (heading error: F(1,5)=0.508, p>0.5; cumulative error: F(1,5)=0.06, p>0.8), indicating that amnesics and comparisons performed similarly when search duration was taken into account. Then, return performance was analyzed as a function of how long the immediately preceding search of that trial was. Specifically, returns were split separately for amnesics and comparisons, by the trial’s search duration, into three categories (short/medium/long search). Average short, medium, and long searches are longer in amnesics than in comparisons, and was significantly so for medium and long searches (amnesic vs. comparison: short: 3.1s vs. 0.25s, t(3)=2.52, P=0.09; medium: 15.9s vs. 0.59s, t(3)=4.25, P=0.02; long: 53.3s vs. 15.2s, t(3)=4.65, P=0.02). For both initial heading error and cumulative error measures, 3x2 (short/medium/long search × amnesic/comparison) repeated measures ANOVA revealed that, amnesics’ return performance was worse than that of comparison across all search durations (main effect of group: initial heading error: F(1,3)=10.8, P<0.05; cumulative error: F(1,3)=13.7, P<0.05), in line with the finding that amnesics did worse throughout Presentations 1 to 6. Additionally, performance declined as the search duration increased (main effect of search duration: initial heading error: F(2,6)=8.36, P<0.02; cumulative error: F(2,6)=13.9, P<0.01). Importantly, there was no group × search duration interaction (initial heading error: F(2,6)=0.58, P>0.5; cumulative error: F(2,6)=0.13, P>0.5), suggesting that time is reducing performance in a similar manner for both amnesic and comparison subjects, and so it is not poor memory for the start locations per se that reduced performance in amnesics, but rather just longer duration because of the longer search. Amnesics’ relatively intact within-trial return performance therefore stands in stark contrast to their large search impairment, which manifested as being less efficient (percent coverage) and therefore required more time to identify the target (longer duration).

Figure 3. Amnesia disrupts across-trial search performance more than within-trial return performance.

(a,b) Log-transformed initial heading error of amnesics (black) and comparison participants (gray), during search (a) and return (b) across six presentations. Error bars indicate SE. (c,d) Distribution and frequency of initial heading error of all trials pooled across participants from each group during search (c) and return (d) across six presentations. 0, 90, −90, and ±180 represent deviation in degrees from the correct heading direction.

Figure 4. Cumulative error scores show the same pattern of relatively more impairment in across-trial search versus within-trial return performance for amnesics relative to comparisons.

(a,b) Log-transformed cumulative error of amnesics (black) and comparison participants (gray), during search (a) and return (b) phases across six presentations. Error bars indicate SE.

Across-trial search performance

Next, we assessed how well amnesics and comparisons learned the locations of start objects and goal objects across the six trials for each scene. The same metrics of heading error and cumulative error were calculated for the trajectory taken from the start to the goal object. The two metrics showed very similar results and so are reported together. Amnesics exhibited impaired learning relative to comparisons (Fig. 3A, Fig. 4A). Error scores differed significantly between these groups across the six trials, indicating different learning rates (group-by-trial interaction: heading error: F(5,40)=19.7, P<0.001, ε=0.29; cumulative error: (F(5,40)=65.0, P<0.001, ε = 0.33)). As expected, the two groups showed similar error for the first trial, when performance could not be guided by learning (heading error: t(8)=0.83, P>0.4; cumulative error: t(8)=1.36, P>0.2). Errors were significantly greater for amnesics by the second trial (heading error: t(8)=5.49, P=0.001, Fig. 3A; cumulative error: t(8)=6.79, P<0.001, Fig.4A), leading to an overall higher level of error for amnesics (heading error: F(1,8)=69.8, P<0.001; cumulative error: F(1,8)=96.6, P<0.001).

To better assess learning rate, difference scores were calculated between the first trial and the n^th trial for each scene. Of particular interest were the difference scores between the first and second and the first and sixth presentations; that is, (1) did one-trial learning differ and (2) was there any overall difference in learning across the six trials? For amnesics, error levels for the second trial versus the first were not significantly different from zero (heading error: t(3)=1.53, P>0.2; cumulative error: t(3)=1.83, P>0.1), indicating no learning. In contrast, error levels were significantly less for comparisons’ second trial versus first (heading error: t(5)=5.70, P=0.002; cumulative error: t(5)=10.9, P<0.001), indicating single-trial learning. Importantly, the first-trial vs. second-trial difference scores were significantly lower for amnesics compared to comparisons (heading error: t(8)=3.50, P<0.01; cumulative error: t(8)=5.80, P<0.001), showing reliable impairment in single-trial learning.

Comparing difference scores between the first and sixth scene presentations indicated that amnesics showed no learning across all trials (heading error measure (t(3)=3.32, P=0.05; cumulative error measure (t(3)=2.26, P>0.1). On the other hand, comparisons showed difference scores markedly greater than zero (heading error: t(5)=8.54, P<0.001; cumulative error: t(5)=19.9, P<0.001), although this learning effect essentially began at the second trial, as indicated above. Amnesics were markedly impaired relative to comparisons by the sixth search attempt, as the first-trial vs. sixth-trial difference scores were significantly lower for amnesics compared to comparisons (heading error: t(8)=5.69, P<0.001; cumulative error: t(8)=10.4, P<0.001). Visual inspection of the distribution of initial heading error supports these statistical conclusions (Fig. 3C).

One possibility is that, because amnesics showed significantly impaired search learning, it also took them longer than comparisons to search for the goal object for all trials after the first. Therefore, there was a greater average delay for amnesics between successive trials for the same scene (105.1s vs. 37.7s, t(8)=5.41, P=0.001). Increased delay could thus have contributed to impaired across-trial learning for amnesics, and it remains to be determined if comparisons would show similar performance levels if similar delays were introduced. However, search duration was approximately equivalent for the first trial (105.7s vs. 89.7s, t(8)=1.11, P>0.2) and, therefore, the delay was approximately matched for amnesics versus comparisons for the first presentation of a trial to its second presentation. Comparisons showed robust learning by the second trial, whereas amnesics did not, despite similar delay (first vs second trial difference score: heading error: t(8)=3.50, P<0.01; cumulative error: t(8)=5.80, P<0.001). Nonetheless, delay could have been a factor that contributed to overall impaired learning for amnesics across subsequent trials. It is nonetheless striking that amnesics never showed learning between any consecutive scene presentations comparable to the learning evident for comparisons after the very first scene presentation.

Discussion

Assessment of the exploration behavior of amnesics during the search attempt provided direct evidence for hippocampal contributions to very short-term processing. Amnesics showed less organized search during the very first search attempt with no retention interval. Because this deficit was expressed on the very first search trial, differences in learning across successive trials for comparison versus amnesics could not have been responsible for the amnesic deficit. Furthermore, amnesics retained the start location during the entire search attempt relatively well, as indicated by successful return performance after the goal was identified, and therefore disorganized search occurred while scene location memory was preserved (confounding factors such as reduced motivation during search, general confusion regarding the task, or rapid memory decay can therefore be dismissed). Amnesics thus displayed ineffective search behavior qua behavior that was not secondary to object-location memory deficits, indicating a real-time deficit in the use of spatial knowledge to guide behavior.

Amnesics were severely impaired in learning goal locations across successive presentations of the same object-scene configuration, but retained the start location relatively well within a trial. This selective impairment for delayed retention is consistent with a long history of findings showing relatively worse performance for long retention intervals relative to brief retention intervals following hippocampal damage (Buffalo, Reber, & Squire, 1998; Cohen & Eichenbaum, 1993; Eichenbaum & Cohen, 2001; Holdstock, Gutnikov, Gaffan, & Mayes, 2000a; Holdstock et al., 2000b; Owen, Sahakian, Semple, Polkey, & Robbins, 1995). Notably, a very similar deficit to that observed here was recently found in humans with transient global amnesia using a modified version of the Morris water maze (Bartsch et al., 2010). Selective deficits for delayed retention in our experiment may have resulted from amnesics’ poor representations of the absolute spatial location of the goal, of the location of the goal within each scene, of the spatial relationship between the start object, the goal, and the scene, or any combination of representation qualities. Such poor representations could have stemmed from the inefficient search for the goal. Overall, relatively impaired across-trial learning in this paradigm supports the notion that the hippocampus is more involved in long-term than short-term memory representation, and not that the hippocampus is specifically important for one particular type of representation.

Amnesics produced efficient, systematic search paths less frequently than did comparison subjects. Systematic search via “snaking” is a valuable strategy in unconstrained search tasks because it minimizes the memory demand for remembering visited locations. Furthermore, snaking is one of few possible ways to systematically explore an entire scene without backtracking or path-crossing, two inefficient search behaviors that provide no new information. Given these characteristics, snaking paths were associated with greater search efficiency during the first search (and in other phases), but not with lower cumulative error. This discrepancy probably emerged because search efficiency and cumulative error measure different aspects of task performance. Cumulative error measures proximity to goal location over time, and indicates how much the actual search path deviates from a direct path from start to goal, penalizing greater distances from goal to a greater extent. Critically, efficient search does not require a direct path from start to goal, but instead requires dense search paths and reductions in time spent reviewing previously explored areas. There might be individual differences among comparisons in how quickly they learned to adopt the snaking strategy, but the large proportion of trials demonstrating clear snaking (Supplementary Material) suggests that all comparisons learned the strategy rather quickly.

We consider several possible scenarios by which amnesics’ search was less efficient than in comparisons. First, amnesics may have had poor memory for recently explored regions and so revisited already searched locations more often. Second, they might have had poor memory for how long they had been searching in a particular portion of the display. Third, they might have had poor memory for current task goals, that is, whether they were searching or returning (although their largely intact within-trial return performance suggests that they did not confuse search with return, at least during their return, and so this account is less likely). Fourth, they might have not been able to appreciate the value of a snaking strategy because of their poor memory. Fifth, to the extent that a coherent scene representation facilitates subsequent search, amnesics’ inefficient search may reflect impairment in the integration and relational binding of successive scene glimpses into coherent scene representations.

In support of the hypothesis that failures in scene integration may underlie poor amnesic search performance, IT neurons have been found to use temporal contiguity as a cue to bind separate retinal images into a single object representation (Li & DiCarlo, 2008). Taken with the findings that some properties of IT neurons depend on the MTL, e.g., pair-coding (Naya, Yoshida, & Miyashita, 2001, 2003; Higuchi & Miyashita, 1996), our results might reflect a similar mechanism: the hippocampus binds distinct retinal images (here, discrete glimpses) into the same object (here, a coherent scene). By this binding account, another dimension by which the task nature in search and return varied may be memory load. It may be that as binding failed in amnesics, the many glimpses could not be integrated into a large unit, and so the memory capacity of amnesics was exceeded. It is interesting, nonetheless, that amnesics maintained the information needed to support return performance across the search period and despite any interference from information accumulated during search. This seems at least partially inconsistent with a purely load-based account of their deficits. The binding interpretation is speculative, however, as our current design did not permit us to test whether a coherent scene representation based on recent viewing and current location was created as a by-product of exploration, and if it was, whether such a representation was useful for planning the upcoming exploratory movement. It could be that in the exploration of real-world scenes, having a representation of the scene is beneficial because it provides a context for exploration, whereas in our arbitrary scenes, the utility of such a representation is more limited.

Notably, our amnesic subjects have no impairments that would suggest failures to appreciate or plan effective exploration strategies. They have no reliable impairments in executive function, as indicated by neuropsychological test scores and prefrontal cortex integrity (Konkel et al., 2008), and therefore no gross deficits in planning or strategic thinking, as these capabilities are commonly attributed to frontal cortical function. Indeed, inspection of scan paths (Supplementary Material) indicates that this strategy was adopted frequently by comparison subjects, but very infrequently by amnesics. However, occasional adoption of this strategy by amnesics indicates no fundamental inability to devise such a strategy.

We therefore suggest that amnesics’ disorganized exploration behavior was not caused by poor executive planning capability per se, but that the apparent deficit in executive planning was due to poor memory for what happened during the just preceding search. It could be that poor scene representation brought about by failure to integrate and relate scene information across successive glimpses, or poor memory for the already visited locations, was not useful for sustaining an effective plan, and so systematic exploration failed. In essence, we propose that poor memory caused amnesic subjects to “get lost” during exploration, including during periods when they were attempting to execute effective search using the snaking strategy (see Supplementary Material), therefore disrupting effective search strategies. Of course, the tendency to “get lost” could have resulted in less consideration of snaking as an effective exploration strategy (i.e., amnesics did not fail to think of snaking as a strategy due to poor executive function per se, but because their memory impairment kept them from appreciating the value of snaking). Although our investigation highlights the intriguing possibility that amnesics may be impaired in strategy execution due to impoverished relational representations, we acknowledge the difficulty of distinguishing between impaired strategy execution and an impaired ability to devise an effective strategy based only on exploration behavior. A deficit in devising a strategy might manifest as a higher level of inconsistency from trial to trial in terms of successful execution of the strategy, and so both interpretations are possible.

Interestingly, some reports have suggested that rats with hippocampal lesions show abnormal exploratory behaviors (Faraji, Lehmann, Metz, & Sutherland, 2008; Packard, Hirsh, & White, 1989; Riedel et al., 1999) that are similar in some ways to those we see here in amnesia, although these deficits have rarely been quantified in the animal literature. For instance, rats with hippocampal inactivation persistently used an ineffective thigmotaxic search strategy (wall-hugging) during training in the Morris water maze (Riedel et al., 1999). It is likely that the mal-adaptive wall-hugging strategy also stemmed from a poor representation of the environment constructed online, similar to what we observed in human amnesics here. Some aspects of our paradigm might have served to emphasize these deficits. Notably, our use of novel scenes eliminated amnesics’ ability to rely on knowledge regarding the normal structured organization of environments that could be used to aid navigation in familiar real-world environments. Furthermore, the restricted viewing window eliminated peripheral information that could be used to facilitate scene representation, thus requiring that any knowledge regarding the scene be acquired via integration across sequential glimpses.

We suggest that disorganized search during restricted viewing and abnormal eye movements during naturalistic scene exploration may stem from a common cause. Undoubtedly, obvious differences exist between scene viewing in a restricted mode and in real-life. We do not scan scenes in a snake-like fashion because our eyes are drawn to meaningful regions depending on the type of scene and our goals. However, our design may resemble real-life scene viewing in that it may require binding across discrete sensory samples. Our results suggest that for both processes, the hippocampus might be integrating successive glimpses and use the resulting representation to guide upcoming behavior. Indeed, other groups have reported amnesic deficits in processing of realistic scenes (Graham et al., 2006; Lee et al., 2005a, b), and the deficits we report in strategic processing could contribute to these real-world deficits. Deficits of this nature have been observed in amnesic patients who performed a complex visual search task (Warren et al., 2011). In that study, patients found the target item much less often than healthy comparisons despite a continuously-available sample of the target and unlimited search time. Poor strategic organization of visual search or an inability to adjust on-line performance to account for memory deficits may have played a role in those search failures. Failures of strategic visual exploration might also underlie amnesic deficits in other visual tasks that do not explicitly require maintenance of any information (Warren et al., 2012). Additional experiments will be needed to examine whether briefly glimpsed visual information is integrated into coherent scenes, or whether it is merely a visually sparse record of where previous glimpses occurred and is only sufficient to support efficient search.

Impaired search behavior in amnesics suggests that the hippocampus plays a critical role in executing strategic behaviors in real-time. This is consistent with the proposal that the hippocampus translates learning into adaptive behavior rapidly (Bast, 2007; Bast, Wilson, Witter, & Morris, 2009). It is important to consider that the anatomical connectivity of the hippocampus (Lavenex & Amaral, 2000; Squire & Zola-Morgan, 1991) shows that it is the ultimate convergence zone for many of the brain’s functionally distinct information-processing pathways. We thus speculate that the hippocampus may be critical for binding the output of distinct functional systems in the real-time service of behavior, such as when search strategies must be integrated with representations of locations traversed moments ago, with motor plans for arriving at future locations, etc., to enable systematic search. This proposal is consistent with recent evidence that the memory performance of hippocampal amnesics is not improved by providing them with strategic control over their study behaviors, which significantly improves memory in healthy individuals (Voss, Gonsalves, Federmeier, Tranel, & Cohen, 2011a), and that part of this lack of ability to benefit from control derives from their lack of implementing effective strategies when given control (Voss et al., 2011b). Notably, in these studies the hippocampus was associated with strategic control through its participation with prefrontal and other cortical regions more closely aligned with executive/strategic planning. Deficits in moment-to-moment hippocampal relational binding and the impacts that these deficits have on the ability to execute effective behavior could thus be a primary cause of the working memory and long-term memory deficits observed in amnesia. The current results provide an example of this, as ineffective and disorganized search behavior can easily be appreciated as a factor in the impaired across-trial goal-location learning.

In conclusion, by providing an experimental analog of the way in which the world is normally perceived, glimpse-by-glimpse, we were able to show that the role of the hippocampus is far more immediate than would be suggested by previous findings of impaired long-term relational memory. Our results indicate that the hippocampus is needed for the execution of efficient search behavior in the moment. Additional research will be needed to determine whether the role of the hippocampus in online behavior occurs in addition to its role in long-term memory formation, or whether a fundamental role in relational binding underlies its involvement in both online processing and long-term memory.