Observing degradation of visual representations over short intervals when MTL is damaged

Abstract

Medial temporal lobe (MTL) contributions to the brief maintenance of visual representations were evaluated by studying a group of patients with MTL damage. Eye movements of patients and healthy comparison subjects were tracked while performing a visual search for a target among complex stimuli of varying similarity to that target. Despite the task having no imposed delays, patients were impaired behaviorally, and eye-movement measures showed abnormally rapid degradation of target representations in the patients. Eye-movement data showed a modulation of the duration of fixations as a function of the similarity of fixated array lures to the target, but the effect was attenuated in patients during long fixation paths away from the sample target. This effect manifested despite patients’ shorter searches and more frequent fixations of the sample target. Novel techniques provided unique insight into visual representation without healthy MTL, which may support maintenance of information through hippocampal-dependent relational binding.

Introduction

Medial temporal lobe (MTL) structures play an acknowledged role in the formation and retrieval of new enduring declarative or relational memories (N. J. Cohen & Squire, 1980; Eichenbaum & Cohen, 2001; Scoville & Milner, 1957), but recent findings from neuropsychological and neuroimaging studies indicate that MTL components may also contribute to representations that need only be maintained or manipulated across a brief interval, a function conventionally associated with prefrontal, parietal, and inferior temporal cortices (cf. Jonides et al., 2008; Postle, 2006; Wager & Smith, 2003). Deficits in patients with MTL damage have been observed at a variety of delays ranging from minutes (Ryan, Althoff, Whitlow, & Cohen, 2000) to seconds (Buffalo, Reber, & Squire, 1998; Ezzyat & Olson, 2008; D. E. Hannula, Tranel, & Cohen, 2006; D. E. Hannula, Ryan, Tranel, & Cohen, 2007; Holdstock et al., 2002; Nichols, Kao, Verfaellie, & Gabrieli, 2006; Olson, Moore, Stark, & Chatterjee, 2006; Olson, Page, Moore, Chatterjee, & Verfaellie, 2006; Shrager, Levy, Hopkins, & Squire, 2008), for a range of stimulus materials including scenes (D. E. Hannula et al., 2006; D. E. Hannula et al., 2007; Hartley et al., 2007; A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005; Ryan et al., 2000; Ryan & Cohen, 2004b), faces (A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005; Nichols et al., 2006; Shrager et al., 2008), and other objects (Barense, Gaffan, & Graham, 2007; Buffalo et al., 1998; A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005; Olson et al., 2006). Likewise, neuroimaging studies show activation not only of prefrontal regions but also of the MTL during maintenance across short intervals of faces (Olsen et al., 2009; Ranganath & D’Esposito, 2001), novel objects (Ranganath, Cohen, & Brozinsky, 2005), and scenes (D. E. Hannula & Ranganath, 2008). These findings converge to imply that the MTL supports the ongoing representation of new information, but the nature and timecourse of that information in the absence of MTL processing remains unspecified.

Taking the visuospatial domain as an example, it has been shown that MTL damage impairs memory performance for photographic or computer-rendered scenes even at the shortest of delays. Focal MTL lesions changed the manner in which patients looked at relationally manipulated versions of previously studied scenes (involving rearrangement of scene elements) at both long (Ryan et al., 2000) and short delays (Ryan & Cohen, 2004a; Ryan & Cohen, 2004b), disrupted the ability to represent spatial and associative relations among stimuli even when the time between study and test was best measured in seconds (D. E. Hannula et al., 2006; D. E. Hannula et al., 2007), and impaired simultaneous comparison of scenes viewed from different perspectives (Hartley et al., 2007; A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005). Deficits after short delays have been observed not only with the rich visual scenes in the above-cited experiments, but also with arrays of simple shapes (Holdstock et al., 2002; Olson et al., 2006; Shrager et al., 2008).

These results indicate that the MTL plays a crucial role not just for enduring memories but also for representations that need persist only long enough to satisfy the demands of ongoing processing employing on-line representations (i.e., what have been called on-line representations, see Barense et al., 2007; Ryan & Cohen, 2004a). Scenes or other relationally complex displays are of particular interest in this context because, unless all that is needed is a rapid, holistic appreciation of a scene’s gist (e.g., identifying a briefly-presented image as a beach or an office, see Intraub, 1980), processing of scene-like materials entails the serial identification of the various constituent scene elements and their spatial arrangement (i.e., piecemeal construction and maintenance of an on-line representation which may also be necessary for processing complex objects, see Biederman, 1987; Marr & Nishihara, 1978). Memory of those elements is therefore critical even when all the information necessary for successful performance is available in the display and no delays are imposed by the experimenter, and the MTL is a likely neural substrate for that memory.

In support of this theoretical role, reports have indicated that MTL damage causes impaired performance in discrimination paradigms that simultaneously present all stimuli, including odd-man-out tasks requiring identification of the object that does not belong with the others (Barense et al., 2007; A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005). Deficits in such paradigms occur in the absence of any imposed delays, and the originating authors have suggested that MTL structures play a critical role in perception and not just in memory (Graham, Barense, & Lee, 2010). But, in light of the above considerations, we would argue that the absence of imposed delays does not rule out mnemonic contributions to ongoing processing (see also Shrager et al., 2008). In the odd-man-out paradigm, although all the objects in a given trial are available for inspection at all times, the size and complexity of the objects evoking deficient performance ensure that processing entails the serial inspection of each separate object and multiple comparisons among them while searching for some feature(s) that discriminate(s) one from the rest (see also A. C. Lee & Rudebeck, 2010). Each object, upon being fixated, must be compared with representations being held on-line of the other objects, necessitating the use of on-line representations despite the absence of any imposed delays. One means of evaluating whether MTL patients can maintain information normally during such tasks would be to index the quality of their maintained representations at multiple time points.

In order to evaluate the role of MTL in on-line representations we used a paradigm in which it was possible to examine the memory representations guiding performance, and to assess any changes over brief intervals during on-line processing. Patients with MTL damage and healthy comparison subjects (henceforth, “MTL patients” and “comparison subjects”) were tested on a complex visual search task. On each trial they saw a display with a centrally placed sample stimulus and 72 peripherally-arranged stimuli, all of which remained visible throughout the trial (see Fig. 1). The task was to indicate whether or not any of the peripheral stimuli matched the sample exactly. On one third of the trials, there was one exact match (i.e., the target), while the remaining stimuli were lures, differing from the target by varying degrees. Target-absent trials contained only lures. Subjects were able to visually inspect any peripheral item, and to re-fixate the sample item, as often as they wished, until making their behavioral response. Eye movements recorded throughout each trial afforded a highly sensitive index of the representations being used to guide performance.

Figure 1.

Illustrations of selected items and sample displays. (a) Three sample items illustrating the three-wedge design of all targets and lures, and the three components that occurred in the top left, top right, and bottom portions of the items. (b) and (c) Illustrations of target-present and target-absent test displays, respectively. Subjects searched each test display for an item that matched the central sample, responding “Yes” if they found a target and “No” if they did not. In (b) the target is located in row 2, column 7.

Three measures, one behavioral and two based on eye movements, provided critical insights. First, MTL patient task performance was above chance but still impaired despite the lack of imposed delays. Second, the duration of fixations to peripheral lure stimuli was found to be a function of the similarity of any given lure to the sample, with longer fixations to lures bearing greater resemblance to the sample, revealing the quality of the maintained representation (also see Warren, Duff, Tranel, & Cohen, 2010). MTL patients exhibited the same function relating fixation duration to lure-sample similarity as comparison subjects, but only when a modest number of fixations intervened between last viewing the sample item and fixating a given lure. As the number of intervening fixations increased, the modulation of fixation duration by lure-sample similarity was attenuated in MTL patients relative to the comparison subjects. Third, all subjects frequently re-fixated the sample, providing them with an opportunity to refresh their representation before sampling further peripheral stimuli. However, over trials, comparison subjects gradually returned to the sample fewer times per trial and fixated more peripheral stimuli before returning to the sample, while MTL patients continued to refresh their representations by fixating the sample at a constant rate across trials. Taken together, these three findings reveal the more rapid degradation of MTL patients’ representations and greater dependence on refreshing their representations during on-line processing of complex displays, along with impaired behavioral performance, despite all information needed to succeed being readily available in the display and the absence of any experimenter-imposed delays. These findings indicate that the MTL participates in the active construction of novel representations even during a superficially non-mnemonic task.

Methods

Subjects

Five amnesic patients with extensive bilateral hippocampal damage participated in the experiment, all of whom were drawn from the Patient Registry of the Division of Behavioral Neurology and Cognitive Neuroscience at the University of Iowa. Three of the patients were amnesic secondary to an anoxic episode (1606, 1846, and 2363), and the MTL damage in these patients was relatively circumscribed; atrophy of bilateral hippocampus has been reported for each individual (Allen, Tranel, Bruss, & Damasio, 2006). Two of the patients were amnesic secondary to herpes simplex encephalitis (1951, 2308), and both exhibited lesions including hippocampus bilaterally and extending well into the temporal lobes (for details of 1951, see Feinstein et al., 2009). Damage to extrahippocampal MTL structures was quite severe in both patients and the volume of the remaining tissue was substantially less than normal in both when it could be identified and measured at all. Detailed neuropsychological and demographic information for the participating patients is available in Table 1. Five healthy comparison subjects recruited from the Champaign-Urbana community also participated, each matched to a participating patient on sex, age, years of education, and handedness. All subjects granted informed consent before their experimental session.

Table 1. Demographic and neuropsychological description of MTL patients.

Neuropsychological and demographic details of participating neurological patients. WAIS, Wechsler adult intelligence scale-III; FIQ, full-scale IQ; WMS, Wechsler memory scale-III; GMI, general memory index; DRI, delayed recall index. For the WMS-III, the DRI is an average of the auditory delayed index and visual delayed index. All tests yield mean scores in the normal population of 100 with an SD of 15.

Patient ID	Etiology	Sex	Hand	Onset	Age	Ed.	WAIS	WMS
							FSIQ	GMI	DRI
1606	Anoxic	M	R	1990	61	12	91	66	61
1846	Anoxia	F	R	1993	45	14	84	57	62
1951	HSE	M	R	1980	56	16	106*	75	53
2308	HSE	M	L	1999	52	16	98*	45	48
2363	Anoxia	M	R	1998	52	16	98	73	74

^*reflects a re-evaluated FSIQ for this participant.

Apparatus

All stimulus displays were presented on an NEC MultiSyncLCD2190UXi LCD monitor with a diagonal measurement of 54 cm, a native resolution of 1600×1200 pixels, and a refresh rate of 60 Hz. The eye-tracking apparatus was an Applied Sciences Laboratory (ASL) R6/VHT-2 remote eye-tracking system with a video-based head-tracking system for enhanced stability; the system sampled point of gaze at a rate of 60 Hz.

Stimulus materials

This experiment employed novel visual stimuli composed of three segments each. Each segment was a wedge resembling one-third of a circle, and when presented in the correct configuration three segments comprised a roughly circular stimulus item (samples are provided in Fig. 1a). Stimuli were approximately 115 pixels and 2.3° of visual angle in diameter.

Procedure

The subject was seated at a comfortable viewing distance from the monitor, approximately 75 cm. During each trial, the subject performed a visual search task with no time limit. After searching an array for the target item, the subject responded “Yes” (i.e., target present), or “No” (i.e., target absent); responses were either made verbally or by button press per individual preference.

Lure items and sometimes a target item filled much of the screen. Specifically, the screen displayed a 9×9 grid centered on the sample item (see Fig. 1b-c). Only the 8 spaces adjacent to the sample were unfilled, leaving 72 peripheral items. Center-to-center, adjacent stimuli in the array were separated by at least 2.6° of visual angle. Peripheral items varied in the number of segments that matched the sample item’s segments, and in a target-absent trial: 24 items had no such segments; 24 more items had one such segment; and the remaining 24 items had two such segments. During target-present trials, the target item randomly replaced one lure item, and the only constraint on its position was the location of the preceding trial’s target item (if any occurred); that location and nearby locations were ruled out.

81 trials were completed, 27 containing targets (Fig. 1b-c). Target incidence was constrained so that the first trial did not contain a target item, no more than 2 trials in a row contained a target item, and each third of the experiment contained 9 target-present trials, but was otherwise random. Positioning of items within the peripheral array was constrained to enforce an approximately equal density of zero-, one-, and two-match lures throughout, but was otherwise random. 27 unique sample items were used three times each, once in each third of the experiment (i.e., 27 of 81 trials).

Eye-movement data were collected while the task was being performed, and a calibration screen was presented before each trial to ensure that point of gaze was being gauged accurately. Trial phase was advanced from the intervening calibration screens by the experimenter while central fixation was maintained. During test trials, free viewing appropriate to search was encouraged.

Eye Movement Data

Fixation data were generated from 60 Hz samples of point of gaze using Applied Science Laboratories’s EyeNal application (version 2.91), and this process considered raw eye-movements falling within 0.5° during intervals of 83ms or greater to be fixations. These fixation data and the transitions between them were analyzed according to generally accepted principles (e.g., D. E. Hannula et al., 2010), and some specifics are reported below.

In our task, the positions of a subject’s fixations were recorded and later binned into pre-determined regions of interest (ROI) containing one stimulus item each. ROIs for the peripheral items were necessarily small owing to the dense layout, but the central sample item was granted a larger ROI because of its isolation and its importance for many of the analyses presented here. Each ROI was square and centered on its corresponding item. Peripheral ROIs were 130 pixels square (~2.5° visual angle), and the central sample ROI was 260 pixels square (~5° visual angle). The ROI and duration of each fixation were coded, and transitions between ROIs were also noted.

Eye-movement data were included in the set to be analyzed only if they exceeded an objective criterion for eye-tracking quality. All trials included in further analyses had fixation data accounting for at least 50% of the trial duration (i.e., elapsed time from display onset until subject response). Trials with a lesser proportion of fixation data were considered unlikely to contain accurate measures of gaze position. Across all reported subjects, 13 of 810 trials were excluded based on this objective quality criterion.

One important concern when testing severely amnesic patients using a complex visual search task was whether their memory deficit would allow them to complete a search of the entire array. Before beginning any other analyses, we evaluated this question both qualitatively and quantitatively. Based on those evaluations (described in the Supplemental Materials) we were satisfied that MTL patients searched the arrays as thoroughly as comparison subjects.

Statistical Analyses

All statistical analyses were conducted in the R (version 2.12.2), SAS (version 9.2), or Python (version 2.7) programming languages. Data were subject to one or two levels of inferential analysis: first, t-tests or ANOVAs as appropriate, using the mean observed values of each subject as the dependent variables; and second, model-based inference using generalized hierarchical linear models (GHLM). For the first level, test results noted in the Results section indicate evaluation of the group-level differences based on only per-subject mean values, while in the second case, Type-III fixed effects tests indicate the reliability of parameter values associated with predictors in models that used each datum as an outcome.

Behavioral

Accuracy in detecting a target among lures was summarized using d to control for potential criterion differences between subjects. Calculation employed a method that adjusts for ceiling and floor performance when necessary (Macmillan & Creelman, 2005). Analysis of behavioral performance using proportion correct was congruent with the reported d analyses.

Group-level differences in response time RT across the entire experiment were evaluated using t-tests of per-subject averages of RT during correct-rejection trials. RT changes over time during the experiment were analyzed using a GLHM implemented with the GLIMMIX SAS procedure: predictors included were MTL status, ordinal value of test trials, and an interaction term; subjects were treated as a random effect to account for individual variation in RT; RT was assumed to follow a log-linked gamma distribution; ordinal value of test trial was treated as a continuous predictor; and MTL damage was a discrete predictor with two levels.

Eye Movement

Fixation duration

Data from correctly rejected target-absent trials included 48736 fixations, and the distribution of fixation durations had several general properties that directed the analyses we conducted. Observed minimum fixation duration was 83 ms (i.e., our enforced minimum value), maximum fixation duration was 2085 ms, and visual inspection of the distribution revealed pronounced positive skew. A log transform was applied to fixation duration data before any averaging in order to address the positive skew, and our plotted data reflect that transformation. Initial evaluation of fixation duration data therefore involved first log-transforming the raw values, and then calculating per-subject, per-condition means as the outcome variable for a 3×2 repeated-measures ANOVA that included predictors for the number of features that a lure shared with the target (i.e., 0, 1, or 2), MTL status (i.e., damaged or healthy), and an interaction term. Next, in order to investigate whether the observed fixation-duration difference varied over time between groups, we introduced a viewing-latency predictor to indicate whether a given fixation was made early or late in search (i.e., within 6 fixations since sample fixation versus 7 or later). This analysis was inspired by visual inspection of the group-mean data, as was the selection of the criterion (see Supplemental Materials for details). We chose to characterize latency as a binary value in order to avoid unwarranted assumptions about a specific relationship between the ordinal value of latency and fixation durations, and also because it provides a convenient shorthand. Once again, we averaged log-transformed per-subject, per-conditions fixation durations, then subtracted the mean fixation durations for 0-match lures from those for 2-match lures, creating a difference score for each subject corresponding to the cells of a 2×2 repeated-measures ANOVA, with factors of MTL status, viewing latency, and an interaction term. Retaining data relevant to only 0- and 2-match lures for this second analysis left 33768 fixations.

Transitions to sample

Changes to transitions to the sample item per correct-rejection trial over time were analyzed with a GHLM including fixed effects for trial as an ordinal value and MTL status as a two-level factor; unique subjects were entered as a normally-distributed random effect. As the outcome variable consisted of counts, its distribution was assumed to be negative binomial. The best-fit model, reported below, employed both predictor variables and an interaction between them.

Maximum search path length

Two predictor variables were used to model any changes in the maximum search path lengths observed during correct-rejection trials across trials: ordinal value of trial was treated as a continuous predictor, and MTL status was employed as a discrete predictor. As the outcome variable consisted of counts, its distribution was assumed to be negative binomial. The best-fit model, reported below, employed both predictor variables and an interaction between them.

Results

Behavioral Performance

Subjects made a yes-no judgment on each trial indicating whether or not they thought the display contained a target exactly matching the sample. Analysis of behavioral responses using d as an index of accuracy indicated that MTL patients performed better than chance responding would have allowed (T₄ = 6.263, p = 0.003), but they were impaired relative to comparison subjects (T₈ = 3.640, p = 0.007) with all patients scoring below the range of comparison subjects’ performance (see Fig. 2a). Incorrect responses by patients were most often misses (see Table 2). Etiology and extent of MTL damage were not reliably related to any aspect of behavioral performance; although anoxic patients all had substantially less temporal lobe tissue loss than HSE patients, the performance of the two groups was mixed and indistinguishable (see Fig. 2a). Performance was not correlated with either of two general measures of memory performance (see Table 1), the Wechsler Memory Scale Third edition (Wechsler, 1997) General Memory Index (r = −0.087, p = 0.889) or its Delayed Recall Index (r = −0.161, p = 0.796).

Figure 2.

Summary of the behavioral data. (a) MTL patients were reliably impaired at search relative to comparisons, and their scores are individually labeled for reference. (b) All subjects searched more quickly as the experiment continued, although patients were slower than comparisons overall. Individual response times and best-fit regression lines are shown.

Table 2. Participant performance and response times.

Per-subject hit and correction-rejection rates and mean response times (RT). Patients were much more likely to miss stimuli than comparisons, but both groups correctly rejected most target-absent displays. Patients were also marginally slower than comparisons overall. Statistical analyses of both measures are available in the Results section.

		Hits		CRs
ID	Group	Rate	RT (s)	Rate	RT (s)
1606	MTL	0.556	19.597	0.759	30.959
1846	MTL	0.741	24.244	0.981	52.073
1951	MTL	0.704	26.857	0.981	59.359
2308	MTL	0.667	46.612	0.963	57.457
2363	MTL	0.630	28.911	0.963	60.311
1606c	Comp	0.963	29.036	1.000	40.799
1846c	Comp	1.000	25.714	1.000	44.037
1951c	Comp	0.778	18.789	1.000	28.431
2308c	Comp	0.926	26.932	1.000	45.861
2363c	Comp	0.741	19.556	1.000	34.137

Subjects were asked to perform their search with both speed and accuracy, but displays were presented until a response was made. Initial analysis of response times (RT) for correctly-rejected (target-absent) test trials (see Fig. 2b) suggested that MTL patients were marginally slower than comparison subjects (T₈ = 2.181, p = 0.061). We investigated both that difference and changes to RT during the experiment in more detail using a GHLM which also suggested that overall MTL patients were slower than comparison subjects in responding (here reliably; T₄₉₅ = 2.230, p = 0.026), but indicated that both subject groups showed decreased RT across trials (T₄₉₅ = 6.820, p < 0.0001), while the lack of a significant interaction suggested that both groups benefitted from repetition at the same rate.

Eye-Movement Measures

Subjects’ eye movements were recorded during all trials, and several measures of eye-movement behavior were extracted for each viewer. Two sets of measures were of primary importance:

One concerned the duration of fixations to the various peripheral items as a function of each item’s similarity to the central sample. Items varied in having from 0 to 3 features in common with the sample (lures had 0-2 features matching the sample; a target-present trial’s single target stimulus was a 3-feature match; see Fig. 1a-c). To the extent that the duration of fixations to lures was positively related to similarity to (i.e., the number of features they shared with) the sample, the measure provided an index of the quality or integrity of the on-line representation of the sample.

The other concerned the pattern of viewing of the sample item. During the course of each trial, subjects were able to return their gaze to the sample item as often as they liked, which they tended to do interspersed among their search of the peripheral stimuli. We examined the number of times each subject transitioned back to the sample item from anywhere among the peripheral stimuli, as well as the length of the search paths (i.e., the number of fixations to peripheral stimuli before returning to the central sample) on each trial. These measures of viewing directed at the sample provide an index of subjects’ need to refresh their representation of the sample so as to guide their search of the display.

Fixation Durations

Lure items were fixated by subjects for varying amounts of time, and that variation was reliably different for lure items sharing different numbers of features with the target (F_2,16 = 152.787, p < 0.0001), and no difference was evident between groups (F_1,8 = 0.989, p = 0.349) or in the fixation duration effect between groups (F_2,16 = 1.402, p = 0.275). All pairwise differences between levels of shared features were reliable (maximum Bonferroni-corrected p = 0.0168; see Fig. 3a). The same fixation duration data were next examined more closely for evidence of group-level differences during search by calculating the mean difference in durations between fixations to 0- and 2-match lures and by introducing a factor indicating whether a fixation occurred early or late in search (i.e., within 6 or fewer fixations since the sample was last fixated vs. 7 or more fixations; see Methods and Supplemental Materials for details). The interaction of MTL status by fixation latency was reliable (F_1,8 = 6.471, p = 0.0345), while neither MTL status nor fixation latency alone were reliable (respectively: F_1,8 =2.718, p=0.138; F_1,8 = 0.961, p = 0.356). Post hoc tests confirmed that MTL patients showed less difference in fixation durations in late search than early search (paired T₄=3.209, p = 0.033), while healthy comparison subjects did not differ across latency (paired T₄ = 0.935, p = 0.403). This suggests that both groups maintained a good representation of the sample early in search, but that representation was maintained better by comparison subjects while searching (Fig. 3b). Notably, the same interaction was reliable both in the fixation data from the entire experiment and from data collected within correct rejection trials during the first third of the experiment before any sample targets were repeated (F_1,8 = 8.904, p = 0.0175), and the same pattern of decreased fixation duration differences for patients at longer search latencies is evident (Fig. 3c). Even when analysis was limited to correct rejections during the first nine trials of the experiment, the same effects are evident (F_1,8 = 9.861, p = 0.0138), suggesting that limited experience of the stimulus materials still permits the effect to be observed.

Figure 3.

Modulation of fixation duration by lure-sample similarity. Note that the y-axes of all three panels use ln(ms) units, but that (a) uses a different scale than (b) and (c), which both plot differences in fixation durations rather than whole fixation durations. (a) Both comparisons and MTL patients fixated items that resembled the sample for longer than those that did not. Whiskers indicate standard error of the mean. (b) and (c) However, as more fixations intervened between the last viewing of the sample item and the fixation of a given lure, this effect was attenuated in patients and exaggerated in comparisons; these plots illustrate the effect using the difference in fixation durations between 2- and 0-match lures early and late in search. Group means are presented as bars, while the difference observed for each subject is plotted as a point, and patient values are individually labeled for reference. (b) summarizes data from the entire experiment, while (c) summarizes data from only the first third of the experiment (i.e., 27 trials). * indicates reliable differences at p < 0.05: differences between levels of lure-sample similarity in (a); and reliable MTL status by latency interactions in (b) and (c).

Transitions to Sample

Fixations to the central sample item were frequent for all subjects in early trials; given the nature of the task and the continuous availability of the sample, this was not surprising. Overall, MTL patients made marginally more fixations to the central sample item than comparison subjects during correct rejection trials (T₈ = 2.170, p = 0.062). However, over the course of the experiment there were numerical changes in the number of transitions to the sample for comparison subjects that were not apparent for patients. A regression model indicated that later trials were associated with fewer transitions to the sample item for all subjects (T₄₉₅ = 11.060, p < 0.0001), but a reliable interaction with MTL status (T₄₉₅ = 6.570, p < 0.0001) indicated that this effect was stronger in comparison subjects and attenuated in patients (see Fig. 4a). There was no main effect of MTL status (T₄₉₅ = 1.31, p = 0.192).

Figure 4.

Evidence of learning across the course of the experiment as reflected in eye-movement measures. Both plots show individual values and best-fit regression functions. (a) Comparisons fixated the sample item less and less frequently as the experiment continued, while MTL patients continued fixating the sample item at approximately the same rate throughout. (b) Likewise, comparisons made longer and longer forays from the sample item across the course of the experiment, fixating more items before returning. The lengths of patients’ search paths were stable across the entire experiment.

Longest Fixation Path

The search paths of MTL patients were shorter than those of comparison subjects (T₈ = 2.770, p = 0.024). Paralleling the preceding analysis, the greatest number of fixations a given subject made per trial before returning to the sample was also modeled, and again differences between patients and comparison subjects were observed. Later trials were associated with increasing path length in comparison subjects (T₄₉₅ = 9.000, p < 0.0001), and the interaction between MTL damage and trial was also reliable (T₄₉₅ = 6.670, p < 0.0001). That interaction indicated that patients did not increase the length of their search paths from the sample item across the experiment (see Fig. 4b). There was no main effect of MTL status according to this model (T₄₉₅ = 1.34, p = 0.183).

Taken together, these analyses of fixation path length and transitions to sample indicate that comparison subjects were able to exploit knowledge about the search-array items in ways that patients were not. Specifically, despite repeated massed exposure to the set of items, patients did not change their search patterns much if at all. In the context of the fixation-duration data it seems likely that patients were intentionally limiting the length of their search paths to preserve some degree of performance, suggesting some adaptation to failures of on-line representation.

Discussion

The current findings indicate that MTL structures aid in the maintenance or construction of on-line representations used during ongoing processing. Patients with damage to the MTL were impaired when performing a task that had no imposed delays, that is, when all information necessary for successful performance was available in the display. Moreover, eye-movement measures sensitive to the quality of viewers’ on-line representations and to the use of those representations in guiding ongoing task performance showed abnormally rapid degradation of such representations in MTL patients, and the behavioral deficits seen here did not require MTL damage outside of the hippocampus. Consequently, the performance of MTL patients in this experiment permits inference about the timecourse of visual representations supported by structures outside of the MTL.

A critical measure here was the duration of fixations to lure stimuli, which varied systematically with lure-sample similarity for all subjects (see also Warren et al., 2010), providing us with an index of the on-line representations used by viewers. MTL patients showed a reduced modulation of fixation duration by lure-sample similarity during long fixation paths away from the sample, revealing directly the degradation of their on-line representations. Abnormally rapid changes in the nature or quality of these representations in the absence of hippocampus could be caused by a variety of mechanisms. Temporal decay is one possibility, and evidence supporting this explanation has been observed in MTL patients tested with simple visual stimuli (Olson et al., 2006; Prisko, 1963; Sidman, Stoddard, & Mohr, 1968; Warren et al., 2010). One previous report (Warren et al., 2010) demonstrated that when MTL patients performed a much simpler visual search task, fixation-duration effects were normal when a sample target was available during search, but the same fixation-duration effects were abnormal if a 6 s delay was imposed between sample presentation and search. However, delays in all of the cited reports were always unfilled, whereas the current task involved ongoing search while the sample representation was maintained. Rapid sequential fixation of multiple lure items might interfere with the sample target’s representation, weakening or distorting it. Follow-up experiments could adapt our paradigm to more tightly control search experience (i.e., search history) while preserving elements of organic search behavior, and this approach would allow temporal decay and interference to be convincingly disentangled. These experiments would also address the underlying relationship between latency and representational integrity, which we characterized as binary (i.e., early or late) for convenience, but could reasonably be linear, binary, or still more complex.

Despite their impairment in search performance and in certain measures of eye-movement behavior, MTL patients did exhibit some learning over time that was similar to healthy comparison subjects (i.e., faster response times), and this can be attributed to intact non-declarative learning mechanisms that have been demonstrated in many other tasks (e.g., Brooks & Baddeley, 1976; Cavaco, Anderson, Allen, Castro-Caldas, & Damasio, 2004; Milner, 1962). One prominent investigation of visual search task behavior in amnesic patients (Chun & Phelps, 1999) employed a widely-used, relatively simple variant (i.e., searching for L among Ts) to test contextual, relational learning. While subjects performed many trials of the task, half of the search arrays were repeated many times without any forewarning. As in the current study, both amnesic patients and comparison subjects showed learning about the task by some measures (e.g., completing searches more quickly), but in that experiment only comparison subjects exhibited a further enhancement of response speed associated with the repetition of specific search arrays. This pattern of sparing and impairment in search behavior is generally congruent with our results, although our more complex stimuli and task taxed patients in different ways and yielded different behaviors. In the Chun & Phelps experiment, maintaining a simple, consistent search instruction (i.e., find the L) presented no difficulty for patients across many trials, while in our task the more complex representations of targets that varied across trials proved more difficult, and our fixation-duration analyses indicate that even while searching within a trial patients had difficulty maintaining the sample-target representation.

MTL patients have been shown to perform poorly in other tasks involving judgments about complex visual stimuli (Barense et al., 2007; A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005), but the current results are unique in several important respects. First, our design allowed us to assess the quality of the representations supporting task performance and their degradation in MTL patients in an unprecedented manner, that is, by providing a sample of the target throughout each test trial and using parametric variation of lure-sample similarity and observation of eye movements. Second, we observed impairment in MTL patients regardless of whether the damage was MTL-wide or was more focally hippocampal. Lesions limited to the hippocampus have not been reported previously to impair the discrimination of complex objects, although damage to that structure has been associated with impairments in scene discrimination (A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005). Third, while it has been suggested that similar impairments in amnesic performance solely reflect dysfunction of a well-characterized long term memory system (Shrager et al., 2008), our results demonstrate that irrespective of the underlying mechanisms, apparent attempts by MTL patients to adjust their behavior to compensate for their impairment did not succeed completely. Even with all necessary material available for indefinite inspection their performance was reliably impaired.

Neuropsychological investigation has revealed deficits in short-delay performance by MTL patients (e.g., Barense et al., 2005; Barense et al., 2007; Buffalo et al., 1998; Ezzyat & Olson, 2008; D. E. Hannula et al., 2006; D. E. Hannula et al., 2007; Hartley et al., 2007; Holdstock et al., 2002; A. C. H. Lee et al., 2005; A. C. H. Lee, Bussey et al., 2005; Nichols et al., 2006; Olson et al., 2006; Ryan et al., 2000; Shrager et al., 2008), but it is only one of several converging methods pointing to a role for the MTL in maintenance or integration of stimulus materials. Functional neuroimaging of healthy participants has also contributed substantially to this field (e.g., D. E. Hannula & Ranganath, 2008; Olsen et al., 2009; Ranganath & D’Esposito, 2001; Ranganath, Cohen, Dam, & D’Esposito, 2004; Ranganath et al., 2005; Stern, Sherman, Kirchhoff, & Hasselmo, 2001) by demonstrating that MTL regions are particularly active while information is maintained. One notable example (Olsen et al., 2009) required participants to perform a delayed-match-to-sample task using a pair of faces as the stimuli to be maintained. High-resolution functional images indicated that two MTL regions (entorhinal and perirhinal cortex along with anterior hippocampus) showed continuous activation throughout the delay period when a correct response would be made immediately afterward, while other MTL regions (parahippocampal and fusiform cortex) responded in an anticipatory fashion, simply showing more activation as the test phase approached irrespective of the outcome. This pattern of results might suggest that information is actively maintained in MTL regions during delay intervals, although other reports of MTL activations predicting subsequent memory after delays of seconds suggest that continuous activation of these structures may not be necessary (e.g., D. E. Hannula & Ranganath, 2008). In either case, the relationship between MTL activation and performance across short intervals appears to be robust across several experimental paradigms and approaches.

The results reported here suggest that damage to the MTL results in visual search deficits that are principally mnemonic in nature, and this explanation can address many other findings of behavioral impairment in MTL patients. We identify these deficits as mnemonic because MTL patients show normal fixation of lures at short latencies, suggesting that there is an early representation of the sample item, and therefore structures outside the MTL must be sufficient to briefly maintain that representation. Likely candidates for this early maintenance include portions of prefrontal and parietal cortex, potentially engaged in cooperative interactions: portions of prefrontal cortex has been associated with executive control processes and enhancement of maintenance across delays (Barcelo, Suwazono, & Knight, 2000; Gazzaley, Rissman, & D’Esposito, 2004; Rossi, Pessoa, Desimone, & Ungerleider, 2009; Zanto, Rubens, Thangavel, & Gazzaley, 2011); and parietal cortex has been implicated in visual working memory using a wide variety of tests and cognitive neuroscience methods (Berryhill & Olson, 2008b; e.g., Berryhill & Olson, 2008a; Koenigs, Barbey, Postle, & Grafman, 2009; see also Wager & Smith, 2003). If these regions are sufficient to maintain a representation briefly, then the MTL (and perhaps the hippocampus in particular) may enhance the resilience of the representation, making it less susceptible to interference or degradation over time, although whether any enhanced resiliency is necessarily tied to long-term representation (e.g., Shrager et al., 2008) remains an open question. Meanwhile, if the patients had defective perception of complex items (cf. Graham et al., 2010), then presumably these effects would not manifest normally regardless of whether viewing took place earlier or later. Instead, it seems that a normal (or at least sufficient) representation is formed initially, but that it is degraded or disrupted by time or interference. Our results leave open the possibility that the representation initially formed by patients may be different in kind from those formed by comparison subjects, even if that representation is initially similar enough to drive normal eye-movement effects. A lack of rapid, durable pattern separation between the sample and lure items under consideration might explain the evanescence of the representations that patients can form, and certain theories of MTL function predict poor pattern separation after hippocampal damage (Norman & O’Reilly, 2003; Norman, 2010).

Relational memory theory (N. J. Cohen & Eichenbaum, 1993; Eichenbaum & Cohen, 2001) may also provide a useful framework for interpreting the current results, as it already addresses a variety of deficits caused by hippocampal damage (D. E. Hannula et al., 2007; Konkel, Warren, Duff, Tranel, & Cohen, 2008; Ryan et al., 2000; Ryan & Cohen, 2004a). In the course of performing our search task, subjects needed to maintain a representation of the sample stimulus while exploring many similar items, and one conceptualization of this process is that the maintained representation is associated or related to a distinct, abstract concept of “goal” or “target”. If the association between sample representation and the “goal” concept was disrupted, search could not be successful, and a potential disruption would have been the accidental substitution of a lure representation (i.e., an item previously considered during search) for the sample representation, which would have produced a continuing search for the wrong item, at least until the sample was fixated again. Such a failure of relational memory binding in ongoing processing would concur with a recent report in which patients with bilateral hippocampal sclerosis were impaired in a task requiring maintenance of scenes that would later need to be discriminated from highly similar lures, performance for which was shown via magnetoencephalography to require strong coupling of theta-frequency activity in the temporal and occipital lobes in healthy subjects during memory maintenance (Cashdollar et al., 2009).

Taken altogether, the current findings demonstrate that the contributions to memory of the hippocampus and related MTL structures extend beyond the formation of enduring memory representations that support later retrieval of memory for facts and events to include the rapid creation of representations that immediately influence ongoing cognition and behavior.