The Hippocampus Supports High-Resolution Binding in the Service of Perception, Working Memory and Long-Term Memory

Abstract

It is well established that the hippocampus plays a critical role in our ability to recollect past events. A number of recent studies have indicated that the hippocampus may also play a critical role in working memory and perception, but these results have been highly controversial because other similar studies have failed to find evidence for hippocampal involvement. Thus, the precise role that the hippocampus plays in cognition is still debated. In the current paper, I propose that the hippocampus supports the generation and utilization of complex high-resolution bindings that link together the qualitative aspects that make up an event; these bindings are essential for recollection, and they can also contribute to performance across a variety of tasks including perception and working memory. An examination of the existing patient literature provides support for this proposal by showing that hippocampal damage leads to impairments on perception and working memory tasks that require complex high-resolution bindings. Conversely, hippocampal damage is much less likely to lead to impairments on tasks that require only low-resolution or simple associations/relations. The current proposal can be distinguished from earlier accounts of hippocampal function, and it generates a number of novel predictions that can be tested in future studies.

The distinction between long-term memory and other cognitive abilities such as working memory and perception is one of the most fundamental in cognitive psychology. This distinction is based on behavioral studies showing that long-term memory can be functionally dissociated from other cognitive abilities (e.g., Baddeley & Dale, 1966; Levy & Murdock, 1968), as well as studies of amnesic patients such as HM, which show that damage to the medial temporal lobe (MTL) results in severe deficits in long-term memory despite relatively preserved cognition (e.g., Scoville & Milner, 2000; Warrington & Baddeley, 1974). This distinction, however, has recently been challenged by results showing that under certain conditions, patients with hippocampal damage exhibit deficits not only in long-term memory but also in working memory and perception (for reviews see Cowell, Bussey, & Saksida, 2010; A. C. H. Lee, Yeung, & Barense, 2012; Ranganath & Blumenfeld, 2005). The aim of the current paper is to examine these new findings in light of our current understanding of the role of the hippocampus in supporting long-term memory via recollection of episodic details, and to argue that the hippocampus is essential in representing complex high-resolution associative information in service of cognitive functions ranging from memory to perception.

In the current paper, I will first argue that recollection supports long-term recognition memory judgments for complex high-resolution bindings and that this ability is critically dependent on the hippocampus. I then suggest that the same hippocampally-dependent bindings also play a role in various other cognitive tasks ranging from perception to working memory. Recent human patient studies of perception and working memory are then reviewed and are found to support the proposal. The high-resolution account of hippocampal function will then be contrasted with several earlier accounts.

Recollection: The phenomenon and its neural underpinnings

What is Recollection?

Recollection reflects the retrieval of qualitative information about a prior study event (Yonelinas, 1994; 2001), such as where or when an event took place, as well as specific details about the event itself, including information like the content of a specific conversation, the tone of voice of the participants, and internal emotional states elicited by the event. Recollection can be contrasted with familiarity-based recognition, in which an object is judged as having been recently studied on the basis that it seems familiar without the retrieval of any specific qualitative details. Familiarity is quantitative in the sense that it can vary in strength from weak to strong, but it lacks the qualitative details that are inherent in recollection. The distinction between recollection and familiarity has a long history (e.g., James, 1890; Atkinson & Juola, 1974; Mandler, 1980; Jacoby, 1991; Yonelinas, 1994) and it is supported by an extensive literature demonstrating that these two processes are behaviorally, electrophysiologically and neuroanatomically distinct (for reviews see Diana, Reder, Arndt, & Park, 2006; Eichenbaum, Yonelinas, & Ranganath, 2007; Rugg & Curran, 2007; Yonelinas, Aly, Wang, & Koen, 2010; Yonelinas, 2002).

What makes recollection so useful is that it can support the creation and retrieval of complex high-resolution bindings that represent ‘specific events’ or ‘moments in time’. Recollection therefore has two key properties (see Figure 1). First, it is associative or relational, meaning that it can bind multiple aspects or features together to represent an event. These bindings can vary from being very simple (e.g., a word was presented on the left side of the screen, or it was paired with some other arbitrary word) to very complex (e.g., a word was presented in red, on the left, in large Helvetica letter font, in a specific experimental context; Yonelinas & Jacoby, 1996). Second, recollection is high-resolution, meaning that individuals can report quite precise information about prior events. For example, in addition to remembering that an item was on the left or right side of the screen, we may be able to indicate with high spatial precision the specific location or the specific color that was associated with a studied word (e.g., Harlow & Donaldson, 2012; Harlow & Yonelinas, in prep; also see Zhang & Luck, 2008). Thus, what is recollected from memory can vary, ranging from a simple low-resolution binding to complex high-resolution bindings.

Figure 1.

Variations in the resolution and associative nature of binding. Each shape reflects a different item (e.g., word, object, odor), whereas color is used to represent a quality of that item (e.g., hue, location, pleasantness). Representations vary from low-resolution (e.g., orange vs blue, left vs right) to high-resolution (e.g., precise color, precise location), and from simple (e.g. object-color, object-object) to complex bindings (e.g., color-object-object-color). Complex high-resolution representations are expected to be the most hippocampally dependent (dark end of gradient), whereas simple low-resolution bindings will be the least hippocampally dependent (light end of gradient). The extent to which specific bindings will be dependent on the hippocampus or cortex varies across this gradient, and varies as a function of temporal delay inherent in various long-term memory (LTM), working memory (WM) and perception tasks.

Note that recollection is often found to reflect a threshold process such that individuals are able to recollect qualitative information for some items whereas recollection fails entirely for others (for a review see Yonelinas et al., 2010). The threshold finding might seem surprising given that recollection can vary with respect to complexity or resolution. However, the threshold finding tells us that there are some trials in which recollection fails to produce qualitative information, and others in which recollection is successful. Importantly, items that exceed the recollection strength threshold can differ in any number of ways, including their complexity and resolution (Yonelinas & Jacoby, 1995).

The associative aspect of recollection makes it particularly useful in supporting episodic memory discriminations, for example, deciding if an item was presented in a specific experimental context or remembering specific details of an event. Nevertheless, recollection is not unique in supporting memory for associations. That is, other types of memory, including familiarity-based recognition and implicit memory can support memory for simple associations, such as word-word or object-color associations (e.g., Diana, Yonelinas, & Ranganath, 2010; Diana, Yonelinas, & Ranganath, 2008; Giovanello, Keane, & Verfaellie, 2006; Quamme, Yonelinas, & Norman, 2007; Rhodes & Donaldson, 2008; Rhodes & Donaldson, 2007; Graf & Schacter, 1985; Schacter & Mcglynn, 1989; Musen & Squire, 1993; Yonelinas, Kroll, Dobbins, & Soltani, 1999). For example, recognition memory for word-color associations and word-word associations can be supported by familiarity under conditions in which the two components (i.e. the word and the color, or the two words) are encoded as a single unified object (e.g., “an elephant that is red”) rather than being processed as two separable aspects of an event (e.g., “an elephant beside a red stop-sign”) (e.g., Diana et al., 2008; 2010). Similarly, implicit memory for word pairs (as measured on word association tasks) can be observed when pairs are treated as single items (Graf & Schacter, 1985; Schacter & McGlynn, 1989), and memory for word-color associations can be observed in implicit word naming tasks when words are presented in colored fonts (and thus the color becomes a bound feature of the item; Musen & Squire, 1993). Thus, tests of associative memory do not serve as process-pure measures of recollection, because other memory processes can also support simple associative learning. The unique aspect of recollection is that it is capable of supporting memory for complex, multifaceted bindings of high-resolution information, so tests of simple binary associative memory are only partially dependent on recollection.

The Hippocampus is Critical for Recollection in Long-Term Memory

Patient studies of long-term memory have established that the hippocampus is critical for recollection. Damage to this MTL structure leads to selective deficits in recollection, and does not impair familiarity-based recognition (e.g., Aggleton et al., 2005; Bastin et al., 2004; Bird, Vargha-Khadem, & Burgess, 2008; Brandt, Gardiner, Vargha-Khadem, Baddeley, & Mishkin, 2008; Jager et al., 2009; Quamme, Yonelinas, Widaman, Kroll, & Sauve, 2004; Peters et al., 2008; Turriziani, Serra, Fadda, Caltagirone, & Carlesimo, 2008; Yonelinas et al., 2004; but see Wais, Wixted, Hopkins, & Squire, 2006). In contrast, damage that includes both the hippocampus and the surrounding MTL leads to deficits in both recollection and familiarity (e.g., Verfaellie & Treadwell, 1993; Knowlton & Squire, 1995; Blaxton & Theodore, 1997; Schacter, Verfaellie, & Pradere, 1996; Schacter, Verfaellie, & Anes, 1997; Yonelinas, Kroll, Dobbins, Lazzara, & Knight, 1998). Additional evidence linking the hippocampus to recollection comes from studies examining the effects of damage to the fornix, a major fiber tract connecting the hippocampus to the thalamus. Several studies examining patients with fornix lesions have indicated that these patients exhibit selective recollection deficits (e.g., Carlesimo et al., 2007; Gilboa et al., 2006; Vann et al., 2009). Moreover, fornix white matter microstructural integrity, as measured with diffusion weighted imaging, is correlated with recollection, but not familiarity (Rudebeck et al., 2009).

In addition to lesion studies, reductions in hippocampal volume in healthy aging are associated with declines in recollection, but not familiarity (Yonelinas et al., 2007). Conversely, differences in cortical volume within the entorhinal/perirhinal cortex are related to familiarity, but not recollection. A similar double dissociation was reported in a study using a source memory procedure to estimate recollection and familiarity (Wolk, Dunfee, Dickerson, Aizenstein, & DeKosky, 2011). Further support for these results comes from a patient with damage to the perirhinal cortex that did not impact the hippocampus, who exhibited a selective deficit in familiarity, but preserved recollection (Bowles et al., 2007; Martin, Bowles, Mirsattari, & Köhler, 2011).

Similar double dissociations linking recollection to the hippocampus and familiarity to the perirhinal cortex have been reported in the neuroimaging literature (for reviews, see Diana, et al., 2007; Eichenbaum et al., 2007; Skinner & Fernandes, 2007; Wais, 2008), and have been well-supported by lesion and neurophysiological studies in rats and nonhuman primates (for reviews see Eichenbaum et al., 2007; Eichenbaum, 1992; Aggleton & Brown, 1999).

The Role of Recollection in Working Memory and Perception

Studies of recollection have focused largely on tests of long-term memory, in which there is a delay of minutes to days between the initial study phase and the test phase. However, if recollection depends on the creation of high-resolution bindings, then it should be possible to find evidence for these bindings even under conditions that do not involve a long delay. For example, in working memory tasks where individuals must actively maintain information over a period of a few seconds, there is growing behavioral evidence that recollection- and familiarity-like processes also contribute to performance (e.g., Yonelinas & Jacoby, 1995; Feredoes & Postle, 2010; Goethe & Oberauer, 2008; McElree & Dosher, 1989; Oberauer, 2005; Öztekin & McElree, 2007; for related electrophysiological work see Danker et al., 2008). In one such study, individuals studied a list of 4 to 8 letters then were given a yes/no recognition test immediately afterwards, and the process dissociation procedure (Jacoby, 1991) was used to separate the contribution of recollective search and automatic retrieval processes (Yonelinas & Jacoby, 1995). A small set of letters was used repeatedly across the experiment such that all of the letters were highly familiar within the experimental context and the task required subjects to indicate if the test item was in the most recent study list. Set size and response speed were found to influence the controlled search process, but to leave automatic processes unaffected. These results suggest that working memory reflects the operation of two functionally separable processes, and the results are similar to those seen in studies of long-term recognition in which list length and response speed influence recollection but not familiarity (e.g., Yonelinas & Jacoby, 1994).

In another working memory paradigm, Oberauer (2005) found that aging was related to deficits in the recollection of item–context bindings rather than familiarity, and that measures of working memory capacity were directly related to the efficiency of recollection, but not of familiarity. These results also parallel those seen in long-term recognition memory where it has been shown that aging selectively impairs recollection and leaves familiarity relatively preserved (for a reviews see Light, Prull, La Voie, & Healy, 2000; Spenser & Raz, 1995; Koen and Yonelinas, under review).

Whether recollection- and familiarity-like processes contribute to perception is less well studied, but there is some evidence that a dual-process distinction may also be necessary. For example, Jacoby and colleagues have shown that in visual perception tasks, individuals can base their responses on consciously controlled or more automatic processes (e.g., Debner & Jacoby, 1994). In addition, in a recent series of studies, Mariam Aly and I (Aly & Yonelinas, 2012) examined same/different visual discriminations made to pairs of objects or scenes in healthy individuals in order to characterize the processes involved in visual perception (see Figure 2A). On each trial, individuals were presented with two items and indicated whether they were the same or different using a 6-point confidence scale. On some trials, the two items were identical, whereas on other trials, one item was slightly altered. The confidence data were used to plot receiver operating characteristics (ROCs), which are functions that relate the hit rate to the false alarm rate for each level of confidence. These ROCs were used to examine the processes underlying overall performance. Across experiments, we examined presentation rates ranging from 180 ms to 1.5 seconds, and various materials including scenes, faces, fractals and simple objects; all of these conditions led to remarkably similar results.

Figure 2.

(A) An illustration of the ‘same/different’ perceptual discrimination task. Individuals indicate their confidence that the scenes are the same or different. The two scenes are either identical or have been slightly altered. (B) Receiver operating characteristics for controls and patients with medial temporal lobe damage (left), along with estimates of state and strength-based perception in each group (right). Hippocampal damage reduced strength-based perception, but did not impact state-based perceptual responses (Aly, Ranganath, & Yonelinas, 2013).

Analysis of the ROCs revealed that perceptual judgments were associated either with a discrete state in which individuals became consciously aware of specific details that differentiated the two similar images, or assessments of a strength signal reflecting the degree of relational match or mismatch between pairs of stimuli (i.e., the individuals reported that the two scenes just seemed different but could not report how they were different). State- and strength-based perception were functionally independent, in that state-based perception played a larger role in performance when specific, local details differentiated pairs of stimuli (e.g., part of an object was added or modified), while strength-based perception played a larger role in performance when stimuli differed in relational/configural information (e.g., one image was expanded or pinched in, as in Figure 2A). Moreover, these functional differences were accompanied by different subjective experiences; subjective reports of state-based perception were associated with access to local, specific details, whereas subjective reports of strength-based perception were associated with a general feeling of overall match/mismatch. In addition, alternative models based purely on strength information such as the unequal variance signal detection model (Swets, Tanner, & Birdsall, 1961) were found to provide insufficient accounts of these findings.

Whether the sets of processes contributing to working memory, perception and long-term memory are related in any direct way to one another is not yet known. However, the evidence is now quite clear that in each of these domains there are separable processes contributing to overall performance. Moreover, there are some striking similarities in the functional nature of these different processes across domains, suggesting that it is worth considering the possibility that there may be some common underlying processes. One such possibility that I consider here is that the hippocampus may contribute to these perception, working memory, and long-term memory tasks by virtue of its role in the generation and utilization of complex high-resolution bindings.

When is the Hippocampus Critical for Working Memory?

Many early studies examining amnesic patients with MTL damage showed that working memory was normal as measured on tasks such as digit span (e.g., Scoville & Milner, 1957) and as indexed by normal recency effects in word recall tasks (e.g., Baddeley & Warrington, 1970). In general, working memory for single items such as digits, words, visual locations and fractals is found to be well-preserved in amnesia (e.g., Holdstock, Gutnikov, Gaffan, & Mayes, 2000; Scoville & Milner, 1957; Drachman & Arbit, 1966; Warrington, 1981; Warrington & Baddeley, 1974). However, more recently there has been a focus on working memory for associative information, and more of a focus on examining patients with relatively selective hippocampal lesions. As described below, a number of these studies have shown that hippocampal patients exhibit deficits on associative working memory tasks, but paradoxically, other studies that have used similar procedures have found no evidence of a deficit. A careful examination of the existing literature suggests that a key to understanding when working memory will be impaired in hippocampal patients is knowing whether the test requires complex high-resolution bindings.

One instrumental study that implicated the hippocampus in working memory was by Olsen et al., (2006; also see Olson, Moore et al., 2006) who presented a series of 3 simple objects in different locations, and then after a 1 or 8 second delay tested recognition memory for objects, locations or object-location pairings. Hippocampal patients performed poorly at the long delay condition, arguably because of impairments in long-term memory. Most critically, they were impaired at the 1 second delay for the object-location pairs, but were unimpaired on memory judgments for simple objects and locations. These results suggest that the hippocampus plays a role when the working memory task requires relational or associative binding, and that it is not involved when the task requires memory for simple items. Note that several studies examining working memory in patients with more extensive MTL damage have also reported evidence of a disproportionate deficit in associative compared to item tasks (e.g., Kinke et al., 2008e.g., Kinke et al., 2011; Olsen et al. 2006; Ryan & Cohen, 2004; for related neuroimaging results see Ranganath & Blumenfeld, 2005; Hannula & Ranganath, 2008).

The associative account of hippocampal function in working memory, however, has been challenged by several subsequent studies showing that hippocampal patients exhibit normal working memory even when the task requires the retrieval of associations (e.g., Baddeley, Allen, & Vargha-Khadem, 2010; Annette Jeneson, Mauldin, & Squire, 2010; Jeneson, Wixted, Hopkins, & Squire, 2012; Shrager, Levy, Hopkins, & Squire, 2008). For example, patient Jon, who has selective hippocampal damage, was tested on a series of working memory tasks assessing memory for colors, shapes, color-shape associations, spatially separated color-shape associations, and associations between objects and spoken color names (Baddeley et al., 2010; also see Baddeley, Jarrold & Vargha-Khadem, 2011). Jon performed normally on all of these tests, suggesting that there are conditions in which the hippocampus is not critical for item or associative working memory. The finding that color-location memory was normal might be explained because the colored objects were treated as single units and thus might not require true associative memory (Cohen, Poldrack, & Eichenbaum, 1997; Diana et al., 2008; Yonelinas, 1999). However, associative memory was also unimpaired when the associated features were not from a single object (e.g., separated color-location and object-spoken color name tests).

One potential concern with this study was that Jon had hippocampal damage early in life, and thus his preserved performance might reflect neural reorganization. Other studies, however, have shown preserved associative working memory in patients suffering hippocampal damage later in life (e.g., Jeneson et al., 2012; Shrager et al., 2008). For example, in a change detection task, individuals were briefly presented with an array of 1 to 6 colored squares, then after a 1-8 s delay were presented with a second array, to which they indicated whether a specified square had changed color (Jeneson et al., 2012). At the longer delays, the hippocampal patients were impaired, but at the 1 second delay the hippocampal patients performed normally across all set sizes, indicating that the hippocampus was not necessary for associative working memory in this task. Importantly, preserved performance was observed across variations in set size and thus across a range of performance, indicating that the normal performance in the patients could not be attributed to ceiling effects. Furthermore, the same patient group was tested on an object-location working memory task like the one developed by Olsen et al (2006), and was found to perform normally in the 1 second delay condition (Shrager et al., 2008), further showing that hippocampal damage does not always lead to associative working memory deficits.

These studies suggest that although hippocampal damage can lead to a more pronounced deficit in associative than item-based tests of working memory, it is not simply that the hippocampus is necessary for working memory tasks that require binding or associative information because not all tasks that require binding are disrupted by hippocampal damage. One reason for the inconsistent findings may be that the tasks did not draw heavily enough on binding of high-resolution information. In support of this idea, recent studies have indicated that the hippocampus does play a critical role in working memory for object-location associations when individuals are required to remember high-resolution location information, rather than simply indicate if a change occurred between the study and test arrays. One such study presented individuals with an array of 1 to 7 objects and then after a 1 second delay had them place objects in their precise studied locations (Jeneson, et al., 2010; also see Watson, Voss, Warren, Tranel & Cohen, 2013). Hippocampal patients were impaired even with set sizes as small as 2, 3 and 4 items (Experiment 2, although note that no statistical comparisons were provided). Moreover, the working memory errors that the patients made were not random, but rather they were often small displacement errors, in the sense that the patients were able to remember the approximate locations of the objects but they could not do so as precisely as healthy controls. The results indicate that hippocampal damage did not entirely eliminate working memory for object-location associations, but it did lead to less precise memory.

In addition, Warren et al. (2010) examined working memory for simple shapes, and required individuals to remember precise visual information, for example, the precise color and the exact object orientation. The patients were numerically impaired at the task, and eye movements were abnormal in the hippocampal patients, suggesting an involvement of the hippocampus in working memory tasks that require high-resolution information.

Further evidence for the involvement of the hippocampus in high-resolution binding comes from working memory studies for complex scenes which have quite consistently revealed that hippocampal damage impairs performance. For example, Hannula et al. (2006) presented a series of complex indoor scenes in which some scenes were repeated after either 0, 4 or 8 intervening scenes, and individuals had to indicate if an object within the scene was in its original position or if its position was altered. They found that even at the immediate repeat condition, hippocampal patients were significantly impaired. Similar deficits were also seen in a second experiment that tested associative memory for face-scene pairs (Hannula et al., 2006). Similarly, Jeneson et al. (2012) examined associative working memory performance in an object-scene binding task and reported a small but nonsignificant deficit in the patients in the immediate repeat condition. Finally, Hartley et al. (2007) conducted a working memory test in which a complex topographic image was followed by four similar looking images and individuals were required to identify the image that was consistent in spatial layout (but from a different view) to the studied item. The hippocampal patients were found to be significantly impaired at this task, further verifying that the hippocampus is critical for associative working memory for scenes.

The fact that hippocampal damage leads to impairments in working memory for scenes is consistent with the idea that the hippocampus is supporting complex high-resolution bindings. That is, scenes almost invariably involve multiple objects or features, and the tasks have required discriminations about high-resolution bindings. That is, in all of these studies, individuals were required to remember high-resolution associative information such as where an object was within a scene or how sets of objects were configured with respect to one another. Thus, the tasks can't be solved using simple low-resolution associations.

Further support for the notion that the hippocampus is involved in supporting high-resolution binding comes from a recent study conducted by WeiWei Zhang (in preparation; Society for Neuroscience Abstracts), in which we made use of a color wheel task developed by Zhang and Luck (2008). On each trial, individuals were first presented with 4 colored squares for 250 ms, followed by a 1 s blank screen, then a location cue indicating that the individual had to remember the color of the square that had appeared in that location. Individuals were presented with a continuous color wheel that they used to indicate the precise color of the cued square. The results of that study indicated that patients with hippocampal damage as well as those with more extensive MTL damage were impaired at overall measures of working memory accuracy. Critically, the deficits were found to reflect reductions in the precision of the memories rather than in the capacity to remember a given number of items. That is, the patients were no less likely to remember an object's general color, but when they did remember a color, their memory was significantly less precise than that of healthy controls.

These latter results indicate that the hippocampus is necessary for working memory when the task requires the retrieval of high-resolution information. In contrast, previous color-location working memory studies that did not require retrieval of high-resolution information (Jeneson et al., 2012) found that hippocampal patients were not impaired. The result is also important in showing that the hippocampus can play a critical role even with very simple stimuli (colored squares), and thus the hippocampus is not limited to supporting working memory for complex scenes. Finally, the results indicate that it is not the capacity of working memory that is disrupted by hippocampal damage, but rather the resolution of the information held in working memory.

Overall, the existing working memory results are consistent with the hypothesis that hippocampal damage leads to a deficit in the formation of high-resolution bindings. As expected, hippocampal deficits are more common in working memory tasks that rely on associative information rather than item or feature information (e.g., Olsen et al., 2006). Importantly however, the requirement to retrieve associations, in itself, is not sufficient to predict when the hippocampus will or will not be involved in a task, as there are conditions in which object-color associative working memory tests are not dependent on the hippocampus (Baddelely et al., 2010; Shrager et al., 2008). The hippocampus appears to play a critical role specifically under conditions in which complex high-resolution associative information is required, such as in working memory for complex scenes and tasks requiring memory for precise locations or precise object-feature information (Hartley et al., 2007; Zhang & Yonelinas, 2012; Jeneson et al., 2010).

When is the Hippocampus Critical for Perception?

A number of recent studies have indicated that the hippocampus is critical for making visual perception judgments (e.g., Lee, Barense, & Graham, 2005; Barense, Gaffan, & Graham, 2007; for a consideration of related neuroimaging and animal lesion results see Lee et al., 2012), but results from other similar studies have led to conflicting conclusions (Shrager, Gold, Hopkins, & Squire, 2006; Kim et al., 2011). As was the case with working memory, an examination of the perception results suggests that a key to understanding when perception will be impaired in hippocampal patients is knowing whether the test requires complex high-resolution bindings.

Perceptual tasks are distinguished from working memory and long-term memory tasks in the sense that individuals are required to make discriminations about items under conditions in which there is little or no delay between stimuli. For example, using an “oddity judgment” task, Lee et al., (2005; also see Barense et al. 2007; Hartley et al., 2007) presented individuals with small sets of scenes, objects, faces or colors, and required individuals to identify which item was different from the others in the set. They found that hippocampal patients showed impairments for scenes, but not objects, faces, or colors. In contrast, patients with larger MTL lesions including the perirhinal cortex were impaired on all materials except colors. Importantly, the scenes, faces and objects were constructed such that simple feature detection strategies would not be particularly useful in solving the task. For example, the scene views were all taken from slightly different angles such that all of the scenes were different from one another, but there was one scene in which the configuration of features was altered. Moreover, the tasks were designed to be equally difficult for controls so that any differences in performance between patients and controls could not be attributed to differences in task difficulty or ceiling effects.

Another task that has been used to examine perceptual abilities in amnesics is the “perceptual matching” task. For example, in one such task, a target scene or face was presented along with two choice items, one of which more closely matched the target item (Lee et al., 2005). Hippocampal patients were impaired for scenes but not faces, whereas patients with more extensive MTL damage were impaired on both scenes and faces (for similar results using a figure-ground segregation task see Barense, Ngo, Hung, & Peterson, 2011).

However, subsequent work with the perceptual matching task has suggested that hippocampal damage does not always lead to deficits on this task. For example, in other studies using very similar procedures, hippocampal patients were found to perform normally on perceptual matching tasks for scenes, faces and objects (Shrager, Gold, Hopkins, & Squire, 2006; Kim et al., 2011). There are several potential reasons for the discrepancies seen in these perceptual matching studies (e.g., Baxter, 2009; Jeneson & Squire, 2012; Kim et al. 2011; Lee et al., 2012). One of the critical factors appears to be that in the perceptual matching task, the materials have been such that accurate discriminations can be based on searching for a single feature that differentiates the two choice items from one another, rather than processing the overall conjunctive or relational information in the scenes (Baxter, 2009; Lee et al., 2012). Thus, if individuals adopt a feature search strategy, there is no need for the high-resolution representations of the hippocampus, and thus the hippocampus may not be involved in task performance. In contrast, if they adopt a configural matching strategy, this will require the representations of the hippocampus, and thus the task will be dependent on the integrity of this structure. In line with this interpretation, in the oddity judgment task described earlier (Lee et al., 2005; Hartley et al., 2007), the scenes could not be discriminated on the basis of a single feature, and the hippocampus did appear to be critical.

Although results from some of these initial studies suggest that the hippocampus plays a critical role in perception of scenes, but not objects, several studies have indicated that the hippocampus also plays a role in perceptual judgments for complex objects. For example, Warren, Duff, Tranel, and Cohen (2011) found that hippocampal patients were impaired at identifying line drawings of objects embedded within static visual displays, and at identifying objects that were presented in a fragmented form. In another study examining the detection of possible and impossible line drawings of geometric objects (Lee & Rudebeck, 2010), a patient with hippocampal damage exhibited abnormal eye movements compared to controls, suggesting that the hippocampus may play some role in normal perception of complex objects. Finally, using the oddity judgment task for complex objects, hippocampal patients were found to exhibit significant impairments (Knutson, Hopkins, & Squire, 2012).¹

Overall, the results from studies of perception suggest that the hippocampus does contribute to perception under some conditions, but not under others. One suggestion has been that if individuals adopt a feature search strategy in a perceptual matching task, and there are individual features that discriminate between the target items, then there is no need for the configural representations supported by the hippocampus, and thus the hippocampus may not be involved in task performance (Baxter, 2009; Lee et al., 2012). A shortcoming of these previous perception studies then, is that they have invariably measured perception as a unitary phenomenon, and have failed to separate what may be qualitatively different kinds of information or strategies that contribute to perceptual judgments. For example, as mentioned earlier (Aly & Yonelinas, 2012), perceptual discriminations can be based either on state-based identification of featural differences, or assessments of the strength of overall relational match information. If hippocampal damage selectively disrupts only one of these perceptual processes, then patients will be impaired in conditions that rely heavily on that one processes, but will be unimpaired in conditions that rely heavily on the other process.

In a recent study, we examined how the hippocampus contributes to these different types of perceptual judgments using a same/different confidence task for complex scenes (see Figure 2A) (Aly & Yonelinas, 2012; Aly, Ranganath & Yonelinas, 2013). Importantly, in that study, the scenes were manipulated not by adding or removing individual objects within the scene (which would allow a simple feature-matching strategy to support performance), but rather by expanding or contracting the images in such a way that altered the relational information within the scenes. Confidence responses were collected in order to plot ROCs, and the ROC data were used to estimate the contributions of state- and strength-based perception. Patients exhibited a significant reduction in overall perceptual sensitivity, as illustrated by lower ROCs (Figure 2B). In addition, the deficits were found to be specific to strength-based responses (reflecting relational match/mismatch information), while the state-based responses (reflecting use of local feature information) remained unaffected. The hippocampal involvement in this task was verified in a subsequent neuroimaging study in which a group of healthy individuals were scanned while taking part in a similar perceptual change detection paradigm (Aly, Ranganath & Yonelinas, 2013). The fMRI data showed that activation in the hippocampus linearly tracked strength-based perception and was not disproportionately increased for state-based perception. The results from these experiments indicate that the hippocampus is directly involved in making perceptual discriminations for scenes, and more specifically that the hippocampus tracks confidence in strength-based responses reflecting overall relational match between scenes, rather than supporting responses in which specific, item-level differences of the scenes are identified.

The results of that study highlight the importance of examining the separate processes underlying perception, using methods such as ROC analysis, rather than simply treating perception as a unitary phenomenon. The ROC analysis revealed that hippocampal damage does not disrupt high confidence perceptual responses. Rather, it selectively disrupts the ability to make lower confidence perceptual judgments associated with assessments of subtle relational changes. Thus, in perception, it appears that the hippocampus supports those responses in which we simply sense that the two images were altered, but for which we are unable to identify any specific feature that has been changed. If only binary same/different judgments were collected and individuals had adopted a strict response criterion, the results would have failed to show a hippocampal impairment in perception (i.e., the leftmost points on the ROCs in Figure 2B), whereas if they had adopted a more liberal criterion there would have been significant impairment (i.e., the midpoint of the ROCs). These latter findings show that studies that use binary same/different judgments may or may not find an impairment in patients' performance depending on the response criterion used by individuals.

In sum, the perceptual literature shows that perceptual impairments following hippocampal damage tend to occur in situations where high-resolution relational information is required to support performance, and impairments are less consistent when the task does not require this information. For example, perceptual impairments are more consistently observed for complex scenes than for objects, faces or colors (e.g., Lee et al., 2005). As discussed earlier, scenes are invariably complex and the tasks have often required judgments about high-resolution binding information, so these results are to be expected. Importantly, however, there is evidence that hippocampal damage does not always disrupt scene perception, and this occurs specifically in those situations in which scene discriminations can be based on the detection of single feature differences (see Baxter, 2009). In addition, hippocampal damage can disrupt perceptual judgments for non-scene materials such as complex objects, as long as the task requires retrieval of high-resolution configurations of object features (e.g., Warren et al., 2012). Finally, the perceptual impairments that are observed in hippocampal patients appear to reflect selective deficits in strength-based judgments that involve assessments of relational match, rather than state-based judgments involving the identification of specific feature changes (Aly et al., 2013).

Relating the high resolution account of hippocampal function to earlier models

Overall, the results from studies of patients with hippocampal damage provide support for the idea that the hippocampus supports the generation and utilization of complex high-resolution bindings. In addition, these bindings appear to be useful in supporting not only long-term memory, but working memory and perception.

The high-resolution account of hippocampal function presented here builds on earlier dual-process models of recollection and familiarity (e.g., Atkinson & Juola, 1974; Yonelinas 1994). The claim that recollection is involved in generating and retrieving complex high-resolution bindings is based on one of the core assumptions of these models, which is that recollection reflects the retrieval of qualitative information about prior events. It is on the basis of this idea that one expects that the complexity and high-resolution aspects of different tasks should be so critical. Equally important, the idea that recollection is dependent on the hippocampus is a critical component of many dual-process frameworks (e.g., Aggleton & Brown, 1999; Eichenbaum, Otto, & Cohen, 1994; Norman & O'Reilly, 2003; Yonelinas, 2002).

The high-resolution approach is closely related to models that assume that the hippocampus is critical in forming relational memories, or in binding aspects of an episode together (e.g., Cohen, Poldrack & Eichenbaum, 1997; Eichenbaum, Ranganath & Yonelinas, 2007). The current approach, however, more heavily emphasizes the complex nature of the relations or bindings that make up an event rather than focusing on simple associations. But even more importantly, the current claim is not that the hippocampus is necessary for any type of relational information or association (cf., Konkel, Warren, Duff, Tranel, & Cohen, 2008), but rather that it will be particularly critical for high-resolution associations. A simple relational or associative account in and of itself is not adequate to account for the existing literature because it is not the case that hippocampal patients have deficits on tasks that require relations or associations per se; they often perform normally on associative tests that require only low-resolution associations.

The idea that the hippocampus supports high-resolution binding is also broadly consistent with current neurobiological and computational models of hippocampal function (e.g., Norman & O'Reilly, 2003; Kesner, 1998; Rolls, 1996). For example, the Complementary Learning Systems (CLS) model (Norman & O'Reilly, 2003) assumes that the pattern separation abilities of the dentate gyrus and CA3 subfields of the hippocampus allow for the rapid formation of distinct representations for complex configurations. Such representations are essential in long-term memory in that they allow the system to pattern complete unique episodic information based on partial cues. I would argue, however, that the same representations are available to influence performance even in tasks that do not impose a long-term memory delay, and thus they could contribute to working memory and perception.

In fact, it has been suggested that CA1 serves as a comparator of stored representations in CA3 with the incoming signals from entorhinal cortex (e.g., Duncan, Ketz, Inati, & Davachi, 2011; Hasselmo & Wyble, 1997; Olsen, Moses, Riggs, & Ryan, 2012; Vinogradova, 2001). As such, it is in an ideal position to derive a global match signal between what is currently being perceived and what was just presented. Preliminary simulations with the CLS model (Elfman, Aly & Yonelinas, in preparation) have suggested that this model's representation of hippocampal functioning is able to account for some critical findings from both memory and perception studies. For example, the model accounts for the fact that in long-term memory, the hippocampus exhibits a thresholded signal in which pattern completion sometimes fails completely, whereas in perception tasks (where items are immediately repeated) pattern completion rarely fails and the hippocampus provides continuously-graded strength signals. Future work will be necessary to determine if the same model parameters are capable of accurately accounting for other aspects of the working memory and perceptual literature.

Importantly, the current approach suggests that the hippocampus is not necessary in all long-term memory, working memory or perception tests. Rather, it is only when the tasks rely on complex high-resolution associative information that the role of the hippocampus becomes obvious. For example, in long-term memory tasks where performance can be based largely on familiarity (e.g. item recognition tests), the role of the hippocampus will be less obvious than in tests like source memory where recollection becomes more relevant. Similarly, the hippocampus is not expected to be involved in all working memory or perceptual tests, but its role is expected to become more obvious when complex high-resolution binding is required.

Overall, the extent to which the hippocampus is critical in different tasks increases as the complexity and high-resolution demands of the tasks increase, as illustrated in Figure 1. In addition, however, the extent to which the hippocampus is critical depends critically on whether the tasks involve long-term memory, working memory, or perception. That is, in long-term memory tests, simple associations typically do require the hippocampus (e.g., simple source memory tests), whereas in working memory, simple associations do not necessarily require the hippocampus. In addition, in perception tests, the hippocampus is most often involved only when the materials are highly complex (e.g., scenes). This likely reflects the fact that the hippocampus is unique in supporting complex high-resolution bindings over long delay periods, but that as the delay is decreased, other cortical processes can begin to support performance, at least temporarily. For example, visual binding mechanisms might support color-location associations over brief delay periods (Seymour, Clifford, Logothetis, & Bartels, 2010; Treisman & Gelade, 1980), thus reducing the demand on the hippocampus in perception and working memory tasks. However, as the delay increases, the cortical regions that support these perceptual bindings are no longer able to maintaining those representations, presumably because of interference from other sensory input. Thus, the hippocampal contribution to performance will become more apparent as the temporal delay increases. Similarly, other brain regions such as the perirhinal cortex can support perception of configural objects and faces (Lee et al., 2005), and so in these cases the role of the hippocampus can be relatively reduced. Thus, in understanding the role of the hippocampus it remains important to distinguish between long-term memory, working memory and perception, because the extent to which the hippocampal representations contribute to performance varies as a function of whether other regions can support performance. After long delays there are few other cortical regions that can support memory for complex high-resolution bindings, but with shorter delays various other regions can support such bindings (Figure 1).

The current approach is quite different from several traditional memory systems models, particularly those that assume that the MTL is specialized for long-term memory whereas other cortical systems are responsible for working memory and perception (e.g., Squire, 1994; Schacter & Tulving, 1984; Tulving, 1982). The current theory suggests that the hippocampus is not just a component of a specialized long-term memory system, but rather it supports high-resolution bindings that are important for long-term memory as well as working memory and perception. Moreover, regions outside the hippocampus are assumed to be critical for other forms of long-term episodic/declarative memory such as familiarity.

The current approach also differs from accounts that assume that the hippocampus plays a selective role in spatial processing (e.g., Lee et al. 2005; O'Keefe & Nadel, 1978). Although space/scene information often does rely on complex high-resolution associative information, and working memory and perception tasks that involve scenes often are hippocampally dependent, the role of the hippocampus is not expected to be limited to tasks that involve spatial materials. In line with these expectations, the hippocampus is necessary for working memory and perception tasks that involve colored squares (Zhang et al., 2012), shapes (Lee & Rubebeck, 2010; Warren et al., 2012) and objects (Jeneson et al., 2010; Knutson et al. 2012; Olsen et al., 2006; Watson & Cohen, in press; Warren et al., 2012). Similarly, in tests of long-term memory, amnesic patients exhibit pronounced deficits for nonspatial materials such as abstract words, sounds, sequences and odors (e.g., Konkel & Cohen, 2008, 2009; Milner, 1972; Squire, 1987; Mayes, 1988).

The high-resolution account of hippocampal function may also provide a new way of viewing the debate about whether the hippocampus is involved specifically in allocentric spatial coding or is more generally involved during relational coding (Nadel, 1991; Cohen & Eichenbaum, 1991). Hippocampal lesions typically do not impair performance on egocentric tasks, such as navigating to a brightly colored cue card over the goal location (Morris et al., 1983), which involve only simple bindings (i.e., self-position to a single landmark). In contrast, lesions to the hippocampus can impair function on allocentric tasks, or behaviors involving associating two locations with a third one, which would be consistent with the idea of having to bind a complex set of associations together (e.g., landmarks and path-integration information, Morris et al., 1983). Importantly, however, hippocampal lesions, in humans at least, do not impair all forms of allocentric memory, particularly ones that can be solved with relatively simple bindings between objects and a to-be-learned location (Bohbot et al., 2002; Bohbot et al., 1998). Furthermore, fMRI studies in humans suggest that it is the degree of spatiotemporal binding, rather than whether the task is allocentric or egocentric, that determines hippocampal involvement during spatial navigation (Zhang & Ekstrom, 2012). Consistent with these findings, the current proposal does not limit the hippocampal contribution to allocentric tasks, rather it is only if the task requires high resolution information and complex bindings that it is sure to be involved.

One important issue that we have not focused on in the current paper is how the current ideas relate to semantic long-term memory or remote memory. On the surface, it would seem that individuals can also develop high-resolution semantic or remote memories. For example, we may be able to remember very precise qualitative information about our childhood homes or grade-school friends. This information would appear to be both highly complex and high-resolution. Whether these forms of memory are hippocampally dependent is not yet clear (e.g., Moscovitch, 2008; Squire & Bayley, 2007). One critical difference may be that this information is encoded gradually over time, whereas in the paradigms discussed above, the hippocampal learning that takes place is rapid and based on a single presentation. How high-resolution sematic representations emerge and how they are related to recollection and the hippocampus is an important topic of future study.

The high-resolution account opens up a number of other important questions that will need to be considered, and it leads to several novel predictions that can be directly tested in future studies. For example, it is worth reconsidering some of the earlier findings that suggested that hippocampal damage did not impair working memory and perception, and ask if those tests may have shown hippocampal involvement if only they had required higher resolution associative information. For example, would the hippocampus play a role in phonological working memory tasks in which there is a requirement for high-resolution retrieval (e.g., recall of the precise information about the articulation of specific phonemes)? Similarly, would the hippocampus play a critical role in recency effects when the recall test requires high-resolution discriminations (e.g., free recall of precise color information using a color wheel)?

Another crucial question for future studies will be to determine how the processes that contribute to long-term memory, are related to the processes found to support working memory and perception. The hippocampal results, that have been the focus of the current paper, provide some important insights into this question, but they open up several additional questions. For example, in long- term memory the hippocampus is critical for recollection but not familiarity, whereas in working memory, the hippocampus is necessary when the task requires high-resolution bindings, but it is often not involved when the task involves simple item decisions or low-resolution bindings. Presumably the working memory tasks that require memory for high-resolution bindings involve a form of recollection that is in some way similar to that involved in long term memory. However, can recollection in a working memory task occur without the contribution of the hippocampus? And if so what are the brain regions that give rise to recollection in these conditions? The existing results are not yet entirely clear, but there is indirect evidence that recollection in working memory can arise independently of the hippocampus. For example, as described earlier, the hippocampus does not appear to be required in working memory tasks for simple items. However, behavioral studies have suggested that performance on these types of working memory tests involves two processes that are similar to recollection and familiarity (e.g., Yonelinas & Jacoby, 1995). So, at these short delays, it seems that recollection can arise in the absence of the hippocampus. Studies that directly examine the contribution of recollection and familiarity to working memory tasks for high and low-resolution information will be critical in determining the relationship between long-term and working memory.

In addition, it is worth considering cognitive domains other than perception and working memory that might also benefit from having high-resolution bindings. For example, certain language tasks would appear to rely on developing a complex mental model of the text that in some cases does require the maintenance of quite precise information. It would be interesting to know if the hippocampus plays a critical role under those conditions (for related work on language see Duff & Brown-Schmidt, 2012). Similarly, reasoning tasks sometimes require the construction of highly complex representations that may also benefit from a hippocampal contribution.

Other questions that are raised by the current proposal include whether expertise can influence whether a stimulus characteristic is processed as a high or low-resolution feature. Presumably, phonemes in one's own language no longer require high-resolution processing, because they can be categorically perceived, but unfamiliar phonemes might. Would the hippocampus be necessary for unfamiliar high-resolution stimuli, but become less involved as the materials become familiar? In addition, to what extent is the contribution of the hippocampus to performance on various tasks contingent on individuals' strategies? In a task that can be based on the hippocampus or on other cortical representations, can individuals orient more to one than the other? What are the strategies that affect the extent to which different processes are utilized?

Conclusions

The finding that performance on working memory and perception tasks can be disrupted by hippocampal damage argues against models that view the role of the hippocampus as being limited to long-term episodic or declarative memory, and shows that the functional role of this region is much more general. However, what is the role of the hippocampus if not just to support long-term memory? I suggest that the role of the hippocampus is to support the generation and utilization of complex high-resolution bindings, and this function will be important across the domains of perception, working memory, and long-term memory. The hippocampus is therefore responsible for representing moments in time that join together the many qualitative aspects that make up the complex events we perceive and remember. In tests of long-term memory, those representations will be essential to support subsequent recollection of prior episodes, but will not be necessary for supporting other forms of long-term memory such as familiarity-based recognition. In working memory and perception, those representations will be critical for the ability to perceive, maintain over the short term, and/or retrieve precise associative information. Traditional measures of working memory, such as digit span, require neither associative nor high-resolution information, and thus preserved performance following hippocampal damage is to be expected. In addition, perceptual judgments about single features, including features in complex scenes, can be supported by regions outside the hippocampus. It is specifically working memory and perception of complex high-resolution bindings that depends on the hippocampus, and this is true across objects, scenes, or even simple stimuli. Thus, the current approach suggests that the unifying principle behind the contribution of the hippocampus to long-term memory, working memory, and perception is the demand for the representation, maintenance, and retrieval of complex high-resolution bindings.

Research Highlights.

• Whether the hippocampus plays a role in working memory and perception is controversial

• I propose that the hippocampus supports complex high-resolution bindings

• Current patient literature supports this proposal

• Hippocampal damage impairs tasks that require complex high-resolution bindings

• Tasks requiring only low-resolution or simple associations are less impaired