Precision, binding, and the hippocampus: Precisely what are we talking about?

Abstract

Endel Tulving’s proposal that episodic memory is distinct from other memory systems like semantic memory remains an extremely influential idea in cognitive neuroscience research. As originally suggested by Tulving, episodic memory involves three key components that differentiate it from all other memory systems: spatiotemporal binding, mental time travel, and autonoetic consciousness. Here, we focus on the idea of spatiotemporal binding in episodic memory and, in particular, how consideration of the precision of spatiotemporal context helps expand our understanding of episodic memory. Precision also helps shed light on another key issue in cognitive neuroscience, the role of the hippocampus outside of episodic memory in perception, attention, and working memory. By considering precision alongside item-context bindings, we attempt to shed new light on both the nature of how we represent context and what roles the hippocampus plays in episodic memory and beyond.

When Endel Tulving first proposed the idea of episodic memory, he suggested that it involved a fundamentally different memory system from semantic memory, one’s memory for facts about the world. Episodic memory, he argued, hinged on the idea that memories for events share key components related to their embedding in time, thus changing the cognitive process by which we can access and work with such memories (Tulving, 1985; 2002; 2005). In particular, Tulving focused on three key components of such memories: their spatiotemporal uniqueness, our ability to travel back and forth through these memories (which he termed “mental time travel”), and the fact that we are aware of this process of moving through time when we review memories (termed “autonoetic consciousness”). Subsequent work attempted to connect Tulving’s ideas of episodic memory with prefrontal cortex and hippocampus, including HERA (Habib, Nyberg, & Tulving, 2003; Nyberg, Cabeza, & Tulving, 1996) and HIPER models (Lepage, Habib, & Tulving, 1998; Lepage, McIntosh, & Tulving, 2001). While Tulving’s ideas about episodic memory as a distinct memory system have received wide support in the literature and continues to influence how we conceive of the neural basis of episodic memory, key questions remain in terms of understanding the neural processes and representations underlying such memories.

An important characteristic of episodic memories, according to Tulving, involved binding an item-related signature with some kind of “source” from which the memory originated (also termed “context”). For Tulving at least, context was what distinguished a specific occurrence of an item, such as a word or picture, as unique from all the other occurrences of it (Wheeler, Stuss, & Tulving, 1997). One example would be recollection of what one was thinking when seeing the word “cat” in a list of words during encoding, such as “‘cat’ makes me think of Endel’s ‘cat’ that got lost in Davis” or “‘cat’ makes me think of my friend’s tabby.” Retrieval of contextual details was also critical to Tulving’s ideas about episodic memory because it allowed recovery of the unique encoding experiences associated with that word and thereby facilitated autonoetic consciousness.

Interestingly, temporal context, which has received considerably more interest and attention since the work of Tulving, was considered largely a by-product of “executive” functioning by the frontal lobe (Wheeler, et al., 1997). In this way, temporal context was not something intrinsic or specific to context but emerged from other aspects of cognition. Theoretical considerations of mental time travel as part of episodic memory similarly considered time as important but did not explicitly define how it might be represented or interleaved with such representations one might time travel through (Suddendorf & Corballis, 2007). We think that considering the nature of “context,” particularly its temporal nature, in significantly more depth will help to advance our understanding of episodic memory by allowing us to link Tulving’s ideas about episodic memory to more recent work on context, binding, and precision.

Like consideration of “context,” another area that has seen significant development since Tulving’s work is in the neural basis of episodic memory. Intriguingly, Tulving pointed out, somewhat presciently for our purposes here:

“An operating component of a system [e.g., episodic memory] consists of a neural substrate and its behavioral or cognitive correlates. Some components are shared by all systems, others are shared only by some, and still others are unique to individual systems. Different learning and memory situations involve different concatenations of components from one or more systems.” We think that the point that “some components are shared by all / some memory systems” in particular is important to help better understand what brain systems may be engaged and in what manner during episodic memory encoding and retrieval. As we will argue here, “precision” can be thought of as a more general property shared by nearly all memory systems, as more precise representations generally will be of higher fidelity and usefulness in many domains (e.g., visual and auditory). We provide a more detailed definition of precision, which includes both resolution and dimensionality (complexity), in a later section.

Finally, we will also argue that the “binding” function of episodic memory – associating a novel context with an item – is an operation on representations that may be partially or even uniquely supported by the hippocampus. The idea of binding thus includes both the operation of association and the idea that associated context must be high dimensional to be effective (Cowell, Barense, & Sadil, 2019). Precision and binding help to shed light on some important puzzles in memory research: in what ways does drifting context relate to episodic memory and what role does the hippocampus play in cognitive processing outside of episodic memory, such as perception and working memory?

Spatiotemporal context: Why is it important to episodic memory?

To successfully retrieve an item or other information that was encoded in episodic memory, one needs a cue specific to the encoding situation, which is often referred to as the context or source under which an item was encoded. For example, recovering the thoughts that we had when we encoded the word “cat” could provide sufficient information to cue recovery of the word “cat,” like the image of a tabby or other cats that got lost in Davis. In this way, context itself might not be unique, but could still be just good enough for recovering enough of a memory to remember the word “cat.” In this case, though, we would not think of this form of context as “episodic,” at least based on Tulving’s considerations, in that it does not necessarily index a unique instance of “cat” that was encoded at a specific time point. Tulving also acknowledged the importance of both space and time, and time in particular, in that time provides a unique code for potentially recovering a memory. As we will discuss shortly in more detail, it follows that space and time must be of sufficiently high precision to serve as a unique cue to retrieve the item paired with context.

In practice, remembering an item based on encoding it 3.4 seconds compared to 3.5 seconds seems unlikely as candidate for how we cue items that we encoded (Friedman, 1993, 2007). Although Tulving did not explicitly define how time might be stored as part of mental time travel, subsequent considerations suggested that recency effects and changes in events (such as temporal boundaries, an issue we will explore more shortly) could provide a means of doing this (Suddendorf & Corballis, 2007). In contrast, we think that a “drifting” representation of temporal context may be particularly important to how we represent time. In this way, “drifting” refers to the type of gradual change typical of time. Changes in spatial context or event boundaries, in contrast, provide a “shift” that can result in more dramatic dimensional changes in context. Shifting context in this way would be a more rapid type of change that results when you move from one location to another. Drifting and shifting context provide a means by which context is sufficiently different, and possibly unique, from all the other contexts that have been associated with an item and neighboring items.

To serve as an effective cue during retrieval, a context representation must be associated with an item during encoding, a process often referred to as binding, or spatiotemporal binding. Binding of items to a unique spatiotemporal context allows an episodic memory to be differentiated from the myriad of others we experienced in an experiment and/or in a day. We can think of binding “demands” increasing as a function of the dimensionality of the stimuli themselves, with context an example of a particular complex type of stimulus with multidimensional features (Yonelinas, 2013). As we have noted earlier, however, episodic memory, at least for veridical retrieval of the encoded stimulus content, should contain at least some unique temporal tags that relate to the encoded context (Polyn, Norman, & Kahana, 2009). It is important to think in more detail, then, about exactly what is context and why, to serve as an effective retrieval aid, must it be high-dimensional and unique? In other words, exactly what is being bound to the item representation in terms of context that allows recovery of spatiotemporal details?

One possible definition is that context is whatever the item is not. While this could include a coarse (low dimension, low resolution) representation of drifting time, this is not something one would directly perceive, and without some definition of time, such a definition does not provide much traction with Tulving’s conception of episodic memory. Another definition often used of context relates to how we represent space (Nadel & Willner, 1980). While space undoubtedly is a powerful cue for encoding and retrieving episodic memories (Hupbach, Gomez, Hardt, & Nadel, 2007; Robin, Wynn, & Moscovitch, 2016), the extent to which space alone allows us to distinguish one item from another, particularly if we are in the same location during an experiment, is somewhat doubtful. Space also suffers, unfortunately, from some degree of circularity in terms of how we think about it. As we have argued elsewhere, space is often defined as whatever a brain structure like the hippocampus does (Ekstrom & Ranganath, 2017) or how we imagine space to be when we remember locations (Ekstrom, Harootonian, & Huffman, in press). Although we provide a stricter definition of space later in this paper related to 2-D/3-D topological axes, space itself might be expected not to change significantly over repeated exposures. Thus, while space could provide the types of “shifts” important to context change, it would not provide a unique code necessarily for episodic memory.

Here, we think it is instead useful to adopt a more holistic definition of context that can include both internal and external events that drift and shift over time (Howard & Kahana, 2002; Polyn, et al., 2009; Watrous & Ekstrom, 2014; Yonelinas, Ranganath, Ekstrom, & Wiltgen, 2019). According to this definition, context changes gradually over time based on the properties of diffusion drift (Howard & Kahana, 2002; Long, Danoff, & Kahana, 2015; Polyn, et al., 2009), with external input, such as changing spatial location providing input that shifts the contextual representation. In this way, context can vary considerably during a period of even 45 minutes of an experiment involving encoding a list of words. Critically, the resolution and dimensionality, and its uniqueness in terms of temporal drift, will help to determine how well it can serve as a cue during retrieval. By binding each item on a list to this slowly drifting and changing context, then, unique associations form the basis for potentially recovering words or objects that were encoded. In this way, item-context bindings provide unique signatures for differentiating encoded information, with the challenge being the extent to which such bindings provide sufficiently different representations for them to be retrieved from all of the other bindings that occurred during that list learning episode.

Precision and context

Given the importance of a relatively unique contextual representations to allow differentiation from other memories, it seems reasonable to consider in more depth what one might mean with regard to “unique.” This is an instance in which we think considering the precision of contextual representations becomes particularly important. To differentiate an item on a list from all other occurrences of that item, there needs to be sufficient information embedded in the contextual representation to serve as an identifier for that item. This is where the idea of precision first becomes apparent when we consider episodic memories. We will later consider the importance of precision with regard to the hippocampus in a later section.

Precision is important in many different ways to context, although the extent to which context is differentiable from other aspects of drifting context does not necessarily have to relate to its precision. We can think of precision as involving both resolution and dimensionality (complexity), which are helpful to think about through two cases. A precise representation of context could involve a high-resolution and high-dimensional representation which drifts over time (Figures 1). A higher precision representation would have more elements (the matrix representing context has more elements), meaning that the resolution of itself, much like a picture of 512x512 vs. 1024 vs. 1024 pixels, would be higher. Additionally, such elements could be higher dimensionality (complexity) such that a decomposition of the elements that make up the matrix would reveal a higher number of linear-independent basis functions (Figure 1). This would be an instance, which we think would be relatively common for well-encoded and retrieved memories, in which resolution and dimensionality are correlated. Importantly, higher precision (resolution + dimensionality) would allow for a greater number of items to potentially be different, any of which could be usefully exploited for binding to an item.

Figure 1: Precision in context representations.

Four different possibilities for precision. Top left: high-resolution and high-dimensional representations, as shown both by the high-resolution image (matrix) representing the spatial environment and the principal components breakdown (inset), which indicates relatively high dimensionality (several components explaining significant variance). In contrast, as shown on the top right, the matrix is the same size (resolution) but is blurred and thus of lower dimensionality. On the bottom left, the same image but in black and white is of lower resolution but comparable dimensionality to the original image. Finally, a low-resolution, low dimension image involves a blurred version of the black and white image.

We could also imagine a situation in which a relatively low-precision contextual representation could nonetheless serve well in memory. For example, if we get up and move to a different place, the low precision vector would shift significantly, thereby providing differentiable representations at the times of the shift. Such significant changes in context likely relate to boundary effects in which either the narrative structure or high-order aspects of an experience change sufficiently to induce a sense of “shift” (Ben-Yakov & Henson, 2018; Zacks & Swallow, 2007). Under typical situations in which significant changes do not happen during the experiment, though, a high-dimensional, precise contextual representation would be most advantageous for encoding and retrieving memories. Therefore, under situations of extensive shift, a lower dimensional representation would suffice but when in the same spatial context (e.g., experiment room), a higher resolution representation is optimal.

Note that the critical component here is the extent to which individual elements of the matrix that make up context change over time, which can happen for a variety of different reasons. According to the original conceptualization of context in the temporal coding model, context itself steadily drifts in a time varying fashion based on diffusion drift properties such that more distant time steps result in less similar random drifts (Howard & Kahana, 2002; Long, et al., 2015; Polyn, et al., 2009). In the instances in which almost everything else stays the same in an experiment (e.g., minimal changes in internal and external states) each word in a list we are trying to learn becomes associated with a slightly different context vector in which only a small subset of elements change over time. Such drifting in temporal context helps explain primacy and recency effects in free recall as temporal context is different from the beginning and endpoints of the experiment, with overall less memory for words in the middle of the list due to the only slight changes in drifting context. In contrast, if the participant gets up and moves to a new room in the middle of the list, this will induce shifts in multiple differentiable elements (the dimensionality) of the context matrix, resulting in greater memory for the middle parts of the list (Polyn, et al., 2009).

Another interesting example to consider here is the method of loci, well known to serve as a mnemonic aid. Let’s consider a situation in which we have a particularly precise representation of locations of objects in our home, for example, we can readily point accurately to the locations of objects within our house. While such a representation would not change much once we have learned it well and can be thought of as primarily semantic, it is useful to think how such a representation could nonetheless serve as an aid for episodic memory. Specifically, with the method of loci, we typically imagine placing words in spatial locations as each word is read to us in the list (Bower, 1972; Yates, 1966). If we imagine our apartment in color, this would refer to a high-resolution type of representation as there are many elements, like couches and chairs that are brown and beige. If we think of our apartment as involving unique dimensions, like the positions and orientations of the furniture, the color of the walls, and smells from the kitchen, then this would involve higher dimensionality (complexity). Then, if we use our kitchen as a scaffold on which to remember items a list of words, a more precise representation will in turn be more useful for encoding and retrieving words because we have increased the number of dimensions over which we are binding with an object we are trying to remember. In this case, interestingly, it is the mental movement within our house that provides for binding with context, and the richer the context, the more the dimensionality and the more effective the binding.

We can also consider the precision of temporal representations and the role that precision would play. The degree to which we can better differentiate a given moment in time from another could be useful for binding to a specific item. For example, we recently demonstrated that a precise representation of one’s mental “lifeline” could serve comparably to a spatial scaffold for anchoring episodic memories (Bouffard, Stokes, Kramer, & Ekstrom, 2017). What about representing time in the moment? While participants can readily judge temporal durations, (e.g., was the item on the screen for 4 vs. 5 seconds (Ekstrom & Isham, 2017; Meck, Church, & Matell, 2013)), it is not clear that time drifts sufficiently within several seconds to differentiate one item from another. While there is certainly evidence for neural representations of time (Kraus, Robinson, White, Eichenbaum, & Hasselmo, 2013), with neural representation of drifting temporal context contributing to episodic memory encoding and retrieval (Manning, Polyn, Baltuch, Litt, & Kahana, 2011), it remains unclear whether we use such precise representations of time to differentiate individual items on a list. In this way, context “shifts” like event boundaries and changes in spatial context are likely to be more effective in binding than changes in temporal context alone, which do not provide the same degree of dimension change.

How does “precision” emerge from representation of context?

While we have considered the computational properties of temporal context to involve diffusion drift (Howard & Kahana, 2002; Long, et al., 2015; Polyn, et al., 2009), we have not defined the spatial aspects of spatiotemoral context. Past work considering spatial context has argued that such representations should be metric (e.g., Bellmund, Gardenfors, Moser, & Doeller, 2018). Like a piece of graph paper, such a representation contains an underlying organization that follows Euclidean rules of geometry, for example, symmetry (AB=BA) and that the angles of a triangle must equal 180°. With a m etric representation, precision is a particularly easy dimension to imagine, as it would simply relate to the scaling of the graph paper (finer = higher resolution) or the noise of the representation (less noise = more unique elements = higher dimensionality). In this way, there appear to be many commonalities between how we might think about spatial navigation, contextual representation, episodic, and semantic memory (Buzsáki & Llinás, 2017; Buzsaki & Moser, 2013). The issue here, however, is that storing a 4-D representation of space time, particularly a high-resolution one, would require an enormous amount of “disk” space. It is also not clear how internal changes, such as one’s mood, could be mapped onto some kind of “metric.”

Instead, we do not think it is necessary to assume any specific kind of metric or underlying organization to spatial context, and instead, assume that this can also vary. There may be instances in which a nearly metric representation of our kitchen could be useful for encoding items. We think, however, that this aspect of context is often topological, in other words, things relatively more similar/closer in space are stored nearby but lack any specific metric on which they are encoded (Ekstrom, et al., in press; Ekstrom, Huffman, & Starrett, 2017; Warren, 2019). This is shown in Figure 2 in which the exact spacing of objects in topological spatial context do not matter as long as the relative positions are preserved. Notably, items metrically or topologically spaced will have the similar precision and be of similar efficacy in binding and representing items in memory. Importantly, topological space allows for the idea that the scaffold itself may be of equal dimensionality and instead that what matters would be the dimensionality of different elements (e.g., how many and what background features) we store rather than their position in the context.

Figure 2: Comparison of metric and topological representations.

Top panel shows the original spatial context the participant learned. A metric representation involves the same physical arrangements of landmarks (A-G), preserving their spacing. A topological representation involves preserving the relative positions of the landmarks. Note that both metric and topological representations are similar dimensionality and resolution here although the topological representation would appear to involve less need for irrelevant information.

Neural basis of contextual precision and binding within the hippocampus

Binding, or the process of associating a high-dimensional context with an item during encoding, depends primarily on the hippocampus, consistent with arguments from amnesia, neuroimaging, and other methodologies (Davachi, Mitchell, & Wagner, 2003; Davachi & Wagner, 2002; Diana, Yonelinas, & Ranganath, 2007; Eichenbaum, Yonelinas, & Ranganath, 2007; Hamann & Squire, 1997; Insausti, Annese, Amaral, & Squire, 2013; Lee, Yip, & Jones-Gotman, 2002; Lepage, et al., 1998; Milner, Corkin, & Teuber, 1968; Scoville & Milner, 1957; Sherman, et al., 2011; Stark & Squire, 2000; Yonelinas, Kroll, Dobbins, Lazzara, & Knight, 1998). As argued above, we can think of binding as involving an association between the context present during item encoding and the item itself, with context assumed to be sufficiently precise and unique compared to other stored contexts to allow recovery during retrieval – assuming sufficient cues can be recovered. Thus, for veridical recall of the item, we would expect a relatively high-match between the retrieved and encoded context, similar to the idea that encoding and retrieval involve a certain degree of match between neural activity during encoding and retrieval (Gelbard-Sagiv, Mukamel, Harel, Malach, & Fried, 2008; Oedekoven, Keidel, Berens, & Bird, 2017). In this case, while precision will certainly be helpful in distinguishing one context from another, it only needs to be sufficiently different from competing contexts to allow completion to the correct trace. If context is imprecise, then the system will enter an unstable attractor and potentially retrieve a different item or return no solution at all.

Such considerations additionally highlight the importance of two computational mechanisms to hippocampal-mediated item-context binding, pattern completion and separation. These involve making two similar representations more different vs. making a different representation more like a matching one (Cowell, et al., 2019; Levy, 1989; McNaughton & Morris, 1987; Yassa & Stark, 2011). In particular, pattern separation should serve to reduce interference between high resolution contextual representations to allow distinguishing of competing contexts. This idea is also critical during encoding, as the item-context binding must be sufficiently different from others such that the correct association is subsequently retrieved. Pattern completion, in contrast, would be important during retrieval as it essentially serves to match a retrieved trace to one during encoding. Note that if context involves low-precision representations, pattern completion may occur to a different list or incorrect item, and thus the importance of the precision of the context representation. Overall, such ideas fit with prior proposals about episodic memory that have emphasized the importance of both high-dimensional representations within the hippocampus as well as the importance of pattern separation and completion to this process (Cowell, et al., 2019).

Contextual precision and episodic memory outside of the hippocampus

The hippocampus is not the only brain area in which contextual processing and pattern completion/separation occur (e.g., Cowell, et al., 2019). Consistent with this idea, recent work has also highlighted the roles of areas outside of the hippocampus in episodic memory. In support of this idea, both imaging and lesion evidence in humans suggest that posterior parietal cortex, medial prefrontal cortex, precuneus / retrosplenial cortex, and other parts of the “core recollection network” are also critical for episodic memory (Berryhill, Phuong, Picasso, Cabeza, & Olson, 2007; Blumenfeld & Ranganath, 2007; Duarte, Ranganath, & Knight, 2005; Kim, 2011; Rugg, Otten, & Henson, 2002; Thakral, Wang, & Rugg, 2016; Uncapher, Otten, & Rugg, 2006; Wagner, Shannon, Kahn, & Buckner, 2005; Zeithamova, Dominick, & Preston, 2012) and their interactions (Fornito, Harrison, Zalesky, & Simons, 2012; Geib, Stanley, Wing, Laurienti, & Cabeza, 2015; King, de Chastelaine, Elward, Wang, & Rugg, 2015; Schedlbauer, Copara, Watrous, & Ekstrom, 2014; Watrous, Tandon, Conner, Pieters, & Ekstrom, 2013). Most compelling are findings that lesions to both prefrontal cortex and posterior parietal cortex, as well as other areas like the mammillary bodies in the thalamus and retrosplenial cortex, produce deficits in episodic memory, suggesting the necessity of these areas to episodic memory function (Berryhill, et al., 2007; Duarte, et al., 2005; Gadian, et al., 2000; Newsome, et al., 2018; Simons, Peers, Mazuz, Berryhill, & Olson, 2010; Valenstein, et al., 1987). The question remaining, which we discuss only briefly here, is whether brain regions in the core recollection network play specific roles in episodic memory that work in an additive manner or whether their function can be better conceived as a non-additive.

The additive perspective on the roles of brain regions within the core recollection network suggests that each brain region contributes something “unique” to episodic memory. Tulving, for example, favored the idea that prefrontal cortex provided “executive” control functions important to episodic memory and this idea has certainly been retained in other formulations as part of source monitoring (Johnson, 2006; Van Petten, et al., 2004; Wheeler, et al., 1997). According to an additive perspective on PFC function in episodic memory, PFC performs monitoring and interference reduction functions but not the item-context bindings that the hippocampus provides. Therefore, damage to the PFC should affect functions such as how well a participant can hold a cue in memory and use this information to cue item-contextual bindings within the hippocampus but does not contribute to the retrieval of the item-context bindings themselves. Additive models are strongly consistent with double dissociations in which damage to one brain area impacts performance on one task and not another, while the opposite patterns occur for damage to a different brain area like the hippocampus (Baddeley, 2003). Thus, areas like prefrontal cortex and posterior parietal cortex have specific, circumscribed roles in episodic memory, and in this way, do not store a “trace” of item-context memory. Instead, their function is “added” to that of the hippocampus (along with others), and together, the emergent behavior is episodic memory.

In contrast, non-additive models, which have gained increasing traction in graph theory, suggest that “episodic memory,” as a cognitive construct, emerges from distributed interactions between brain regions such as the core recollection network (Schedlbauer & Ekstrom, 2017; Schedlbauer & Ekstrom, 2019). Consistent with the non-additive framework, cellular responses to context are also present in brain areas outside of the hippocampus, such as place cells in the rodent retrosplenial cortex (Mao, Kandler, McNaughton, & Bonin, 2017), visual cortex (Haggerty & Ji, 2015; Ji & Wilson, 2007), prefrontal cortex (Fujisawa, Amarasingham, Harrison, & Buzsaki, 2008), claustrum (Jankowski & O’Mara, 2015) and even the human amygdala (see Figure 2C in Miller, et al., 2013). The argument for non-additive coding, which we have made previously with regard to both episodic memory and spatial navigation (Ekstrom, et al., 2017; Schedlbauer & Ekstrom, 2017), posits that interactions across multiple brain hubs operate such that the role of a hub cannot be distilled to a single function in cognition (Bassett & Gazzaniga, 2011; Finger, Koehler, & Jagella, 2004). As such, binding depends on the interactions of multiple brain regions, and cannot be distilled to a single region such as the hippocampus (Schedlbauer & Ekstrom, 2017). To bring back our earlier example, this conceptualization would suggest that both PFC and hippocampus play a role in both trace storage as well as episodic memory more generally. This does not mean, however, that all regions within the core recollection network contain identical neural architecture. PFC may be slightly biased neuroanatomically toward executive-type functions and hippocampus to item-context bindings. While different brain regions undoubtedly contain partially unique and partially overlapping neural patterns of connectivity and computational capacities, the property of binding, according to the non-additive framework, only emerges “normally” and collectively when these areas can interact.

Currently, we do not think there is sufficient evidence to support either the additive or non-additive perspective on the neural basis of episodic memory. While there is evidence that areas of the core recollection network can be dissociated based on different dependent measures (Bonnici, Cheke, Green, FitzGerald, & Simons, 2018; Richter, Cooper, Bays, & Simons, 2016), we suspect that at least some of such effects could be task specific. Any form of task-specific effects of activation patterns or perturbations within the core recollection network would seem to relate better to a non-additive conceptualization, suggesting that different episodic memory tasks may place different emphasis on components of the core recollection network. We also think the idea of interactions is a key and often overlooked aspect of models of episodic memory, although more recent models do place an emphasis on such phenomenon (Ranganath & Ritchey, 2012). Such intermediate versions of additive/non-additive network models involve some segregation of function in anterior to posterior brain networks with evidence for dynamic interactions between the two (Cooper & Ritchey, 2019). Future experiments will be needed to address the extent to which item-context bindings, a key component of memory, is additive or non-additive.

Precision, context, and hippocampal function

To better understand context, we also need to consider how such an entity might take shape in the first place. As we have discussed, many aspects of context are likely built on semantic knowledge, what could be termed a scaffold or a script (Bartlett, 1932; Schank & Abelson, 1977). One issue, however, as discussed above, is that the more static elements present in a contextual representation the less effective it will be at distinguishing different encoded items at retrieval. One way of producing different elements within this larger semantic scaffold then could be based shifts in external input and drifting temporal context. For example, using the method of loci will be facilitated by a more precise representation of the spatial environment you employ as a scaffold and remembering what you had to eat at your favorite restaurant will be facilitated by a more precise representation of the layout of the restaurant (e.g., such that you could distinguish sitting in different places at different times). This process will involve perception, attention, and working memory, all of which can happen at varying degrees of precision.

As we have described above in terms of computations, we can think of precision as critical to the success of recall because it adds dimensionality and resolution to contextual representations important to computational functions within the hippocampus like pattern separation (Hindy, Ng, & Turk-Browne, 2016). As far as how precision manifests at the representational/behavioral level, we can think of it in a manner consistent with that proposed in the Precision and Binding Model (PBM) (Kolarik, Baer, Shahlaie, Yonelinas, & Ekstrom, 2017; Yonelinas, 2013). Accordingly, precision can be thought of as a continuous measure of the level of detail in a perceptual or memory representation (Aly, Ranganath, & Yonelinas, 2013; Barense, et al., 2012; Kolarik, et al., 2016; Richter, et al., 2016; Yonelinas, 2013). This is consistent with the idea that when we encode the color, orientation, or location of a stimulus, the greater the resolution and dimensionality of attributes, the better differentiated from other competing “source” information we have also encoded. In this way, precision is much like what is often supposed in working memory and attention research as involving narrowly tuned attributes directly related to encoding of that feature. With more features (orientation + color) to be encoded, resource allocation results in increasingly noisy representations, decreasing precision at the expense of greater memory for distinct stimulus features (Ma, Husain, & Bays, 2014).

Given the importance of the hippocampus to episodic memory, and our proposal of the additional importance of precision to perception, working memory, and attention, it may seem surprising to suggest that the hippocampus plays a role beyond the widely agreed upon role in item-context bindings. We think, however, that ample evidence now supports the idea that the domain of the hippocampus extends beyond episodic memory alone. Even the original reports with patients H.M and E.P. noted some deficits in both perceptual and working memory processes (Hamann & Squire, 1997; Insausti, et al., 2013; Milner, et al., 1968; Olson, Page, Moore, Chatterjee, & Verfaellie, 2006; Scoville & Milner, 1957; Stark & Squire, 2000). These findings, which typically have manifested in deficits in neuropsychological tests related to working memory and perception, could potentially have origins in some of the patients’ heterogenous lesion locations. Additional evidence, however, from tests providing a more detailed assay of working memory and perception suggests this is likely not the case and that such deficits arise, in part, from effects related to hippocampal lesions.

In one example, Aly et al. tested patients with medial temporal lobe lesions some of which were largely circumscribed to the hippocampus, and healthy controls, on complex scene images (Figure 3). Participants made perceptual judgments indicating whether two simultaneously presented images were the same or different. Importantly, the perceptual changes did not involve adding or removing specific objects in the scene but rather pinching or expanding of the images such that there was only a slight mismatch between them. This was important because this would have otherwise involved changes in discrete features that would have readily helped participants identify perceptual differences. Strikingly, the patients exhibited significantly reduced perceptual sensitivity on this task indicating that the hippocampus contributed to perceptual discriminations when the task required the detection of very subtle perceptual differences. For other examples of the importance of the hippocampus (and medial temporal lobe more generally) to perception also see: (Barense, Gaffan, & Graham, 2007; Erez, Lee, & Barense, 2013; Hindy, et al., 2016; Warren, Duff, Tranel & Cohen, 2011).

Figure 3: Empirical work supporting the role of the hippocampus in precision.

A. Aly et al. showed that hippocampal lesion patients, compared to controls, showed decrements in strength-based but not state-based perceptual judgments. Example image shows a strength-based difference between two scenes. B. Kolarik et al. showed that hippocampal lesion patients, compared to controls, showed decrements in the precision with which they searched for a hidden location in a virtual environment. Example VR image shows different precision windows surrounding a memorized hidden location. Asterisks indicate significant differences between patients and controls

The Aly et al. experiment, however, did not involve explicit manipulation of precision and could best be considered a test case for whether the hippocampus plays a role in perception of continuously changed features. To address this issue, Koen et al. tested patients and controls who viewed a small set of colored objects and then after a 2 second delay were given a forced-choice test for object-color (i.e., “which of the two colors was this object presented in?”) or object-location (“which of the two locations was this object presented in?”). This aspect of the design has commonality with that used by Richter, et al. (2016) in the domain of episodic memory. To explicitly manipulate precision, half of the test trials necessitated high precision representations of color (e.g., one option was red and the other was a slightly different shade of red) and half were low precision (e.g., one option was red and the other was yellow). Overall difficulty was matched across high and low precision trials by varying set size. Importantly, patients showed greater deficits relative to controls for more precise but not less precise memory trials, which was true for both the location and color conditions. These findings suggest the human hippocampus plays a necessary role in in working memory, and in particular, in the resolution with which the items were stored and retrieved (Koen, Borders, Petzold, & Yonelinas, 2017).

Another way to think of the manipulations in the Koen et al. study might instead be to relate such representations to the precision of object-feature bindings. For example, remembering the color and orientation of a stimulus could instead be thought of as involving how this information is bound to the representation of the object. Overall, we think this idea relates directly to the precision of information represented not only in the hippocampus but in other structures as well, such as perirhinal cortex and other ventral stream structures. As we noted earlier, however, precision is also important to contextual representations. There is also evidence in patients with hippocampal lesions to demonstrate that the hippocampus plays a necessary role in precision for context. In this case, we will think of precision as important to the issue of spatial context, but as our examples before demonstrated, precision is an important consideration for many different forms of representation, including time, emotional valence, and other relevant dimensions.

To address precision related to spatial context, Kolarik et al. tested patients with lesions to the medial temporal lobe, which included two patients with bilateral hippocampal lesions (Kolarik, et al., 2017; Kolarik, et al., 2016). All participants navigated a large-virtual arena (Figure 3) in virtual reality by searching for a hidden target. During acquisition, if the participant did not find the target after 30 seconds, it was displayed on the screen, as is often done in assays involving the virtual Morris Water to ensure that participants learn the hidden location (Astur, Taylor, Mamelak, Philpott, & Sutherland, 2002). During probe trials (retrieval), participants searched for the hidden target location, with no feedback provided. In this way, the study assayed the precision of searches both during encoding (acquisition) and retrieval (probe trials).

To better understand the precision of the spatial searches, Kolarik et al. employed a novel analysis involving squares to determine how much of the search occurred in the near or far vicinity of the target area. Note that such information would be difficult to obtain with the conventional quadrant measure used in many past studies of the Morris Water Maze. Similarly, total distance and distance from target could obscure accurate searches that tended to be slightly more distant from the hidden target or more meandering but still “on target.” Thus, the dependent measure used in this study involved the percent of time spent in a 2-D area surrounding the hidden target, with such “windows” at different distances from the target. Patients spent significantly less time searching closest to the hidden target compared to controls, but more time in the distant areas compared to the controls, a finding true for both immediate and delayed testing. Together, these findings support the idea that hippocampal lesions impair the precision of representations for context within the hippocampus, but that such patients can still perform search strategies (allocentric) that are appropriate and partially accurate for the task.

These studies converge in showing the importance of considering precision when examining the role of the hippocampus in episodic memory, working memory and perception. We believe that many of the inconsistencies reported in prior studies regarding whether the hippocampus is or is not involved in different long term, working memory, and perceptual tasks can be explained by the extent to which the tasks requires high precision representations. The results also highlight the fact that precision of both the item information and the context information can be critical. For example, the navigation and the object-location working memory results discussed above indicate that the hippocampus is particularly important in supporting memory for precise contextual information, whereas the object-color working memory results suggest that the hippocampus is important in supporting precise item information.

The neural basis of precision

Somewhat unlike the operation of binding of item and context, precision refers to the quality of a representation that would appear to be shared across many different brain networks and regions (Cowell, et al., 2019). We can readily talk about the idea of precision, for example, in the sensory domain. When we perceive a scene, this necessitates some form of representation within primary visual cortex. If we hear a sound and remember it, this requires some form of representation within primary auditory cortex. Importantly, we would typically think of such representations of varying precision, which will depend on factors like how well we fixated the item on our retina or how the sound waves hit our cochlea, as well as attentional factors related to encoding the stimulus. For example, a patient with a medial temporal lobe lesion might still be expected to have a fairly precise representation of different pitches as part of language provided by auditory cortex, even if their ability to effectively bind such information in memory might be impaired.

In fact, it is probably reasonable to think about precision as a phenomenon that would be important to representation in many different brain areas. The extent to which we can recognize a face likely depends on its resolution/dimensionality in brain areas central to this function. Because we can think of the hippocampus as a convergence zone, it likely receives much of this information through interactions and input from both primary sensory and secondary/tertiary association areas. The question then is that given that the hippocampus would not appear directly involved in perception of the stimulus, why would lesions to this area affect perception?

As we have suggested, precision is likely a phenomenon supported by many different brain regions, each of which may contribute certain dimensions (e.g., visual cortex) but together which interact to produce what we think of as an aggregate on-line representation of the current context/item. In this way, hippocampal activity, although a small part of a much larger sum, contributes overall to the precision of the representation. In the case of space, this is relatively easy to see at the neural level. Place cell activity, from which the location of an animal can be partially decoded (Jensen & Lisman, 2000; Wilson & McNaughton, 1993), could contribute to representations of spatial context. One example would be place cells changing with temporal and other task related variables, thereby creating a dynamic form of contextual representation of varying precision (Shapiro & Eichenbaum, 1999; Wood, Dudchenko, Robitsek, & Eichenbaum, 2000). In a similar vein, the distributed nature of time cells within the hippocampus (Kraus, et al., 2013), which also exist in other brain structures (Buhusi & Meck, 2005), would likely contribute to the overall resolution of a representation for time.

How would hippocampal neural responses relate to the issue of item representation, which might appear to be the case in the Koen et al study? Single neuron studies in the human hippocampus have also identified item responses, like those to famous actors and animals (Kreiman, Koch, & Fried, 2000; Quiroga, Reddy, Koch, & Fried, 2007; Quiroga, Reddy, Kreiman, Koch, & Fried, 2005) and these would also be likely to contribute to the overall precision of any representation for an item. Notably, such cells responded to concepts (Jennifer Aniston) rather than specific instantiations of the concepts (such as a 90 degree or 180 degree oriented Jennifer Aniston). The tasks themselves in these studies, however, did not involve detection of such differences and rather questions related to whether the object was a house or not. Given that other single neuron studies have suggested firing rate differences for different targets (Wixted, et al., 2014) and conjunctive responses related to goals, landmarks, and locations (Ekstrom, et al., 2003), it seems likely that tasks that require such “precise” types of neural coding would also demonstrate such capacities in the human hippocampus.

It seems surprising, however, to attribute a brain region like the hippocampus to a function in perception, which is more often relegated to brain areas like primary visual cortex. As Tulving noted, though, some aspects of cognition are likely shared by many if not all brain regions. In this way, we believe that the overall precision of a representation in perception and memory emerges from the interactions of numerous areas across the brain. The critical role of the hippocampus would be in linking the representation between other brain areas more directly involved in perception, such as primary visual cortex, auditory cortex, and multimodal areas like fusiform gyrus. In this way, simpler forms of representation would certainly be possible with a lesioned hippocampus but would be impaired in terms of how well such representations were overall integrated across domains.

Why, however, would we need such representations across the brain? For one, redundancy is almost certainly important to something as fundamental as perception and having many processing modules that can contribute would overall increase one’s ability to “max out” on this important function. In addition, almost all brain regions would require some form of item representation in order to perform more specific computations. Without a representation of a face, there can be no memory for the face, and thus in the process of receiving the input for a face, the hippocampus could also be contributing to the perception of it as well via distributed interactions. Finally, as we argued above, the hippocampus, unlike parts of neocortex, would be critical in linking these multimodal and disparate representations together via functional interactions. For examples of such interactions between working memory and perception, please see Teng and Kravitz (2019); for examples of such interactions between visual cortex and the hippocampus, please see: Hindy, et al. (2016).

We note that this conception of precision as a distributed phenomenon shared by many different brain regions, with the hippocampus as one of many different “cogs” in this function yet serving a linking function, goes against classic conceptions of hierarchical processing (Kravitz, Saleem, Baker, Ungerleider, & Mishkin, 2013; Ungerleider & Miskin, 1982). The idea of parallel processing amongst brain regions, even those who appear to serve “deeper” visual functions, however, is gaining increasing traction in the fields of perception and attention. For example, visual perception, rather than preceding hierarchically behaviorally, shows several instances in which some steps, like figure ground segregation, occur after object perception (Peterson, 1994; Peterson & Gibson, 1994). Similarly, neural accounts of attention increasingly assume distributed roles across multiple brain regions such that no one brain primarily modulates or controls attention and instead, this emerges across interactions across many different “nodes” (Shipp, 2004). Together, these ideas suggest that perception relies on the interactions of multiple brain regions, many of which may share similar neural architecture involved in representing aspects of context or items.

Novel explanatory and predictive power

We hope that our proposal here regarding episodic memory and hippocampal involvement in memory and beyond will be helpful in generating new experiments. Theoretically, we think the somewhat ubiquitous role of the hippocampus in areas outside of memory has remained a bit of a puzzle, and classic theories of declarative memory do not have a clear explanation of how this could be so (Squire, 1992; Squire, Stark, & Clark, 2004). Yet, the evidence that hippocampal lesions impact perception, working memory, and even language function, is considerable (Barense, et al., 2007; Borders, Aly, Parks, & Yonelinas, 2017; Graham, Barense, & Lee, 2010; Konkel, Warren, Duff, Tranel, & Cohen, 2008; D. E. Warren, Duff, Jensen, Tranel, & Cohen, 2012; Warren, Duff, Tranel, & Cohen, 2010; D. E. Warren, et al., 2011). This suggests that the hippocampus cannot be a module exclusively dedicated to episodic memory. By casting hippocampus as contributing to representational precision as one of many different players in the brain, and such on-line representation emerging through dynamic interactions across many different brain regions, our model helps solves the puzzle of how the hippocampus can play necessary, although perhaps more minor roles in areas outside of episodic memory.

Yet, the fundamental role of the hippocampus in episodic memory is undeniable and bolstered by decades of work on the topic. By casting item-context binding as a primary role of the hippocampus, with a few other areas (like the core recollection network) also contributing, our model is consistent with this long tradition arguing for the centrally of medial temporal lobes to amnesia. At the same time, by suggesting that areas of the core recollection network also play necessary (and possibly non-additive roles) in episodic memory, our model helps solve another potential puzzle regarding medial temporal lobe lesions. Past work in memory research indicates that although damage to the hippocampus severely impairs episodic memory encoding and retrieval (Corkin, 1984; Rempel-Clower, Zola, Squire, & Amaral, 1996; Scoville & Milner, 1957; Yonelinas, et al., 1998), performance is rarely at chance in such patients, suggesting some intact function (Gold, et al., 2006; Helmstaedter, Grunwald, Lehnertz, Gleissner, & Elger, 1997; Zola-Morgan, Squire, & Ramus, 1994). While this could be due to residual hippocampal tissue, we also think that compensation is another viable alternative that our model provides for. According to this idea, assuming that other brain areas within the core recollection network play important roles in binding, it could be that such regions can partially compensate for lost function in the hippocampus, particularly if their computational role is distributed and non-additive.

Thus, our model provides for key yet untested predictions. Following hippocampal lesions, we predict that level of impairment behaviorally should be a function of demands on both item-context binding and representational precision. Thus, there may be cases in which simple bindings are possible but precision is impaired, and vice versa, depending on the extent to which the two must work in tandem (Yonelinas, 2013). In addition, we expect that, over time following a lesion to the hippocampus, other brain areas within the core recollection network may be able to compensate for lost binding function. As one example of this, a recent manuscript by Froudist-Walsh, et al. (2018) found that hippocampal lesions in non-human primates result in degradation amongst connected and interacting brain areas like precuneus and parts of prefrontal cortex shortly after the lesion. Interestingly, however, over time these same areas also increased connectivity with each other, suggesting changes that could relate to neural compensation. Similarly, a recent study by Argyropoulos, et al. (2019) suggested that functional connectivity patterns within areas of the core recollection network explain greater variance in delayed memory performance in amnesiacs than gray matter loss within the hippocampus. As one possible area of future investigation, the converse approach of what we typically do with episodic memory and patients with medial temporal lobe damage could help resolve some of the issues discussed in this manuscript. Specifically, identifying patients with complete episodic memory loss and then determining their patterns of brain damage could help resolve the extent to which binding functions are distributed across the core recollection network and how such lesions also affect precision.

Conclusion

We have elucidated on the important concept of representational precision here to attempt to explain both the role of the hippocampus in item-context bindings and its contributions to representation more generally. The first area we explored, item-context binding, is widely recognized as important to episodic memory in particular and involves associating a unique context with an item representation. We suggest here that binding relies primarily on the hippocampus, with other brain regions within the core recollection network also playing necessary, but still unclear roles. Precision, in contrast, relates to both the resolution and dimensionality of a representation and helps predict the extent to which a brain region like the hippocampus will be necessary for cognition outside of episodic memory. Here, we conceive of precision as important to both item and contextual representation and something that will tend to be distributed across the brain. In this way, precision will emerge from interactions of shared neural machinery across many different brain regions. Thus, lesions to almost any “cog” within this larger machinery will impair the precision of such a representation, although lesions to the recollection network would be needed to impair binding. By considering both binding (as an operation) and precision (as related to the resolution and dimensionality of a representation), these two aspects of can help better explain both lesion, behavioral, and fMRI findings related to memory and perception.

Highlights.

• Precision as adding to Endel Tulving’s original conceptualization of episodic memory

• Precision as involving resolution and dimensionality (complexity)

• Precision as important to context and item-context binding

• Context as involving drifts (time) and shifts (space)

• Representational precision helps explains hippocampal role outside of episodic memory

• The hippocampus involved in networks involving perceptual representation and binding

• Proposed additive and non-additive components for representation and binding