Memory and decision making

The only reason we remember things is to make better decisions.

What is memory? Memory can be defined as any physical change that carries information about the historical past. Typically, in animal systems, memory is stored in physical changes inside and between neurons (Engert & Bonhoeffer, 1999; Kandel, 2006; Malinow & Malenka, 2002; Silva, Kogan, Frankland, & Kida, 1998). How these physical changes affect information processing depends on how those systems compute information processing. In practice, memory needs to be encoded in a representational form easily accessed by specific computational processes. There will be tradeoffs in these representational forms between generalization and specificity, between detail and accessibility, and between storage size and these other issues (Cormen, Leiserson, & Rivest, 1992; McClelland & Rumelhart, 1986; O’Reilly & McClelland, 1994). These tradeoffs suggest that there should be multiple memory systems, each with representational forms optimized for different aspects of these tradeoffs (O’Keefe & Nadel, 1978; Redish, 1999, 2013; Schacter & Tulving, 1994).

Similarly, we can ask What is a decision? Following the definitions in Redish (2013), in order to operationally define decision-making such that it can be easily recognized and observed, we define decision-making as the process of selecting an action. At its most general, an action is anything that physically affects the world — thus muscle movements (Grillner, 2003; Llinas, 2001) and social speech acts (Searle, 1965) are both decisions, as are physiological processes such as salivation (Pavlov, 1927). Because we are physical beings, a decision that changes one’s internal (computational) state can also be considered an action. And, of course, choosing not to act is also a decision-process.

This means that any process that leads to the selection of an action from a set of possible actions is a decision. As with memory, decisions depend on tradeoffs between factors such as generalization and specificity, and between computational speed and flexibility.

Therefore, as has been found to be the case with memory, there are likely to be multiple decision-making systems, each with computational processes optimized for different aspects of these trade-offs (Cisek & Kalask, 2010; Daw, Niv, & Dayan, 2005; Keramati, Dezfouli, & Piray, 2011; O’Keefe & Nadel, 1978; Redish, 1999, 2013). These computational processes select actions that reflect an interaction between one’s needs and desires (goals, motivation), external cues (information about the current state of the world), and internal representations of one’s historical experience (i.e. memory).

These two definitions imply a close relationship between memory and decision-making systems, particularly in their multiplicity of computational components. Where decision-making processes fall in terms of their tradeoffs is going to depend in large part on the computational availability of memory representations—a memory representation that provides quick generalization but little specificity is going to produce decisions that are fast, but inflexible, while a memory representation that provides many details, but requires extensive processing to unpack (and reconstitute) those details into amemory will produce decisions that are slow, but flexible. It follows, then, that the same underlying neural systems that are critical for memory are going to be critical for decision-making.

The idea that memory is not unitary traces itself back to the declarative versus procedural distinction first seen in the late 1970s and early 1980s (Cohen & Eichenbaum, 1993; Cohen & Squire, 1980; O’Keefe & Nadel, 1978; Redish, 1999; Squire, 1987). It was observed that quickly-learned, factual information (such that it could be “declared”) depended on one set of structures (such as the hippocampus), while slowly-learned procedural information depended on other structures (particularly specific dorsal and lateral aspects of the striatum). Over time, it was recognized that declarative memory did not depend on language itself, but rather on a ubiquitously-learned cognitive model of the world (a “cognitive map”) (Johnson & Crowe, 2009; O’Keefe & Nadel, 1978; Redish, 1999; Tolman, 1948). In contrast, procedural memories depended on a learning algorithm that only learned the cues that were important to predict outcomes (Berke, Breck, & Eichenbaum, 2009; Jog, Kubota, Connolly, Hillegaart, & Graybiel, 1999; Schmitzer-Torbert & Redish, 2008; Sutton & Barto, 1998).

Similarly, the idea that decision-making is not unitary traces itself in the animal learning literature back several decades to different effects of training on decision-making processes, particularly differences in latent learning and devaluation processes (Balleine & Dickinson, 1998; Bouton, 2007; Mackintosh, 1974). In latent learning, pre-exposure to a condition enables very fast changes in action selection when that condition affects the decision (such as adding a new goal location once one knows the structure of a maze) (Tolman, 1932; Tse et al., 2007). In devaluation, changing the value of one of two rewards (for example, by pairing it with a negative stimulus in another context) changes the response to that reward immediately on re-exposure (Adams & Dickinson, 1981; Balleine & Dickinson, 1998; Schoenbaum, Roesch, & Stalnaker, 2006). In contrast, slow, regular experiences led to decision-making processes that were insensitive to devaluation or to changes in the contingencies of cue-reward interaction (Balleine & Dickinson, 1998; Coutureau & Killcross, 2003). Changing the training presumably led to differences in memory-storage representations, which led to differences in decision-making behaviors. These two processes depended on the same brain structure differences as the non-unitary memory processes reviewed above (Yin, Knowlton, & Balleine, 2004).

In the 1990s, a similar set of differences appeared in the computational literature. Building on the ubiquitous temporal-difference reinforcement-learning (TDRL) model (Sutton & Barto, 1998), computational analyses showed that there were fundamental differences between algorithms that searched through potential futures and algorithms that selected actions based on recognition of the current state of the world. An algorithm that searched through models of the world to construct hypothetical states, which could then be evaluated in the context of the animal’s current situation, depended on knowing the structure of the world, was flexible, but computationally slow (Daw et al., 2005; Johnson, van der Meer, & Redish, 2007; Keramati et al., 2011; Niv, Joel, & Dayan, 2006; van der Meer, Kurth-Nelson, & Redish, 2012). In contrast, an algorithm that categorized the situation and recalled a single generalized action that had been learned to be optimal within that situation would be inflexible, but computationally fast to execute (Johnson et al., 2007; Niv et al., 2006; van der Meer et al., 2012; Yang & Shadlen, 2007). More recently, it has become clear that a full description of memory and decision-making will require additional components including affective memory systems, Pavlovian action-selection systems, reflexive systems, as well as cognitive and cue-recognition components (Dayan, 2012; Gershman, Blei, & Niv, 2010; Montague, Dolan, Friston, & Redish, 2012; Phelps, Lempert, & Sokol-Hessner, 2014; Redish, 2013; Redish, Jensen & Johnson, 2008; Redish, Jensen, Johnson, & Kurth-Nelson, 2007).

There are excellent reviews of the similarities and differences between these multiple memory systems, multiple decision-making systems, and multiple computational components, and so in this special issue, the eleven papers delve into specific issues related to these relationships, showing that decision-making abilities correlate with measures of memory abilities, and identifying the computational and neurophysiological processes that underlie these parallel memory and decision-making abilities.

The first set of papers examines the computational and neurophysiological processes that underlie these two primary systems (slow, flexible, based on searching through the future [termed model-based, because it requires a search through a model of the world] versus fast, inflexible, based on recall of primary situation- action pairs [termed model-free because it requires only categorization of current states of the world]). Doll, Shohamy, and Daw (this issue) review the key dichotomy here from a computational perspective and argue that the memory process distinction underlies the decision-making differences. They report experiments finding correlations between flexible, relational memory with model-based strategies but not with model-free strategies. Schacter, Benoit, De Bridgard, and Szpunar (this issue) review the concepts of episodic future thinking that are critical to searching through models of the world and suggest that episodic future thinking depends on the ability to construct counter-factual and hypothetical scenarios through imagination. They suggest that these abilities depend on a common neural network involving the hippocampus and prefrontal cortex. Wang, Cohen, and Voss (this issue) propose a conceptual framework in which prefrontal cortex polls the hippocampus for hypothetical scenarios covertly and rapidly, suggesting that the necessary simulation cycles explain the slower speed of some decisions, even those that occur without explicit awareness. While Wang et al. concentrate on interactions between prefrontal cortex and hippocampus in humans, Yu and Frank (this issue) examine those interactions in the other species in which they have been most studied, the rat.

The second set of papers follows these by delving deeper into those interactions. Dahmani and Bohbot (this issue) examine a task that differentiates spatial (model-based) and stimulus–response (model-free) strategies, which are known to separately activate hippocampal systems and caudate systems, respectively. They find that different aspects of prefrontal cortex are involved in these two systems, suggesting a dichotomy within prefrontal cortex itself. Similarly, Burton, Nakamura, and Roesch (this issue) review data on striatal subdivisions, finding a similar heterogeneity within striatum.

The third set of papers examines interactions between memory and decision-making systems beyond this single dichotomy. In particular, Shimp, Mitchell, Beas, Bizon, and Setlow (this issue) examine sensitivity to risk and find underlying correlations between working memory abilities (memory) and discounting rates (decision), suggesting underlying functional components, which likely relate to those discussed in the Schacter et al. and Wang et al. papers. Hutchinson, Uncapher, and Wagner (this issue) show that these representations of risk are represented in subregions of posterior parietal cortex, and note how retrieval of memory is itself a decision-process. Hart, Clark, and Phillips (this issue) examine the role of dopamine in risk taking behavior, finding that with dopamine signals in the rat correlate with both reward prediction errors and the expected variance of reward itself (similar to what has been found in monkeys). Their data suggests that these informational components change with experience on the task. Redila, Kinzel, Jo, Puryear, and Mizumori (this issue) take the computation one step further back, identifying the role and firing patterns of lateral dorsal tegmental (LDTg) neurons (which drive dopamine bursting in the ventral tegmental area [VTA]), and comparing them to other VTA afferents such as the pedunculopontine nucleus [PPTg]. The LDTg monitors ongoing behaviors perhaps to increase the accuracy of predicting future reward encounters, and the PPTg provides current sensory information that VTA needs to calculate reward prediction error signals.

Finally, Erdem, Milford, and Hasselmo (this issue) show that in a robot model capable of navigating in the world, memory is critical to correct formation of the cognitive map, particularly in recognition of situations, allowing the map to correctly reflect the world. Maps that better reflect the world provide a better substrate for decision-making processes. This brings this interaction together, showing how memory and decision-making necessarily interact in order to produce successful behavior.

These eleven papers provide new insights into the relationship between memory and decision-making. We hope you enjoy this special issue.