Covert rapid action-memory simulation (CRAMS): A hypothesis of hippocampal-prefrontal interactions for adaptive behavior

Abstract

Effective choices generally require memory, yet little is known regarding the cognitive or neural mechanisms that allow memory to influence choices. We outline a new framework proposing that covert memory processing of hippocampus interacts with action-generation processing of prefrontal cortex in order to arrive at optimal, memory-guided choices. Covert, rapid action-memory simulation (CRAMS) is proposed here as a framework for understanding cognitive and/or behavioral choices, whereby prefrontal-hippocampal interactions quickly provide multiple simulations of potential outcomes used to evaluate the set of possible choices. We hypothesize that this CRAMS process is automatic, obligatory, and covert, meaning that many cycles of action-memory simulation occur in response to choice conflict without an individual’s necessary intention and generally without awareness of the simulations, leading to adaptive behavior with little perceived effort. CRAMS is thus distinct from influential proposals that adaptive memory-based behavior in humans requires consciously experienced memory-based construction of possible future scenarios and deliberate decisions among possible future constructions. CRAMS provides an account of why hippocampus has been shown to make critical contributions to the short-term control of behavior, and it motivates several new experimental approaches and hypotheses that could be used to better understand the ubiquitous role of prefrontal-hippocampal interactions in situations that require adaptively using memory to guide choices. Importantly, this framework provides a perspective that allows for testing decision-making mechanisms in a manner that translates well across human and nonhuman animal model systems.

Overview

Although people often consider memory as merely the capability that allows us to relive past experiences and recognize whether other individuals or places are familiar, the utility of memory is far greater. Knowledge of past experiences is critical for deciding how to behave in the present and perhaps how plan for the future. Indeed, many current theories emphasize the role of memory in constructing, imagining, and preparing for future events (Addis and Schacter, 2011; Buckner, 2010; Buckner and Carroll, 2007; Eichenbaum and Fortin, 2009; Schacter et al., 2012; van der Meer et al., 2012). Memory thus allows us to tailor our current and future behavior based on our personal history. Indeed, damage to structures that support memory, such as the hippocampus, can seriously impair the ability to plan for and imagine the future (Maguire et al., 2010; Race et al., 2013, but see Squire et al., 2010). This review will extend this line of thinking a step further by proposing a mechanism whereby memory is critical to the translation of past experience into current adaptive behavior. We propose that the hippocampus makes necessary contributions to the determination of how to behave “in the moment”, and that it does so via iterative interactivity with prefrontal cortex (PFC) that unfolds as choices are made to determine immediately forthcoming behavior. Further, as described below, we propose that this process can occur covertly, without conscious awareness of memory retrieval or the feeling of deliberate control of choices, such that memory can be rapidly accessed and used to make in-the-moment choices without encumbrance by conscious memory recollection or by deliberation.

The essence of our proposal is that, when faced with uncertainty about how to behave, such as when one cannot decide whether to take a leftward versus a rightward path to a goal location, the PFC (particularly anterior and/or lateral PFC) bombards the hippocampus with possible plans for action (e.g., turn left or turn right). Subsequently, these action plans serve as signals to hippocampus that cue simulated outcomes of each possible action, in the form of covertly retrieved previous experiences or associated memories. For example, simulated outcomes might be based on memory that a leftward turn from the current point led home once before, on memory for other experiences that occurred in proximity to home and therefore provide relevant directional information, or on general spatial or experiential knowledge regarding the location of home. The behavior that is selected is thus based on the match between the outcome of simulated experiences and current desired goals (e.g., getting home in a reasonable time). Furthermore, because most real-world behavioral choices involve ongoing selections among numerous competing options, each with many associated past experiences, this process would entail extremely rapid simulation of many options before arriving at the choice. Our proposal is thus that PFC and hippocampus interact via covert, rapid action-memory simulation, or “CRAMS” in order to arrive at ongoing behavioral choices, and that this process is automatic, obligatory, and covert (or “implicit”) in most circumstances. We particularly emphasize the covert nature of this simulation. Many have previously considered the role of PFC and hippocampus in various forms of simulation, including self-projection and construction of future events using elements of previous experiences in memory (Buckner and Carroll, 2007; Schacter and Addis, 2007; Schacter et al., 2008). In CRAMS, “simulation” is covert, including hippocampal-dependent covert retrieval and construction of possible event outcomes based on PFC action cues. Therefore, in contrast to a psychological experience, we thus refer to “simulation” as the covert activation of neural ensembles involved in episodic memory that can be rapidly generated and evaluated on the short timescales necessary to support decisions during ongoing behavior. CRAMS thus provides an account of why the hippocampus is so crucial to the performance of memory-guided behaviors transpiring entirely within these brief timeframes that occur more rapidly and with different dynamics than the psychological experience of imagining future events. In addition, it encapsulates how the PFC is able to judiciously exert “top-down” control of behavior when contextual or rule-based responding is required.

Substantial evidence implicates the hippocampus and PFC in guiding ongoing behavior. We will first review this evidence and offer some suggestions about how the properties of these structures are particularly well suited to the kind of rapid, automatic, obligatory, and covert processing that we envision as necessary for CRAMS. We will then explicate the CRAMS proposal and give specific examples of the adaptive behaviors that it can support in humans and in nonhuman animals. Finally, we will describe several ramifications of the CRAMS proposal for theories of memory and decision-making and describe possible experimental routes for testing the key tenets of CRAMS.

The hippocampus: Obligatory relational binding and retrieval supporting multiple cognitive functions

The hippocampus has long been known as a critical site for processing relevant to explicit, or declarative, memory (Baddeley and Warrington, 1970; Cohen and Squire, 1980; Scoville and Milner, 1957). The relational theory of hippocampal function (Cohen and Eichenbaum, 1993; Eichenbaum and Cohen, 2001; Konkel and Cohen, 2009; Konkel et al., 2008) provides a neurobiological mechanism for the role of the hippocampus in memory and has received broad experimental support. In short, relational memory theory proposes that the hippocampus binds the elements of experience into relational/associative representations. That is, discrete, perceptually distinct components of experience (such as the features of a scene or the individual occurrences that comprise an episode) are linked into a network of arbitrary associations that can be partially or fully reconstituted at a later time (Konkel and Cohen, 2009; for similar theories, see Mayes et al., 2007; Ranganath, 2010; Yonelinas, 2013). Because of its fundamental role in relational binding, the hippocampus is important for all manner of cognitive functions that require handling of relations among elements, including language, attention, high-level perception, etc. (reviewed in Olsen et al., 2012).

Recent experiments have demonstrated an important extension of the role of the hippocampus in relational binding that is relevant to our proposal of hippocampal involvement in ongoing control of behavior and the nature of its interaction with PFC. These experiments have indicated that the hippocampus is necessary for short-term or “in-the-moment” memory processing. Early work in this area showed the role of the hippocampus in expressions of short-term memory. For instance, hippocampal activity correlates with maintenance of new associations over short delays (Ranganath and D’Esposito, 2001), and hippocampal lesions produce relational memory deficits even for delays as brief as several seconds (Hannula et al., 2006). More recent evidence has shown critical hippocampal involvement at even shorter delays and evidence for “in the moment” impairment. Warren et al. (2010) tested MTL amnesic patients and matched controls on a perceptual search and match task with simple visual stimuli. They showed that when the target object and search array were presented simultaneously, amnesics performed no differently from control subjects. However, when a brief 6-second delay was introduced between the target object and search array, amnesics were significantly impaired compared to controls, suggesting that MTL contributes to the maintenance of even very simple stimuli over short delays and that this maintenance begins to decay almost immediately.

Further work by Watson et al. (2013) indicated that the short-term deficits of hippocampal amnesics are especially pronounced for arbitrary relations among features of an experience. Amnesic subjects displayed profound deficits in spatial reconstruction of object configurations a mere four seconds after study. By analyzing the different types of errors that could be made during reconstruction, Watson et al. found evidence for disproportionate impairment in “swap” errors that indicate poor relational memory. That is, errors during the reconstruction of the spatial layout of several objects can include misplacing objects such that they occupy generally incorrect spatial configurations (e.g., if three objects were studied with locations that form the vertices of a right triangle, general location errors could include expansion of the size of the triangle or placing objects to form an equilateral triangle). Hippocampal amnesics were mildly impaired in general spatial configuration reconstruction relative to controls. In contrast, “swap” errors occurred when configurations were generally correct, but two or more objects switched locations. For instance, if objects A, B, and C were studied at triangle vertices V₁, V₂, and V₃, respectively, reconstruction of vertex locations would be accurate such that the same triangle configuration resulted, yet object A would occupy V₂ and object B would occupy V₁, thus indicating a “swap” of objects A and B. This error type is especially indicative of a deficit in relational memory because the overall spatial layout is intact, yet the subject fails to demonstrate memory for the arbitrary binding between each object and each possible location (e.g., between A and V₁ and between B and V₂, despite accurate memory for V₁ and V₂ locations). Swap errors were significantly disproportionate relative to general configuration errors in hippocampal amnesics even when only two objects were studied and then tested four seconds later. In contrast, control subjects rarely made swap errors. Taken with the evidence reviewed above, these findings constitute compelling evidence for the necessary role of the hippocampus and MTL in the short-term maintenance of relational information.

An additional aspect of hippocampal function relevant to our proposed role of the hippocampus in ongoing control of behavior is that all manner of event elements are bound automatically and obligatorily by hippocampus into relational representations, and are then retrieved automatically and obligatorily when given appropriate cues. For instance, Wimmer and Shohamy (2012) found that the hippocampus can support object preferences via “guilt by association.” That is, objects that were never rewarded but had been covertly associated with other objects that were rewarded were later valued more highly than if these objects were covertly associated with other unrewarded objects. Hippocampal activity supported the binding and retrieval of this covert associative value, despite lack of awareness on the part of participants that there was any association with reward. Thus, the hippocampus bound value to objects despite only indirect association, and this value was retrieved when given the associative cue, despite no possible intention on the part of the subject to retrieve this information (due to lack of awareness). Importantly, partial cues (i.e., one fragment of the several information sources that were bound into a relational representation) were sufficient to retrieve any and all associated information, presumably via pattern completion (Norman and O’Reilly, 2003; Rolls, 1996). Other similar demonstrations of automatic and obligatory hippocampal processing have been demonstrated for the binding and covert retrieval of hidden statistical regularities (Schapiro et al., 2012; Turk-Browne et al., 2010) and associations between arbitrary stimulus pairings (Hannula and Ranganath, 2009; Hannula et al., 2007; Hannula et al., 2006). The emerging conclusion from these studies is that the hippocampus binds all associative/relational information automatically and obligatorily, and then retrieves these associations when given a cue, without any necessary intention or awareness on the part of the experimental subject (Hannula and Greene, 2012).

Here we propose that the automatic, obligatory, and covert binding and retrieval processing by the hippocampus results in its critical participation in many aspects of ongoing behavior. Hippocampal-dependent memory, accumulated from past experiences, can be brought to bear on the current behavioral choice that must be made. Furthermore, because relevant cues, even partial ones, can elicit this information from hippocampus obligatorily (via processes such as pattern completion) and because retrieval can occur covertly, the information needed for simulation of future possible events can be obtained in a manner that is seamless with behavior and without deliberate retrieval of conscious experience. Indeed, it is unlikely that hippocampal contributions to ongoing behavior rely on deliberate choices made based on overt retrieval, given that choices must be made on timescales too rapid for such deliberation based on conscious awareness of memory retrieval. Although the hippocampus can rapidly and covertly provide access to relevant information during ongoing behavior, appropriate signals must be established elsewhere to automatically and obligatorily cue this information, and next we consider the role of PFC in providing such signals.

Role of PFC in rule-based, contextual, and other adaptive behaviors

The lateral and anterior/frontopolar PFC (referred to here as “PFC”) has long been recognized as the seat of “executive” functions such as decision-making (Fuster, 2008), working memory (Goldman-Rakic, 1987; Miller, 2000; Miller and Cohen, 2001), supervisory attentional control (Shallice and Burgess, 1991) and inhibitory control (Shimamura, 1995), temporal ordering and organization (Fuster, 1995, 1997; Kesner et al., 1994; Milner et al., 1985), organization and monitoring of information (Johnson et al., 1993; Petrides, 1996), and the implementation of context-dependent or abstract rules (Wallis et al., 2001; White and Wise, 1999). Further, PFC has been heavily implicated in maintenance of task-relevant information in a manner resistant to distraction irrelevant to behavioral goals (Miller et al., 1996). Lesions of PFC cause deficits in tasks that require cognitive flexibility, such as set shifting (Dias et al., 1996; Moore et al., 2009; Rossi et al., 2007). Patients with PFC damage do not generally have severe memory impairments, but deficits are apparent when memory for target information must be obtained under a variety of conditions of interference or distraction (Kuhl and Wagner, 2009).

Miller and Cohen (2001) proposed an influential theory that PFC supports context-dependent or rule-based responding by exerting context-specific top-down influence on other brain regions via selective excitation. This context-specific excitation can sustain task-relevant information and filter task-irrelevant information processed by other more specialized systems. For instance, PFC deactivation via cortical cooling attenuates the response of parietal cortex to task-relevant cues (Chafee and Goldman-Rakic, 2000), indicating that PFC normally enhances parietal responsivity to these cues (Lee and D’Esposito, 2012). Context-specificity and top-down control by PFC is important for the acquisition and expression of context- and rule-based responding that shapes perception (Rainer et al., 1999), action (Asaad et al., 2000), and cognitive choices (Asaad et al., 2000; Wallis et al., 2001; White and Wise, 1999). For instance, in an experiment conducted by White and Wise (1999), a rhesus monkey was trained on two types of cued target-selection trials, one governed by a spatial rule (cue location predicted target location) and the other guided by an associative rule (cue identity predicted target location, regardless of cue location). A significant fraction of neurons recorded in the PFC showed selectivity for the rule condition, indicating that PFC represents abstract rules of behavior rather than specific stimuli or motor acts (Asaad et al., 2000).

Substantial evidence thus indicates that PFC can guide context- or rule-dependent behavior; i.e., the PFC is critically involved in sorting among the many possible behaviors available in order to select the action most likely to produce a positive outcome in a given circumstance. It is important to emphasize, however, that in studies of rule- and context-based responding by lateral PFC neurons, nonhuman animals require extensive training (typically months in nonhuman primates) to establish these response properties. Likewise, rule- and context-based responding in humans is generally demonstrated in circumstances with which the individual has substantial prior general experience. Some evidence suggests that PFC learns rule- and context-based responding through integration of action-outcome relationships over multiple experiences, such as when subjects are required to integrate across either time or multiple outcomes in a goal-oriented task or when making a decision (Asaad et al., 1998; Cohen et al., 1997). Integration over multiple contingencies allows the PFC to discern statistical regularities in order to extract general principles or rules. Some regions of PFC (e.g., frontopolar cortex) are particularly involved in integrating information across multiple examples of a similar event in order to support this kind of learning (Badre and Wagner, 2004; Crescentini et al., 2011; Green et al., 2006; Yarkoni et al., 2005a; Yarkoni et al., 2005b). Indeed, PFC networks appear to be in a constant state of spatiotemporal evolution, capable of integrating and processing incoming information in an online fashion in the service of goal-relevant decision-making (Buonomano and Maass, 2009).

Nonetheless, there is relatively little work showing how PFC functions before rule- and context-based action-outcome associations have been learned via extensive training. That is, how does PFC contribute to the online guidance of behavior before context-dependent rules have been strongly learned? Thus, we next consider a putative means by which PFC processing could contribute to ongoing behavior without prior establishment of rules for responding in a given situation (i.e. in a relatively novel situation).

Although PFC is often discussed in light of executive function and deliberate “top-down” responding, we emphasize that, as an extension of the cortical motor control system, it is also involved more directly in the planning and execution of complex actions (Fuster, 2008; Tanji and Hoshi, 2008). It is in this capacity that we consider PFC to take part in ongoing behavioral control without any strong prior establishment of rule- and context-based responding. Lateral PFC neurons are critically involved in generating actions that lead to goal states (Mushiake et al., 2006; Tanji and Hoshi, 2001), and action-planning capabilities have been proposed as the core “building blocks” by which executive functions are supported (Hoshi and Tanji, 2004; Tanji and Hoshi, 2008). Indeed, there is strong evidence for organization of human PFC according to the representation of actions, with a posterior to anterior gradient along the lateral surface reflecting increased complexity and/or novelty of action complexity (Badre and D’Esposito, 2009; Wood and Grafman, 2003). Critically, during the preparatory period preceding a complex, sequential action, lateral PFC neurons display sequential activity of each action component (i.e., firing related to the movements required at each step of the action), and furthermore this activity reflects many possible forthcoming actions, not just the one action that will be ultimately performed (Mushiake et al., 2006). This response profile indicates that action-planning neurons of lateral PFC generate possible behavioral options during periods of forced quiescence before behavior, suggesting that, under normal circumstances, PFC neurons engage in the generation of multiple possible behaviors prior to a behavioral choice.

We propose that this ability to plan and generate activity patterns of potential forthcoming actions prior to actual behavior provides a means by which PFC can coordinate ongoing behavior. That is, even as rule- or context-based responses are in the process of being established via the reinforcement that occurs during extensive training, PFC likely must simulate potential actions from the large set of possibilities afforded by the situation. When faced with several competing behaviors that could be performed in order to obtain a goal (e.g., moving left versus right into a corridor), PFC neurons can generate relevant action plans before the behavior occurs (as in Mushiake et al., 2006; see also Saito et al., 2005). Before execution, in order to arrive at a decision for action, evidence must be used to evaluate and compare the multiple generated action plans in terms of how likely they are to produce the desired goal. In the case of a laboratory task involving a small set of actions that have been performed previously and individually associated with different rewards or outcomes, these experiences provide a repertoire of knowledge regarding action-outcome contingencies that could be easily used to select the best action (see Kurth-Nelson et al., 2012 for an account of search among options in action-reward learning). In circumstances with higher novelty or uncertainty, another source of evidence is the collection of past experiences that involve similar/associated events, contexts, and actions. Therefore, in these cases, the generated action plans would need to interact with memory of experiences that are only indirectly related to the current situation in order to support the selection of the best action.

The CRAMS proposal

Our CRAMS proposal brings together these two lines of thinking about the role of hippocampus and PFC in adaptive behavior as follows: By virtue of its automatic and obligatory role in binding relations and covertly retrieving any and all relevant relational representations in response to a cue (or partial cue), the hippocampus serves as a vast resource of knowledge relevant to the choices that must be made during behavior. The PFC solicits this information by providing the hippocampus with multiple cues that are relevant to the current choice in the form of potential action plans generated prior to action execution. In a situation requiring a choice among multiple alternatives, the PFC sequentially generates the action plans that would be needed in order to pursue each alternative (see above). These multiple action plans serve to sequentially bombard the hippocampus with cues that activate any related information in memory, due to the automatic and obligatory nature of relational retrieval by hippocampus. Further, because hippocampal reactivation can be covert, i.e., without awareness, experiences can be retrieved covertly. Thus, they serve as covert event simulations for the various alternative action plans that can be used to select the one that is most advantageous in terms of goal obtainment, without requiring awareness. The main aspects of this proposed rapid action-memory simulation cycle are outlined in Figure 1. As reviewed below, we propose that this cycle is rapid (occurring many times per second) and covert (occurs without necessary intentionality or awareness), making it distinct from other accounts of simulation-related hippocampal function, such as processing in the service of the generation of conscious mental imagery and future event construction (Addis and Schacter, 2011; Bird and Burgess, 2008; Buckner, 2010).

Figure 1. Overview of CRAMS.

(A) CRAMS is initiated by response conflict, such as occurs when choices must be made between various options (i.e., decision-making). This conflict generates action plans that could be executed given the constraints of the environment (i.e., affordances). One action plan is covertly generated, leading in turn to covert retrieval of memory related to experiences that were similar to the simulated action in the current situation. The contents of covert retrieval constitute simulated action outcomes that are then evaluated relative to current goals, an “action-memory simulation.” This process of action generation, covert retrieval, and evaluation of the simulated outcome continues iteratively until the value of a simulation exceeds a threshold of proximity to the goal, at which point the cycle ends and the currently selected action is thus performed. (B) The number of CRAMS cycles that would be needed in a given situation is hypothesized to depend on three primary factors: familiarity, uncertainty, and environmental affordances. In highly familiar situations, situations with low uncertainty (i.e., high contrast among simulated outcomes), and when the environment affords few possible options, CRAMS proceeds for relatively few cycles before reaching a satisfactory selection. In contrast, in highly novel situations, situations with high uncertainty (i.e., low contrast among simulated outcomes), and when the environment affords many possible options, relatively more CRAMS cycles are required to systematically test many options before arriving at a selection.

Functional and structural properties of PFC, hippocampus, and their interactivity are consistent with the CRAMS proposal. PFC is extensively and reciprocally connected with hippocampus and surrounding structures of the medial temporal lobe (Amaral and Witter, 1989; Fuster, 2008; Simons and Spiers, 2003). Furthermore, hippocampus and PFC are often co-active during learning (Simons and Spiers, 2003), and long-term potentiation and depression of hippocampal-prefrontal connections can be induced during learning (Doyere et al., 1993). The lateral PFC, which is proximal to motor cortex, is most heavily involved in the generation of action plans (Fuster, 1997; Mushiake et al., 2006; Saito et al., 2005; Tanji and Hoshi, 2008). It can thus generate multiple action plans prior to action execution (Mushiake et al., 2006; Saito et al., 2005). Dorsolateral PFC could provide this information as input to the hippocampus via its direct connections with retrosplenial and parahippocampal cortex (Simons and Spiers, 2003) and could also cue hippocampus indirectly with action plans via efference copy signals reaching hippocampus through connections with the dorsal neocortical “where” stream (Fuster, 1997; see Cullen, 2004 for overview of efference copy for action; see Bird and Burgess, 2008 for a proposed role of PFC efference copy in the generation of mental imagery by hippocampus). Likewise, action plans generated by ventrolateral PFC could cue hippocampus through direct connections with perirhinal and parahippocampal cortex (Simons and Spiers, 2003) or via efference copy through its extensive connectivity with the “where” pathway. Importantly, the hippocampus is a convergence zone for the dorsal and ventral “where” and “what” pathways (Insausti et al., 1987; Squire and Zola-Morgan, 1991; Van Hoesen et al., 1979) and therefore could receive action cues from lateral PFC cortex through multiple routes. Given these cues, the hippocampus can then covertly retrieve related experiences, comprising action-memory simulations.

The simulated outcomes of actions in CRAMS are presumably based on two sources of memory: (1) previous experiences in the same environment and associated outcomes (i.e., to prevent exploring the same region of the environment twice or making the same incorrect choice again), and (2) similar experiences that have occurred previously in different environmental contexts but are nonetheless associated. As described above, two aspects of hippocampal function make it well suited for supporting this kind of memory simulation. First, the hippocampus is important for short-term expressions of memory (e.g., brief retention intervals), especially for complex and/or novel relational information (Hannula and Greene, 2012; Hannula and Ranganath, 2009; Hannula et al., 2007; Hannula et al., 2006; Jeneson et al., 2011; Ranganath and D’Esposito, 2001; Warren et al., 2010; 2013), allowing it to participate in “in the moment” decision-making processes. Second, hippocampal representations are richly relational and associative. Given a cue or partial cue, even distantly associated or goal-irrelevant information (that nonetheless involve at least surface-level similarity to the current environment) could be obligatorily retrieved (Turk-Browne et al., 2010; Wimmer and Shohamy, 2012). Further, awake replay of previous experiences have been found in the rat hippocampus during quiet waking states (Karlsson and Frank, 2009), in the absence of overt environmental input (but possibly due to internally generated action cues from PFC). Thus, given action plans as cues, the hippocampus is able to generate simulations of potential future actions; i.e., information retrieved covertly in response to each cue. We refer to these as “action-memory simulations” to distinguish them from psychological experiences involved in various forms of prospection (Buckner and Carroll, 2007; Hassabis and Maguire, 2009; Schacter and Addis, 2007; Schacter et al., 2008; Zeithamova et al., 2012).

In order to select among the multiple hippocampal-generated simulations cued by PFC during a decision period, simulation information is likely evaluated by medial PFC. Medial PFC receives major input from hippocampus via fornix (Jay et al., 1992) and potentially also monosynaptic input (Parent et al., 2010), and therefore could readily receive the contents of simulated action outcomes produced by hippocampus in response to action plan cues. The medial PFC, especially ventromedial PFC, is densely interconnected with other brain regions that could represent and signal current goals and value representations, such as various neuromodulatory systems, ventral and medial striatum, and amygdala (Hoover and Vertes, 2007; Rudebeck et al., 2008; Rushworth and Behrens, 2008). Indeed, the medial PFC has been characterized as a site of action-outcome prediction (Alexander and Brown, 2011). CRAMS could act in service of this predictive function, as prediction is one outcome of action-memory simulation (as discussed below). This hypothesized contribution from medial PFC also accords with functional neuroanatomical studies in rodents. That is, ventromedial PFC is densely interconnected with the rostral anterior cingulate area (Brodmann Area 32) that is the hypothesized homologue of rodent prelimibc cortex (Wise, 2008). Studies of prelimibic cortex in rodents have identified critical roles in complex working memory tasks and value-based decision-making (Kesner and Churchwell, 2011), providing a potential rodent homologue of the evaluative functions of medial PFC proposed for CRAMS.

In summary, our CRAMS proposal can be tied to the functional neuroanatomy of PFC and hippocampus as follows (Figure 2): When a conflict exists regarding the behavior that should be performed at any moment (i.e., a choice or decision must be made among alternative possible behaviors), lateral PFC sequentially generates various relevant action plans (Saito et al., 2005). Signals of these potential action plans are transmitted either directly or indirectly to hippocampus via the connectivity of lateral PFC to hippocampus described above. Each potential action plan signal serves as a retrieval cue causing hippocampus to automatically, obligatorily, and covertly retrieve relevant contents of relational memory representations stored in distributed cortical modules, via the dense and reciprocal interconnectivity of hippocampus with widespread cortical and subcortical structures (Insausti et al., 1987; Munoz and Insausti, 2005; Van Hoesen et al., 1979). These retrieved experiences comprise covert action-memory simulations of the outcome for each potential action, which reach medial PFC via direct inputs from hippocampus. Medial PFC can then interface with other regions that represent goals and rewards in order to compare action-memory simulations to desired goals, allowing selection of the action that brings the organism into closest alignment with goals. Choice among multiple action-memory simulations by medial PFC could furthermore be aided by frontopolar cortex, which is involved in integrating information over time (Badre and Wagner, 2004; Crescentini et al., 2011; Green et al., 2006; Yarkoni et al., 2005a; Yarkoni et al., 2005b) and is densely and reciprocally interconnected with most of the PFC (Liu et al., 2013). Finally, the selected action generated by lateral PFC drives behavior via standard pathways for motor output.

Figure 2. Hypothesized functional neuroanatomy for CRAMS.

Regions hypothesized to contribute to each portion of the CRAMS cycle are shown superimposed on medial and lateral views of the human brain. Response conflict initiates CRAMS, causing lateral PFC (green oval) to generate action plans that cue hippocampus (blue oval) to engage covert memory retrieval of relevant experience. Medial PFC (red oval) supports evaluation of the action-memory simulation by comparing the simulated outcome to current goals. This cycle is repeated for multiple generated actions until the goal threshold is reached (yellow diamond), at which point CRAMS terminates and motor systems (black oval) are engaged to perform the selected action. Putative contributions from other regions include inputs from striatum and amygdala to medial PFC that aid in goal maintenance and evaluation, as well as interactions between lateral PFC and frontopolar cortex that aid in maintenance of the outcomes that have already been simulated across time. Anatomical images adapted with permission from (Martin, 1996) (Permission pending).

Importantly, we do not envision these interactions as the direct transmission of information from one region to another; rather, the phrase “information transmission” is intended to only metaphorically describe the nature of the interactions of these brain regions. The same function described could be readily obtained via a combination of neuronal properties that are well documented in PFC (Miller and Cohen, 2001), including excitation, inhibition, and threshold setting. For instance, inhibition of lateral PFC motor output until a suitably large reward signal is generated in medial PFC would provide opportunity for excitatory interactions of lateral PFC action-related activity and covert hippocampal retrieval, and lifting of inhibition when the reward threshold is reached would allow performance of the appropriate action. Generation of multiple possible action plans given action uncertainty is also envisioned as an automatic process that derives from the theorized auto-associative and attractor-dynamic properties of PFC circuits (Durstewitz et al., 2000; Wilson and Cowan, 1972). When there is conflict between multiple possible response options, network activity that is subthreshold for producing motor output could be generated until task-related influences temporarily shift network tuning profiles to permit suprathreshold motor signaling (Buonomano and Maass, 2009; Duncan, 2001). There are several possible sources of influence on PFC response selection. Recent theoretical accounts of decision-making emphasize dopaminergic neuromodulation (Lo and Wang, 2006), most likely provided via cortico-striatal connections and the medial PFC regions that compare simulated outcomes to goal states, as proposed above. In general, synaptic facilitation of both afferent pathways and recurrent intra-network circuits likely affect the evolution of PFC networks toward attractor states in the absence of sensory stimulation (Wang, 2008) in a process of cognitive search (Kurth-Nelson et al., 2012). Thus, the initiation of CRAMS need not be under deliberate control, but instead could come about simply by virtue of the automatic response of lateral PFC to conflict among different possible actions due to uncertainty regarding action-outcome associations. Indeed, other findings have also emphasized the ability for PFC, including especially lateral PFC, to generate motor control signals without an individual’s intention or awareness (Lau and Passingham, 2007; van Gaal et al., 2010). Because hippocampal representations can be cued automatically given such cues, complex information processing operations could be solved by relatively simple dynamics within and between PFC and hippocampus within the CRAMS framework.

Connecting CRAMS to ongoing choices: Behavior and neurophysiology

In this section, we further explicate the CRAMS proposal by describing its relevance to a behaviorally simple form of decision-making that can be observed experimentally at choice points among various behavioral options. Vicarious trial-and-error behavior (VTE) was first described by Tolman and Meunzinger (Muenzinger, 1938; Tolman, 1932, 1938, 1939), and comprises an iterative, back-and-forth viewing pattern when confronted with two behavioral choices. They hypothesized that VTE reflects covert simulation of the two possible choices before action (hence “vicarious”), and is thus similar in outcome to the process we hypothesize for CRAMS. VTE is high early in learning and when the choice is most uncertain, reflecting difficulty at arriving at a conclusion based on simulation, and becomes less frequent with learning and increased certainty (Tolman, 1932, 1938, 1939). Furthermore, several experiments have shown that VTE is reduced with lesions of the hippocampus (Hu and Amsel, 1995), especially when VTE is generated in tasks requiring the use of memory of previous episodes to select between the two behavioral choices, such as in two-arm maze tasks as opposed to two-choice perceptual discrimination tasks (Bett et al., 2012). Johnson and Redish (2007) found that hippocampal CA1 neurons generate non-local firing patterns at the choice point in a T-maze, where VTE is generated. This non-local firing reflects “preplay” or “prospective coding” of the two possible forthcoming routes. Furthermore, preplay of one potential route was uncorrelated with preplay of the other, indicating distinct hippocampal simulation of each possibility, and there was a bias for higher preplay of the route that had been previously paired with reward and that was subsequently selected following the preplay events. More recently, hippocampal place cells have been shown to encode spatial trajectories predicting future paths taken toward a goal location in an open arena (Pfeiffer and Foster, 2013). Taken together, these data provide compelling evidence that goal-relevant simulation is occurring prior to behavior, consistent with the CRAMS proposal.

In humans, an analogous behavior to VTE was identified recently (Voss et al., 2011b). Participants explored an array of objects on a computer screen through a controllable semi-transparent viewer that allowed viewing of one object at a time. Individuals exhibited spontaneous revisitation of recently seen objects during visual exploration, a behavior highly reminiscent of VTE behavior seen in rodents in terms of the “back-and-forth” pattern of viewing. In this particular paradigm, advantageous choices concerned what to study for a future memory test, and therefore the VTE-like revisitation strategy correlated with better memory performance upon subsequent test. Furthermore, spontaneous revisitation covaried linearly with activation in hippocampus and PFC. Even more telling, hippocampal amnesics who performed the same memory task rarely exhibited spontaneous revisitation, and it did not improve later memory when performed. Thus, in addition to the surface-level similarities of VTE in rodents and this revisitation strategy in humans, there were also strong similarities in functional properties (i.e., “strategic” and beneficial to performance) as well as relevant functional neuroanatomy (hippocampal dependence). These deeper similarities suggest that the same processes that support VTE in the rodent could be operative in support of decisions about how to explore novel environments in humans.

Notably, the paradigms used to study VTE in rodents and revisitation in humans differ fundamentally, especially in terms of the number of options for behavior that are available and thus the degrees of freedom inherent in the choice. A novel hypothesis that derives from our CRAMS proposal is that the amount of action-memory simulation iterations that must be performed before a choice is reached will scale with the number of response options available as well as with the uncertainty involved in weighing the simulation outcomes against one another (Figure 1B). These two factors, the number of response options and the uncertainty, operate independently and derive from different sources of information. The number of possible response options is determined by environmental or external constraints (i.e. number of paths at a choice junction), while uncertainty or difficulty of weighing outcomes derives from either inadequate previous experience (highly novel) or inadequate information on which to base discrimination among the choice options (highly uncertain). Thus, early in learning in a highly novel situation involving even a small number of choices, such as in the VTE testing apparatus, no single simulation outcome would rise above the rest, and therefore the animal would be expected to engage in many action-behavior simulations in a (futile) attempt to select the best action (Figure 1B). As learning progresses, previous experiences would provide information for weighing different options, thereby decreasing uncertainty, and less simulation iterations would be performed. Interestingly, this decline in VTE is exactly what is observed as animals acquire task knowledge, and, furthermore, the amount of initial VTE scales with the difficulty of the discrimination and hence with uncertainty (Tolman, 1932, 1938, 1939).

Likewise, we would expect that more complex decision situations, with more possible actions than in two-choice tasks, would involve more simulation, but to date there are few relevant datapoints on more complicated discrete-choice tasks. However, open-ended exploration of novel environments is a scenario that requires ongoing choices and that is likely heavily dependent on CRAMS. Even though this situation lacks discrete (e.g., experimenter-defined) choice points, we hypothesize that CRAMS occurs continuously during exploration. During exploration, choices must be made continuously regarding where to explore, when, and for how long. Figure 3 highlights that CRAMS could be similarly involved in simple, discrete selections at choice points as in the moment-to-moment decisions that are necessary for effective exploration. Indeed, we have found that the hippocampus makes a necessary contribution to exploration of novel information even over brief timescales (Yee et al., 2013). In this study, amnesic individuals with bilateral hippocampal damage generated inefficient visual search paths of novel scenes, whereby they frequently engaged in path-crossing and repeated visitation of the same locations while searching for hidden targets. Consistent with our CRAMS proposal, this suggests that the hippocampus must be called upon during the course of exploration to provide information regarding what has been explored in the recent past in order to support optimally efficient continued exploration.

Figure 3. CRAMS varies among environments based on the richness of options.

(A) The standard VTE apparatus involves a choice point with two options. Rodents thus iteratively simulate the two options before choosing. Early in training, alternation between the two options is high because of high novelty and high uncertainty, whereas alternation decreases with training (Figure 1B). (B) Similar choice behavior occurs in a more complex environment such as an arena, although the number of options available at every moment is much higher. Iteration thus occurs continuously among many options. Previous experiences with the same arena and with similar arenas, objects, and situations leads to more distinction among the various options during simulation (indicated by darker versus lighter arrows) and thus more efficient testing of the various options via CRAMS. (C and D) Humans are faced with similar challenges at discrete choice points as well as in more complex environments, and we hypothesize that similar CRAMS mechanisms guide choices.

The rapid and iterative nature of PFC-hippocampal interactions that are a central component of CRAMS is indirectly supported by findings of rhythmic activity of PFC and hippocampus in situations that require selection among behavioral choices. Theta-modulated phase-locking between PFC and hippocampus occurs during exploration (Brockmann et al., 2011) and, importantly, at choice points after task rule acquisition (Benchenane et al., 2010). Further, comodulation of cortical gamma and hippocampal theta predicts memory retrieval for recent spatial exploration events (Shirvalkar et al., 2010). This rhythmic coupling of PFC and hippocampal neurons could reflect the CRAMS iterative cycle of action-memory simulation. Interestingly, each “sweep” of hippocampal preplay in the Johnson and Redish (2007) study lasted for 0.15 s, and therefore would generate an approximately 7-Hz rhythm (in the range of hippocampal theta) if cued iteratively for difficult and or multiple-possibility choices. Furthermore, exploratory behaviors such as eye movements tend to occur at theta frequency (Landau and Fries, 2012) and theta-band hippocampal activity occurs with exploratory eye movement in the primate hippocampus (Jutras et al., 2013). Thus, iterative covert simulation of these exploration events would also be expected to occur with similar timing (Mushiake et al., 2006).

We speculate that most decision-making tasks fall somewhere between the two-choice VTE scenario and open-ended environmental exploration in terms of the number of simulations that are required for each choice. This is due to the different levels of constraints on the number of choices that must be considered imposed by the environment and by previous learning (i.e., constrained vs. unconstrained environments, familiar vs. novel situations, and high vs. low uncertainty, as described above and emphasized in Figures 1B and 3). Consequently, we would expect theta to vary accordingly to the extent that it is an indicator of CRAMS. Although hippocampal theta has been characterized as unreliable and difficult to identify in primates, Guitart-Masip et al. (2013) recently identified evidence for theta synchrony of PFC and hippocampus in humans during a decision-making task. Furthermore, Stokes et al. (2013) found that PFC transitions from a high-activity stimulus-specific network state to a low-activity network state that is associated with the rule that will be subsequently applied within 150–300 ms of the rule cue. The temporal evolution of this effect suggests that each of the two possible rule options could have been simulated at theta frequency before the rule-specific network state was achieved. In rodents, hippocampal theta is a dominant frequency, although it is important to note that rodents spend most time engaged in exploratory movements, and exploration is a strong modulator of hippocampal theta in rodents (Buzsaki, 2011; O’Keefe and Nadel, 1978). Emerging evidence from primate eye-movement studies suggests that likewise hippocampal theta is robust during visual exploration (Jutras et al., 2013; Killian et al., 2012). These studies provide tentative links showing that expected variation in the amount of CRAMS cycles that would be necessary in a given situation (e.g., Figure 1B) corresponds roughly to the amount of theta that is generated in that situation, providing indirect support for the notion that theta is a neurophysiological marker of simulation cycles during CRAMS.

It is important to clarify that although we provide the example of decision-making during the exploration of physical space, the CRAMS framework proposes no special role for spatial processing by either PFC or hippocampus. We propose that the processes involved in CRAMS are as relevant when one is making a cognitive decision among multiple competing mental representations as when making a choice among multiple competing routes or directions to explore in space. Indeed, there are many parallels between exploration of physical and cognitive spaces. For instance, Hills and colleagues (2008, 2010) have shown that the mode of searching physical space (e.g., for a visual target) alters the mode of searching cognitive space (e.g., searching for a word to retrieve). Furthermore, Metcalfe and Jacobs (2010) drew several interesting parallels between students’ allocation time during studying verbal materials and animals’ food-foraging behaviors. For instance, animals utilize external cues such as scent markings to tag already-visited locations, whereas humans use self-assessment cues for similar purposes. In general, although experimental paradigms often lend themselves to testing spatial questions, especially in nonhuman animals, considerable evidence and theorizing suggests that similar processing can be used to support decisions in both physical and cognitive spaces (Shapiro and Eichenbaum, 1999). Indeed, with respect to lateral PFC control of action, Fuster has suggested that similar mechanisms are used to generate and select thoughts (cognitive actions) as motoric actions (Fuster, 1997, 2008; see also Wood and Grafman, 2003). We therefore propose that the action-memory simulation processing involved in CRAMS operates similarly for cognitive decisions as decisions regarding where and how to traverse space (Figure 4).

Figure 4. Similar mechanisms for exploration of cognitive and physical spaces.

A depiction of the mental search involved in name recall given a face is provided in order to highlight that CRAMS could be as relevant for choices among mental possibilities as it is for choices among action possibilities. For instance, face-cued name recall can involve search among different places, events, and situations in which the person could have been encountered. Similar to choices encountered in a physical space, one can encounter constraints in choices and variability in novelty and uncertainty imposed by the “cognitive” environment that can dictate the number of logical options to be considered, as well as the amount of time required to converge on a solution. Thus, choices that are considered more purely “mental” than choices regarding how to act could nonetheless rely on similar CRAMS mechanisms.

Relationship to other theoretical frameworks

Our CRAMS proposal offers a way of conceptualizing how “top-down” control by PFC is established. Although many findings have confirmed that PFC influences other brain regions in order to select behaviors that are context-dependent and rule-based (Chafee and Goldman-Rakic, 2000; Lee and D’Esposito, 2012; Miller and Cohen, 2001), and neurophysiological studies have shown that this is based on context-dependent responsivity of PFC neurons (Asaad et al., 2000; Stokes et al., 2013; Wallis et al., 2001; White and Wise, 1999), little information has been obtained regarding how this top-down control comes to exist. One ramification of CRAMS is the repetition of a “dumb” (i.e., automatic, obligatory, and covert) iterative process that arises whenever conflict occurs among various choices could produce over time what appears to be very “smart” behavior (i.e., rule-based decisions). That is, without effort or intention, CRAMS leads the organism to make its best guess using all available information in moments of choice conflict, and, over time, learning refines the set of options in order to produce adaptive rule-based or context-dependent responding. Based partially on our evidence for similar neural mechanisms for VTE in humans (Voss et al., 2011b) as compared to animals (Johnson and Redish, 2007; Johnson et al., 2012; Muenzinger, 1938; Tolman, 1932, 1938, 1939), we suggest that processes involved in CRAMS might serve as common mechanisms for the development of complex rule-based decisions that are analogous across humans and other mammals. That is, despite the seemingly vast differences in how decisions appear to be made and experienced by humans versus nonhuman animals, critical underlying mechanisms involving CRAMS could be present in both cases. The disproportionately greater PFC in humans might thus confer greater ability to make and test a larger set of appropriate actions during choice periods, but perhaps mediated by very similar PFC-hippocampal mechanisms.

Brain regions and mechanisms other than those we have described are certainly important for adaptive function. For instance, Hills (2006) reviews the substantial evidence for the role of basal ganglia in the strategic decisions involved in foraging, primarily obtained in studies of rodents. Indeed, basal ganglia contributions to reward learning have also been emphasized in human decision-making (Delgado and Dickerson, 2012). With respect to CRAMS, it is possible that basal ganglia networks (via their interactivity with PFC) help compute the relative costs and benefits associated with simulated actions (Hwang, 2013). As an animal learns the mappings between simulated actions and outcomes, decisions that initially rely on CRAMS-related contributions from PFC and hippocampus could become automatic and dependent on basal ganglia. Thus, in rodents, many strategic decisions could be supported by basal ganglia, consistent with the relative inflexibility and paucity of novel behaviors generated in the face of novel challenges in rodents versus mammals with more highly developed PFC. Indeed, increased PFC development is generally associated with greater consideration and performance of a wider range of behaviors in response to novel challenges. For instance, cats will make varied attempts to escape from a puzzle box whereas rats make less-varied attempts, nonhuman primates attempt novel actions with objects in order to acquire goals, and humans display greater and more varied attempts to solve novel problems as PFC increases throughout childhood development. Action-memory simulation involved in CRAMS could be central to these varied instances of adaptive behavior and problem solving generated to varying degrees in accordance with the degree of PFC and hippocampal development. A key distinction between the PFC-hippocampal interactivity proposed for CRAMS versus contributions that other brain regions might make to adaptive function is that CRAMS is especially relevant to novel situations. That is, the flexible, relational representations supported by hippocampus are particularly suitable to simulating potential outcomes in situations that are only mildly related to past experiences. In this case, hippocampus is able to simulate outcomes based on its ability to retrieve many aspects of associated experiences given a cue that is only partially related to those circumstances (as described above). In contrast, structures such as basal ganglia would be better suited to providing specific action-outcome associations in response to action cues from PFC, and are likely more involved when strong action-outcome associations already exist for a given circumstance.

As mentioned above, CRAMS bears some similarity to other considerations of how hippocampal processing is related to the generation of visual mental imagery and future planning (Bird and Burgess, 2008; Buckner, 2010; Buckner and Carroll, 2007; Eichenbaum and Fortin, 2009; Hassabis and Maguire, 2009; Schacter and Addis, 2007; Schacter et al., 2008). The main novelty of CRAMS is the notion that simulation in the service of behavioral choices and decisions is highly rapid and iterative as well as covert/implicit. CRAMS in no way requires that any form of deliberate or executive control need be intentionally exerted by an individual. That is, the iterative PFC interactions with hippocampus do not need to be initiated by a conscious or willful central executive. Just as we conceive of the role of hippocampus in retrieving relational information as automatic, obligatory, and covert, so too we propose that the action generation operations of PFC are likewise automatic, obligatory, and covert. Covert simulation of hippocampal activity (e.g., “preplay” by hippocampal primary neurons) has been considered as a precursor or component of visual mental imagery and the construction of future scenarios based on memory (Bird and Burgess, 2008; Buckner, 2010). However, our characterization of hippocampal involvement in CRAMS differs from its hypothesized contributions to mental imagery and mental simulation of future events based on memory (as in Addis and Schacter, 2011; Bird and Burgess, 2008; Buckner, 2010; Eichenbaum and Fortin, 2009). We emphasize prefrontal-hippocampal interactivity, and propose that PFC cues retrieval, and hence preplay-like phenomena, by providing signals of possible actions to the hippocampus. In the case of VTE in rodents, this is accompanied by iterative viewing of the two choices. In most decision-making circumstances, myriad possible choices for action are possible, and thus the PFC iteratively bombards hippocampus with many possible actions and obtains simulated outcomes until the best choice is reached. Thus, there is little room for mental imagery in this scenario, given that rapid generation of multiple mental images would be disorienting, and it is not commonly reported as part of psychological experiences in cases of ongoing decision-making in situations such as exploration. Instead, we propose that action-cued hippocampal simulation is a covert process, whereby the contents of hippocampal retrieval are not experienced consciously, but instead can be used without necessary conscious awareness to guide the choice. In the case of VTE at a two-choice decision point in a well-learned apparatus, the possible response options have been limited (through experience and by the nature of the apparatus) to only two, and so simulation options have been considerably restricted. Thus, the animal alternates between them until a decision is reached. In this limited scenario, iterative simulation occurs for a relatively small number of options, which could promote conscious awareness in some circumstances (for instance, because repeat simulation could enhance stable and synchronous neural responses that have been associated in other contexts with visual awareness, e.g., Pollen, 1999). In contrast, covert simulation is especially relevant for situations in which there is need to rapidly test multiple possible options, such as during more open-ended exploration.

CRAMS is thus also distinct from other proposals of PFC in predictive function (Henson and Gagnepain, 2010). Although one ramification of CRAMS is that PFC supports predictive function (i.e. likely forthcoming actions and their outcomes will be generated in advance of behavior), CRAMS is mechanistically distinct from other proposals in that predictive function is based on simple iterative communication with hippocampus. Due to its iterative component, CRAMS has surface-level similarities with other iterative models such as T.O.T.E. (Miller et al., 1960). However unlike existing models of iterative PFC functions for decision making, such as the proposed iteration of goal setting, planning, plan execution, and monitoring (Mushiake et al., 2009), CRAMS considers the critical role of memory-retrieval functions of hippocampus, which are essential to evaluating different possible actions before they are performed based on previous experiences.

Future directions, summary, and implications

Future experiments will be needed to test various aspects of the CRAMS proposal. One simple prediction of CRAMS is that the number of action-memory simulations will vary according to factors such as the number of possible alternative choices presented at decision points as well as the relative strength of any one alternative with predicted outcomes (i.e. uncertainty). We therefore consider exploration of the problem space between two-choice VTE-like scenarios and open-ended exploration as a promising experimental route for testing CRAMS. We expect that as the number of possible options or outcome uncertainty increases, the iterative nature of the CRAMS processes will result in higher cognitive load and therefore increase the time needed to make a decision or decrease the relative accuracy of selecting the optimal decision. Manipulating the relative proportion of uncertainty associated with different specific actions might be expected to also affect the ability to simulate future outcomes and arrive at optimal decisions. Furthermore, we expect that lesions to critical areas such as lateral PFC or hippocampus will profoundly affect expression of CRAMS and the timing of interactions between these regions. The role of theta frequency in mediating these interactions also warrants further investigation.

We have proposed a framework of learning and decision-making that incorporates and is supported by key recent findings regarding strategic and adaptive behavior. Our model easily captures various aspects of exploratory learning such as the benefits of volitional control in humans (Voss et al., 2011a; Voss et al., 2011b) and animals (Song et al., 2005) to later memory performance. An important feature of CRAMS is the ability to account for decision-making despite lack of necessary conscious awareness or a central executive, allowing it to extend beyond human cognitive theory and encompass animal models of behavioral control and adaptation. We also emphasize the importance of iterative interaction between PFC and hippocampus during action selection, rather than localized functions employed separately. More detailed exploration of CRAMS is important for understanding the detailed mechanisms of these functions and, based on the considerations we review, could motivate powerful comparisons of higher-order cognition in humans and animal models.