Latent learning, cognitive maps, and curiosity

Abstract

Curiosity is a desire for information that is not motivated by strategic concerns. Latent learning is not driven by standard reinforcement processes. We propose that curiosity serves the purpose of motivating latent learning. While latent learning is often treated as a passive or incidental process, it normally reflects a strong evolved pressure to actively seek large amounts of information. That information in turn allows curious decision makers to represent the structure of their environment, that is, to form cognitive maps. These cognitive maps then drive adaptive flexible behavior. Based on recent data, we propose that orbitofrontal cortex (OFC) and dorsal anterior cingulate cortex (dACC) play complementary roles in curiosity-driven learning. Specifically, we propose that (1) OFC tracks intrinsic value of information and incorporates new information into a cognitive map; and (2) dACC tracks the environmental demands and information availability to then use the cognitive map from OFC to guide behavior.

Main Text

The natural environment offers a plethora of rewards to most foragers but acquiring these rewards requires knowledge [1,2,3,4]. For example, red knots (arctic shorebirds), feed on bivalves that are patchily distributed and buried in the mud [5]. Notably, the locations of these prey cannot be guessed based on visual inspection, but can be inferred based on a rich knowledge of likely patch structure and distribution of other prey. When foraging, the birds reside in patches longer than predicted by simple foraging models; their overstaying can explained by modified models that include a bonus for the information that the extra residence time provides. Knots are typical of many natural decision makers, which are constantly starved for information. Notably, this is an area where laboratory experiments tend to differ most starkly from natural decision-making contexts; in the lab relevant information is typically made available and, if obscured, simple.

Curiosity, which we can define as a drive for non-strategic information, is a major driver of learning and determinative of the success of development in humans and other animals [6,7,8,9,10,11]. Its features are preserved across species and over the lifespan. It appears to be associated with at least somewhat discrete neural circuits [12,13,14].

Latent learning

Classical concepts of learning held that all learning is driven by reinforcement contingencies. These ideas are fundamental to the “Law of Effect” and to Hebbian learning [15,16]. That work, in elaborated form, is central to reinforcement learning, one of the most successful psychological theories and the basis of a generation of systems neuroscience, and to much of machine learning.

However robust learning can occur in the absence of reward [17,18]. This idea poses a challenge to simple reinforcement learning models, which Tolman termed the “stimulus response school.” In a classic latent learning setup, a rat is released into a large maze with no reward. Naive rats typically amble around the maze, ostensibly with no purpose. Later, the experimenters introduce a reward to a specific location in the maze. The rats with maze exposure learned to locate the reward much more quickly than ones who were naive to that maze. The rats learned the maze structure – and formed a cognitive map – latently.

Curiosity and cognitive maps

Any forager placed within a complex natural environment must naturally trade off between the costs and benefits of exploration. In addition to the metabolic costs of locomotion, sensory processing, and learning, active exploration carries opportunity costs: that time could be better spent searching for food, courting and reproducing, or avoiding predators. For example, in the case of the knot, the delay in travel time imposes an opportunity cost in the form of foregone large rewards at new patches. Even motivational processes driven by distal reward seeking must necessarily discount future rewards and uncertain rewards, and the benefits of exploration are unavoidably delayed beyond the temporal horizon and, individually, infinitesimally unlikely. So reward-maximizing calculation is unlikely to sufficiently motivate search. Instead, evolution must endow the decision maker with intrinsic motivation to learn and ultimately to map its environment [8,19,20,21].

Indeed, curiosity would seem to go hand in hand with the learning of cognitive maps. Cognitive maps refer to detailed mental representations of the relationship between various elements in the world and their sequelae [22,23,24]. Having a cognitive map allows a decision maker to not just guess what will happen but also to deal with unexpected changes in our environment. The classic idea about cognitive maps - also attributable to Tolman - is that they allow us to respond flexibly when contingencies change (e.g. when the layout of a maze changes, [18,25]. That kind of flexibility is very difficult to implement with basic reinforcement learning processes [18,22,23,24,25]. Instead, it requires a sophisticated representation of the structure of the world.

Critically, cognitive maps typically require a rich representation of the world- they require a level of detail that is not normally available from reinforcement learning processes. That detailed representation of the linkages between adjacent spaces allows for vicarious travel along those linkages. Because it is so much more detailed, it requires orders of magnitude more information than standard reward-motivated reinforcement learning can give. Getting that information cannot occur if it needs extrinsic rewards - those rewards simply are not available in the environment.

We propose, therefore, that latent learning is motivated and enabled by curiosity. However, Tolman conceived of latent learning as a fundamentally passive process, one that took place during apparently purposeless exploration - almost as if by accident. We propose, instead, that latent learning in practice tends to be more actively driven. However, this purposive exploration must be e driven by the evolutionary advantage brought by curiosity and ultimately by the extreme information gap experienced by foragers in the natural world.

The analogy to artificial intelligence

The problems faced by a naturalistic decision-maker or forager are similar in many ways to the problems faced by artificially intelligence (AI). When AI performs classic Atari games, it uses straightforward RL principles [26]. But those games, especially the ones that AI is good at differ from natural situations. The real world – and some games like Pitfall and Montezuma’s Revenge - are what is known as hard-exploration problems [27,28,29]. Rewards are sparse (they require dozens or hundreds of correct actions), so gradient descent procedures are nearly useless. For example, in Pitfall, the first opportunity to gain any points comes after ~60 seconds of perfect play involving dozens of precisely timed moves. Moreover, rewards are often deceptive (they result in highly suboptimal local minima, so getting a small reward promotes adherence to a suboptimal strategy). RL agents that do well at relatively naturalistic hard-exploration games tend to have deliberate hard-coded exploration bonuses [28,29].

The AI domain provides a good illustration of how cognitive maps can be crucial for the success of curiosity. The optimal search strategy in sparse (natural) environments is typically to identify a locally promising region and then perform strategic explorations from that spot to identify subsequent ones [30]. That exploration will not be random, but will take place along identified high-value destinations. AI agents suffer from the problem of detachment, that is, when they explore the environment, they leave the relatively high-reward areas of space to explore lower-reward ones [28]. Most such areas are likely to be dead ends, and, when a dead end is detected, the agent ought to return to the high reward area and pursue other promising paths. However, the basic curiosity-based approach, which gives intrinsic rewards for novelty, repels the agent from returning to the promising region of space, precisely because it’s the most familiar and least intrinsically rewarded (it’s also not extrinsically rewarding, because any extrinsic reward has been consumed on the path there, and does not replenish in the meantime). This in turn requires making some kind of internal map of space so that the agent can return to the locus of high potential reward and explore more efficiently than a wholly random path.

A closely related problem that AI agents - and real-world agents as well - face, is the problem of derailment [28]. To explore a space efficiently, an agent must be able to return to promising states and use those as a starting point for efficient exploration. From there, the agent must engage in random search. However, in real environments, returning to a promising state may require a very precise sequence of actions that cannot be deviated from - so stochasticity must be controlled until that state is achieved, at which case it must begin again in earnest. As such, a stochastic search must be carefully controlled depending on one’s place in the larger environment - which requires basic mapping functions, and cannot be done with simple RL-type learning. Moreover, important factors governing the exploration process, such as detecting an information gap, deriving the value of information itself, and directing exploration towards potential sites that might be low in external reward but high in information/entropy, simply cannot be supported by only experienced reward history. The key to achieve this, again, is to have a mental map, or internal model, of what is available, and what is novel and potentially offer high information content (high entropy).

Operational definition of curiosity

Developing these ideas about curiosity, latent learning, and cognitive maps holds great potential in neuroscience. However, it faces several problems from the get-go. We and others have defined curiosity as a motivation to seek information that lacks instrumental or strategic benefit [6,7,8,10]. By this definition, many explorative and playing behaviors qualify as a demonstration of curiosity [9,31]; even risk-seeking and other learning behaviors may be explained that way [32,33,34,35]. But this definition is vague and does not readily lend itself to many laboratory contexts. In an effort to remedy these drawbacks, we developed a conservative operational definition that combines three criteria: (1) a curious research subject is willing to sacrifice primary reward in order to obtain information; (2) the amount of reward a subject is willing to pay scales with the amount of potentially available additional information; and (3) additionally gained information provides no obvious instrumental or strategic benefit.

We devised a more complex task [37] that would circumvent published criticisms [36] of the observing task. This task is based on the observation that monkeys seek counterfactual information - information about what would have happened had they chosen differently [38,39]. In the counterfactual curiosity task, monkeys choose between two risky offers. During testing, monkeys are sometimes given the opportunity to choose an option that will provide valid information about the outcome that would have occurred had they chosen the other option. Monkeys are willing to pay to choose this option, indicating that they are curious about counterfactual outcomes. Moreover, monkeys paid more for options that provide more counterfactual information. We speculate that this curiosity-driven information-seeking helps monkeys to develop a sophisticated cognitive map of their task environments [37].

Functional neuroanatomy of curiosity in the frontal lobes

Our ultimate goal is to understand the neural circuitry underlying curiosity-driven choice. Here we summarize the tentative picture, with a focus on two prefrontal regions, the orbitofrontal cortex (OFC) and the dorsal anterior cingulate cortex (dACC). Both regions are implicated in neuroimaging studies of curiosity [40,41,42]. The neuroanatomy of curiosity is more complex and includes other areas such as hippocampal areas [12,43,44] and basal ganglia [33]. But we would like to highlight OFC and dACC for their potential involvement that bridges curiosity, latent learning, and cognitive maps.

Orbitofrontal cortex:

We propose that OFC serves to (i) track the intrinsic value of information, (ii) maintain a cognitive map of state space, and (iii) update that map when new information is gained. The clear role of OFC in cognitive mapping has been one of the major intellectual advances of the past decade, and is demonstrated in rodents, monkeys, and humans [23,24,45,46]. This theory, for example, integrates economic findings in OFC with evidence that it carries non-economic variables [46,48,49,50]. This extends to curiosity [51]. In that study, we found that OFC neurons encode the value of information and (confirming much previous work) the value of offers. Critically, OFC uses distinct codes for informational value and more standard juice value. This distinction presumably reflects the role of that information in updating the cognitive map -- even though this information may not offer immediate strategic benefit. In other words, OFC doesn’t use a single coherent value code across contexts, but rather, represents task-relevant information in multiple formats, as would be expected in a map rather than a simple reinforcement learning situation. Of course, OFC does not achieve this alone. Studies using similar paradigms show that information is signaled by other systems, including the midbrain dopamine system (e.g. [52,53,54]).

Dorsal anterior cingulate cortex:

We propose that dACC plays a distinct and complementary role to OFC. Specifically, it appears to track both information delivery and level and task demands for use by OFC in updating the cognitive map and applying it to instrumental use. This idea is motivated by the observation that dACC tracks informativeness, counterfactual information [33,38], environmental demands [55,56], as well as various economic variables (e.g. [57,58,59,60]. It is further motivated by observations about the relative hierarchical positions of the two regions and their relative contributions to choice [61,62,63]. Perhaps most relevantly, in a recent study, White et., al. [33] trained monkeys to associate juice rewards with various reward probabilities with different fractals. Single units in dACC showed increased firing rates to increased uncertainty, and thus to higher expectation of information (when the uncertainty resolved). Moreover, dACC firing rates ramped up to the anticipation of the information that came with the resolution of the uncertainty. In other words, dACC neurons did not simply encode different levels of uncertainty which remained at a constant level for each trial; nor did they ramp up firing rates in anticipation to reward delivery (see also our own results, which paint a somewhat similar picture, [64]).

Conclusion and future directions

Curiosity has long been treated mystically, as if it is impenetrable to scholarly study. Even when treated as a regular psychological phenomenon, curiosity is often studied in an ad hoc manner. More recent work has made great progress in developing formal approaches to understand the phenomenon systemically study its neural substrates. That formal approach, aided by remarkable progress in AI, has in turn allowed neuroscientists to tentatively start to understand the circuity of curiosity. That work in turn will likely be critical for understanding naturalistic decision-making, which is marked by the need to make quick decisions with orders of magnitude less information than would be ideal.

Figure 1.

A virtual maze our lab has used for monkeys based on the classic alley maze of Tolman. Tolman and his graduate students placed rats in mazes like this one and found that they would explore the maze unrewarded and would demonstrably learn the features of the structure of the maze in the absence of rewards, a result that is difficult to explain using then-dominant simple stimulus-response learning theories. Tolman proposed that the rats generated a cognitive map that instantiated features of the maze and could be consulted to drive flexible behavior.

Figure 2.

Monkey in a tree, illustrating the problem of derailment in curiosity research. The monkey must learn foraging strategy through trial and error, which requires a highly variable exploration of the environment. But getting to the end of a branch is somewhat risky and requires suppressing stochastic variability. To successfully deploy curiosity the monkey must have a cognitive map of where variability is good and where it is bad.

Figure 3.

In our curiosity task, subjects could choose between risky options for juice rewards. In some trials, they could also gain information about what would have occurred had they chosen differently. By analyzing preference curves on such trials, we could quantify their subjective value of counterfactual information. We found a small but significant positive valuation of counterfactual information in both subjects tested (Wand and Hayden, 2019).

Figure 4.

Responses of an ensemble of OFC neurons to offers varying in informational value and reward value (Blanchard et al., 2015). We find that individual neurons encode both variables (horizontal and vertical axes indicate tuning coefficients for the two dimensions respectively). However, those codes are themselves uncorrelated, as indicated by the lack of a significant slope between the two dimensions.