Neural circuits for inference-based decision-making

Abstract

In novel situations, where direct experience is lacking or outdated, humans must rely on mental simulations to predict future outcomes. This review discusses recent work on the neural circuits that support such inference-based behavior. We focus on two specific examples: 1) using knowledge about the associative structure of the world to infer outcomes when direct experience is lacking; 2) inferring the current value of options when the desirability of the associated outcome has changed since the original learning experience. These two examples can be studied in the sensory preconditioning and devaluation tasks, respectively. We review results from studies in animals and humans suggesting that the orbitofrontal cortex (OFC), together with the hippocampus and amygdala, is necessary for inference in both of these tasks. Together, these findings suggest that the OFC is a critical hub in the brain network that supports inference-based decision-making.

Introduction

Imagine deciding where to go for your winter holidays. You might consider traveling to a village in Vermont, as you’ve traditionally done with your family. To evaluate this option, you can use your direct experience from previous years. However, what if your family proposes to instead spend the holidays at Oahu’s North Shore? Since you’ve never been to Hawaii, you need to evaluate this option based on indirect knowledge, mental simulation, or inference [1–4]. Inferring future outcomes to guide decision-making requires knowledge about the associative structure of the environment; a so-called cognitive map [5,6]. Using this map, we can mentally simulate the likely consequences of decisions and evaluate different options, even if we have never encountered them before.

This review focuses on the brain regions and networks that support inference-based decision-making, taking into account correlational and causal experiments in humans and animals. Specifically, we focus on two examples of inference-based decision-making. First, we discuss decisions in novel situations, which require mental simulation to predict future outcomes. Second, we consider cases where the value of outcomes has changed since the original learning experience, which requires their current value to be inferred. These two forms of inference have been widely studied in rodents and non-human primates using two classical behavioral tasks, sensory preconditioning [7] and devaluation [8], offering well-validated paradigms for studying inference in humans in a way that facilitates cross-species comparisons. We propose that inference in both of these tasks depends on the flexible use of learned associations, and we will highlight data suggesting that such processing is supported by a set of brain areas which includes the orbitofrontal cortex (OFC), hippocampus (HC), and amygdala.

Using associative structures to infer novel outcomes

To mentally simulate a vacation on the island of Oahu, you might use associative knowledge that you have acquired in the past. For instance, let’s say you know that Barack Obama’s family spends Christmas at Oahu’s North Shore, and that you also agree with his policy decisions and how he represented the U.S. as the 44^th president. Based on these relationships, you could now make an inference that you will likely enjoy spending the holidays on Oahu. This inference process can be more formally studied in the lab using the sensory preconditioning task, which consists of learning value-neutral and value-related associations and using these associations to infer outcomes that have never been directly experienced.

For instance, in a recent study conducted in our lab [9], human subjects learned associations between different visual cues during a preconditioning phase (A-B and C-D, Fig. 1A). Then, in a conditioning phase, they learned that cue B predicted a reward whereas cue D was not followed by a reward. Finally, they were asked to predict the likely outcomes associated with cues A and C in a probe test phase. Subjects were more likely to predict a reward after cue A than C (Fig. 1B), indicating that they used the association between A and B, and their knowledge that B leads to reward, to predict that A would also lead to reward. Importantly, subjects never experienced a reward after cue A, meaning that they could not make this prediction based on their direct experience. As such, this behavior must result from a novel inference that involves mentally connecting the associations between the two cues and between cues and rewards. Importantly, this ability to infer outcomes is not unique to humans but has widely been observed in rodents [3,10–12], suggesting it is a fundamental mechanism of behavioral control.

Figure 1. Sensory preconditioning task.

A. During preconditioning subjects learned cue pairs (A→B, C→D) and then during conditioning they learned associations between the second cue in each pair and one of two outcomes (B→reward, D→no reward). During the probe test, subjects made outcome predictions for all cues in the absence of outcomes. B. Relative to sham, OFC-targeted TMS before the probe test disrupts reward predictions based inference (responding to cue A) but not based on direct experience (responding to cue B).

Studies using neural recording and imaging techniques in rodents and humans have shown that inference in the sensory preconditioning task recruits HC and OFC [9,10,12]. For instance, our study referenced above showed that cue-cue associations emerged in HC and OFC during the initial learning phase [9]. That is, fMRI activity patterns in HC and OFC evoked by paired cues (A-B, C-D) became more similar over the course of preconditioning. Comparable effects were observed in precentral gyrus, middle occipital gyrus, and insula. These results are consistent with findings in rodents, showing that responses of OFC neurons to paired but not unpaired cues become more similar over the course of preconditioning [10], suggesting that these neurons encode the associative structure of the task during preconditioning. Moreover, OFC and HC were also involved at the time of decision making. Specifically, during the probe test, activity in medial and lateral OFC in response to cue A reflected the inferred outcome in the same way as it reflected the expected outcome in response to cue B. Interestingly, although we did not find neural correlates of the inferred outcome in HC, the functional connectivity between medial OFC and HC increased in trials that required subjects to infer outcomes (cues A and C) relative to trials in which subjects could respond based on previous experience (cues B and D) [9]. This increase in connectivity could suggest that HC may help to bind associative information in the OFC to form an integrated representation of the task that can be used for inference.

Furthermore, studies across species have shown that HC and OFC are necessary for inference-based responding in this task. For instance, correct inference in the sensory preconditioning task is disrupted by pharmacological inactivation of the OFC in rats [3], optogenetic inhibition of HC output neurons in mice [12], and by transcranial magnetic stimulation (TMS) targeting central-lateral OFC networks in humans [13]. Moreover, hippocampal damage in humans and optogenetic inactivation of HC in rodents affect behavior in other two-step decision-making tasks that also require model-based inference [14,15]. Importantly, impairments in these experiments are typically selective for decisions requiring inference, and are not observed when the response could be based on direct experience. For instance, in a study from our lab [13], network-targeted TMS of the central-lateral OFC impaired subjects’ ability to infer the outcome for cue A, but not responding to cue B, for which subjects had directly learned the cue-outcome association (Fig. 1B). Thus, disruption of OFC activity does not lead to a general impairment in decision-making. In summary, these findings show that activity in brain networks involving the OFC and HC is critical for supporting inference-based decision making across species.

Inferring the state-dependent value of outcomes

Inference is also necessary when the value of outcomes has changed since we originally learned about them. For instance, the value of travel destinations changed with the COVID-19 pandemic, and using experiences acquired in previous years would have resulted in maladaptive decisions in 2020. To avoid this, we had to update the value of these destinations. Accessing updated values represents a novel inference, because it diverges from what we have directly experienced in the past (e.g., holiday travel in previous years). This inference is structurally similar to what is probed in the sensory preconditioning task described above. However, instead of flexibly combining learned associations, it requires using associations between cues and specific outcomes to infer their new value based on the current context. Such inferences can be studied using the Pavlovian outcome devaluation task [16].

In a non-human primate version of this task [17], animals first learn that choices of object A and object B lead to the delivery of two different food rewards (e.g., A → peanuts and B → M&Ms). Animals are then given ad libitum access to only one of the two foods (e.g., M&Ms) before they are asked to again make choices between the two objects. Intact animals stop choosing the object that was associated with the food they had just eaten (e.g., B), and instead choose objects that are associated with the other food (e.g., A). This suggests that they mentally travel through the association between the objects and the specific foods to access the current values of the objects. In contrast, monkeys with inactivated or lesioned OFC continue to choose the two objects equally often [16–18], as if they were using the values they had originally learned. Importantly, this effect is only observed when animals have to decide between the conditioned objects but not when they could choose directly between the two foods themselves. This indicates that OFC is not required for satiety-dependent changes in valuation, but that OFC is specifically necessary for accessing or inferring this new value of the object and using this updated value in decision-making. Similar effects can be observed with OFC lesions and inactivation in rats [19,20], and with amygdala lesions in rodents and non-human primates [21,22].

Disruptions of adaptive behavior in the Pavlovian devaluation task have also been reported in humans after a temporary perturbation of central-lateral OFC activity using TMS [23]. In a study from our lab [23], hungry subjects learned associations between visual cues and two equally preferred food odors (Fig. 2A). After a baseline session of choosing between the two cues to receive the odors, subjects underwent TMS or sham stimulation targeting the central-lateral OFC network. Then, they were asked to eat a meal corresponding to one of the two food odors. For instance, if the two food odors were pizza and strawberry, subjects could be asked to eat as many strawberry wafers as they wanted. Afterwards, they could again choose between the cues, conducted under extinction conditions (i.e., no odors were delivered, regardless of choice) to prevent any further learning. Whereas subjects in the sham group reduced their choices of cues associated with the food odor matched to the meal they had just eaten (e.g., strawberry), subjects in the TMS group continued to choose both cues equally often, just as they had before the meal (Fig. 2B). In both groups, pleasantness ratings decreased after the meal, but only for the meal-matched odor, indicating a specific impairment in the ability to access the updated value of the specific outcome predicted by the cue.

Figure 2. Devaluation task.

A. Subjects learned associations between cues and food odors before choosing between the two cues to receive the corresponding odor. Afterwards, subjects were provided with a meal corresponding to one of the food odors (devaluation). Finally, they again chose between the two cues. B. Relative to sham, OFC-targeted TMS before the meal disrupts adaptive responses in the probe test, as indicated by a lack of a normal reduction of choices for cues predicting the sated odor.

Inference can also be studied using instrumental versions of the devaluation task. This task requires forming associations between actions and outcomes, and inferring the state-dependent value of action outcomes after devaluation. Some studies in rats and non-human primates directly compared the role of OFC in Pavlovian and instrumental devaluation, and showed that OFC was only required for adaptive behavior in the former but not the latter task [24,25], or only when there was a shift in instrumental contingencies [26]. However, the involvement of OFC in Pavlovian and instrumental devaluation may depend on the specific subregion, as other studies have shown that medial OFC and its connections to amygdala are critical for instrumental outcome devaluation [27–30], whereas lateral OFC is necessary for outcome devaluation in Pavlovian but not instrumental tasks [24]. These findings are corroborated by converging results from human neuroimaging studies, showing a role of medial OFC in instrumental [31] and lateral OFC in Pavlovian devaluation tasks [32].

Together, these findings show that OFC, in concert with the amygdala, is critically involved in inferring the current value of choice options. This ability allows us to optimize behavior when making decisions involving specific outcomes whose values have changed since the original learning context. The critical role of the OFC in this inference process is in line with findings from rats [33–35], non-human primates [36–38], and humans [32,39–41] showing that OFC represents specific outcomes, and suggests that such information is part of the associative structure, or cognitive map, that underlies inference-based behavior.

Summary and conclusions

The ability to flexibly utilize prior knowledge and experiences to mentally simulate probable outcomes is critical for making adaptive decisions. Here we reviewed studies on inference-based behavior using two classic behavioral tasks. In the sensory preconditioning task, direct experience is unavailable (e.g., a vacation on Oahu) and outcome predictions rely on knowledge about the associative structure of the task and the ability to mentally connect these associations to make novel inferences. The OFC and HC are shown to be involved in this process, and disruption of activity in either region leads to selective impairments in the ability to infer outcomes. In the devaluation task, previous experience with the association between cues or actions and the value of outcomes is not informative due to a change in the internal (e.g., satiety) or external (e.g., pandemic) state, and one must infer the current value of outcomes when evaluating options. This inference process may involve the flexible use of learned associations between cues or actions and specific outcomes and is shown to rely critically on the OFC-amygdala circuit.

Although the contribution of OFC to inference-based processing appears consistent across species, the rodent-primate homology of these regions is still somewhat controversial. In addition, the role of OFC subregions (i.e., medial vs. lateral) in different species requires further work. Some studies suggest that both medial and lateral OFC are involved in inference [9], whereas others highlight differential roles of these subregions [17,24,28,42]. This discrepancy could reflect differences in the experimental tasks used across species or real species differences [43,44]. Moreover, even within species, OFC subregions are sometimes defined differently across studies, which further complicates interpretation [44]. Using similar tasks across species and more detailed reporting of anatomical locations will help addressing these challenges in the future.

Taken together, the work reviewed here suggests that the OFC occupies a central position in a network of brain regions that is critical for inference-based behavior. Depending on the type of inference, the OFC may fulfill this role in coordination with other brain areas. Whereas inference of novel outcomes appears to recruit OFC-HC circuits, inferring the updated value of specific outcomes seems to rely on OFC-amygdala circuits. It is likely that the neural mechanisms identified in these simple laboratory experiments are also relevant for complex real-life decisions involving outcomes that may lie in the distant future or only affect future generations, such as the decision to go college or signing up for a renewable energy plan.