Anterior cingulate cortex and the value of the environment, search, persistence, and model updating

Abstract

Anterior cingulate cortex (ACC) carries a wealth of value-related information necessary for regulating behavioral flexibility and persistence. It signals error and reward events informing decisions about switching or staying with current behavior. During decisions it encodes the average value of exploring alternative choices (search value), even after controlling for response selection difficulty, and, during learning, the degree to which internal models of the environment and current task must be updated. ACC value signals are in part derived from the history of recent reward simultaneously integrated over multiple time scales thereby enabling comparison of experience over the recent and more extended past. Such ACC signals may instigate attentionally demanding and difficult processes such as behavioral change via interactions with prefrontal cortex. However, the signal in ACC instigating behavioral change need not itself be a conflict/difficulty signal.

Despite many prominent reports relating dorsal anterior cingulate cortex activity (dACC, or rostral cingulate zone: RCZa¹; figs.lb,2,3e) to behavior and cognition both in health and disease², a general theory of its function remains elusive because there is no single factor linking change in stimuli or behavior to neural activity. According to most current theories, dACC plays a key role in behavioural flexibility but there are disputes about its specific contribution. Imagine you are exploring a complex environment when you encounter a valuable item (e.g. an employment offer for a job-seeker or fruit for a foraging monkey). You may either engage with that item or ignore it if the environment is sufficiently rich to make trying elsewhere tempting or more valuable. In such situations, we argue dACC signals information such as the average value of the environment (“search value”), influencing whether you continue your search, potentially entering a sequence of new actions, or remain with the item encountered³. DACC activity also reflects other information determining behavioral change such as how well things have been going (average reward rate) recently over multiple time scales^4–6. DACC activity also occurs when animals^7,8 or people⁹ update models of the current task or environment so that new patterns of behavior can emerge. While evidence for encoding of such information in dACC activity is comparatively recent, there is already broad consensus that dACC activity in many species reflects outcomes of decisions - successes and errors - and whether such feedback indicates a need for behavioral change^10–18.

Figure 1. Comparing dACC and pgACC in humans and macaques.

(a) Every brain region has a distinctive “fingerprint” of connections. To compare brain areas in humans and macaques we first identify the fingerprint of the human area. This is estimated from its fMRI-derived resting state activity correlations with other brain areas (left). There is strong positive coupling with the area marked on the circumference when the green line is close to the circumference. The fingerprint can then be compared with fingerprints of every frontal area in the macaque. The best matching fingerprint from the other species is shown in red on the right. Comparison of fingerprints suggests (b) dACC and (c) pgACC similarities in humans and macaques¹. In each case task-related human brain activity is shown on the left. Activity in dACC activity from Behrens and colleagues¹² is shown in b. Panel c shows activity in pgACC covarying with participants’ general willingness to forage amongst alternative choices despite costs recorded by Kolling and colleagues³ (far left) and activity recorded by McGuire and Kable⁴⁸ (to its right) also in pgACC and adjacent dorsomedial prefrontal cortex that is related to moment-to-moment variation in the value of persisting in a choice through a time delay. The anatomical names used by the two sets of authors differed but the activations’ proximity highlight the fact that it is the same region that is active in both studies. In each case the center shows fingerprints for the same areas based on a set of 23 key brain regions for the human (green) and best matching macaque area (red). On the right heat maps show the strength of fingerprint correspondence for all voxels in the macaque frontal lobe (red indicates strong correspondence and arrows indicate peak correspondence). DACC and pgACC are associated with different patterns of resting state connectivity but in each case corresponding areas are found in the macaque.

Figure 2. The derivation of value signals in dACC and the presence of model updating signals in dACC.

(a) Deriving value signals from the history of past rewards in macaque and human dACC over multiple time scales. (left) The value of a choice can be estimated from the history of I rewards associated with it. A choice may be associated first with high value (many coins, red line), low value (green), or medium value (blue). Changes in reward rates over time mean that red and green option values reverse over time. (right) The activity of neurons in macaque dACC reflects the history of rewards received over different time scales allowing the simultaneous representation of value estimates over different time periods^4,6. A neuron sensitive to reward over longer time scales will be more active, all other things being equal, when a choice is initially associated with high levels of reward (bottom) than low levels (top). A neuron sensitive to short term reward histories, all other things being equal, will be more active when recent experience has been good (top) rather than bad (bottom). (b) Human dACC also reflects reward history over different time scales simultaneously. The relative weight and sign assigned to more recent and more distant reward history suggest a comparison that effectively allows for the projection of future expected reward trajectories (has reward been encountered more frequently recently than over the longer term average) that could guide decisions to keep with a default or to change⁵. (c) DACC is active when internal models are updated not just when task difficulty increases because surprising events occur. Imagine a naturalist who has only ever observed white swans. On first visiting a new country they come across a black swan for the first time. Should they treat this new swan as an outlier and continue to expect that the next swan they see will be white as usual? Alternatively should they update their model of the new environment and expect to see more black swans? In the first case it may be difficult to know how to respond to the surprising new event but the neural representation of the environment remains constant. In the second case the neural representation is reconfigured. (d) Whole-brain cluster-corrected fMRI analysis indicated a region spanning dACC and adjacent pre-SMA in which there was a significant effect of model updating (contrast shows all voxels with a parametric effect of D_KL). The ROI denoted by the yellow line is the dACC region of interest analysed in the lower part of the panel to show mean effect size for surprise (I_S) and updating (D_KL) (error bars are SEM). Adapted from⁹.

Figure 3. Comparing vmPFC and dACC values signals and decision-making processes.

Both vmPFC (left column) and dACC (right column) are anatomically distinct from adjacent areas. In both areas multiple signals are present but can be explained in relation to biophysically plausible neural network models. (a) Whole brain cluster’corrected value difference (chosen’unchosen option value) signal in vmPFC⁵⁰ (orange) is anatomically distinct from reward activity in more lateral OFC¹ (blue on coronal section) and pgACC (sometimes also called “vmPFC”⁴⁸, magenta oval on sagittal section). (b) fMRI time course analysis reveals vmPFC activity reflects both decision value (green) and confidence (white) [adopted from⁵⁵]. (c) MEG recordings show vmPFC activity first reflects the sum of the values (black) of the possible choices and then the difference (blue) in the choices’ values (solid lines are correct trials, dashed lines are errors; adapted from⁶¹) but these different signals can be explained by (d) a neural network model⁶⁰. P_A and P_B are two pools of neurons in which activity is a function of the value of options A and B respectively. There are recurrent excitatory connections within both pools but between pools interactions are inhibitory and mediated by pool PI. The inhibitory interneurons instantiate a competitive “value comparison” process that leaves a single pool in an attractor state and a decision is made. (e) Left panel shows yellow dACC ROI from which signals were extracted. The region it is anatomically distinct from the location of difficulty effects in or near pre’SMA (green)²⁰. Right panel shows the whole brain cluster’corrected effect of search value in dACC in red, even after controlling for difficulty and log(RT)³ (peak MNI,x=‘4mm;y=36mm;z=26mm). (f) fMRI timecourse analysis of dACC reveals effects of search value (red) followed by engage value (blue) even after controlling for later effects of other factors (g) such as logRT (red) and difficulty (blue). Insets in panel f show the BOLD signal binned by different levels of search value illustrating a search value signal emerges early and is sustained until late in the trial but insets in panel g show that, using a similar binning approach, difficulty effects emerge only later. Arrows linking insets to timecourses indicate approximate time of binning analysis. (g) Network model of dACC explaining the sequence of activity in f. Here, similarly to the network model in d distinct neural populations receive different value input and interact with each other via mutual inhibition and excitation. However, we believe that compared to the symmetric representation of different option values in d, this network model of dACC, has a larger population that represents the value of the environment and is sensitive to environmental context and meta changes such as volatility (Referred to as Ps for a population that can represent search value). Due to dACC’s well established signals related to costs such as effort and pain, we believe such representations to interact with neural population here referred to as P_c (i.e. a population signaling costs). Furthermore, the value of sticking with a default option is also implemented here as P_D (i.e. population signaling a pull or bias toward a default), as a self-sustaining neural population that inhibits populations representing the value of exploration or the overall environment Ps. Note however, that this inhibitory impact on dACC might not be implemented as a symmetric interaction and might originate from remote regions. Furthermore, as for panel d, one very important remaining question is how those neural population trigger appropriate responses after a decision has been reached. In this model population PS might simply initiate behavioral adaptation and exploratory behavior, as well as a mode in which there is increased plasticity, model updating, and learning.

By contrast a prominent theory has proposed dACC “diversity can be understood in terms of a single underlying function: allocation of control”¹⁹. Arguably it is non-trivial to identify the controlled process in many naturalistic settings. To return to our example, one could argue that continuing to explore and ignoring a tempting item encountered requires control, or equally that precisely the opposite behavior, engaging with the item encountered and ignoring distracting influences of potentially valuable alternatives, requires control. A third, more recent suggestion is that dACC signals the need for control when both options are almost identical in value²⁰. Consequently, in one version of this account¹⁹ foraging-related value signals are discussed as determining the value of exerting control¹⁹. However, in a more recent version²⁰ the existence of such signals in dACC is questioned and instead it is proposed “dACC activity can be most parsimoniously and accurately interpreted as reflecting choice difficulty alone”²⁰.

Our argument is not that difficulty/conflict does not modulate dACC activity. Indeed such modulation is seen in most brain regions concerned with decision-making. Rather we argue that difficulty or control allocation is insufficient to account for all dACC activity. Moreover, we argue below that difficulty’s/conflict’s impact on dACC signals may be a side effect of its role in evaluating behavioral change and model updating, not the other way around.

Anatomy of ACC and medial frontal cortex in primates and rodents

Although human dACC has been suggested to be unique²¹, its somatotopy^22,23 and activity coupling with other brain areas¹, which reflects anatomical connections²⁴, suggest important resemblances with monkey dACC (fig.1). A region’s connectivity fingerprint is critical in constraining its function because connections determine the information regions receive and the influence they wield over other areas. Some of the areas dACC interacts with, such as frontal pole, have changed during evolution^25,27, but dACC’s overall connectivity fingerprint remains similar in humans and other primates. Although there is no exact equivalent of primate dACC in rodents there are similarities between the anatomy of area 24, of which dACC is a part, and area 24 in rodent ACC²⁸. While rodent’primate ACC correspondences are not precise they are stronger than for any granular prefrontal area²⁹.

In both humans and macaques dACC is distinguished from adjacent medial frontal cortex such as the pre’supplementary motor area (pre’SMA). Although both pre’SMA and dACC share connections, for example with dorsolateral prefrontal cortex (dlPFC)^30,31, dACC is more strongly connected with subcortical regions coding reward and value including amygdala³¹, much of striatum³², dopaminergic and serotonergic transmitter systems³³, and adjacent ACC areas such as perigenual ACC (pgACC) which in turn is special in its ability to influence dopamine^31,34,35 via connections to the striosome. Also, unlike pre’SMA, dACC may exert direct influences over motor output; dACC projects to primary motor cortex and spinal cord³⁰. Therefore, while we might expect dACC and pre’SMA to sometimes be co’active, perhaps with dlPFC, the different connections suggest this will not always be true. Furthermore, if we are interested in how value signals are translated into behavioral change and persistence then we should focus on dACC.

Value signals in frontal cortex and dACC

When monkeys make decisions, dACC neuron activity reflects choice value in terms of potential rewards and effort costs^36,42. Sometimes value signals arise later than in other areas such as orbitofrontal cortex but other times they are more prevalent and arise earlier in _dACC ^36,42.

We are beginning to understand how value signals arise within dACC; they reflect the recency-weighted history of previously chosen rewards. DACC neurons have activity reflecting reward history with different time constants (fig.2a,b)^4,6. Such reward history signals reflecting different time constants are also detectable in human dACC where they can be compared to predict future rewards and guide decisions to persist or change behavior⁵. Independent of difficulty effects, dACC can compute the value of persisting in the current environment, rather than with switching away from it⁵. Furthermore, dACC lesions impair the use of reward history dependent value to determine the balance between behavioral persistence and change⁴³.

Several additional lines of evidence demonstrate value-related activity in dACC neurons is unrelated to response selection difficulty. For example, response selection becomes easier as monkeys progress through a sequence of actions towards a reward [accuracy increases and reaction times (RTs) decrease] but many dACC neurons increase their activity⁴⁴. Moreover, despite repeated attempts it has been impossible to identify dACC single neuron activity encoding difficulty in monkeys^10,11,45,46.

DACC neurons responsive to conflict or cognitive load have been claimed in humans⁴⁷. However, arguably the key contrast of behavioral conditions supporting the claim might not just reflect conflict but the possibility of alternative courses of actions; increased neural activity is predicted by most theories if the contrast is between conditions varying in number of response associations, number of distracting alternative courses of action, and effort costs. However, after careful testing of a large sample of dACC neurons in monkeys in an experiment focusing directly on difficulty, not a single neuron actually coding response difficulty per se was found⁴⁵. Instead each neuron with activity related to a particular response became active whenever there was even partial evidence for that response. Therefore, neurons encoded actions or action values and may also have signaled alternative task goals, but never difficulty. However, as a result of such coding many neurons became active in conflict situations and their aggregate activity gave the impression of a difficulty signal that could not be dissociated from that expected from true difficulty neurons by a technique such as fMRI.

Similarly Ebitz and Platt⁴⁶ pointed out that while dACC neurons reflected a potentially valuable goal that might become an alternative focus of monkey behavior (a function for which we argue dACC is critical) “action conflict signals were absent”. As already noted, however, we are not really concerned with whether conflict/difficulty signals are present but instead our claim is that any such effects are insufficient to explain away evidence of other signals in dACC. What is clear is that, in monkey dACC, value signals exist without clear relation to difficulty.

Of course value signals are also found beyond dACC: in parietal cortex, the dopaminergic system, striatum, and amygdala. There has been a surprising tendency, however, to assert that, within frontal cortex, value signals exist only in ventromedial prefrontal cortex (vmPFC)⁴⁸. However, at least three different “vmPFC” regions show specific value and decision-related activity and have distinct roles in behavior: areas 13⁴⁹, 14^50,51, and pgACC^48,52,53 (fig.3a). It should therefore come as no surprise if value signals exist in other frontal areas such as dACC. It is, however, likely that any value signals in dACC will, as elsewhere, have distinctive features.

Human dACC and the value of behavioral change

Neurophysiological experiments suggest individual dACC neurons carry value signals but that aggregate population activity also reflects difficulty. Two recent reports explain how an fMRI experiment should be conducted when there may be multiple influences on a brain area’s activity^54,55. Both focused on vmPFC rather than dACC. One factor was value and the other was decision confidence (approximately the inverse of difficulty^20,55). Both factors influenced brain activity and temporal evolution of their effect was visualized with a General Linear Model (GLM) timecourse analysis⁵⁵(fig.3b).

An analogous approach can be used to identify activity related to the value of exploring an environment, a behavior called foraging³. Participants decided whether to engage with a default option or whether they preferred to explore alternative options. The decision to explore depended on search value (average value of alternatives) and engage value (value of the default options).

The experimental design and schedule ensured search value and difficulty shared little variance (<2.5%). Therefore brain activity could be securely related to either difficulty or search value. Using whole brain GLM analysis, including control regressors of difficulty and log(RT), search value was linked with dACC (fig.3e). In addition there was a smaller and later negative effect of engage value on dACC (fig.3f). Therefore dACC has just the signals needed for determining value of behavioral exploration. Marginally significant effects of difficulty and log(RT) occurred in dACC towards the end of the decision period (fig.3g). An equivalent analysis in which dACC activity was binned by search value or difficulty suggested similar conclusions. Difficulty effects may be stronger in or anterior to pre-SMA (fig.3e)²⁰; many other experiments link pre-SMA and adjacent SEF to difficult response selection^56,57.

Interpreting multiple signals in dACC

It can be difficult to know what conclusions to draw when different experiments provide evidence for presence or absence of signals in dACC but again we can find inspiration in the manner in which controversies regarding vmPFC activity have been resolved. Various experiments had suggested that during decision making vmPFC activity reflects the sum of values of possible choices, the choices’ difference in value, or simply the value of the choice ultimately taken^50,58,59. By using a biophysical neural network model of the decision process⁶⁰ it was possible to reconcile these claims and demonstrate they may correspond to signals in a decisionmaking circuit generated at different time points during evolution of a decision⁶¹ (fig.3c,d). Pools of neurons code for the value of each potential choice and become active in proportion to its value. Recurrent excitation between neurons in each pool and inhibition between pools ensures the network moves to an attractor state in which one pool, representing one option, remains active and a “choice” is made. Model activity first reflects the sum of choice values and then difference in choice values. High temporal resolution recordings show the same is true of vmPFC⁶¹.

Variants of this model are likely applicable in many cortical areas concerned with selection. Drawing on work linking comparator processes to dACC⁵⁹ and our own studies^3,25,26 we propose a neural network model of dACC (fig.3h) predicting presence of both value signals and difficulty effects. In such a hypothetical network variance in activity is related to search value first and then, slightly later, to engage value. When these values are closer together the decision is difficult and the network takes longer to move into an attractor state. This means that later in time during decision making, variance in activity correlates with difficulty and RT even though there are no units explicitly signaling “difficulty”. Difficulty correlates arise because difficulty affects the temporal dynamics of the comparison process. Interestingly, this is the time’varying activity pattern observed in dACC (fig.3f,g). Note that, in the absence of high temporal resolution measurements, the properties and sign of predicted difficulty effects depend on assumptions made about the model and how it is reflected in the time’integrated fMRI signal. For example, if high firing attractor states last after reaction time, then one might see a negative effect of difficulty in fMRI, whereas if activity diminishes as soon as a threshold is crossed one might expect a positive correlate. Either way it is clear that merely measuring a correlate of difficulty does not mean an area’s primary function is to signal difficulty.

Another prediction of such models is that if a decision is very easy, because options have very different values, then it may not be possible to detect value signals in the network’s aggregate activity with a low temporal resolution technique such as fMRI. In some experiments examining very easy decisions²⁰, the brevity of the comparative process makes any detectable value effects in fMRI unlikely. Moreover, we know that faster and more efficient alternative selection mechanisms can be used if choices can be made with very simple heuristics⁶¹. In other words, no decision-related value signals are expected if there is no real decision to make.

Foraging, task switching, and updating of internal models

“Search value”, the average value of an environment, is signaled in dACC. In many natural situations animals don’t choose between simultaneously presented options but instead decide whether to engage with sequentially presented options as they are encountered⁶². Engaging incurs an opportunity cost because potential opportunities to pursue better options are lost. We have therefore pointed out that search value (indexing potential opportunity costs) signals in dACC could guide foraging.

Importantly, our theory is not that dACC activity is simply synonymous with foraging value. Although like others⁶³, we have drawn inspiration from consideration of foraging problems primates evolved to solve it is important to appreciate that many neuroeconomic decisions people make in modern environments involve similar factors. A job-seeker considering one position and foregoing alternatives is making a decision about opportunity costs. Similar signals could guide task switching. The opportunity cost of alternatives makes maintaining engagement in a particular task difficult and so it should be possible to integrate search value-related ideas into models of cognitive control that focus on dACC-dlPFC interactions^64–66.

By the same token, not every task with a link to naturalistic decision-making can be performed by dACC alone. PgACC activity is also related to participants’ general willingness to forage amongst alternative choices despite costs³ (fig.1c). In another recent experiment⁴⁸ without a requirement for search value to guide behavior (and therefore perhaps not surprisingly no dACC activity) the need to persevere or to continue engaging with the current environment again led to pgACC activity. Homologous regions are also involved in cost/benefit decision making in monkeys and rodents^35,52,67.

DACC is also implicated in attention switching when it is driven by the updating of internal models of behavior⁶⁸. In an fMRI experiment, two types of unexpected events occurred. On model update trials subjects responded to a target in an unexpected location and its color indicated future targets would appear nearby. However, on surprise only trials, differently colored targets in an unexpected location indicated one-off events and no need to update internal models of where future targets would appear. We quantified and carefully dissociated model updating and difficulty of responding in this experiment⁹. Difficulty is equivalent to the surprise associated with a particular target location characterized by its Shannon information I_S. Model updating is captured by the Kullback-Leibler divergence (D_KL) between the posterior and the prior probability estimates of target location. DACC activity occurred on update trials, as a function of D_KL, but not on surprise only trials even though both were associated with RT increases (fig.2c,d). Once again behavioral change-related activity is found in dACC but difficulty itself has little explanatory power. Detailed descriptions of rodent ACC neuron activity during discarding and updating of internal models have also been reported^7,8.

Lesions and inactivation of dACC

There might be selection difficulty during decisions between sticking or switching to an alternative. However, investigations of dACC disruption in humans and macaques have revealed complicated impairments that cannot be related in a simple way to difficulty^69–72. Unlike lesions to other frontal areas, dACC lesions have little impact on cognitive control; instead impairments are most prominent when decisions concern assessment of the relative value of behavioral persistence versus change^43,73,74. Perhaps the most striking deficits seen after bilateral lesions, even when circumscribed to the cingulate sulcus’s ventral bank, however, are failures to act at all⁷⁵; it is difficult to account for such dramatic effects with subtle arguments about detecting difficulty. Such profound failures are, however, expected if an average sense of the value of possibilities afforded by the environment is lost.

Summary and Future Directions

DACC is active both during decisions and when decision outcomes are assessed. Value, model update, and outcome-related activity in dACC have in common that they all regulate behavioral adaptation and persistence. Although behavioral adaptation may, in turn, entail difficult response selection, dACC activity reflects more than just need or value of control. Moreover, the actual process of behavioral adaptation may be implemented not just in dACC but through dACC’s interactions with dlPFC^64–66.

Surprisingly, despite considerable debate about dACC, there have been few attempts to understand adjacent ACC regions. Understanding the precise anatomical arrangements of activity patterns is important because sometimes differing views of function can be reconciled if they are focussed on different subdivisions of medial frontal cortex. As we have noted (fig.3) value and difficulty effects are prominent in adjacent but different areas. Understanding dACC in the context of interactions with pgACC, adjacent gyral ACC, and posterior cingulate cortex, not just in relation to dlPFC, will be important. Ecological foraging theory suggests additional ways of thinking about the decisions we have evolved to take and this, together with neuroeconomic analysis methods, can be exploited to design new decision’ making paradigms that may clarify ACC activity. Interpreting the activity that we find, however, will only be possible in the context of computational descriptions (fig.3). Importantly, such models not only make qualitative predictions about presence or absence of neural activity but suggest specific patterns of temporal dynamics strongly suggestive of particular choice mechanisms (fig.3). Finally, detailed neurophysiological and lesion/inactivation studies will be needed to aid interpretation of human neuroimaging studies of dACC.