CALCULATING THE COST OF ACTING IN FRONTAL CORTEX

Abstract

In order to be able to make informed and successful decisions, it is vital to be able to evaluate whether the expected benefits of a course of action make it worth tolerating the costs incurred to obtain them. The frontal lobe has been implicated in several aspects of goal-directed action selection, social interaction and optimal choice behavior. However, its exact contribution has remained elusive. Here, we discuss a series of studies in rats and primates examining the effect of discrete lesions on different aspects of cost-benefit decision making. Rats with excitotoxic lesions of the anterior cingulate cortex became less willing to invest effort for reward but showed no change when having to tolerate delays. Orbitofrontal cortex-lesioned rats, by contrast, became more impulsive, yet were just as prepared as normal animals to expend energy to obtain reward. The sulcal region of primate anterior cingulate cortex was also shown to be essential for dynamically integrating over time the recent history of choices and outcomes. Selecting a particular course of action may also come at the expense of gathering important information about other individuals. Evaluating social information when deciding whether to respond was demonstrated to be a function of the anterior cingulate gyrus. Taken together, this indicates that there may be dissociable pathways in the frontal lobe for managing different types of response cost and for gathering social information.

Introduction

Over the past decades, significant strides have been made in our understanding of the mechanisms underlying goal-directed actions. From the background of a stringent theoretical framework, systematic research is starting to uncover the brain regions and neural mechanisms involved in learning and representing the connection between responses and their consequences ¹. Obviously, the availability of such information is essential in order to be able to make beneficial decisions when confronted with multiple options as it allows organisms to begin to construct a notion of value of the available alternatives given their current motivation.

However, one failing with applying such an approach to decision making is that it has tended to assume that actions lead directly to reward with few intervening costs or negative events in between the choice and the outcome. As has been appreciated for many years in other behavioral sciences such as behavioral ecology and economics, animals, including humans, do not just make decisions and select actions on the basis of an expected reward but also weigh up the potential costs of the different courses of action that might be pursued ^2-5. Such costs may involve the investment of time or effort, a willingness to tolerate a risk that a reward may not be forthcoming or to endure pain in the pursuit of a goal. Similarly, selecting one particular option may also come at the expense of gathering other significant information such as the proximity and intentions of other competing animals. To date, though, it is far from clear how the brain incorporates calculations of such costs with expected reward in order to guide response selection.

Neuropsychological studies have demonstrated that parts of the frontal lobe are integral for goal-directed action selection, strategy-implementation and social behavior, all of which are essential components of optimal decision making ^6-9. More recently, the direct contribution of these regions to cost-benefit choices has been investigated, but there remains a large degree of confusion over which regions are critical and what their exact contributions are. Patients with damage to parts of prefrontal cortex can simultaneously exhibit prolonged deliberation about choices accompanied by subsequent irresponsible, risky behavior ^10-13. Similarly, several studies have shown that such lesions can cause both symptoms of apathy and indifference as well as poor impulse control ^14-16.

Such combinations of apparently contradictory deficits may seem surprising given behavioral evidence for potential partial dissociations between the processing of different response costs ¹⁷. However, the damage in most prefrontal patients encompasses several neuroanatomically-separable areas. By contrast, by making experimental lesions in animals, it is possible to examine the direct contribution of discrete regions to aspects of cost-benefit decision making. Here, we discuss recent evidence from our laboratory which indicates that different parts of the frontal lobe – in particular, the orbitofrontal cortex (OFC) and anterior cingulate cortex (ACC) – play dissociable roles in action selection depending on the type of response cost and environmental information that is encountered.

The ACC and effort-based valuation

If an animal is faced with a choice between two courses of action, one of which leads to a larger reward than the other, then it is simple to work out which is more valuable and therefore which it should choose. However, if the same choice is presented except that now the larger reward is only obtained after a period of time has elapsed or an amount of effort has been expended, then the optimal decision is no longer immediately self-evident. This diminution of the value of a reward by the cost incurred to achieve it is known as discounting and has been shown consistently to influence the way in which animals and humans make choices ^{3, 17, 18}. If we assume that the basic machinery of motivation is designed to bias animals towards courses of action that result in more certain, easily obtained and immediate reward ¹⁹, then the question arises of what mechanisms exist to resist such temptation when more taxing options may result in greater overall utility.

Such situations can be modeled in animals by presenting them with a choice between an easily obtained low reward (LR) option and a high reward (HR) option attained only after overcoming some type of additional response cost. Several studies in rodents have implicated parts of the frontal lobe in allowing animals to overcome response costs to earn greater reward. Using a cost-benefit T-maze paradigm originally designed by Salamone and colleagues ²⁰, Walton and colleagues demonstrated that medial frontal cortex is essential for allowing an animal to put in extra work for greater reward ²¹. Rats chose between investing effort by surmounting a large barrier to obtain the HR or selecting the LR which did not incur any additional response cost. Following excitotoxic lesions of this region, animals became profoundly cost averse, switching from selecting to climb the barrier for the HR on the majority of trials pre-surgery to choosing the more easily obtained LR option on almost all occasions. However, when the response costs were equated by adding an identical barrier into the LR arm, meaning that the rats were now required to put in the same amount of effort to obtain either size of reward, all animals returned to choosing the HR. This implies that the deficit was not caused by any gross motor deficits or spatial impairments, but is instead primarily one concerned with making optimal decisions.

Subsequent experiments have localized this effect to the ACC, with lesions of this region (including pre- and perigenual Cg1 and Cg2, Figure 1A) causing a comparable switch to low effort responses whereas lesions of adjacent prelimbic cortex or to the OFC having no effect on the allocation of responses in this task (Figure 1B) ^22-24. However, the ACC lesions do not affect response selection or make animals cost-averse in all decision making situations. When a rule has to be learned and applied in order to work out what action to make, such as in a delayed matching-to-position task, animals with ACC lesions perform comparably to control animals ^{22, 25}. It had previously been shown in an operant cost-benefit decision making task where rats chose between an immediately available LR or a HR presented after an increasing delay that neither ACC nor prelimbic cortex lesions caused any change in the pattern of responses ²⁶. Similarly, in a T-maze analogue of the barrier task described above, Rudebeck and colleagues also found no change in delay-based decision making following identical ACC lesions to those which affect effort-related choices ²⁴. The role of rodent ACC in guiding decisions in uncertain situations or in tasks with probabilistic reinforcement has yet to be tested.

Figure 1.

(A) Representation of ACC and OFC regions in the rat brain that were lesioned in the study by Rudebeck and colleagues ²⁶. ACC includes pre- and perigenual Cg1 and Cg2 and dorsal ACC (ACd). OFC includes medial (MO), ventral (VO) and lateral orbital regions (latter not depicted). (B) Choice performance of ACC- and OFC-lesioned animals and controls in two pre-operative and a single post-operative testing session of the effort-based decision making task. In each block, rats chose between climbing a 30 cm barrier for the HR (4 food pellets) or selecting the unoccupied arm for the LR (2 pellets) (denoted on the figure as “1 × 30cm Barrier”). Adapted from Rudebeck et al. ²⁶.

The OFC and delay-based valuation

While OFC lesions did not make animals cost averse on the T-maze barrier task, several neuropsychological studies in humans and animals have implicated the OFC in mediating certain types of cost-benefit decision making, in particular in aspects of impulsivity and foresight ^{11, 12, 27}. Recent studies have found cells in the OFC that respond to the amount of time before reward presentation, with activity appearing to reflect a discounted value of the delayed reward ^{28, 29}. However, just as the human literature can appear contradictory in highlighting both impulsive choices and long deliberation times of patients with damage to this region, the results from studies of delay-based decision making in rodents are similarly conflicting, with some showing an increase in impulsive choices following OFC lesions ^{30, 31} and another the opposite effect ³². Moreover, all of these studies were performed in operant boxes which have markedly different response selection and cue-control elements to a T-maze, making direct comparison with the effort-based decision making findings tricky.

To investigate this inconsistency, Rudebeck and colleagues also tested animals with excitotoxic lesions of the OFC on a T-maze delay-based decision making task where they could choose between a delayed HR or an immediately available LR option ²⁴. Prior to surgery, all animals preferred to wait for the HR. However, post-operatively, in contrast to the animals with ACC lesions, the OFC-lesioned rats became cost-averse, switching to the LR option on the majority of trials (Figure 2). This was not caused by a general hyperactivity as the same animals showed no increase in spontaneous locomotor activity. Moreover, as with the ACC-lesioned animals on the T-maze barrier task, when the costs were equated by requiring the animals to wait for an identical amount of time for both the HR and LR options, the OFC group returned to choosing the HR. This again implies that the increase in impulsive choices was a consequence of an alteration in the way the costs and benefits of the options were processed rather than being a simple impairment in spatial or reward magnitude processing.

Figure 2.

Choice performance of ACC- and OFC-lesioned animals and controls in two pre- operative and a single post-operative testing session of the delay-based decision making task. In each block, rats chose between an immediately available LR (1 pellet) or waiting for 15sec for the HR (10 pellets) (denoted on the figure as “1 × 15s Delay”). Adapted from Rudebeck et al. ²⁶.

What are the contributions of the ACC and OFC to overcoming response costs?

The studies discussed above clearly demonstrate that both the ACC and OFC are crucial to allow animals to make optimal cost-benefit decisions. More importantly, they also illustrate that there is anatomical separation in choice behavior depending on whether animals are required to integrate predicted energetic expenditure or delay information with expected reward magnitude. However, their respective roles in guiding different types of cost-benefit decision making are not unique or categorical. Inactivating the basolateral amygdala (BLA) with bupivacaine or lesions of the dopamine terminals in the nucleus accumbens, for example, make rats comparably cost-averse when choosing whether or not to invest effort for greater reward ^{20, 33}. Similarly, excitotoxic lesions of the BLA or nucleus accumbens also render them impulsive ^{26, 32}. This indicates that extended interconnected, but partially dissociable, frontal-subcortical circuits are required in order to calculate the value of the available options and, where appropriate, to resist the temptation of the most easily available reward. Moreover, the fact that either ACC- or OFC-lesioned animals were able to return to choosing the HR option when either the effort or delay respectively was equated for both response options demonstrates that the surgery, rather than rendering them completely unable to tolerate response costs, caused them instead to be biased away from the high cost option.

Interestingly, even this bias does not appear to be permanent. In Rudebeck and colleagues' study ²⁴, the animals on each cost-benefit decision making task were re-tested after the equal cost manipulation with the original choice contingencies of a high cost HR and a low cost LR restored. On both the effort- and delay-based tasks, the group which had previously been impaired immediately after surgery now performed comparably to the control animals. This meant that the ACC-lesioned rats were now just as likely to choose to surmount the barrier for the HR as the sham or OFC-lesioned groups (Figure 3A) and the OFC-lesioned rats opted to tolerate the delay for the HR at comparable levels to the other two groups (Figure 3B). Further analysis and testing revealed that this was not caused by simple recovery from the lesion as there was no correlation between rats' choice performance and the length of time from surgery, and the same animals were impaired on further social and locomotor experiments conducted several weeks later.

Figure 3.

Post-operative choice performance of ACC- and OFC-lesioned animals and controls in the equal cost condition (“2 × 30cm Barrier” or “2 × 15s Delay”) and a re-test of the original cost-benefit choice contingency on either (A) the effort-based (“1 × 30cm Barrier”) or (B) delay-based decision making task (“1 × 20s Delay”). Adapted from Rudebeck et al. ²⁶.

The question therefore arises as to what is the exact contribution of the rat ACC and OFC that allows each to bias animals to tolerate investing effort or time respectively for greater reward. It is known that the OFC, in particular, plays a crucial role with interconnected structures such as the BLA in learning associations between stimuli and rewards, signaling expected outcomes and representing incentive value ^34-37. Moreover, it has been demonstrated that lesions to the OFC cause a reduction, though not abolition, in expected reward-related neuronal activity in the BLA ³⁸. Primate ACC too has been shown to contain cells which track the progress towards an expected reward across a sequence of actions ³⁹, though no such evidence yet exists of how lesions to this region affect the responses in other interconnected structures. Based on this evidence, it may be that the ACC and OFC play crucial and dissociable roles in representing the anticipated goal of a course of action across effort- and delay-based costs, especially in the absence of mediating cues and extensive pre-operative training. Lesions to these regions could degrade this representation, pushing animals away from being sufficiently motivated to select the costly HR in the face of the easily obtainable LR. However, following the equal cost condition in which animals are repeatedly exposed to the HR option, animals are able to reevaluate their decision reference, perhaps using their intact BLA and nucleus accumbens, meaning that they can continue to make optimal choices when re-exposed to the original cost-benefit choice contingencies. Further experiments will be required to validate these hypotheses.

Weighing up what to do in an uncertain world

All of the experiments described above deliberately use paradigms in which the contingencies are static and fully learned. However, in more naturalistic settings, outcomes are often uncertain and vary as a function of the patterns of choices made by both the agent in question and other organisms in the environment. Being able to dynamically integrate the expected magnitude of a reward with the uncertainty of its availability is vitally important in order to maintain up-to-date and accurate representations of the value of available options. This is particularly the case for a foraging animal having to choose whether to stay with what is a depleting patch of food or to move away to try to explore other potentially more fruitful sources of nourishment ^{3, 4, 40}

There is an increasingly voluminous literature implicating ACC, particularly in primates the sulcal region (ACCs) which has direct projections to the motor system ^{41, 42} (Figure 4A), in learning and representing the value of available actions ^43-45. In order to investigate the role of this region in allocating responses in an uncertain environment, Kennerley and colleagues ⁴⁶ taught monkeys a version of the “matching” task, originally devised by Herrnstein ⁴⁷ and used more recently in macaque monkeys by Sugrue and colleagues ⁴⁸, which captures several of the key features faced by foraging animals described above. In this task, animals can choose between one of two joystick movements – lift or a turn response – which are rewarded with unequal probabilities. Importantly, these rewards are assigned independently to each action on each trial and remain available until captured. As only one of the two actions is possible on each trial, this means that the cumulative probability of the ignored option increases the more trials on which it is not chosen. Therefore, it is not optimal for an animal continually simply to select the option with a higher probability as there will come a point when the likelihood of a reward being available on the less profitable option is actually greater than for the more profitable response.

Figure 4.

(A)Schematic of the location of the ACCs lesion. (B) Number of trials required to reach and sustain performance within 97% of the optimal rate when the outcome for each choice was probabilistic. Control and ACCs lesion data are shown by white and grey bars respectively. All data come from the post-operative testing period. The optimal rate was defined as

$$r_{\mathit{opt}}=\frac{p-\mathit{pq}}{p+q-2\mathit{pq}}$$

where p and q are the probabilities of a new reward being assigned to the lift or turn responses respectively. (C) Number of trials required to reach and sustain performance within 97% of the optimal rate when the outcome for each choice was deterministic. Adapted from Kennerley et al. ⁴⁶.

Normal monkeys discover that they need to sample both options to develop a sense of the yield of each alternative and usually learn within 150 trials how often it is advantageous (within 97% of the optimal) for them to switch away the more profitable option to the one that normally leads to a poorer revenue of reward. However, by contrast, macaques with discrete lesions to the ACCs took significantly more trials to approach the optimum response ratio with each of the three sets of probabilistically rewarded contingencies (Figure 4B). This was not caused by an inability to work out which response was better or to sustain a response as the lesioned animals were just as good as the controls when the outcomes were deterministic (correct or incorrect) (Figure 4C). Instead, coupled with a previous error-guided switching experiment by Kennerley et al. ⁴⁶, this suggests that the ACCs may be one crucial cortical component for integrating the extended history of an animals' choices and payoffs to guide action selection.

The value of social information

While it is more common to associate the term reward with homeostatic necessities such as food and liquid, other information which can guide decision making can also be intrinsically valuable. For a social, competitive animal, for instance, choices outside the laboratory will need to take account of the presence and position of other individuals who may either guide it to a fruitful source of food or challenge it for its provisions. In humans, a whole branch of mathematics has been devoted to describing the optimal way of making decisions in a world populated with intelligent competitors ^{49, 50}. There is also a range of evidence to suggest that monkeys find the acquisition of social information rewarding and scale their valuation depending on the how important the information would be to them, even going so far as to sacrifice fluid in order to gain access to a picture of a high-status male or female perineum ^{51, 52}.

It has been long been known that damage to ventromedial parts of prefrontal cortex in humans, encompassing both the ACC and OFC which result in decision making impairments, can also cause profound changes in personality and disrupt normal patterns of social interaction ⁶. To investigate the involvement of these regions in evaluating social information, Rudebeck and colleagues ⁵³ gave monkeys a task in which they were presented with a appetizing food item at the same time as either a neutral or a socially-interesting stimulus. Their latencies to pick up the food item served as an index of how much they valued obtaining further information about the stimulus against the rewarding incentive of the food. Normal animals delayed reaching for a foodstuff in the presence of images of male or female monkeys, becoming increasingly retarded as the stimuli became more socially-interesting (Figure 5A). However, animals with selective lesions to the ACC gyrus (ACCg – Figure 5B) were seemingly not interested by the social information, retrieving the food rapidly regardless of whether an image of a monkey or a neutral picture was presented. By contrast, no such alterations in performance were seen in animals with either lesions encompassing lateral orbital and ventral prefrontal cortex or to ACCs, with these monkeys showing an analogous scaling as controls with the social stimuli. The lack of concern shown by the ACCg group was not simply caused by a basic impairment in flexibly assigning value as they were just as proficient as controls on a stimulus reversal learning task. This finding is consistent with several recent imaging studies requiring complex social interaction or use of social knowledge which have shown activations in a potentially homologous region of the ACC ^54-57. However, Rudebeck and colleagues' data indicates that the crucial role of this area may be in integrating the value of the social information with the other incentives in the environment in order to decide whether to act. How these social functions interact with other cognitive and autonomic roles of the ACC in decision making is presently unexplored.

Figure 5.

Response latencies to pick up a food item in the presence of either a social stimulus (5 left-hand columns) or two neutral stimuli (2 right-hand columns). Symbols indicate scores for each individual. Redrawn, based on Rudebeck et al.⁵³.

Conclusions and outstanding questions

Making advantageous decisions involves a complex process of weighing up the potential benefits of a particular course of action given current motivational state against the costs of achieving that goal and the value of gathering information from other possible sources. A series of neuropsychological studies in rats and monkeys has indicated that parts of the frontal lobe play essential and dissociable roles in integrating the costs and benefits of available alternatives. Rodent ACC is necessary to allow an animal to invest effort for greater reward whereas the OFC is needed to avoid impulsive choices. This separation in the processing of costs may map onto differences seen in human patients, where dysfunction in an ACC-subcortical circuit can result in apathy whereas damage to the OFC-subcortical circuit may cause disinhibition and agitation ¹⁵. Neither decision making deficit was insurmountable, however, with the lesioned rats changing their behavior after extensive experience of choosing to tolerate the cost to obtain the reward. This suggests that neuropsychiatric patients may also be aided in their decision making with practice and by providing mediating cues to help them make favorable choices. Primate ACCs was also shown to be vital for learning about the extended history of choices and outcomes and guiding behavior appropriately. Finally, the ACCg appears crucial for evaluating social information relating to other individuals in the environment when deciding whether to respond.

While the ACC and OFC are clearly vital for guiding optimal decision making, their exact contributions in concert with interconnected regions is as yet far from clear. As previously discussed, the basolateral amygdala, nucleus accumbens and monoamines such as dopamine and serotonin all play roles in allowing animals to avoid being cost-averse, with some having dissociable functions depending on the cost the animal has to overcome ⁵⁸. Similarly, cells in dorsolateral prefrontal cortex and the caudate, both of which are connected with the ACCs​ ^59-62, also appear to represent the current action value based on the history of outcomes ^{63, 64}. Other regions, such as posterior cingulate cortex and parts of parietal cortex, are also sensitive to reward probability and value ^{48, 65}. However, it is also the case that even invertebrates such as gastropods and locusts with much more simple nervous systems can be shown to integrate costs and benefits and develop state-based valuation systems ^{66, 67}. It is a priority for future research to try to discover what the specific contributions of the ACC and OFC to the process of valuation and deciding are.

Part of this will involve improved definition of what constitutes a response cost and how animals learn to mediate them. There have been descriptions which partially equate aspects of risk-taking and impulsivity, with the former being aided by myopia for potentially negative consequences in the future ^{68, 69}. Similarly, it is possible to conceive of an effort task with multiple unrewarded steps leading to an eventual predicted goal as having a probabilistic component (a fixed ratio [FR] 4 schedule and a reward probability of 0.25 should result in a comparable rate of rewarded and unrewarded responses). Moreover, in an uncertain and changeable environment, it will be vital to learn when it is worth persisting with a mode of responding even if it is temporarily unfruitful and when it would be better to switch to a different course of action and explore other possibilities.

While the dissociations described in the present article are striking, it will also be important to try to find whether there are also commonalities in the way such costs are dealt with. One possibility is that this will require a closer connection to be made between the types of cost-benefit decision making discussed here and other types of goal-directed action. It is known, for instance, that both the ACC and OFC play roles in types of associative learning ^{35, 70}. Behavioral ecologists have advanced an associative learning hypothesis to explain how rats discount the value of outcomes as a function of the cost, with the strength of the attribution between the choice and its consequences being scaled as a function of the cost between the two (i.e., the longer the delay between the choice and the outcome, the weaker the causal connection between the two which can in turn lead to the propensity to discount delayed rewards) ⁷¹. Similarly, frontal-striatal circuits have also long been associated with aspects of rule-based learning ^{72, 73} which may provide a way of generating heuristics for the contexts in which it is worth investing time or effort, taking risks or exploring the environment. Further experiments will be required which explicitly manipulate the instrumental and Pavlovian elements of the tasks and which examine whether animals are devising strategies or are making choices based more simply on their history of reinforcement.

It is also an intriguing question how the changes in social behavior observed following ACC lesions relate to the other types of cost-benefit decision making discussed above. In monkeys, there appears to be an anatomical dissociation between the parts of the ACC concerned more closely with integrating reward history to guide action selection (ACCs) and those involved in social valuation (ACCg) ^{46, 53}. However, there is recent evidence that the same ACC lesions in rats that cause alterations in effort-related decision making also disrupt the acquisition and retention of social information (Rudebeck et al., in preparation). While there might also be finer-scale anatomical distinctions in the rat ACC, this latter finding raises the possibility that parts of the ACC might play a common role in integrating many different factors to decide whether one course of action is worth choosing over the other possibilities. A similar connection can be made between the OFC's role in processing fearful stimuli and in aspects of reversal learning ^74-76, with both potentially related to an underlying representation of the reinforcement value of stimuli ³⁵. By investigating how such crucial determinants of decisions as the availability of social information interact with goal-directed action selection, it should be possible gain a better understanding of how exactly frontal regions guide animals to make optimal choices.