Prefrontal Cortex and Impulsive Decision Making

Abstract

Impulsivity refers to a set of heterogeneous behaviors that are tuned suboptimally along certain temporal dimensions. Impulsive inter-temporal choice refers to the tendency to forego a large but delayed reward and to seek an inferior but more immediate reward, whereas impulsive motor responses also result when the subjects fail to suppress inappropriate automatic behaviors. In addition, impulsive actions can be produced when too much emphasis is placed on speed rather than accuracy in a wide range of behaviors, including perceptual decision making. Despite this heterogeneous nature, the prefrontal cortex and its connected areas, such as the basal ganglia, play an important role in gating impulsive actions in a variety of behavioral tasks. Here, we describe key features of computations necessary for optimal decision making, and how their failures can lead to impulsive behaviors. We also review the recent findings from neuroimaging and single-neuron recording studies on the neural mechanisms related to impulsive behaviors. Converging approaches in economics, psychology, and neuroscience provide a unique vista for better understanding the nature of behavioral impairments associated with impulsivity.

Introduction

Decision making refers to the process of choosing a particular action among a number of alternative options that is expected to produce the most beneficial outcome to the decision maker. Given the complexity and diversity of necessary computations, it is perhaps not surprising that many cortical and subcortical areas in the brain are involved in decision making. In addition, there is a substantial variability in the decision-making strategies displayed by different individuals, and many psychiatric disorders, including mood disorders and substance abuse, are thought to result from decision-making abilities impaired in some aspects. Accordingly, understanding the precise nature and neural mechanisms of such variability is an important goal for theories on personality traits and psychiatric disorders (1). In particular, individuals vary substantially in their strategies to deal with time during decision making, and impulsivity often refers to the failures in optimally incorporating temporal factors in decision making (2). In the last decade, substantial progress has been made in uncovering the basic mechanisms of decision making, including how the brain synthesizes incoming sensory information and the decision maker’s previous reward history to select the action that is most beneficial to the subject (3–9). This knowledge is essential for understanding the precise nature of dysfunctions in the decision-making circuitries in the brain, including those responsible for impulsive choice behaviors. This review first describes the range of computations required for optimal decision making. Next, how suboptimal solutions to some of these computational problems can lead to impulsive behaviors is discussed. The rest of the article is devoted to the review of the neural mechanisms for impulsive behaviors.

Computations necessary for optimal decision making

Economic decision making

In general, outcomes from a particular action might vary depending on the state of the decision maker’s environment. For relatively simple types of decision making, the relationship between actions and their outcomes might be fully known, but often the relationship among the environment, action, and outcomes might change dynamically and may not be known completely. Nevertheless, formal analyses of decision making tend to share two common features. First, a set of available actions from which the final action can be chosen must be specified. In most behavioral and neurobiological studies on decision making, this is determined explicitly by the experimenter. In reality, the number of actions that need to be considered by the decision maker can be potentially very large. Second, for each of the actions under consideration, the desirability of outcomes expected from the action must be evaluated. Outcomes of actions can vary in many different dimensions, including not only the type of an outcome, but also its statistical and temporal properties. For example, performing a particular action or a sequence of actions might determine the probability of obtaining a particular amount of money or food, and such an outcome might become available after some delay. In some cases, decision makers might dread the arrival of an aversive outcome, and savor the anticipation of a pleasant outcome. As a result, the preference for reward might increase as it is delayed and produces more opportunity for its anticipation, while the preference for some painful outcome might increase when it happens sooner (10). Such anomaly is referred to as negative time preference (11, 12). In most cases, however, humans and animals are more likely to take an action when the reward expected from that action is delivered sooner than later, and when the aversive outcome that can be avoided by taking that action is more imminent. This is referred to as positive time preference and the process by which the value of a delayed outcome is diminished according to its delay is referred to as delay or temporal discounting (13–15). Similarly, the subjective value of an outcome is commonly depreciated by its uncertainty. Although a standard economic choice theory has assumed that the subjective preference or utility of an outcome is depreciated by its probability (16), behavioral studies have shown that the relative weight given to a particular outcome might be determined by a non-linear function of its probability (17, 18).

Various cues from the environment of decision makers provide information about the probabilities and timing of outcomes expected from each action. In some cases, as in behavioral studies on economic decision making, statistics and temporal properties of outcomes are explicitly indicated to the decision makers, making it possible to calculate the utilities of alternative actions. However, more often, how the state of the decision maker’s environment alters the relationship between an action and its outcomes has to be learned through experience. Accordingly, the utilities of alternative actions cannot be determined precisely, but can be only estimated empirically. To distinguish such estimates from the real utilities of actual outcomes, the empirical estimates of outcomes expected from a particular action are referred to as value functions in the reinforcement learning theory (19). As assumed by the economic theories of decision making, in which rational decision makers select the action with the maximum utility, decision makers or agents in the reinforcement learning theory tend to choose the action with the maximum value function. In contrast to rational economic decision makers, however, reinforcement learning theory requires its agents to behave stochastically, namely explore their environment, to gain the opportunity to improve the accuracy of their value functions. This conflict between the need to choose the action with the maximum value function (“exploitation”) and the need to choose randomly (“exploration”) is often referred to as exploration-exploitation dilemma.

Perceptual decision making

In a stable environment, decision makers might be able to extract sufficient information from sensory stimuli about the action that produces the most beneficial outcome in a given context. In addition, the outcomes of choosing the correct vs. incorrect actions might be fixed and well known to the decision makers. In such cases, the nature of decision making is perceptual, since the outcome of the chosen action depends entirely on identifying the sensory stimulus correctly. However, the task of such perceptual decision making can be challenging, when the sensory inputs are noisy, and their quality fluctuates over time. In the last two decades, important insights into the neural mechanisms of decision making have resulted from the studies that manipulated the difficulty of decision making by changing the signal strength of sensory stimuli (20–22). Results from the research on two-alternative forced-choice paradigms have generated converging evidence that during the time interval between the stimulus onset and motor response, noisy sensory evidence is temporally integrated, analogous to a diffusion process or random walk, and the response is emitted when this accumulated evidence reaches a certain threshold (23, 24). Such models predict, consistent with the findings from many studies on reaction times, that the accuracy of the choice will deteriorate and the reaction time will be prolonged as the sensory stimulus becomes noisier.

Models of impulsivity in decision making

Impulsivity and temporal discounting

Impulsive decision making can be viewed as a failure to take certain types of temporal factors into account appropriately. However, impulsivity is not a unitary concept, and includes several distinct cases in which time is handled suboptimally. For economic decision making, impulsivity refers to the tendency to weigh immediate outcomes strongly and to discount the value of delayed rewards precipitously (13–15, 25). As described above, humans and animals tend to prefer an immediate but smaller reward to a delayed but larger reward, when the difference in magnitude is too small or when the difference in delay is too large. This pattern of preference is often modeled by a temporal discount function, F(t), which describes how steeply the value of delayed reward decreases with its delay t. Although several different types of discount functions have been proposed (25), exponential and hyperbolic discount functions have been used in most behavioral studies on inter-temporal choice (Figure 1, left). Each of these discount functions has a single free parameter that controls its steepness, and therefore, such parameters can be used as measures of impulsivity. First, exponential discount function is given by the following.

Figure 1.

Comparison of exponential and hyperbolic discount functions (left) and their corresponding discount rates (right). Exponential discount functions (black) are given by F_exp(t) = exp(−k_exp t), where k_exp = 0.2 (solid) or 0.8 (dotted), whereas hyperbolic discount functions (green) are given by F_hyp(t) = 1/(1+k_hyp t), where k_hyp = 2.0 (solid) or 0.5 (dotted). For both types of discount functions, larger values of k parameters correspond to more impulsive choices.

$${\text{F}}_{exp}(\text{t})=exp(-{\text{k}}_{exp}\text{t}),$$

where t denotes reward delay, and k_exp corresponds to discount rate r, which is defined as the ratio r=−F′(t)/F(t). Therefore, for a given exponential discount function, its discount rate is fixed and independent of reward delays (Figure 1, right). Second, hyperbolic discount function is given by the following.

$${\text{F}}_{\text{hyp}}(\text{t})=1/(1+{\text{k}}_{\text{hyp}}\text{t}),$$

where k_hyp is the parameter that controls the steepness of the discount function. In contrast to exponential discount functions, the discount rate for hyperbolic discount function decreases with delay, according to k_hyp/(1+k_hyp t) (Figure 1, right). Nevertheless, for any delay t, F_hyp decreases as the parameter k_hyp increases, so the value of k_hyp in a hyperbolic discount function can be used as a measure of impulsivity in inter-temporal choice. Many studies have consistently demonstrated that behaviors of humans and other animals during inter-temporal choice are better accounted for by hyperbolic discount functions (13, 25–28). In addition, patients of substance abuse tend to display steeper discount functions (29–33), suggesting that such maladaptive behaviors arise from impulsive choices.

Impulsivity and response inhibition

In addition to steep temporal discounting, impulsivity also refers to the tendency to produce a habitual or some other default action prematurely. The ability to abort or suppress the movement that was previously prepared but is no longer desired has been studied extensively by using a stop-signal task (34, 35). In this task, the subject is required to respond to a particular sensory stimulus, such as a tone or visual target, but in a small proportion of trials, the subject is also required to cancel the movement in response to a stop signal. Whether the subjects can successfully abort the movement in response to a stop signal largely depends on the interval between the target onset and stop signal onset, which is often referred to as stop signal delay (SSD). For a short SSD, namely, when the stop signal immediately follows the target, the subjects are more likely to cancel the movement compared to when the SSD is longer. The performance of humans and monkeys in stop-signal tasks has been well accounted for by the horse race model (34, 36; Figure 2 top). In this model, the onset of a target and the onset of a stop signal trigger go and stop processes, respectively, and the movement is canceled only when the stop process reaches the threshold before the go process. The duration of the go process can be estimated directly from the observed distribution of reaction times during the trials without stop signals. In addition, the duration of the stop process can be inferred indirectly from the frequencies of cancelling the movement for each SSD, and is referred to as stop signal reaction time (SSRT). A relatively short SSRT implies that the subjects can successfully abort the movement even when the stop signal is somewhat delayed (i.e., long SSD). In contrast, a long SSRT corresponds to a reduced ability to suppress a movement that is no longer appropriate and therefore can be used as a measure of impulsivity. In fact, children with attention-deficit/hyperactivity disorder (ADHD) show prolonged SSRT compared to normal children (37), suggesting that impaired response inhibition might be a core deficit in this disorder (38). However, it is not entirely clear whether longer SSRT is a main characteristic for the behavioral deficit in ADHD, because reaction times in children with ADHD are more variable and this might introduce systematic biases in SSRT estimates (39, 40).

Figure 2.

Models of impulsive behavioral responses. A. Horse-race model for response inhibition in a stop-signal task. Go (green, G₁ or G₂) and stop (red, S₁ or S₂) processes are initiated when the target for the movement and stop signal are presented, respectively. Movements are canceled only if the stop process reaches the threshold (gray line) before the go process. Stop-signal reaction time (SSRT) refers to the time needed by the stop process to reach the threshold. In this example, a short SSRT (S₁) successfully cancels the movement with a slow go process (G₂), whereas a long SSRT (S₂) fails to cancel any movement. B. Random-walk or drift-diffusion model for a two-alternative forced choice (2AFC) task. Noisy evidence from a sensory stimulus is temporally integrated and “up” or “down” response is triggered depending on whether this integrated evidence reaches the upper or lower bound. In this example, the average signal is positive (a gray straight line). For a bound closer to the origin (B_low), an error is committed when the integrated evidence crosses the lower bound (−B_low). In contrast, a correct response is made for a bound farther from the origin (B_high).

Impulsivity and evidence accumulation

In the horse race model used to account for the behaviors in stop-signal tasks, the rate of increase in the activation for each process is variable across trials but remains constant within a given trial (Figure 2A). In contrast, in the so-called random-walk or drift-diffusion models, it is assumed that the sensory inputs are corrupted by noise and therefore display moment-to-moment fluctuations in their signal strengths (Figure 2B). The process of accumulating noisy sensory evidence is therefore considered analogous to the diffusion of a particle in one dimension, and the ultimate choice is determined by which of the two bounds (upper or lower) is crossed first. The accuracy of identification based on such noisy sensory evidence is enhanced by this temporal integration. These models provide an intuitive explanation for speed-accuracy trade-off, commonly observed in choice reaction time tasks (41). When the bounds are set farther away from the starting point, the choices become more accurate, but the reaction times increase (Figure 2B). Conversely, if the bounds are close to the origin, random fluctuations in the momentary sensory signals are more likely to produce errors, but this decreases the average reaction times. Therefore, within this framework, impulsive behaviors result from the bounds that are set too close to the origin (e.g., B_low in Figure 2B).

Neural mechanisms of impulsive decision making

Neural basis of temporal discounting

Inter-temporal choice behaviors are relatively well described by the models of temporal discounting. Therefore, the neural substrates of inter-temporal choice should somehow integrate the information about the magnitude and delay of reward expected from each option. Indeed, the blood-oxygen-level-dependent (BOLD) signals obtained from the medial frontal cortex and ventral striatum in functional magnetic resonance imaging (fMRI) studies are correlated with the temporally discounted values of rewards chosen by the subjects during inter-temporal choice tasks, suggesting that the information about reward magnitude and delay might be combined in these brain areas (42–49). Lesion and neuropharmacological studies have also implicated orbitofrontal cortex (50, 51) and nucleus accumbens (52) in temporal discounting. However, the results from neuroimaging or lesion studies do not reveal how the information about reward magnitude and delay is combined at the level of individual neurons.

In particular, if the activity of a given neuron related to reward magnitude and delay reflect the temporally discounted values, then it should change similarly as the magnitude of a reward is increased and as its delay is decreased. In fact, single-neuron recording studies in rodents and non-human primates have shown that the neurons in the prefrontal cortex and dopamine neurons change their activity according to the magnitude or delay of reward expected by the animal (53–60). When the animal’s choice depends on both magnitude and delay of reward, the activity of neurons in the prefrontal cortex is systematically influenced by both variables, and consequently, prefrontal activity reflects signals related to temporally discounted values (7, 59). In addition, the time course and the nature of signals related to temporally discounted values represented in the prefrontal cortex suggest that the prefrontal cortex is involved in multiple computations during inter-temporal choice. For example, immediately after the magnitude and delay of reward available from each option are specified, neurons in the prefrontal cortex largely encode the signals related to the sum of the temporally discounted values for the rewards available from different options as well as their difference, whereas the signals related to the animal’s choice and the temporally discounted value of the reward chosen by the animal develop more gradually (Figure 3b). These results suggest that the prefrontal cortex is involved in gradual transformation of the signals related to the values of individual options to the animal’s choice.

Figure 3.

Time course of signals related to the temporally discounted values and choice in the dorsolateral prefrontal cortex (DLPFC). A. Signals related to the magnitude (red) and delay (blue) of the reward expected from each choice and the choice of the animal (leftward or rightward saccade; green). B. Signals related to the sum of the temporally discounted values with the two alternative rewards (DV_L and DV_R for the left and right target, respectively; red), their differences (blue), the difference between the temporally discounted values of the chosen (DV_C) and unchosen (DV_UC) rewards (black), and the animal’s choice (green). Left panels, the fraction of neurons that showed significant modulations in their activity according to each variable during a 200-ms sliding window used for a multiple linear regression model. Right panels, coefficient of partial determination (CPD) averaged across the population of DLPFC neurons estimated for each variable in the same regression model (59).

The convergence of signals related to reward magnitude and delay and encoding of temporally discounted values in the prefrontal cortex might also endow it with the ability to monitor and enhance the decision maker’s ability to pursue the long-term benefits by resisting the temptation to accept immediate reward. Consistent with this possibility, some neuroimaging studies have found that the BOLD activity in the dorsolateral prefrontal cortex is enhanced when the subject chooses the larger but more delayed reward during inter-temporal choice (61, 62). Moreover, the involvement of the dorsolateral prefrontal cortex might generalize to other situations in which the decision makers must exert self-control to choose the options that will maximize their long-term benefits. Such abilities have strong implications in health care, since they might prevent over-eating and substance abuse (29–33). For example, a recent study identified a stronger BOLD activity in the dorsolateral prefrontal cortex when the dieters successfully choose healthy rather than tasty food (63). In addition, during the trials in which the subjects successfully exerted self control and chose healthy food, the activity in the dorsolateral prefrontal cortex influenced the activity in the ventromedial prefrontal cortex, which commonly shows the activity related to the subjective value of the option chosen by the subject. This interaction might be indirect and mediated by the Brodmann area 46 in the inferior frontal gyrus (63).

Neural basis of response inhibition

Many of our actions are habitual and they are often generated automatically and rapidly without much deliberation. Often, in order to produce most appropriate behaviors, such automatic and habitual behaviors must be first suppressed. Numerous studies based on lesions, neuroimaging, and single-neuron recording studies have suggested that this ability to suppress undesirable movements might require the prefrontal cortex (64–69) and basal ganglia (70, 71). Currently, however, whether and how specific cognitive processes involved in response inhibition are localized within different subregions of the prefrontal cortex is not completely understood. For example, a number of studies suggested that the right inferior frontal gyrous (IFG) or the ventrolateral prefrontal cortex (VLPFC) might be particularly important for response inhibition during a go/no-go or stop-signal task (64–66). However, some studies have also implicated the involvement of the dorsolateral prefrontal cortex in response inhibition (67–69).

Single-neuron recording studies in non-human primates have focused on the role of the frontal eye field (FEF) and supplementary eye field (SEF) in inhibiting previously planned saccadic eye movements during a stop-signal task (72–74). These studies have found that some neurons in the FEF display differential activity sufficiently early to cause the cancellation of upcoming eye movements during stop-signal trials (72), whereas the neurons in the SEF seem to play a role largely in monitoring the outcome of unwanted eye movements (73, 74). Single neuron recordings in the supplementary and pre-supplementary motor areas during a manual stop-signal task also indicated that the medial frontal cortex might be more involved in monitoring the consequences of animal’s actions and their motivational control (75). These results suggest that the activation in the human medial frontal cortex identified during a stop-signal task might also reflect functions related to performance monitoring (76). In addition, results from diffusion-weighted imaging tractography studies suggest that the network consisting of pre-supplementary motor area (pre-SMA), right inferior frontal gyrus (IFG), and subthalamic nucleus might play an important role in inhibiting unwanted movements (77).

Whereas the medial frontal cortex might not play a major role in response inhibition during a stop-signal task, it might be more involved in overcoming the tendency to produce habitual and automatic response according to various contextual cues (78). For example, neurons in the pre-SMA often increase their activity immediately before the animal switches from automatic, repeated patterns of behaviors to a new pattern of behaviors in response to sensory cues (79, 80). The lack of phasic activity in these so-called switching neurons was associated with the failure to suppress automatic actions. In addition, the activity patterns of switch neurons during a go/no-go task suggest that they are involved in suppressing a particular action as well as facilitating the desired action (80). In contrast, neurons in the subthalamic nucleus contribute largely to suppressing automatic actions that are no longer appropriate (81). These results are consistent with the finding that the disruption of the subthalamic nucleus leads to premature and impulsive actions (78, 82, 83). The subthalamic nucleus is known to mediate a short-latency excitation in the internal segment of the globus pallidus (GPi) induced by cortical stimulation. Since the output from the GPi inhibits the downstream neurons in the thalamus, the subthalamic nucleus is considered a part of the “hyperdirect” pathway (84), capable of suppressing alternative actions and promoting the selection of the most appropriate action (70, 85). Therefore, neurophysiological studies on action switching have highlighted the role of the subthalamic nucleus in response inhibition and controlling impulsive actions. However, whether this function is unique to the subthalamic nucleus or shared by other areas of the basal ganglia, such as the striatum, needs to be investigated further in future studies (86).

Neural basis of perceptual decision making

The random-walk or drift-diffusion models describe how both accuracy and speed of decision making are influenced by the stimulus discriminability (Figure 2b). The same models can also provide a parsimonious explanation for the speed-accuracy trade-off by shifts in their threshold or bounds (Figure 2b). A threshold placed closer to the origin can be crossed more easily by the noisy fluctuations in the integrated variable, leading to shorter reaction times but with more errors. There is a large amount of theoretical as well as empirical evidence for these mathematical models of decision making (5, 87, 88), but the mechanisms for adjusting the threshold are not understood. For example, the onset of saccadic eye movement during a choice reaction time task is tightly coupled to the moment when the activity level of individual neurons in the frontal eye field reaches a particular threshold (89). In the posterior parietal cortex, such as the lateral intraparietal area (LIP), the activity of individual neurons tends to rise at a rate that increases with the stimulus strength during a perceptual decision making task, converging to a particular level shortly before the animal produces its behavioral response (5, 90). These results suggest that the bound or threshold postulated in the drift-diffusion models might correspond to a fixed level of activity that needs to be reached by the neurons responsible for implementing a particular choice or action. Accordingly, impulsive behaviors during perceptual decision making might be accompanied by a lower threshold, causing the movements to be triggered more quickly even when the rate of activity increase in such cortical areas as FEF and LIP remains unchanged. These predictions need to be tested in future studies. Results from neuroimaging studies have also suggested that the cortical and subcortical network involved in switching from automatic to more controlled behaviors, such as pre-SMA and subthalamic nucleus, might also be involved in speed-accuracy trade-off (41, 91–93). However, the detailed underlying neural mechanisms need to be investigated by characterizing the properties of single neurons in these areas.

Conclusions

Impulsivity is often associated with a number of mental illnesses, ranging from attention deficit hyperactivity disorder (94) to substance abuse (95). While this label applies to a set of heterogeneous behavioral features, they are related by the fact that impulsive behaviors are suboptimally tuned along certain temporal dimensions. For inter-temporal choice, impulsivity implies placing too much weight on immediate outcomes, whereas other types of impulsivity result from the premium on speed rather than accuracy of responses or failures to suppress appropriate actions. The data from a number of studies suggest an important role of the prefrontal cortex and basal ganglia in controlling multiple types of impulsivity. Neuroimaging and lesion studies have implicated specific anatomical areas, including the medial frontal cortex and basal ganglia in addition to the prefrontal cortex, in controlling impulsive behaviors. In addition, single-neuron recording studies in non-human primates have also characterized the dynamic properties of neural activity that underlies temporal discounting and the suppression of prepotent but inappropriate behaviors. As we understand better the nature of neural mechanisms underlying various forms of impulsivity, such knowledge will lead to the improvement in the diagnosis and treatment of impulsivity-related behavioral disorders.