Neural dynamics and circuit mechanisms of decision-making

Abstract

In this review, I briefly summarize current neurobiological studies of decision-making that bear on two general themes. The first focuses on the nature of neural representation and dynamics in a decision circuit. Experimental and computational results suggest that ramping-to-threshold in the temporal domain and trajectory of population activity in the state space represent a duality of perspectives on a decision process. Moreover, a decision circuit can display several different dynamical regimes, such as the ramping mode and the jumping mode with distinct defining properties. The second is concerned with the relationship between biologically-based mechanistic models and normative-type models. A fruitful interplay between experiments and these models at different levels of abstraction have enabled investigators to pose increasingly refined questions and gain new insights into the neural basis of decision-making. In particular, recent work on multi-alternative decisions suggests that deviations from rational models of choice behavior can be explained by established neural mechanisms.

Introduction

David Marr, a pioneer in Computational Neuroscience who died at the young age of 35 in 1980, is mostly remembered today for his three-step recipe of brain modeling [49]: first, formulate the problem and identify its normative solution (the way it should be, optimally). Second, search for computational algorithms that accomplish the optimal solution and, third, elucidate implementations of such algorithm(s) in the brain. Yet, it would do Marr injustice to forget his deep roots in neurobiology (and his seminal trilogy of papers on the theories of cerebellum, hippocampus and neocortex). Francis Crick, who interacted extensively with Marr, reminisced in 1991: ‘David’s work clearly falls into two phases: Marr I was concerned with neural circuitry and what it might compute. Marr II (the AI phase) was more functional. The emphasis was on the theory of the process and possible algorithms, with much less attention to realistic implementations. I believe that if he had lived he would have moved to a synthesis of these two approaches.’ [19]. The relationship and interplay between the question of ‘how’ (Marr I) versus the question of ‘why’ (Marr II) of brain functions continues to be a subject of epistemological discussions in Neuroscience today [11].

With the tremendous advances in neuroscience, time is ripe to go back and forth between different levels of Marr’s hierarchy: behavior, computational algorithm and neural circuit mechanism. Recent research on the neural basis of decision-making offers an illustration of this perspective par excellence, as reviewed here.

Interplay between normative theory and neural circuit mechanism

Modern neurobiological studies of decision-making took off around the turn of this century. On perceptual decision-making, pioneer work was done using a random-dots motion (RDM) direction discrimination task. In this task, subjects are trained to make a judgment about the direction of motion (e.g. left or right) in a near-threshold random dot display, and to report the perceived direction with a saccadic eye movement. Neurophysiological studies of behaving monkeys showed stochastic activity of single neurons in the posterior parietal cortex [67, 68, 65, 13] and prefrontal cortex [39] that were correlated with the subject’s judgment. On the other hand, neuroscientists began to examine valuation underlying reward-based choice behavior [57, 63, 71, 3, 24]. Around the same time, in a seemingly unrelated effort, computational neuroscientists were developing increasingly realistic models of neural persistent activity as a brain mechanism for working memory (active short-term memory). Biologically realistic synaptic circuit modeling revealed that a working memory system should not operate as fast switches (between a resting state and memory states); instead, recurrent synaptic excitation underlying self-sustained persistent activity needs to be slow [75]. It was soon recognized that this slow reverberation mechanism is precisely what is needed for decision-making computations, because a deliberate decision requires a temporal accumulation of evidence for or against different choice options (via slow transients), ultimately leading to a categorical choice (through attractor dynamics) [76]. The proposal of a common mechanism for decision-making and working memory is supported by physiological observations that single-neuron activity signals correlated with both processes are found in the same brain regions such as the prefrontal and parietal cortices [30, 77] and by a recent human study showing that disruption (using transcranial magnetic stimulation) of the prefrontal cortex (known to be important for working memory) causally affected decision making [62]. Over the last ten years, recurrent circuit models (RNMs) have been elaborated and extensively applied by computational and cognitive neuroscientists to perceptual decisions [76, 48, 78, 42, 20, 22, 79, 27••, 41•, 23, 77, 81, 9, 53•, 25•], action selection [44] and value-based choice behavior [70, 36•, 5 ••].

The idea of temporal accumulation of evidence through sequential analysis in decision-making has a long history in cognitive psychology. According to the drift diffusion model (DDM) [47,69], a ‘decision variable’ integrates relative evidence for two alternative choices (A and B), a decision is made when the decision variable reaches either a positive (for alternative A) or negative (for alternative B) threshold. DDM has proven to be quite successful with accounting for behavioral data [47, 69], and more recent monkey’s accuracy and reaction time in the RDM task [65, 30]. DDM can be considered as a normative theory, since it is the continuous-time equivalent of the sequential probability-ratio test (SPRT), which is the optimal procedure for making binary choices under uncertainty in the sense that it minimizes the mean decision time among all tests for a given error rate [10, 6].

How does RNM compare with DDM? First, they are at two quite different levels of abstraction. DDM assumes an infinite integration time; whereas RNM strives to identify biological mechanisms for a long but finite integration time. A plausible neural basis of a long integration time is the NMDA receptor dependent recurrent synaptic excitation. This model prediction remains to be tested experimentally. It has also been proposed that neural integrators could be realized with an appropriate network architecture devoid of recurrent connections [31]. Second, the functional benefit of time integration was demonstrated in the model by showing that performance improves when the system is allowed to integrate inputs over a longer time, but eventually plateaus with sufficiently long integration as the system reaches an attractor state representing a categorical choice [76]. This prediction was confirmed in a recent monkey experiment [38]. Third, whereas in DDM evidence shown at different time points has equal weight, RNM asserts that evidence available early on has a larger impact on the ultimate choice than evidence presented later and immediately before a decision is made. This RNM prediction was supported in an experiment where a brief pulse of sensory information was introduced at different time points [35, 79]. However, in more general situations when sensory evidence or attention varies continuously in time, information provided a long time ago may be forgotten, and a commitment may be reversed in the face of newly presented evidence [64]. The biological basis and possible fundamental limitation of integration time in decision-making remain an outstanding subject of future research.

In RNM, categorical formation is realized by attractor dynamics that depend on feedback inhibition. This differs from an implementation of DDM in which feedforward inhibition implements an explicit neural computation of the difference in the sensory inputs [50]. Monkey experiments provided evidence for both feedback inhibition [32] and feedforward inhibition [8]. Note that it is empirically unknown whether accumulation of evidence and formation of a categorical choice take place in a single neural circuit. Alternatively, these two computations may proceed sequentially in distinct brain areas, or simultaneously through cooperative and distributed dynamics across multiple brain areas [17].

Accumulation-to-threshold in time and population dynamics in state space

RNM can be viewed in two different ways. One can plot neural activity of neural pools (each selective for a particular choice) as a function of time (Fig. 1A–B), and a threshold can be introduced for the termination of a decision process (Fig. 1A). On the other hand, alternatively, one can portray the dynamics of a decision circuit in the state space, where the firing rates of neural pools selective for different options are plotted against each other (Fig. 1C). According to this view, different choices are represented by distinct ‘attractor’ states. The mathematical term ‘attractor’ simply means a dynamical system state is stable against small perturbations. An attractor does not have to be a steady state but can be a complex spatiotemporal pattern. Furthermore, a system’s attractor landscape is not rigidly fixed; any relatively sustained input (an external stimulus or top-down cognitive control signal) readily alters the attractor landscape in the state space, as shown in Fig. 1C (middle and right panels).

Figure 1.

Ramping-to-threshold and population dynamics of decision-making. (A) Stochastic ramping activity to a threshold (dashed line) in a RNM (three sample trials are shown). (B) Two neural populations selective for different choices display graded ramping followed by winner-take-all competition, in a simulation of motion direction discrimination task where the task difficulty is quantified by motion coherence c′. (C) The population dynamics of a RNM is displayed in the state space of firing rates r_A and r_B. without external input (left panel), in the presence of a motion stimulus with a low (middle panel) or high (right panel) coherence. Note that the attractor landscape sensitively depends on the input (middle versus right panel). (D) Population dynamics of ~ 65 cells recorded from the posterior parietal cortex in mice performing a virtual-navigation decision task. Trajectories are choice specific (red: right choice trials, blue: left choice trials). Left panel: Sample trial trajectories in correct trials. Middle and Right panels: individual trial trajectories (grey and black) on erroneous right choice and left choice trials, plotted with the mean trajectories for correct right (red) and left (blue) choice trials. Adapted from [76] for (B), from [77] for (C) and from [33] for (D).

The state space framework puts the emphasis on population dynamics rather than single neurons. It offers natural explanations for observations that cannot be readily explained by the pure DDM: time integration has a limit because eventually the system reaches an attractor state when ramping stops; a brief pulse of evidence has a stronger impact on the ultimate choice early on when the system is still in the vicinity of the boundary between the basins of choice attractor states (Fig. 1C, middle and right panels); reaction times are longer in error trials because a wrong choice means that the system has to pass by a saddle point where the dynamics is dramatically slower (Fig. 1C, middle panel). Finally, according to this population dynamics view, the decision is made as long as the system has gone into the basin of a choice attractor state far enough, even though the neural firing rate may not have yet attained a threshold level. Biologically, the notion of decision threshold should be understood as the firing level of decision neurons that is required for triggering a switch-like response in downstream premotor neurons [42]. Consistent with this perspective is the common observation that, in a visual search task, behavioral reaction time co-varies with the time it takes for the firing rates of competing visually responsive neurons to significantly diverge from each other, rather than to reach a fixed threshold level, in FEF [66, 80, 18, 34], SC [52] and LIP [37, 73, 2].

A recent paper provided remarkable support for the population dynamics perspective. Harvey et al. [33 ••] recorded ~65 (ranging 37–94) single neurons in the parietal cortex (using calcium imaging) from behaving mice. In a virtual-navigation decision task, animals were shown a cue indicating, after a delay period, which of two turns (left or right) was correct choice and yielded a reward on a given trial. It was reported that dynamical trajectory in the recorded population’s state space is well separated for two alternative choices (Fig. 1D, left panel). Moreover, in error trials, the animal’s choice decoded from the population activity often started along the path of the correct choice but later somehow veered to be aligned with the path of the erroneous choice (Fig. 1D, grey traces in middle and right panels). Another line of recent work, using simultaneous electrical recording from multiple single neurons, suggests that the population dynamics framework also applies to the motor system [15, 16].

Ramping mode versus jumping mode

In most experiments a spike train of a single neuron is recorded at a time, which is often too irregular for investigators to ascertain that its underlying firing rate actually climbs smoothly and quasi-linearly over time (the ramping mode). Alternatively, a neuron may display a discrete jump of firing rate at a time that varies from trial to trial, so that the trial-averaged firing rate shows a graded time course (the jumping mode). In one realization of the jumping mode, single neurons are endowed with bistability, for instance thanks to the presence of a calcium-activated inward current [75, 72, 26, 61]. If neurons switch from one state to the other randomly, approximately at a constant rate, then the trial-average of a single neuron (or the population average in a single trial) would exhibit a graded rate change [61, 60 •]. Another scenario does not require single neurons to be bistable, but a recurrent network may switch from a low spontaneous state to a high choice state [21, 23, 28, 43, 53 •]. In other words, the resting state of RNM (in the left panel of Fig. 1C) may be still present in the presence of stimulation (Fig. 1C, middle and right panels). The system remains at that state for sometime until noise eventually induces a transition from it to one of the choice attractor states, which is reflected in the temporal domain as a jump in the firing rate of decision neurons. This noise-induced stochastic event takes place at random times in different trials, resulting in a trial-average that may appear to a graded ramping time course [55, 61, 53 •].

Notably, in the jumping mode, without noise the system would remain in the resting state, therefore fluctuations are required for decision making. A poorly understood issue is the sources of noise or stochasticity in a decision process. Perceptual decisions (identification, discrimination, etc) are often hard because sensory information is noisy, and integration of sensory data over time is computationally desirable because it improves signal-to-noise ratio [47, 30]. However, there is also stochasticity intrinsic to a decision circuit, and the Fano factor (the ratio of the variance versus mean of spike counts) of neural integrators may itself increase over time [54, 14 •]. This is likely to be generally true for neural circuits involved in both perceptual decisions and value-based choices, and stochastic neural dynamics of decision systems may play a critical role in indeterminacy of decision behavior [29, 77].

One possible way to differentiate contrasting models is to analyze not only the mean firing rate but also the time course of the variance of firing rates [14 •]; future theoretical work is needed to quantitatively assess differential dynamics of neural variability for the ramping mode and the jumping mode. Furthermore, new methods to simultaneous recording from many neurons [15, 33 ••] will enable investigators to assess whether multiple cells display any synchronous switch, as predicted by the jumping mode of RNM. On the theory side, how the distribution of jumping times is determined by network properties remains to be clarified.

Neural mechanistic explanation of deviations from rational behavior

Often times what constitutes optimality for a given decision task is unclear. Whereas SPRT is optimal for two-alternative forced choice tasks, optimal tests for three or more options are not known. Several ways have been proposed to generalize DDM to multi-alternatives [51, 58, 13•, 40 •], one of them is shown in Fig. 2A. RNM generalized to multi-alternative decision-making [74, 27 ••, 1] typically assumes competition between neural pools with shared inhibition (Fig. 2B). This model fits well with human’s performance data from a 3-choice RDM direction discrimination experiment [58] (Fig. 1C, left panel). The decision behavior of a three-choice version of the attractor network model can be well described by a softmax function, P(1) = exp(σV1)/(exp(σV₁) + exp(σV₂) + exp(σV₃)), where V₁, V₂ and V₃ are the values or strengths of evidence for the three options, and σ is a parameter that quantifies the amount of stochasticity (Fig. 1C, left and right panels).

Figure 2.

Three-choice decision-making. (A) A proposed generalization of DDM to three-choice. Adapted from [58] with permission. (B) RNM for multiple choice. (C) Left panel: Probability of choosing option 1 as a function of the motion coherence in the direction 1. Filled circles: data from three human subjects of the experiment [58] (filled circles); open circles: smulation results of a 3-choice RNM (B); solid curves: best fit of model simulations with a softmax function. Different colors correspond to different pairs of motion coherence levels for options 2 and 3 (blue: 0 and 0, green: 10/10, yellow: 15/5, red: 20/20, black: 25/15). Right panel: Performance data from RNM simulations are plotted against those predicted by the softmax function. Model simulations were carried out by Nathaniel Smith. These results need to be confirmed in future studies. (D) In a value-based choice task [45 •], 3 options are offered in the order of values (1: best, 2: second best, 3: worst). According to normative decision theory, option 3 should be irrelevant and changing its value should not influence the relative probability of choosing option 1 among the first two options P(1)/(P(1)+P(2)). In contrast to this ideal optimality, in the monkey experiment a higher value for option 3 reduces the relative probability for choosing the best of the two better options, which is inconsistent with the softmax decision criterion. Figure kindly provided by K. Louie and P. Glimcher. (E) Similar finding as in (D) in another monkey experiment, when medial orbitofrontal cortex was lesioned. Adapted from [59 •] with permission.

One prediction of the softmax decision criterion is that the relative probability of choosing one of two options (say 1 and 2), P(1 | 1+2)=P(1)/(P(1)+P(2)), is independent of the strength of the third option V₃. This prediction has been shown to be contradicted in economic choice behavior in a surprising way. In an experiment using both monkeys and humans, three choice options were associated with different reward values (1: best, 2: second best, 3: worst) [45 •, K Louie and P Glimcher, personal communication]. The third option has a lower value than both the first and second options, thus is irrelevant and should be ignored. Yet, when the value for the worst option 3 was increased (while remaining lower than those for options 1 and 2), subjects reduced the relative probability for choosing the best of the two better options, contrary to normative models of rational behavior (Fig. 2D). Similar findings were reported in another monkey experiment, but only when the medial orbitofrontal cortex was lesioned (Fig. 2E) [59 •]. These findings highlight the context dependence of neural representation of values [46].

Interestingly, deviations from rational behavior in the monkey experiment of [45 •, K Louie and P Glimcher, personal communication] can be concisely accounted for with the assumption that neural circuit is endowed with divisive normalization, namely the activity of a neuron is divided by the sum of activities of all neurons in the circuit; by virtue of normalization neural firing rates are correlated with the relative rather than absolute action values action values (K Louie and P Glimcher, personal communication). Divisive normalization has been widely observed in a number of cortical circuits [12], and has recently been suggested to play a useful role in decision-making [4, 46, 56]. Therefore, this combined approach using monkey behavior, physiology and model demonstrated how a neural circuit mechanism predicts robust behavioral trends that are not anticipated nor easily explained by optimality-based normative theories.

Further studies will clarify several important issues. First, it remains to be seen why this deviation from rational behavior was found in normal subjects in one experiment, yet only in animals with damaged brain in another experiment. Second, the above reasoning assumes that the parameter σ controlling stochasticity remains the same when the value of the third option is varied. However, σ may not be fixed, because an increased value for options other than the best ones renders the situation less certain, as a result the decision process might become more stochastic (characterized by a larger σ). It would be interesting to design new experimental manipulations to test directly these two contrasting scenarios. Third, although RNM simulations are consistent with behavioral data from a 3-choice perceptual decision experiment (Fig. 2C), additional data are needed to assess whether the third option with the lowest motion coherence does not affect the relative decisions between the other two options, as predicted by the softmax decision rule. New experiments will probe into the possibility that perceptual decisions conform to rational theory of decision-making (at least in well trained animals) whereas value-based choices are more prone to seemingly irrational behavioral effects which nevertheless can be explained by established neural computations such as normalization.

Conclusions

In this short review I focused on two themes of recent neurobiological studies of decision-making. One is the nature of neural decision dynamics. There are two complementary perspectives of a decision process: ramping-to-threshold in the temporal domain and population dynamics in the state space. Moreover, according to RNM, the same model (with modest variations of parameters) can display several different dynamical regimes such as the ramping mode and the jumping mode, suggesting that they could occur in different local circuits of the brain or under different conditions in a single brain area. New experimental approaches such as measurements and analysis of simultaneously recorded single neurons will shed new insights into these issues in the coming years. The second theme is the relationship between normative theories and neural circuit mechanisms. The fruitful interplay between RNM and DDM has shown that research in the field would greatly benefit from cross-talk between theories of models at different levels of abstraction, in close interaction with experiments. Epistemologically, neural circuit mechanisms should be not viewed as mere implementations of normative principles, but can provide a principled explanation of irrational choice effects observed in humans and nonhuman animals, as illustrated by recent findings with 3-choice experiments. Future research that integrates across cognitive, computational, and circuit levels will be especially promising in our quest to understand the neurobiology of decision behavior.