A distributed, hierarchical and recurrent framework for reward-based choice

Abstract

Many accounts of reward-based choice argue for distinct component processes that are serial and functionally localized. In this article, we argue for an alternative viewpoint, in which choices emerge from repeated computations that are distributed across many brain regions. We emphasize how several features of neuroanatomy may support the implementation of choice, including mutual inhibition in recurrent neural networks and the hierarchical organisation of timescales for information processing across the cortex. This account also suggests that certain correlates of value may be emergent rather than represented explicitly in the brain.

Neurobiologists have long been interested in developing mechanistic models to explain how we evaluate options and choose the best course of action^1–3. Many of these models take a modular perspective. That is, they assume, even if only tacitly, that goal-directed choice can be subdivided into a set of discrete component processes, and that the neural implementation of these processes is both serial and localized^4–7. The component processes typically include the evaluation of options, the comparison of option values in the absence of any other factors, the selection of an appropriate action plan, and the monitoring of the outcome of the choice. These component processes are generally assumed to correspond to discrete neural computations that are implemented in distinct neural structures.

An alternative perspective takes features of neural circuit anatomy as a starting point, and constructs circuit-based models that predict both behavioural and neural data while retaining biological plausibility at their core^8–14. Recent research using such an approach emphasizes three overarching principles of reward-based choice. First, decisions may be formed in a distributed fashion across many brain regions that act in concert and perform similar computations. Second, the distributed networks implementing choices are highly recurrent in nature, which affects the kinds of computation that are performed. Third, these distributed and recurrent networks are organised into functional and temporal hierarchies.

Centring behavioural models on neural circuit plausibility builds on several traditions. Its origins can be traced back to mid-century pioneers like Weiner, Hull, Hebb, and McCullough and Pitts^15,16. These scholars took advantage of then-new discoveries about neurophysiology and computer science to propose neurally plausible computational theories. The next generation of scientists — including Grossberg, Hopfield, McClelland, Rumelhart and Hinton — developed these ideas to model entire neural systems. The key insight of this branch, from McCullough and Pitts to the present day, was that neuron-like units that performed biophysically plausible computations and were connected in simple ways could perform astonishingly rich computations^17–20. Such systems do not have dedicated memory and processing subsystems, unlike other computing architectures. Memory and computation are instead interwoven in the system and distributed broadly throughout it¹⁷. This is directly relevant to choice models because evaluation depends critically on memory and comparison is a basic computation for choice.

These traditions continue to influence neuroscience to this day. Their influence on decision making has been expressed by computational neuroscientists such as Wang, Frank, O’Reilly, Rolls and Cisek, among others^8–13. Recently, several studies have tested key empirical predictions from circuit-based accounts of decision-making. Moreover, these advances have coincided with a resurgence of interest in neural networks in machine learning and computer science²¹, and methodological breakthroughs in neuroanatomy²² and circuit manipulation²³. These methods should allow for far more rigorous testing and refinement of circuit-level models in the near future. It is therefore particularly timely to consider the contribution that such models make to our understanding of reward-guided choice.

In this Opinion article, we first outline the key principles behind a distributed, hierarchical and recurrent account of reward-guided choice. We then discuss some empirical motivations for this account. We argue that the known and emerging neuroscience of simple economic choice is consistent with several important properties of circuit-based models.

The distributed, hierarchical, and recurrent approach differs qualitatively from modular explanations of reward-guided choice in that it is eliminative²⁴. This means that the neural implementation of choice does not necessarily recapitulate the steps or modules often used to describe the overall process. Instead, choice algorithms may be thought of as emergent properties of network activity. This change in perspective leads to a different core set of research questions than the modular approach (BOX 1).

Box 1. A change of perspective.

Adopting a circuit-based perspective of reward-based choice reframes the questions that we ask about how decisions are formed, and alters our interpretation of the resulting neural data. Fundamentally, it changes the question from “What is represented?” to “How is the computation implemented?” To be more specific, the modular perspective has led to us asking questions such as how is value computed and represented in the brain? In what regions do the evaluation, comparison and selection steps occur? In what ‘space’ (for example, goods-based or action-based) does value comparison occur? Do the computations performed in a particular brain region precede or follow the decision? What are the qualitative functional differences between different regions of the reward system?

By contrast, the distributed, hierarchical and recurrent perspective of reward-based choice makes a different set of assumptions, and these in turn lead to different research questions. First, it assumes that the component processes of choice may not be localised to particular computations in discrete brain areas. Instead, the components may be distributed across many regions simultaneously — implementing fundamentally similar, canonical computations. What is the nature of these computations, and how do different brain regions interact as choices are made?

The second assumption is that a comparison may not occur in a single decision space: decision spaces may reflect the anatomical connections of a given region, or even be artefacts of the experimental design. How does the hierarchical organization of decision and reward areas lead to effective choices? And how does it explain the observed hemodynamic and neuronal response patterns?

Third, most brain regions are both pre- and post-decisional. Equally, because decision formation occurs gradually, most regions may be better classified as mid-decisional. How does neuroanatomy produce a gradual transformation from offers to choices? What role do the brain’s ubiquitous recurrent and feedback connections play in that process?

Fourth, value, or at least certain correlates of value, may not be represented per se. The implementation of choice need not recapitulate the algorithms that can be used to describe the overall choice process. Instead, the algorithm is an emergent property of the system: certain correlates of value could therefore emerge naturally as a consequence of how neural dynamics unfold across different trials. What is the structure of these dynamics, and why might they give rise to value correlates? Which correlates of value is this true for?

Properties of the framework

Distributed

A distributed decision is one in which separate elements perform subsidiary computations that, when combined, produce the eventual choice. Well-known examples include the actions of individual voters in a national election and the selection of hive sites by swarms of bees²⁵ (BOX 2). In each of these cases, individual elements process a small (and often noisy) fragment of the overall input, possibly including the outcomes of other agents, to make an overarching single decision in the aggregate.

Box 2. Distributed decision-making in bee swarms.

Although modular decision systems are often intuitive — the functions map directly onto the structure — distributed ones are not. The decision process of a bee swarm provides a natural and intuitive example of a distributed decision system.

In late spring, a hive of bees will enter into a swarm state and begin the process of choosing a new hive site²⁵,¹⁵⁹. Individual scout bees make reconnaissance flights to identify and evaluate potential sites. The ideal sites are open, dry cavities of medium volume that are located high up in the canopy, protected from wind, and facing south; thus adaptive decisions need to optimize across many dimensions. On finding a potential site, each scout returns to the swarm and signals its location and quality through a specific patterns of dancing. Dances signalling high-quality sites can recruit other bees to investigate the same site. Subsequent bees evaluate the popularity of a potential site by counting the number of visitors there. When scouts detect a quorum of other scouts at a hive site (around 20 bees), they transmit an activation signal to the swarm. Bee swarms even show a distinct mutual inhibition signal that reduces the chance of costly ambivalence¹⁶⁰.

Beehive decision-making has several notable features that make it a good analogy for distributed decision-making²⁵. First, there is no localized evaluation: no individual bee has more than an extremely limited amount of information about the world, and each bee’s behaviour is remarkably stochastic. Second, there is no central decision-maker: no individual or subgroup makes the decision; instead, it arises in a well-understood emergent manner from the simple rules followed by individuals. Thus, removing any bee or bees would degrade performance in a graded but not all-or-none manner. Third, there is very little stable functional specialization: scouts are drawn at random and serve as site selectors, as observers of other bees’ dances, and as both members and monitors of the hive site quorum. Fourth, information about the value and location of options is firmly linked at every step of the process, thus sidestepping the otherwise difficult binding problem associated with choice and selection⁴⁵.

As a consequence, the steps of evaluation, comparison and selection are clearly performed by bees, but at the same time they are not performed by dedicated subsets of bees or at specific times. No individual bee ever has knowledge of more than one hive site, so no bee performs a comparison¹⁵⁹; instead the comparison step emerges as a consequence of the types of interactions the bees are programmed to perform.

A neuron can be thought of as a small and informationally limited decision-maker^26,27. It non-linearly transforms its dendritic inputs to its firing rate. However, this transformation of inputs to outputs does not have to be completed by a single neuron (or layer of neurons) to be useful. Neurons providing incremental change may instead contribute as part of a well-organised distributed system that includes multiple brain regions. Any neuron whose activity influences the behaviour of the network can be thought of as participating in that behaviour. In a decision-making network, this means that individual neurons may not require a pure representation of decision parameters to contribute to the decision-making process²⁸.

Distributed transformations within cortical circuits

Distributed decisions often involve a single simple computation repeated in each element on different inputs. One strong candidate for that computation in economic choice is competition through mutual inhibition. Mutual inhibition is a common motif that is found throughout the nervous system, and is often considered part of a basic repertoire of neural circuits^29,30. Effective competition via mutual inhibition can be mediated in a biophysically realistic cortical circuit model by an appropriate choice of synaptic weights^11,13 (FIG. 1a). Several recent results indicate that mutual inhibition may be at the core of reward-guided choice^31–42.

Figure 1. Evidence for competition via mutual inhibition during reward-guided choice.

a | A biophysical attractor network model of decision-making¹¹ relies upon effective competition via mutual inhibition between ‘A units’ and ‘B units’, which are neurons selective for a different option. Noise is added as an input to all excitatory units in the network, causing choice behaviour to vary stochastically on different simulations. This model has been used to explain various findings. b | The model predicts variation in correlates of value as a function of time³¹,³². In human magnetoencephalography data³¹, the model predicts a transition from ‘overall value’ correlates to ‘value difference’ correlates. In macaque single units recorded from dorsolateral PFC³², it predicts a sequence of value correlates from the original difference in options available to the eventual categorical choice. c | The model predicts an anticorrelation of regression coefficients between different presented offers. This is observed in single neuron data recorded from ventromedial PFC and ventral striatum during sequential choice³⁴,³⁵ d | In orbitofrontal cortex, different neurons are found that respond to the values of different juice offers, the value of the chosen offer, or the identity of the chosen juice⁴⁹. It has been proposed that these neuronal classes correspond to different identities of neurons in the network model³⁶, whose response properties are shown here. e | The model predicts variation in choice behaviour as the balance between excitation and inhibition is altered. This matches with evidence from human subjects, whose choice consistency is found to correlate with resting concentrations of glutamate and GABA in ventromedial PFC, indexed using magnetic resonance spectroscopy³³.

A recent study using magnetoencephalography demonstrated that human ventromedial prefrontal cortex (vmPFC) and intraparietal sulcus (IPS) expressed a key signature of mutual inhibition during economic choice: a change from encoding the sum of to the difference between chosen and unchosen values³¹ (FIG. 1b). This signature was subsequently also found in a macaque study of local field potentials (LFPs) from several subregions of the macaque PFC, including the orbitofrontal cortex (OFC), the dorsolateral prefrontal cortex (DLPFC) and the anterior cingulate cortex (ACC), confirming the generality of the mutual inhibition principle across species³².

Single-neuron recordings in macaques provide further evidence for a mutual inhibition process. In a binary choice, firing rates of neurons in both the vmPFC³⁴ and the ventral striatum (VS)³⁵ encode the values of the two offers through monotonic changes in firing rates. During the comparison period of the task, the directions of the tuning curves (positive or negative) for the two offers are opposed^34,35 (FIG. 1c). Consequently, the ensemble activity of both areas functions as a comparator between the values of the two offers. The fact that similar effects are seen in both areas, and with largely overlapping time courses, suggests that neither is the sole site of comparison, but that comparison may take place in both regions simultaneously.

Nor are the vmPFC and the VS likely to be unique in this respect; the mutual inhibition model has also been used to capture the dynamics of single-unit activity in the OFC³⁶ (FIG. 1d), the DLPFC³² and the lateral intraparietal cortex (LIP)^39,42. Data from studies in the dorsal premotor cortex (PMd) are also consistent with a mutual inhibition process. In this region, neural responses during economic decisions encode the relative value of targets in their response fields, and show additional sensitivity to the physical distance between those response fields^40,41. This finding suggests that even ostensibly motor areas are part of the distributed choice network.

Distributed transformations across cortical circuits

The body of work described above suggests that decisions occur through repeated mutual inhibition computations occurring simultaneously in both the motor and the abstract value domains. This implies that comparison is not the unique purview of any single brain area^43–45. The distributed account instead suggests that multiple areas may perform a similar non-specialized function: they may all perform a comparison operation like mutual inhibition on the inputs received. Importantly, however, those inputs would differ by region. The nature of the competition occurring in any given area would then depend upon the interaction of the particular demands of the task⁴⁶ and each area’s anatomical inputs^47,48. For example, tuning properties of neurons in the OFC may be relatively specialized for gustatory comparisons⁴⁹, whereas neurons in cingulate cortex may appear specialized for motor cost evaluations⁵⁰.

This idea of multiple distributed comparators may help to resolve differing interpretations of imaging studies in neuroeconomics, which have attempted to localise the regions that are most critical to value comparison. In blood-oxygen level dependent (BOLD) functional MRI (fMRI) studies, debate has particularly centred on different regions of medial frontal cortex. For example, one study used the mutual inhibition model to predict how variation in levels of vmPFC GABAergic inhibition, indexed via magnetic resonance spectroscopy in humans, related to cross-subject variation in both choice stochasticity and value correlates in the vmPFC BOLD fMRI signal³³ (FIG. 1e). A related study argued that because activity within a mutual inhibition model is highest at the end of the choice process and this activity is persistent, then the fMRI signal will be greatest for faster (easy) decisions, as is typically the case in the vmPFC⁵¹. However, other studies have argued that one should only consider accumulated activity until a decision boundary is reached. In this case, the fMRI signal would be higher for slower (difficult) decisions, as is typically found in the dorsomedial PFC⁵².

A potential reconciliation of these results would be that both regions implement a mutual inhibition process but differ in their response properties post-choice after a decision bound has been reached. This explanation is supported by the event-related profile of LFP recordings from multiple subregions of the PFC during reward-based choice³². It is also supported by the simultaneous emergence of single unit choice-related signals across six simultaneously recorded cortical regions in perceptual choice⁵³, and the demonstration that motor output (corticospinal excitability) is already biased as a decision is unfolding⁵⁴.

The question then arises of how different areas interact as choices are made. Here, whole-brain techniques (such as fMRI) come into their element⁵⁵. Although anatomical connectivity is stable, functional connectivity is more flexible. One recent study examined changes in functional connectivity during multi-attribute choices involving integration of stimulus-based and action-based attributes³⁷. A model of choice in which competition via mutual inhibition occurred at multiple levels (stimuli, actions and attributes) best explained the subjects’ choices. BOLD fMRI signal in the intraparietal sulcus (IPS) matched with the model’s predictions from the competition over which attribute was most relevant, with the value difference signal exhibiting opposing signs for the relevant and irrelevant attributes. Notably, changes in IPS functional connectivity to other brain areas depended on which attribute was most relevant to the current choice at hand. Connectivity was increased to either OFC or putamen when stimuli or actions were respectively made relevant. A related study of multidimensional learning demonstrated that the IPS is also particularly active when subjects update their understanding of the relevance of particular dimensions for guiding future choices⁵⁶.

Recurrent

The predominant paradigm in studies of simple economic choice has been N-alternative forced choice, in which two or more options are presented simultaneously to the subject and they select the most valuable. However, real-world choice is often sequential rather than simultaneous in nature^57,58. Even ostensibly simultaneous choices may be made sequential by virtue of limits imposed by attention or noise that needs to be integrated out across time^2,14,59. The relationship between simultaneous and sequential accounts of choice parallels a distinction made between the computational roles of two distinct architectures of neural networks²¹: feedforward and recurrent architectures.

Two different classes of network architecture

Feedforward neural networks — recently popularised through the impressive perfomance of convolutional networks in computer vision (ConvNets)^60,61 — contain units that exhibit activity that is only dependent upon the currently presented input. Such networks are ideally suited to tasks involving classification of multiple, simultaneously presented input features, such as pixel intensities in an image. The design of the ‘local convolution’ and ‘max pooling’ steps in a ConvNet were based on the response properties of primary visual cortex (V1) simple and complex cells, respectively⁶⁰. Modern feedforward ConvNet models strikingly reproduce single-unit responses along the visual hierarchy^62,63.

However, feedforward networks are poorly adapted to tasks that require some form of persistent memory for previous states across time. Recurrent neural networks (RNNs), by contrast, contain units that not only receive inputs from other network layers but also receive their own previous output at time t-1^18,64. This allows RNNs to show sustained memory for inputs long after they have been removed, allowing temporally extended computations to be performed on sequential inputs^65,66. One of the original successes of such networks was in showing how a biologically plausible network could account for working memory responses in the PFC^67,68. The recurrence of these networks recapitulates the high degree of recurrent connectivity observed empirically in prefrontal circuits relative to other parts of the cortex^69,70.

RNNs as a naturalistic substrate for sequential choices

At first glance, the simultaneous simple economic choices between two goods often studied in the lab appear well suited to being solved with a feedforward network. The options in such choices are, after all, presented to the subject simultaneously and statically. However, reaction times in such tasks vary systematically as a function of value, and this implies that the underlying computation is dynamic rather than static⁷¹. Recent models in neuroeconomics have therefore moved away from static economic accounts of choice towards temporally extended algorithms, in which noisy estimates of value are integrated sequentially over time^14,52. These algorithms can be formally related to the model of competition via mutual inhibition, discussed above and in FIG. 1⁷². This network exemplifies how a biologically plausible RNN could implement an evidence accumulation algorithm¹¹.

But why should the brain use a recurrent network if a feedforward architecture appears sufficient? It may be because of the sequential structure of choices that are typically faced in natural environments, such as foraging decisions. These choices probably have shaped the evolution of the frontal cortex most heavily^43,73. Foraging theory emphasizes the fact that nearly all decisions must be made in a strategic manner, and doing so requires a telescoping representation of rewards and their future allocation distribution^74,75. For example, when macaque monkeys perform a foraging task, activity within the anterior cingulate cortex (ACC) accumulates slowly over time in a depleting resource environment until a fixed firing threshold is reached, at which point a change in behaviour is triggered⁷⁶.

The algorithmic structures of economic and foraging models are certainly very different, and one possibility would be that they are implemented by different brain systems, with the former being feedforward and the latter recurrent. We consider it more likely, however, that evolution will have developed a general approach to solving both types of decisions using temporally extended, recurrent implementations. The neural mechanisms of economic choice may even build on the underlying architecture laid down to solve foraging choices using recurrent computations. This fits well with modern ideas in neuroeconomic accounts that argue for temporal integration of noisy value estimates across time^14,52,71, with attention being sequentially allocated to different choice options^59,77.

Understanding recurrent network computations

The majority of neuroscientific studies to date using RNNs have used networks containing hand-tuned synaptic weights to elicit specific behaviours¹¹ (Fig. 1a). Although this is a valuable starting point, it produces an unrealistic homogeneity of neuronal responses that does not match the empirically observed heterogeneity in cortical populations^78–80. Recently introduced automatic training algorithms for RNNs have vastly increased their capacity to perform a wide variety of tasks and also, potentially, to accurately describe neural data^81,82. A major challenge is then to understand the nature of the computations that an RNN is performing after it has been automatically trained.

An elegant solution to this challenge is to perform ‘reverse engineering’ on RNNs that have been fit to data⁸³. This was exemplified in a recent study of PFC population responses, in which the PFC performed both the selection and the integration of relevant information in a context-dependent perceptual decision task⁸⁴. The authors trained a randomly connected RNN to capture key features of the behavioural data; activity in the trained RNN matched with several key features of the neural population data. Reverse engineering of the RNN then revealed the mechanisms whereby such a computation could be achieved within a single cortical circuit. Two stable line attractors were present within the trained RNN; the selection of relevant inputs depended on how the network’s activity relaxed towards these line attractors under different contexts. Similar analyses of population recordings should yield important insights into the recurrent computations supporting reward-based choice in the near future⁸⁵.

Hierarchical

A hierarchical organization is a cardinal feature of the organization of the brain’s reward and decision-making system. Information from one area is converted to a more abstract form that becomes more comprehensive as it increases in complexity. One feature of the hierarchy of brain areas associated with economic choice is that there are new inputs at each point. Thus, for example, the OFC receives gustatory inputs, the vmPFC receives limbic inputs, the subgenual ACC receives hypothalamic inputs, and the dorsal ACC receives motor inputs⁸⁶. These inputs provide a way for different factors to enter into the distributed network subserving choice. They also allow for each area to have a different specific contribution to choice even if the general role of incorporating information into ongoing decisions is similar. This viewpoint differs from that in which all factors that influence choices must come together at a single point to create a single value scale before they can influence choice.

Hierarchical RNNs allow for multiple timescales

Humans excel at tasks that demand online organisation of behaviour across multiple differing timescales. Such tasks are particularly sensitive to PFC damage⁸⁷. It has been hypothesised that the presence of parallel and hierarchical architectures within PFC allow different pieces of information to be remembered and parsed at multiple timescales^88,89. This idea recapitulates recent developments in the design of RNNs in computer science. In particular, by making recurrent networks multi-layered or ‘deep’^21,90, it is possible to dramatically improve performance on tasks that operate over multiple timescales (such as the generation of meaningful sentences to describe images⁹¹, and the recognition of speech⁶⁵). The relevant timescales in speech are unknown to the neural network prior to training, but using automated training of multiple connected RNNs, the relevant temporal structure is extracted to successfully generate the requisite output. In decision-making, the ability of a population of cortical neurons to exhibit multiple timescales of integration over a previous history of rewards has recently been shown empirically⁹². These timescales vary in a manner that is dependent upon the timescale of integration of the animal’s current choice behaviour^92,93.

Hierarchical timescales for information processing in cortex

A hierarchy of timescales across brain regions has recently been explicated in a large-scale network model of dynamical processing in non-human primate neocortex. One study combined a mean-field reduction of a RNN within each cortical area with detailed knowledge of anatomical connectivity between different areas derived from tracer studies⁹⁴. This established a mechanism whereby multiple timescales could coexist within a single anatomical network, explaining the temporal structure of neurophysiological data recorded at rest across different cortical areas⁹⁵. This work suggests an explanation for why early sensory areas possess an inherently transient temporal structure (changing across tens of milliseconds) while higher regions (for example, in the PFC) show a more-sustained, longer-lasting temporal profile (across hundreds of milliseconds or longer) (FIG. 2a).

Figure 2. Hierarchical organisation of cortical timescales and its relationship with reward-guided choice.

a | The ‘intrinsic timescale’ of different cortical regions can be indexed by examining the rate of decay of their spike rate autocorrelation while the subject is at rest⁹⁵. This reveals a hierarchical organisation to cortical timescales, with lower areas having rapidly changing activity, and higher areas having more persistent activity (bottom panel; data points from the same data set are joined). The medial prefrontal cortex (PFC), in particular the anterior cingulate cortex (ACC), has the longest intrinsic timescale. b | The panel shows persistent network activity, and network ‘resets’ during behavioural shifts, in rat medial PFC in a reward-guided exploration task¹⁰⁰. Each element of the matrix reflects the trial-to-trial similarity of neuronal ensemble activity. The lighter and darker elements reflect more similar and more dissimilar network activity, respectively. Large, bright blocks reflect persistent, stable states across trials. Abrupt network transitions, and relaxations into different stable states, can be readily detected within network activity, and are related to behavioural shifts. c | The panel depicts that rostro-caudal hierarchy of temporal organisation in the PFC¹⁰⁴. A similar hierarchy can be seen on medial and lateral surfaces for internal motivation (reward) and task difficulty, respectively. Whereas more anterior regions reflect block-wise changes in reward or in task difficulty, more posterior regions reflect trial-to-trial changes. MT, middle temporal area; LIP, lateral intraparietal cortex; LPFC, lateral PFC; OFC, orbitofrontal cortex; S1, primary somatosensory cortex; S2, secondary somatosensory cortex.

Recent decision-making paradigms have combined evidence accumulation at both slow (inter-trial) and fast (intra-trial) timescales⁹⁶. It might then be predicted that evidence accumulation at slow timescales would be supported by cortical regions with sustained temporal structure. Interestingly, the dACC emerges as having the most sustained temporal structure within the regions that have been characterised thus far⁹⁵. This observation sits well with studies that have linked ACC activity to disengagement from a foreground option in foraging tasks^75,76,97,98 and during exploration⁹⁹. A foreground option may be held in mind over many trials, but disengagments are linked to abrupt and coordinated changes in network activity in the medial PFC¹⁰⁰ (FIG. 2b). Prior to disengagement, ramping activity is observed in both medial PFC–dACC^76,101,102 and another region associated with exploration⁹⁹, the rostrolateral PFC¹⁰².

Relationship to cognitive hierarchies

The observed relationship between anatomical and temporal hierarchies may be related to hierarchical accounts of PFC function in other cognitive paradigms¹⁰³. One such account suggests the existence of two parallel streams of rostro-caudal organization within the PFC, along the medial and lateral surfaces of this region, respectively¹⁰⁴ (FIG 2c). Data from BOLD fMRI studies suggest that a hierarchy relating to states of internal motivation exists along the medial surface ^104–106, with more anterior portions reflecting block-wise changes in reward value and more posterior regions reflecting trial-to-trial changes. By contrast, studies indicate that a hierarchy of cognitive control processes exists along the lateral surface^104,107,108. Again, anterior regions of the lateral surface exhibit sustained changes reflecting the complexity of the current block, whereas more posterior regions of this surface reflect task complexity only for relevant individual trials. Similar rostro-caudal PFC hierarchies have also been found in a related study where activation transferred to the striatum during the course of hierarchical rule learning¹⁰⁹. This observation was well described by a hierarchically organized network model of corticostriatal interactions¹¹⁰ inspired by a particularly successful form of RNN known as ‘long short-term memory’^111,112.

Evidence for the framework

Lesions and component processes

An important property of distributed computing systems is graceful degradation¹⁷. Because information is stored broadly, small amounts of damage to the system is seldom catastrophic; major impairments only come with large amounts of damage. Damage selectively impairs difficult retrieval processes and spares easier ones. The analogue of damage in connectionist networks is brain lesions. Brain lesions have long provided an important source of evidence for functional specialization. In the visual system, lesions to the middle temporal area cause akinetopsia¹¹³ and lesions to fusiform face area cause prosopagnosia¹¹⁴, confirming functional specialization for motion perception and face processing, respectively.

It has been natural for experimenters to design lesion studies that allow conceptually different components of choice to be assigned to different brain regions, and results of these studies have advocated a degree of functional specialisation within the PFC^115–117. However, given how profound the impact of PFC lesions on day-to-day choices can be¹¹⁸, the impairments on choice are often surprisingly mild. PFC lesions sometimes lead to graded deficits in choosing, with difficult decisions impaired but easier ones spared¹¹⁵. It is possible that these deficits are mild because the tasks seek to differentiate component processes of economic choice rather than addressing the temporally extended nature of real-world economic decisions⁸⁷.

The variety of cognitive functions affected by PFC lesions also casts doubt on the idea that choice, or even value comparison, is a specialisation of any single brain region. This doubt is supported by functional imaging data from studies of simple economic choice: different types of comparison signal are observed in different brain areas, depending upon the task at hand^32,46,52,75. Moreover, regions implicated in choice in lesion and fMRI studies are not specific to choice: they are associated with cognitive processes that include working memory, strategic planning, executive control, reasoning and social cognition¹¹⁹. Together, this evidence suggests that many brain regions will collectively contribute to the process of comparison, and also that brain regions subserving choice make their contribution as part of a larger supporting suite of cognitive functions. By way of analogy, reward-guided choice seems to operate less like motion- or face-processing and more like the ability to drive a car, form a political preference or do calculus: such abilities rely on coordinated computations across many brain structures and systems.

Ubiquitous value correlates

Another feature of connectionist networks is that their storage of memories depends on distributed synaptic weight changes across all parts of a network, rather than at a single site¹⁷. Value is closely related to memory: it is a feature of an option that is inferred from associations with reward, which are based on past experiences¹²⁰. In connectionist systems, memories are widely distributed (a feature leading to graceful degradation, discussed above), and traces of those memories can be observed throughout the network¹⁷; one might then predict that the same could also be true of value.

It is indeed the case that value correlates can be found in multiple brain regions. Correlates of value are seen in core reward regions such as the OFC⁴⁹, the vmPFC¹²¹ and the VS¹²², and they are also observed in the amygdala, the insula, the dorsal striatum, the midbrain, the pregenual, subgenual, dorsal anterior, and posterior cingulate cortices, the dorsolateral and ventrolateral PFC, the IPS, and even the sensory and motor cortices ^40,123–130. One recent neuroimaging study found that over 30% of the brain exhibited such signals¹³¹. Although there are certainly differences between the types of value-related information represented in these regions, there are also many overlaps.

Despite the ubiquity of value correlates, it often remains unclear whether activity in any area truly represents value — or what precise definition of value we should use^28,132,133. One common criticism of neuroeconomic studies is that putative value correlates often reflect other correlated variables, such as attention, salience or, even, subthreshold premotor activation. There have certainly been some important efforts to disambiguate value correlates from alternative explanations^28,133,134. Of course, it is possible that one of these value correlates genuinely represents value in the formal sense. However, in distributed systems, representation is often an emergent property; that is, it is driven by the specific pattern of connection between units and is not a property of any particular unit. In other words, value may be a network routing principle rather than a quantity that is represented. This eliminative view of value would be consistent with a subset of both classic studies and recent work in behavioural economics on process models of economic choice^58,135,136.

One particular value signal is probably an emergent property. Recent work indicates that the ‘chosen value’ variable is a by-product of variation in decision dynamics across trials. To understand why, consider the model of competition via mutual inhibition discussed above¹¹ (FIG. 1). Although many units’ activities in this network model correlate with chosen value, this quantity does need not to be represented to form a decision. Instead, it may arise naturally as a consequence of the varying speed at which network dynamics unfold on different trials³¹ (BOX 3).

Box 3. Value representations as epiphenomena of decision dynamics.

One particularly ubiquitous signal during reward-guided decision tasks is a representation of chosen value³⁴,³⁵,³⁹,⁴⁹,¹²⁹. This representation is isolated by correlating some measure of neural activity (firing rates, local field potentials (LFPs) or functional MRI signals) with the value of the option that will eventually be chosen on that trial. In measurements that are time-varying, this correlation can be repeated across many different timepoints, isolating the timepoint at which maximal variance is explained (see the figure, part a (green line)).

Why might this signal occur so commonly? Is it expressly represented by the brain, or is it an artefact? A clue comes from examining brain areas that also carry representations of other decision variables. These include the offer values of the options available, and the eventual categorical choice that the subject will make. Several studies³²,³⁹,¹²⁹ show that the maximal variance explained by chosen value occurs between an initial representation of offer values, and the final representation of choices (see the figure, part a). Chosen value representations emerge as the decision is being formed, rather than after the choice is completed.

Decision formation is, of course, a dynamical process occurring at different rates on different trials⁷¹. Crucially, chosen value influences decision speed. Indeed, at a fixed timepoint in the ‘middle’ of the decision process, the decision may have neared completion on some trials (typically with high chosen value), whereas on others it may be a long way from completion. Activity in any part of the brain that reflects the progression of this dynamical process will thus correlate with the chosen value at the mid-decision point (see the figure, part b). Notably, these dynamics have several possible neural substrates, including ramp-to-threshold accumulation in single neurons², neural population trajectories through a low-dimensional manifold¹³⁷, or bulk activity observed at the level of LFP or magnetoencephalography (MEG) signals³². In each case, correlates of chosen value would naturally emerge as a consequence of varying dynamics across trials. This idea has, for example, been used to explain the origin of a commonly observed ‘unchosen minus chosen value’ signal in dorsomedial prefrontal cortex using functional MRI⁵². Additionally, a recent study estimated the speed at which dynamics unfolded on a trial-by-trial basis using LFP and MEG data³². The authors found this single-trial estimate of decision dynamics explained some of the variance that was previously explained by chosen value – implying that the underlying cause of chosen value correlates was indeed linked to decision speed.

The above findings are in line with those in recent studies of motor control, which indicate that motor cortical activity is better understood as a dynamical system than as one that represents movement parameters^137,138. This line of work suggests that chosen value correlates emerge as a necessary consequence of recurrent network dynamics in mediating competition¹³⁹, and, again, that chosen value may not be represented per se. It remains possible that other ostensible value correlates (such as offer value and experienced value) are also by-products of the computations that underlie choice and are not reified in the activity of dedicated reward neurons or regions¹³⁵.

Elusive pure value

A modular view of reward-based choice predicts that certain brain regions or populations of neurons should be specialised for value, meaning that they respond primarily to values of options. Because value computation is a key intermediate stage in economic choice, the existence of specialized value regions or neurons is an important prediction of modular theories. By contrast, distributed theories do not demand any specialized value computation. Instead, in these theories, value is distributed broadly across a large number of regions, and is predicted in neurons that have other roles unrelated to valuation.

Some meta-analyses of neuroimaging data have argued that certain brain areas are central to ‘pure valuation’¹⁴⁰. However, even in putative core reward regions such as the vmPFC, OFC and VS, a wealth of information processing occurs that is not related to value. For instance, the vmPFC is engaged by several ostensibly value-neutral factors, including autobiographical memory¹⁴¹, spatial navigation¹⁴², imagination¹⁴³ and social cognition¹⁴⁴; OFC is engaged by non-reward processes like conflict, working memory, and rule encoding^145,146. Likewise, factors that modulate individual neurons in these areas may include ‘valueless’ changes in outcome expectancy^147,148, previous outcomes⁹², intention to switch as well as other strategy variables¹⁴⁹, metacognition¹⁵⁰, spatial positions of offers and choices⁴⁵, rules and task set¹⁴⁶, and even irrelevant task variables¹⁵¹. It is possible that these apparently value-neutral signals are observed in these tasks because the tasks nevertheless demand the computation of value. However, it is becoming more clear that neurons in most, if not all, value-relevant regions encode a large number of task-relevant variables simultaneously, a property known as mixed selectivity^152,153.

Neuronatomy

Finally, the strongest evidence for a distributed, hierarchical and recurrent approach to choice comes from neuroanatomy. Of course, the original work that provides the foundation for this approach developed side by side with progress in understanding the anatomy (and physiology) of the nervous system¹⁵. Perceptrons, feed-forward networks, parallel distributed networks, Hebbian learning and ConvNets were all inspired from observations about brain structure and physiology.

In the cortex, anatomical studies have indicated a reciprocal feedforward and feedback structure ¹⁵⁴. This architecture was quantified via detailed tracer studies in macaques¹⁵⁵ and diffusion imaging studies in humans²². Connectivity differences between adjacent prefrontal regions are smaller than is often appreciated, and adjacent regions generally blur into each other gradually rather than showing categorical boundaries¹⁵⁶. Likewise, when considering the organization of subcortical brain regions, it is now accepted that cortico-thalamo-basal ganglia loops are not segregated as once thought¹⁵⁷ but instead show strong functional convergence⁸⁶.

Local intraregional connections in the cortex also argue for a distributed and recurrent organization. This is particularly true of ‘higher’ cortical areas that subserve cognitive functions. In particular, dendritic arbors of PFC pyramidal cells are endowed with many more dendritic spines than pyramidal cells in sensory areas, meaning that a single neuron in prefrontal areas 10, 11 or 12 receives 16 times the number of excitatory inputs of a neuron in V1⁷⁰. The majority of these cortical connections are local rather than long-distance, allowing these circuits to have a highly recurrent organization that is similar to that observed in RNNs.

Discussion

In this article, we have presented an overview of recent work suggesting that reward-based decisions reflect the outcome of a distributed, hierarchical and recurrent computational process. These ideas have their genesis in connectionist and neural network models that have also been used to understand perception and memory, among other processes. More recently, these ideas have been integrated to form detailed models of economic or reward-based decisions^8–14.

According to the distributed view, the implementation of economic choice is dissimilar to a description of how it works at a more abstract level. Choice is an emergent consequence of the interactions of small computational elements¹⁵⁸. By contrast, much recent research into the neurobiology of choice has adopted a modular framework, in which major components of choice map directly onto brain structures and discrete computations. Thus, for example, this framework encourages scholars to look for the specialized sites of evaluation, comparison and action selection. The change in viewpoint we propose leads us to reframe this debate and many other central questions (BOX 1).

The work we describe here brings up a philosophical issue that has long influenced cognitive science: mental representation¹⁶. It has long been unclear whether we ‘represent’ mental concepts. That is, whether specific patterns of brain activity serve to recapitulate a mental version of some external event or object. Although neural correlates of important events and objects are observed, these correlations may be consequences of internal processes. Recent work provides two reasons to doubt that value, at least, is explicitly represented in the brain. First, in regards to chosen value, it seems that although value is decodable, it is possible that it is an artefact of the way neural data are analysed³². Second, in regards to value more broadly, the case is less clear, but connectionist models suggest that is possible to construct networks that make good choices without explicit value representations; these networks at the very least have a similar flavour to neuroanatomy¹⁷.

Paul and Patricia Churchland have articulated the notion of eliminative materialism, which includes a suggestion that natural categories we use to describe psychological phenomena do not do a good job of capturing the organization of brain processes that generate our mental lives²⁴. Economic choice may be one such case in point. Choice as a whole and steps like evaluation, comparison, selection and monitoring stages do not necessarily correspond to discrete anatomical substrates, to discrete neuron types, or even to discrete computations. Instead, they may be emergent consequences of processing units performing simple operations on inputs, ones that are radically different from the operations of the system as a whole²⁴.

Biographies

Laurence Hunt completed his DPhil in Neuroscience at the University of Oxford in 2012, and was then awarded a Sir Henry Wellcome Postdoctoral Fellowship to move to University College London. He is currently a senior research associate in the Sobell Department of Motor Neuroscience and the Wellcome Trust Centre for Neuroimaging at UCL. His research draws together neurophysiological data in animal models with imaging data in humans to understand the neural basis of decision-making. In his spare time, he enjoys singing, both with the London Oriana Choir and occasionally on the terraces of Oxford United.

Ben Hayden is an associate professor of Brain and Cognitive Sciences at the University of Rochester. He got his Ph.D. at UC Berkeley with Jack Gallant in 2005 and did a post-doc with Michael Platt at Duke until 2011. His lab studies the neural bases of economic choice and executive control, with a focus on problems involving foraging, curiosity, and risk. His lab website is http://www.haydenlab.com.