Adaptive neural coding: from biological to behavioral decision-making

Abstract

Empirical decision-making in diverse species deviates from the predictions of normative choice theory, but why such suboptimal behavior occurs is unknown. Here, we propose that deviations from optimality arise from biological decision mechanisms that have evolved to maximize choice performance within intrinsic biophysical constraints. Sensory processing utilizes specific computations such as divisive normalization to maximize information coding in constrained neural circuits, and recent evidence suggests that analogous computations operate in decision-related brain areas. These adaptive computations implement a relative value code that may explain the characteristic context-dependent nature of behavioral violations of classical normative theory. Examining decision-making at the computational level thus provides a crucial link between the architecture of biological decision circuits and the form of empirical choice behavior.

Introduction

Normative choice theories are the foundation of many modern approaches to decision-making, describing how the ideal or optimal chooser should make choices. In economics and psychology, rational choice models assume that choosers act to maximize a subjective measure of satisfaction termed expected utility [1]. In ecology, optimal foraging theory similarly assumes that organisms act to maximize an internal currency ultimately related to reproductive fitness [2]. Despite the rigorous mathematical framework and intuitive appeal of standard choice theories, empirical choice behavior violates the predictions of these optimality models in a wide range of species [3-9]. In particular, biological choosers demonstrate context-dependent preferences, where decisions depend on additional (often irrelevant) information beyond the values of the given alternatives. Here, we review how recent work on the neural representation of value information offers a biological rationale for these apparent violations of rationality. Consideration of such computational principles suggests that choice behavior reflects a utility optimization process operating under intrinsic biological constraints.

Context-dependent violations of rationality

A key principle of nearly all optimal decision-making theories is that preferences rely on a stable, independent valuation of each choice alternative. Given the assumption that individual alternatives are evaluated independently, decisions should be unaffected by uninformative contextual factors such as the previous history or the structure of the choice set. However, many of the documented behavioral deviations from optimality suggest that value is determined in a relative rather than absolute manner, with a characteristic dependence on choice context (Figure 1).

Figure 1. Behavioral and neural context-dependence in decision-making.

Empirical choice studies demonstrate two broad classes of context-dependent choice behavior (top), dependent on characteristics of either alternatives at the time of decision (spatial context) or alternatives in the recent past (temporal context). Both phenomena violate the standard normative assumption that decisions depend solely on the absolute, fixed values of choice alternatives. Increasing evidence indicates that similar contextual modulation governs how decision-related neural circuits represent value information (bottom), implementing a value code computed relative to current and past values.

The most prominent violations of rationality involve modulation by factors present in the choice set at the time of decision (which we term spatial context, drawing an analogy between the choice set in decision studies and visual space in sensory studies). Under spatial context-dependence, the relative preference between two given alternatives changes with the quantity or quality (attributes) of other alternatives. A number of spatial context effects have been described in human choosers, including the attraction [4], similarity [8], and compromise [7] effects, each requiring a relationship between attributes defined in a multi-dimensional space. In the attraction effect, for example, the introduction of a third decoy alternative that is similar, but inferior in quality, to one of the two original alternatives can selectively shift relative preference towards the closest original alternative. Interestingly, analogous phenomena exist in diverse and evolutionarily distant species including insects, birds, and monkeys, suggesting context-dependent choice behavior may be intrinsic to biological decision-making mechanisms [3,5,6].

Context-dependent preferences also vary with the past history of choices and outcomes, or temporal context. While less studied in the literature, both human and animal choice can exhibit a dependence on previously experienced alternatives [7,10]. For example, in [7], a choice between consumer items (e.g. car tires) differing in two attributes (e.g. warranty and price) depends on the distribution of attributes seen in a previous choice. Temporal context-dependence has been identified in various decision behaviors observed in the field, including speed dating, commuting behavior, and housing choice [11-13]. Most spatial and temporal context effects involve multi-attribute choice between alternatives differing in multiple dimensions, with the pattern of context-dependence depending on the specific relationship of the alternatives in attribute space. However, alternatives varying in a single dimension or varying in integrated value can also drive context-dependence, arguing that context-dependence is a general feature of the biological decision process [5,14,15].

Both heuristic and mathematical models have been proposed that can reproduce empirically observed spatial and temporal context-dependent preferences [7-9,16,17]. However, why empirical choosers exhibit such patterns of suboptimal behavior remains unknown. In this review, we focus on the underlying neural systems that implement value representation and decision-making. An understanding of information processing in neural systems, and the constraints under which they operate, can provide a crucial link between neurobiological substrate and resulting choice behavior [5,18,19].

The neural representation of value

A critical step in understanding context-dependent choice behavior is identifying how neural circuits represent value-related information. While many brain areas are modulated by rewards, electrophysiological and neuroimaging studies have identified specific neural circuits that represent the subjective values of choice alternatives (see [19,20] for recent reviews). Value coding is prominent in a network of areas linked to action selection, including sensorimotor circuits in prefrontal cortex, posterior parietal cortex, and the dorsal striatum [21-25]. In these areas, the activity of action-specific neurons varies with action value, a representation believed to be critical for selecting between competing motor actions. In addition, value also modulates neural activity in brain areas without an explicit role in action selection. For example, despite exhibiting little motor selectivity, neurons in the monkey orbitofrontal cortex represent the value of stimuli and choice alternatives [26,27]. Similarly, ventromedial prefrontal cortex and ventral striatum display a consistent valuation signal generalizing over different types of reward in human subjects [20,28]. These brain areas may function as a global valuation system, determining and assigning alternative values in a given choice scenario prior to the decision process.

Contextual modulation in value coding

Context-dependent choice behavior implies that value is constructed in a comparative manner, relative to the spatial or temporal background. Guided by these behavioral effects, recent experiments have begun to examine how contextual factors influence neural activity in identified value coding circuits (Figure 2).

Figure 2. Context-dependent neural value coding.

(A) Modulation by spatial context in monkey lateral intraparietal area (LIP) neurons. Left, average firing rate histograms as a function of time, segregated by the value context of the array of alternatives. Despite a constant reward associated with the neural response field alternative, LIP activity is suppressed by the values of simultaneously presented extra-RF alternatives. Right, average LIP activity as a function of spatial value context during alternative presentation (gray window in left panel). Neural coding for a specific alternative increases with the value of that alternative (different lines) and decreases with the total value of all alternatives (decreasing slopes). Adapted from [34]. (B) Modulation by temporal context in monkey orbitofrontal cortex (OFC) neurons. Left, average OFC value coding activity, segregated by the varying range of values presented in individual experimental sessions. These findings indicate that the representation of value is sensitive to the recent reward history. Right, simple range adaptation model of value temporal context effect. Adapted from [40].

The effect of spatial context on value coding has been primarily examined in sensorimotor regions of frontal and parietal cortex underlying action selection and execution. In these areas, single neurons represent specific actions, allowing observation of alternative-specific neural activity. For example, in the monkey lateral intraparietal area (LIP), individual neurons fire selectively to both visual stimulus presentation in, and saccadic eye movements towards, specific subregions of visual space (the neural response field, or RF). Across a variety of paradigms, sensorimotor neural activity in LIP correlates closely with the integrated decision information necessary for choice. In perceptual discrimination tasks, individual LIP neuron activity closely matches the accumulated evidence for the saccade toward the RF [29-31]. In economic manipulations, LIP activity consistently represents the subjective value of individual saccades, regardless of whether value is determined by reward magnitude and probability, delay discounting, foraging requirements, or strategic valuation in a competitive game [23,25,32,33].

Rather than an absolute representation of individual action values, however, new studies suggest that LIP and other similar regions employ a relative form of valuation [22,34-36]. In a recent study, monkeys were presented varying saccade target arrays, with different rewards associated with specific target alternatives [34]. Despite a constant reward associated with a saccade to the neural RF, LIP activity is nevertheless suppressed by the value of extra-RF target alternatives (Figure 2A). Coupled with excitation in response to RF alternative value, this suppression by spatial context value instantiates a relative reward representation. Notably, this form of value modulation mirrors extensively studied spatial contextual phenomena in sensory processing [37], pointing towards potential unifying neural mechanisms we discuss below in Neural computation and biological constraints.

Similar to behavioral context-dependent phenomena, neural value coding is also sensitive to the recent history of rewards. While thus far unexamined in parietal brain areas, temporal contextual modulation has been observed in other neural circuits in reward-guided decision-making. For example, reward-related responses are prominent in the monkey orbitofrontal cortex (OFC), a prefrontal region with extensive connections to higher-order sensory cortices and limbic system areas. Neurons in OFC encode a number of decision-related variables, including the subjective value of specific alternative classes (e.g. type of fruit juice) and the value of the chosen alternatives [27].

Instead of a fixed relationship between value and neural activity, OFC value coding adapts to the local statistics of recently experienced rewards [38-40]. For example, in monkeys experiencing different ranges of reward magnitudes, narrow reward ranges produce steep firing rate functions of value while wide ranges elicit shallow ones ([40]; Figure 2B). Similar adaptive value coding has been observed in other monkey brain regions including the anterior cingulate cortex [38] and midbrain dopaminergic nuclei [41], as well as in reward-related circuits in the human brain [42]. Such adaptive changes in neural response functions are widespread in sensory processing, and may be crucial for neurons with limited dynamic ranges facing changing contexts in a non-stationary world [43].

Neural computation and biological constraints

The widespread prevalence of both spatial and temporal forms of contextual modulation suggests a conservation of function across different circuits and systems [37]. In particular, contextual processing is thought to be critical for the maximization of information coded in spiking activity. Neural systems face a number of constraints, imposed by factors such as the energetic cost of spiking activity, irreducible noise, and maximum firing rates. To explain how neural systems can maximize information coding in the face of such constraints, Barlow's efficient coding hypothesis proposed that sensory systems exploit widespread statistical regularities in the distribution of the sensory environment [44]. Under efficient coding, regularity-induced redundancies in the incoming information stream are removed by sensory systems, increasing the independence of neural responses to different stimuli (thus maximizing information and increasing efficiency). Contextual modulation may play a key role in the efficiency of neural responses to such spatiotemporal regularities [45-48].

A critical question then is how neural circuits implement the relative information coding evident in contextual modulation. At the neural level, spatial and temporal context effects are evident as a form of gain control, modifying the input-output function that relates value information to firing rates. We focus here on divisive normalization, a prominent form of nonlinear gain control widely observed in early sensory systems [49] and recently characterized in higher order processes such as attention, multisensory integration, and decision-making [34,50,51].

The normalization algorithm computes a ratio between a given neuron's response and the summed activity of a larger pool of neurons [49,52,53]. Consider a pool of n neurons with each neuron i having its own driving input d_i, with the vector d denoting a set of inputs to the neural pool (e.g. luminance at different points in visual space, or the values of alternatives in a choice set). A general form of normalization can be expressed by the function:

$$R_i\left(\mathbf{d}\right)=r_{max}\frac{d_i^{\alpha }+\beta }{{\sigma }^{\alpha }+{\sum }_{j=1}^n{d}_j^{\alpha }},$$

where the response of a given neuron i depends on both the direct driving input d_i to that neuron and the summed driving input from the entire pool (typically including neuron i). The parameters r_max and β govern the maximal response level and the effect of spontaneous activity, respectively, while the exponent α represents the exponentiation of inputs. The semisaturation term σ determines the how quickly the function approaches saturation with increasing input, governing the range of inputs over which the response function is most sensitive.

In the normalization algorithm, division by the pooled activity of other neurons drives spatial contextual modulation. For example, normalization explains phenomena such as cross-orientation suppression and surround suppression in visual cortex [54,55], where the neural response to a stimulus is decreased when superimposed with other stimuli that are ineffective in driving a response on their own. In a recent study, normalization also characterizes how LIP neurons represent relative value in different choice arrays [34]. In particular, LIP value coding under different target and reward conditions were closely predicted by a simplified normalization model:

$$R_i\left(\mathbf{v}\right)=r_{max}\frac{v_i+\beta }{\sigma +{\sum }_{j=1}^n{v}_j}.$$

In this formulation, the value of an action towards the neural response field (v_i) is normalized by a term summing over the values of all available actions (v), incorporating reward characteristics of the current choice set into the neural representation of value. Thus the firing rate R_i representing the value of option i is a function of all alternative values v rather than v_i alone. Notably, divisive normalization outperformed a number of alternative models in explaining the observed electrophysiology data, including models based on value difference or simple fractional value [34]. In the framework of efficient coding, spatial normalization implements a relative representation that preserves discriminability between alternative values while obeying constraints on neural activity (Figure 3A).

Figure 3. Coding implications of spatial and temporal normalization.

(A) Spatial normalization scales firing rates to an activity bound. The start of each vector depicts a pair of value inputs (v₁, v₂), plotted in pre-normalization activity units (i.e. R_i = v_i). The end of each vector depicts the corresponding normalized firing rates (R₁, R₂) resulting from a simple spatial normalization equation with σ=0 and α=1. Choice performance will vary inversely with the distance between (R₁, R₂) and the indifference line (dashed) along the discriminability axis (red), assuming fixed decision noise. Large value inputs (upper example vector, black) will be scaled down to the activity bound, reducing discriminability by the minimum possible amount. Small value inputs (lower example vector, black) will be scaled up to the activity bound, increasing discriminability. (B) Temporal normalization adapts the firing rate function to changing value distributions. Different value distributions (bottom) and the corresponding neural value representations mapping value to firing rate (top). Given a fixed range of neural activity, an efficient neural representation will approximate the cumulative distribution of values, aligning the steepest region of the output function with the most likely value inputs. Under temporal normalization, horizontal shifts in the value distribution (black to green) produce horizontal shifts in the neural mapping function, mediated by changes in σ(). Changes in the variance of the value distribution (black to blue) require a change in the shape of the mapping function, which may be mediated by either a change in exponentiation (α) or a more complex dependence of σ() on recent values.

Normalization can also incorporate temporal contextual modulation via a semisaturation function σ(), which can introduce a divisive term that depends on the recent history of input. For example, a purely temporal form of context-dependent value coding can be written as:

$$R_i(v_{i,t},v_{i,t-1},\rightleftharpoons ,v_{i,t_0})=r_{max}\frac{v_{i,t}+\beta }{\sigma (v_{i,t-1,\rightleftharpoons },v_{i,t_0})+v_{i,t}},$$

where the firing rate representing the value of alternative i at time t is a function of both current and past values (extending back to some t₀ , where t₀ can go to −∞). In this form of temporal normalization, the effect of past value is mediated through the function σ().

In order to adapt to local statistics in a dynamic environment, σ() typically is structured to emphasize recent over distant past values (e.g. an exponentially decaying weighted average of past rewards). Intuitively, σ() can be viewed as an expectation of future inputs, based on samples from past history; because this term governs horizontal shifts of the response function, normalization aligns the widest range of spiking output to the most likely range of input (Figure 3B). In sensory processing, history-dependent normalization explains how neural responses adapt to changing stimulus properties such as luminance and contrast [49,53,56].

The divisive normalization algorithm has yet to be explicitly tested in cases of adaptive value coding, where reward-related activity varies with the range or variance of recent rewards [38-40]. These data suggest that value coding can adapt to higher-order moments of the recent reward distribution beyond the mean. The normalization algorithm can adapt to distributional statistics given the proper choice of the parameters, but the precise role of normalization in adaptive value coding remains a key target of future research.

Both value context effects described above can be combined in a single spatiotemporal normalization:

$$R_i({\mathbf{v}}_t,{\mathbf{v}}_{t-1},\rightleftharpoons ,{\mathbf{v}}_{t_0})=r_{max}\frac{v_{i,t}^{\alpha }+\beta }{\sigma {({\mathbf{v}}_{t-1},\rightleftharpoons ,v_{t_0})}^{\alpha }+{\sum }_{j=1}^n{v}_{j,t}^{\alpha }},$$

but no clear empirical evidence yet exists for such an integrated contextual value representation. In the monkey, orbitofrontal value coding shows strong temporal contextual modulation [39,40], but appears to be unaffected by aspects of the current choice set [57]. On the other hand, prefrontal [22] and parietal [34-36] value coding shows strong spatial contextual modulation, but the effect of temporal context has not been examined. The potential integration of spatial and temporal context effects in both neural processing and behavior thus remains an important open question.

Linking computation and choice behavior

Despite extensive documentation in the ecology and psychology literature, little is known about the neural mechanisms underlying context-dependent preferences. In sensory processing, both spatial and temporal forms of contextual modulation are linked to corresponding perceptual phenomena. Recent evidence suggests a similar linkage may exist between adaptive value coding computations and context-dependent decision-making.

A recent study quantified spatial context effects predicted by the normalization model and compared them to empirical choice behavior in monkey and human subjects [5]. Under value normalization, the overall activity signaling the subjective value of a choice alternative is normalized by a term including the summed value of available alternatives. Because this normalization is simultaneously applied to all alternatives, the composition of the choice set should not alter the relative ranking of alternatives by average firing rate. However, because division scales the difference between firing rates, normalization predicts that the discriminability between alternatives will vary with the choice set. Specifically, the relative preference between two alternatives should be diminished by increases in the value of a third alternative, even if it is never chosen. In value-guided choice experiments, both monkey and human choosers showed this novel form of context-dependent behavior predicted by the value normalization model.

While normalization provides a clear link between neural and behavioral spatial context effects, additional mechanisms can influence the decision process. In a choice task involving a presented, but subsequently unavailable, third alternative, human subjects show diminishing choice performance as context value decreases – an opposite effect from that predicted by value normalization [58]. These differing results suggest that specific aspects of the behavioral task are likely to play an important role in determining the influence of context. In [58], the value of alternatives were determined by subjects from visually-cued attributes (reward magnitude, probability), raising the possibility of additional interactions in the value integration process prior to choice ([59]; see Attributes and the valuation process). Given the brief initial availability of the third alternative, the timing of biophysical processes may also play a crucial role; examination of dynamical models suggest that normalization is a time-varying computation with a stronger influence late in the decision process [60-62].

The temporal context effects observed in electrophysiological experiments on value coding have not yet been directly linked to behavior, but the relationship between the internal representation of decision information and choice has been shown in perceptual decision-making tasks. In a recent study, human subjects viewed a rapid stream of sinusoidal gratings of varying orientations and judged whether the average orientation was closer to the cardinal or diagonal axis [14]. Analogous to economic decisions based on value, observed choices depended on the integrated decision variable (i.e. weighted sum of orientations). When the authors examined the impact of individual samples on choice, they found two effects: higher weights were applied to later samples (a recency bias) and to samples consistent with the average decision information at the time of viewing. Critically, these effects are explained by an adaptive gain process that adjusts the transfer function relating decision information to behavior; moreover, such adjustments aligns the maximal sensitivity of the function with the distribution of samples seen in the recent past. While not modeled explicitly with normalization, this work indicates that adaptive tuning of internal representations of decision information may be a critical feature of temporal context effects.

Attributes and the valuation process

A critical issue for future research concerns how overall decision values are constructed from separate alternative attributes and how contextual modulation affects this process. The electrophysiological and behavioral evidence discussed above focuses on the integrated value of choice alternatives, but the primary examples of context-dependence in the decision literature involve multi-attribute choice, where alternatives differ in multiple dimensions [3,4,6,8-10,63]. However, given the existence of adaptive gain control and contextual modulation at the level of integrated value, similar mechanisms may govern attribute processing prior to integration. Recent computational work suggests that a range normalization process over attributes can reproduce many of the documented context-dependent phenomena [18]. Furthermore, neuroimaging data in human subjects performing a multi-attribute reward-guided decision task are consistent with a hierarchical competition process involving normalization at the attribute level [59]. Because a standard divisive normalization algorithm operating at the attribute level cannot generate the full range of empirically observed phenomena, classic multi-attribute effects likely involve additional neural processes. Thus, at its most complex, choice behavior may involve different adaptive coding mechanisms operating at multiple stages of decision-making.

Conclusion

Understanding how the brain represents behaviorally relevant variables is a key step in linking behavior to the underlying neural mechanisms. Classic approaches to decision-making rely on a characterization of choice behavior, but recent interest has turned to identifying the neural basis of valuation and choice. Emerging evidence for a context-dependent neural representation of value, and its relevance for context-dependent violations of rationality, underscores the role of information processing in neural circuits. Fundamental aspects of information processing – such as the use of canonical gain control algorithms like normalization – occur in both sensory and decision-related systems, suggesting common principles of neural computation. In particular, context-dependent value coding may reflect an adaptive response to the intrinsic constraints of computing with biological circuitry.

Highlights.

• Empirical decision-making often violates normative choice theory

• Suboptimal choice can arise from contextual value coding in neural activity

• Adaptive computations can maximize information coding in constrained neural circuits

• Biological choice reflects a tradeoff between performance and the cost of computation