The hierarchical construction of value

Abstract

Here we argue that the assignment of subjective value to potential outcomes at the time of decision-making is an active process, in which individual features of a potential outcome of varying degrees of abstraction are represented hierarchically and integrated in a weighted fashion to produce an overall value judgment. We implicate the lateral orbital and medial prefrontal cortex in this function, situating these areas more broadly within a hierarchical integration process that takes place throughout the cortex for the ultimate purpose of valuing options to guide decisions.

To survive and prosper, humans and other animals need to choose actions leading to beneficial outcomes. Most modern theories of decision-making presume that individuals accomplish this by computing an expected value (or utility) for different decision outcomes, and, all else being equal, committing to the option that yields the highest expected value [1–3]. Value is not merely an abstract mathematical construct, but is rather correlated with neural activity in the brains of humans and other animals in a manner ultimately predictive of (and in some instances, causally related to) choice behavior [4–8].

However, a fundamental open question remains: how are these value signals computed in the first place? Attempts to answer this question have predominantly involved an appeal to associative learning whereby a cue or an action stimuli acquire value through associations being formed between a hitherto affectively neutral stimulus or action, and an outcome with an extant (perhaps innate) value [9]. However, while past associative history is crucial for accounting for how predictive cues or actions come to elicit outcome representations (including ultimately outcome value) it leaves us with an incomplete picture of how value signals for those potential outcomes are computed in the first place.

This is because value is not a static variable -- instead it can change flexibly and without prior experience depending on both intrinsic and extrinsic factors. For instance, a packet of peanuts may have high value when hungry, but become dramatically less valuable after lunch is consumed. A warm jacket might be desirable when going on a ski trip, but be much less so when planning a vacation in Hawaii. Clearly, the brain is capable of flexibly making value-based decisions on the fly based on current motivational and homeostatic states, the context in which a stimulus is being perceived, and the goal that is currently being pursued. Indeed, it is even possible for values to be produced for stimuli that have never before been experienced [10].

How is it possible for value to be computed so flexibly?

We propose that the brain actively constructs the value of a stimulus by integrating over its constituent attributes or features in a context-dependent way. These features are the components by which a potentially never before seen novel stimulus is evaluated. For instance, a food stimulus consists of odor, texture and gustatory components, visual appearance, caloric density, and different nutritive content such as carbohydrates, protein, fats etc, or more abstract judgments such as health and tastiness [11]. A work of visual art is composed of different color, intensity, textures, shapes, and can be designated as abstract or concrete, dynamic or still and so on. We argue that in order to compute an overall value signal, the brain assigns weights to individual features, which are then integrated in a linear or non-linear manner to compute an overall value for a stimulus (Fig. 1). The weights assigned to individual features reflect an individual’s subjective judgement about the degree to which a particular feature should count toward the overall value of a stimulus. For instance, for food, an individual might assign a high positive weight to protein, meaning that items that are high in protein will be assigned high value and a high negative weight to carbohydrate, such that items high in carbohydrates would tend to be valued low. When making a decision about whether to consume a particular food, the perceived features of the food such as its protein and carbohydrate content get combined with the weights over those features to compute an overall value.

Figure 1. Illustration of hierarchical active value construction process.

An object (here shown an item of clothing) gets broken down into underlying features, which in this case (as the object is presented visually) consists of low-level features of color, shape, visual texture etc. Mixtures of these low-level features construct higher level features including more abstract properties of the item, such as whether the clothing is warm or not, whether the clothing item is formal or casual etc. These features are assigned weights that in turn are integrated over to flexibly compute an overall value judgement.

The active construction of value from a weighted combination of underlying features naturally endows the decision-making agent with the capability to: (a) generalize value judgments across stimuli encountered in the environment, even novel ones, provided judgments about the underlying features can be made and (b) flexibly change the weights assigned to attribute features based on changes in internal motivation/homeostasis and/or external context. For instance, if encountering a new potential partner on a dating app, the potential value of that partner can be rapidly evaluated by considering their attractiveness, social status, career, and so on. Similarly, if a person highly values protein in food but then consumes a large protein-heavy meal, the weight assigned to that attribute can be switched from positive to negative resulting in an immediate change in value for any food that is high in protein.

Hierarchy of features

Features themselves can be organized in a hierarchical manner. To illustrate let’s take a visual image such as a painting. It is known that the visual cortex will extract/detect various low-level features such as the color, texture and shapes present in the image. These features can then be used to construct more abstract, high-level features about the painting, such as how dynamic or still the painting is. These higher level features can then in turn be integrated to compute an overall value judgement [12].

Different possible architectures for context-dependent feature weight integration

How might value be computed flexibly using hierarchically represented features? There are at least two ways to do this (Fig. 2): the first is to have representations of features that are independent of context but where weights are flexibly assigned to those features according to the context. For instance, when choosing clothing to go on a ski trip, a large positive weight could be assigned to the ‘warmth’ of the clothing, while when planning a trip to Hawaii, the feature weight for ‘warmth’ might actually be negative. This approach might have the advantage of being maximally flexible for new situations and contexts, such as when transitioning to a new environment, at the cost of it being a relatively computationally expensive process to dynamically change the weights.

Figure 2. Different ways to construct context-dependent value.

A. Weight adjustment. In this scheme, stimuli features and context features are integrated in parallel. One way to achieve flexible value judgement in such a scheme is to change the integration weights according to contexts. B. Representing mixtures of stimulus X contexts. In this mechanism, stimuli features and context information are mixed nonlinearly before value judgement, and different context x stimuli features representations are activated according to the current context. This enables flexible judgements with fixed integration weights.

Another possible architecture is to have multiplicative representations of “feature X context” whereby weights on each of the feature x context representations are fixed across contexts. Instead, what changes is the activated representations of particular feature x context combinations according to which context is currently active. Although the brain needs to mix features and contexts to generate such representations, this type of implementation might be ideal for decisions in contexts that individuals repeatedly encounter [13]. In this sense, context itself can be viewed as another feature, and high-level features include context-dependent features. In this implementation, learning about the value of a particular outcome in a given context such as in incentive learning [14], would involve training of specific context dependent feature weights. Note that such feature-context dependent weights would still enable generalization to novel stimuli in a given situation, provided that the stimulus shares features with stimuli that have already been experienced.

Determining which of these possible architectures are actually implemented in the brain and how these implementations are reflected by different measures of neural activity (i.e. BOLD fMRI, iEEG, and single-neuron) are important research questions. It is even possible that both architectures exist simultaneously, in which case another important question would be under what conditions does the brain adopt one mechanism or the other. Perhaps the computation of value for novel contexts and feature combinations might rely on a flexible weight adaptation scheme, while repeatedly encountered outcomes and contexts might rely instead on embedded feature x context representations.

Another important consideration is the role of attention in feature-based value computation. If some features are attended to more than others, this could lead to a greater weighting on those features, consistent with previous work on the role of attention on stimulus valuation [15]. Attention could also facilitate gating of which features enter into value construction, so that less relevant features are not considered [16], thereby increasing efficiency and reducing computational complexity.

Incorporating classic decision variables

Classical economic decision variables can be accommodated as features in this framework. Two ubiquitous variables are the magnitude and probability of an outcome. A reward-maximizing strategy would simply multiply these two variables to compute an overall expected value. However, recent behavioral data suggests that at least under some contexts, human behavior deviates from this normative expectation, such that the integration of these features might be better approximated as a sub-multiplicative linear process [17–19].

Another set of decision variables are higher-order properties of an outcome distribution such as its variance. Different forms of variance described in economics include risk and ambiguity. It is well known that individuals vary considerably in their attitude toward these variables when making decisions [20–22]. In the present framework, differences in preference for different forms of outcome variance can be easily approximated by assigning different weights over the components and by integrating over them alongside the mean value, as formulated in mean-variance approximations of expected utility [23,24].

Thus, by treating classic decision variables as yet another set of relevant outcome features and by in turn assigning different weights to each of these features, it is possible to capture variation in behavioral preferences as accomplished in classic decision theories. Next, we briefly turn to where value construction might happen in the brain, with a particular focus on the lateral orbital (lOFC) and ventromedial (vmPFC) prefrontal cortex.

Neural substrates of flexible value computation

It is well established that the lOFC and vmPFC are two key areas that play a central role in enabling the current value of an outcome to guide behavior. Lesions to these structures result in an impairment to alter choice behavior of a stimulus or action in order to obtain a specific outcome, when the value of that outcome has changed, by for instance feeding an individual to satiety on that specific food, or following a rapid change in the associations between stimuli or responses and outcomes [25–28]. Neural activity in the lOFC and vmPFC tracks the current value of a predicted outcome [29–31], which can be updated rapidly following a change in contingencies or outcome values.

Sensory representations of prospective outcomes and outcome identity in lOFC

The sensory features of prospective outcomes per se are also represented in these regions, particularly in lOFC. The lOFC receives inputs from all sensory modalities [32–34] and neurons in this region respond to gustatory, olfactory, visual and auditory, and somatosensory stimuli, consistent with its role as a highly multisensory area [35,36]. More specifically, the identity of experienced outcomes [37], of the cues that predict those outcomes [38] and the cue-elicited identity of predicted outcomes [39], have been found to be represented in this area. Outcome specific responses in this region decrease as a function of a change in the value of an outcome induced via satiation, suggesting that the value of specific outcomes are encoded in this region [40]. This implies that lOFC is involved in linking cues to the sensory identity of outcomes, as well as to the value of those outcomes.

Individual features of prospective outcomes are represented in lOFC

Howard and Gottfried [41] examined changes in component representations of odors at the level of the fMRI BOLD signal while participants were devalued on a specific food associated with a target odor. In this study, representations of specific odor components as well as of the whole odor, showed changes in activity in OFC following satiation. Suzuki et al. [42] examined the extent to which the subjective value of foods could also be predicted from underlying nutritional features. In that study, hungry participants were scanned with fMRI while reporting their subjective valuation of different foods. After the scan was complete, participants saw each of the foods again and were asked to make a judgement about the relative nutritive content of a food, including its carbohydrate, protein, fat and vitamin content among other factors. Using these subjective nutritive ratings (specifically the carbohydrate, protein, fat and vitamin content), it was possible to significantly predict participants’ subjective valuations for each item, suggesting that at least part of the variance in people’s subjective ratings pertain to the underlying nutritive content of that food. Each of the individual nutritive components for a given food was found to be represented in the lOFC (Fig 3).

Figure 3. Neural representation of subjective nutrient features in lOFC.

A. Significant encoding was found for each nutrient factor. B. Nutrient factors were not decodable above chance in mOFC. C. Searchlight revealed sub-regions of lOFC correlating with each nutrient factor. Adapted from [40].

The neural organization of hierarchical value construction

Suzuki et al. also found that while lOFC contained a representation of the individual nutritive features of a food, medial parts of OFC and adjacent mPFC did not. Instead, only subjective value signals could be decoded from the vmPFC, consistent with a large literature implicating this region in encoding the value of potential goals [2,8,43]. Though value signals were also found in lateral OFC, functional connectivity analyses found that lateral OFC areas involved in representing the nutritive components exhibited increased connectivity at the time of decision-making with value signals in vmPFC, suggesting that lOFC->vmPFC interactions may be involved in the weighted integration of sensory features to form an overall value signal.

There is evidence for a necessary role for vmPFC in attribute integration. Vaidya et al., found that vmPFC lesion patients utilized specific visual features differently when making aesthetic judgments [44]. Pelletier trained participants on arbitrary attribute-reward associations embedded in multi-attribute artificial objects and examined whether judgements about the value of those objects was impaired following vmPFC lesions. Although a vmPFC lesion did not impact judgments for single attribute-reward associations, it did impact more complex judgements based on configurations of attributes [45].

There is also evidence for a topographical organization of value within the vmPFC itself. McNamee et al. [46] measured vmPFC activity with fMRI while participants made value judgments about three different categories of goods: consumer goods, food items, and the lotteries for monetary lotteries. Category-specific representations of value were found in the vmPFC, while mid to posterior medial OFC correlated with the value of food but not other categories, and a region of anterior medial prefrontal cortex above the orbital surface correlated with the value of non-comestible consumer items. In addition, a more dorsal region of medial prefrontal cortex was found to contain a category independent representation of value for food, non-comestible goods and the value of monetary gambles [46].

We thus propose a hierarchical organization of value computation in which the representation of individual stimulus features are encoded in the lOFC, these signals are in turn integrated to generate category dependent values (as in specific to types of stimulus such as food) in the ventral mPFC, which are in turn integrated into a category independent value signal more dorsally on the medial wall (Fig. 4).

Figure 4. Hierarchical organization of value construction in prefrontal cortex.

Individual features are represented in the lateral OFC. Category-dependent value is represented in the vmPFC, where category independent value is represented in a more dorsal part of the mPFC. It is possible that category-dependent value incorporates the context-dependent value signals illustrated in Figure 2.

The hierarchical organization of value can be plausibly mapped to an even broader parts of the brain in which relevant lower level features in the earliest sensory cortical areas are transformed into higher order feature representations that ultimately find their way to the lateral prefrontal cortex and these are in turn directly converted into value signals on the medial wall of prefrontal cortex [47]. Furthermore, the amygdala would be a candidate for implementing context x feature computations, given prior reports of context-dependent coding in this area [48–51].

Conclusion

We contend that the value of a prospective outcome is actively constructed from the underlying features of that outcome. This mechanism confers on an organism the means to rapidly alter behavior following a sudden change in either internal motivation or the external context or goal, therefore lying at the core of the adaptive control of behavior. This process appears to depend on a hierarchical cortical organization extending from the earliest sensory regions to the lateral and ultimately medial prefrontal cortex. Thus, rather than being pre-ordained solely by prior associative history, value can be viewed as being actively constructed by the brain in a manner that takes into account the organism’s current motivational states, goals and external context.

Highlights:

• Value judgments for a prospective outcome are proposed to be actively constructed in the brain.

• This is argued to be accomplished by breaking down a prospective outcome into its constituent features and then integrating over those features in a weighted fashion to produce an overall value judgment.

• This confers the capability to rapidly generalize value judgments to new stimuli and to flexibly change values following a change in motivational state and/or context.

• Value judgment involves a hierarchical integration over features of different levels of abstraction.

• The lateral orbitofrontal and ventromedial prefrontal cortex play key roles in the feature integration process to compute overall value judgments for potential goals.