A Push For Examining Subjective Experience in Value-Based Decision-Making

Abstract

Subjective experience is a powerful contributor to value-based decision-making. Not every decision is the same, nor made in isolation. Rather, decision-making relies on historical information and internal states for adaptive control. Hence, it is inherently continuous with respect to time - one decision or action evolves into the next. However, forays into the neurobiological underpinnings of decision-making have too frequently ignored the contribution of such continuous subjective experience, instead tying circuit activity and brain area involvement to discrete averaged behaviors and task parameters. While much information has been gained through these investigations, recent works have demonstrated the potential for a greater understanding of neural mechanisms when the continuous, experiential nature of behavior is integrated into the investigation. Such integration has important implications for disease states with disordered decision-making such as addiction, where subjective experience is a large contributor to the disorder.

Introduction

Subjective experience is well-understood to be an important contributor in value-based decision-making. However, most investigations examining the neural basis of decision-making do not focus on the contribution of subjective experience to associated neural mechanisms. By subjective experience, we mean the individual continuous experience of temporal, contextual, consequential, associative, ethological, and internal factors during decision-making [1–6]. This continuous subjective experience evolves across time as animals interact with their environment. Of note, we are not necessarily invoking individual differences here; even mice of the same strain undergoing the same behavioral probing will show differences depending on experienced factors [7]. Animals are not automatons. Even genetically identical animals trained in precisely the same conditions will show behavioral variability due to differing interactions within that environment. At a base level, trial and error learning through subjective experience [8,9] is involved in many learning theories, including associative learning and reinforcement learning. In the broad context of conditioning or learning, it is appreciated that experience during trial-by-trial learning can lead to the recruitment of different behavioral and neural processes. For instance, ample evidence suggests that the type of schedule that is experienced during operant conditioning can bias the use of either goal-directed or habitual decision-making systems [10-12], even when other aspects of experience such as the amount of training and number of rewards are held constant [13,14]. Similarly, the relative expression of explore vs. exploit strategies has been investigated in the context of the degree of uncertainty and the amount of experience that animals have accrued across learning [15]. Thus, there is something about the experienced contingency or the patterning of experience during learning that leads to the recruitment of different behavioral and neural controllers during later decision-making.

However, the above task-constrained view of subjective experience does not consider the multitude of temporal, contextual, and internal factors concurrently and continuously experienced by individual subjects, which will differ even when exposed to the same task. How does this continuously evolving subjective experience influence ongoing decision-making and how does it affect the involved neural mechanisms? Promisingly, there is a growing call to focus research on examining the continuous nature of decision-making (e.g.,[●●3]); however, we note that it may be impossible to accurately capture the full subjective experience given the complexity of so many potentially interacting factors. Still, recent studies in animal models suggest that inclusion of at least some of the complexity that arises across a continuous experiential space can provide a deeper understanding of how brain dynamics can come to support behavior, and in particular, value-based decision-making.

Ethological works highlight the use and reliance on subjective history

The importance of experiential information has been especially investigated in naturalistic and ethological studies of continuous behavior, which have shown dependencies on subjective experience and related them to associated neural mechanisms. An examination of exploratory behavior that accounted for subjective experience found a novel role for the amygdala in mediating spatial familiarity in exploration [●16], while neurons in dorsal lateral striatum have been found to encode the identity and ordering of natural movement “syllables” [17]. A non-human primate virtual prey-pursuit study revealed that neurons in the dorsal anterior cingulate seem to represent experienced information (e.g., prey’s position, velocity, and acceleration) multiplexed with predictions about future states [18]. Studies examining birdsong have shown that vocal learning displays both local and long-term dependencies on subjective experience [19,20]. Other ethological and cognitively complex behaviors are also influenced by a diverse array of experienced information - for instance, corvids will drop shelled prey items from height in order to break the shells open. Crows are able to keep track of a diverse array of concurrent experiential information including the type of prey, the hardness of the surface, the number of times a specific item has been dropped, and the amount of kleptoparasitism [21].

These ethological findings provide evidence that animals are adapted to utilize subjective experience in guiding their behavior and decision making [22]. Even in highly structured laboratory tasks, and even when the use of experience is suboptimal or actively detrimental (e.g., in most perceptual decision-making tasks, animals should ideally attend only to the current stimulus), there is strong evidence for subjective experience contributing to behavior (e.g., [23, ●24]). Indeed, when Lak and colleagues (2020) classified reward biases in perceptual decision-making, they found that there was a principled mechanism at play in mice, rats, and humans: prior reward influences subsequent decision-making particularly when decision confidence is low. Perhaps a striking indicator of the powerful influence of subjective experience (and the unnaturality of common laboratory tasks) is how long it takes to train animals to perform many of these “simple” perceptual tasks, demonstrating that we need to carefully consider the stimuli, contexts, and tasks we use to measure variables of interest. Further, the natural world is typically volatile. Thus, even when we create stationary tasks in the laboratory, humans and other animals still seek to use their experience to adapt to potentially uncertain conditions. Indeed, even the propensity for exploration and response variability itself can be modified and reinforced based on experienced environmental contingencies [25,26].

Factors limiting the examination of subjective experience in value-based decision-making.

Thus far, the examination of how subjective experience influences continuously occurring value-based decision-making is limited. Aside from the fact that many value-based decision-making paradigms are not designed to address subjective experience or continuously occurring decision-making, prevalent current drawbacks include the reliance on discretization of behavior, and relatively coarse measurements of that discretized behavior. Researchers understandably constrain tasks to isolate specific aspects and mechanisms of decision-making. However, by focusing solely on discrete measures, we may miss the evolution of a decision and its interaction with various factors. Here, we are not making an argument to move away from such experimentally-defined behaviors and tasks, although as mentioned above, exploration into non-constrained behavior can reveal important insights into how subjective experience may be influencing behavior. Instead, we wish to highlight that the current reliance on discrete, binary variables and high-level analyses (which choice was made, was an action performed or not, etc.) is limiting and ignores any additional or analog variables that may influence the behavioral or neural measurement of interest.

First, variables such as lever press, binary choice, and reward are averaged within or even across animals/sessions, thereby removing any contribution the pattern of decision execution may have. It is also common to utilize trials or bins, and treat each incident as independent, ignoring the temporal dependency that value-based decision-making may have. Inter-trial periods are also usually treated as baseline periods, as if no relevant computations or actions could occur during inter-trial periods (but see [27,28]). Secondly, research on instrumental learning has highlighted the strength of assessing control of action-outcome contingency and action consequence over adaptive value-based decision-making, allowing one to probe which rules and what internal representations control behavior [1]. However, often missing is the continuous and interactive nature of such behavior. For example, we have little understanding of how accruing earned rewards (as value-based decision-making tasks often rely on rewards to generate behavior) across an experimental session affects perception of that reward, motivation for that reward, representation of that reward, or behavioral control by that reward. Nor is it common to examine how intervening behaviors (e.g. movement; [29]), that may seem unrelated to the main task, influence the measurement of the primary behavior or the involved neural mechanisms. In addition, for tasks where the decision and a contingent reward is inextricably linked to exploring and experiencing a vast motor space, it can be hard to disentangle the motor component from the decision process itself, if indeed there is a distinction to be made [●●3,30]. While new developments in neuroscience techniques allow for collection of ever-increasing amounts of data easing our ability to capture more information, more data does not necessarily mean better data. As nicely discussed in a prior review [31], without the application of theoretical frameworks and hypothesis testing onto such big data, our ability to garner a greater understanding in mechanisms of animal behavior from big data may be limited.

To give one final example of an understudied and tricky experiential influence on value-based decision-making, we and other animals accrue information across a temporal dimension [32-34]. Yet, we rarely examine how the passage of time may affect choices or actions. Decisions are not made in a static temporal window. Depending on the dynamic temporal relationship between the agent’s experience and its decision, aspects of a choice or action such as desirability and feasibility can play more or less influential roles in the decision process [●35]. If one rushes through consumption of a multi-course meal, dessert selection may differ compared to a more leisurely paced consumption experience. This temporal dynamic can also be affected by the agent’s internal affective state or experienced external environment, where coming to and implementing a decision more quickly can be more (or less) optimal. Furthermore, in continuous decision-making, the timing of when the decision is implemented can affect the outcome and thus the decisions that follow. While it is difficult to experimentally control for the varying influences of passage of time, more directed efforts could be made to account for the temporal aspect of subjective experience as it pertains to value-based decision-making. For example, investigators could include time as a dependent variable in existing tasks; for instance, in self-paced tasks one can examine how endogenous variation in the timing of behavior (e.g., bouts vs. distributed responding) influences both the behavior itself and its neural representation.

What we lose by neglecting subjective experience

By not considering concurrent and continuous experiential information, investigations into associated neurobiological mechanisms may be hampered. Recent works have reported widespread neural representation of action initiation in a simple visual discrimination task [36] and of task engagement in a simple Go/No-Go odor task [37], though not all the activity provides obvious functional contributions. One interpretation of similar activity levels in relation to measured behavior is that such activity reflects common representation of information or global coordination. However, it may also be that similar activity will be observed when neglecting to account for experiential, contextual, and temporal information. For example, there is activity modulation in orbitofrontal cortex (OFC) during lever pressing that can be differentiated based on its behavioral controllers [38]. Modulation of OFC activity is still apparent even when contextual information biases use of non-OFC dependent behavioral controllers. A recent report finds that neural activity related to un-instructed movements dominates cortical activity during task engagement [29], supporting the necessity to include subjective experience (movement and otherwise) to the interpretation and investigation of brain mechanisms.

Neglecting how the continuous experiential nature of value-based decision-making is represented and governed by the brain may also hamper progress toward understanding disrupted value-based decision-making in psychiatric disease. There has been growing interest in individual variability across animal models of psychiatric disease because such variability has the potential to provide a deeper understanding of involved mechanisms. While a constellation of genetic, metabolic, developmental, and social factors may drive substance use disorders (SUDs), there is also clearly an important role for subjective experience. An example that demonstrates the importance of subjective experience in value-based decision-making is in how one decides to consume alcohol. One may decide to sip a single glass of wine slowly, with drinking bouts spread more or less evenly across a couple of hours. Alternatively, one could gulp the wine, with ¾ of the drinking bouts more closely spaced within a relatively short time period, and the remainder spread across several hours. Although the same amount of alcohol is consumed, the latter pattern is likely to produce a steeper rise in blood alcohol concentration (BAC) and a different intoxication experience. Further, any decision-making during and after this consumption experience is going to occur in the context of, and in relation to, this BAC curve. Intake patterns have been shown to differentially affect associated neural function. For example, relative to extended access to cocaine self-administration, intermittent access to cocaine led to increased dopamine uptake, release, and behavioral sensitization whereas extended access induced tolerance in rats [39]. In addition to shorter-term pattern influences, research in those suffering from SUDs suggest longer-term experiential factors such as withdrawal experience can differentially alter subsequent decision-making [40]. Animal models of alcohol self-administration behaviors have also shown consumption patterns depend upon prior experience. For example, with experience, long-term heavy drinking and relapsing non-human primates reduced the number of drinking bouts yet increased the amount of alcohol they drank within a bout, essentially “gulping” to rapidly increase BACs [●●41]. Mice allowed to self-administer alcohol during acute, but not protracted withdrawal, showed negative reinforcement, and increased alcohol self-administration on subsequent days [42]. These and similar findings have led to an increased appreciation for the powerful role drug taking patterns may play in SUDs [●43,44].

However, efforts to understand neurobiological mechanisms responsible for increased drug seeking and consumption often rely on self-administration models where seeking is quantified as the number of lever presses made, and consumption is the total consumed or administered over that session [45]. Hence when the above wine glass scenario occurs in alcohol self-administration studies (i.e., intense bouts vs. evenly distributed responding), researchers are often left observing similar overall seeking and consumption behaviors, and may miss relevant information about experiential interactions, such as the pattern of consumption, that can influence decision-making and introduce variability to the measured responses. For example, studies that have investigated experiential factors and the corresponding changes in involved circuits found novel mechanisms protecting against increased drug use (e.g., [46]). Thus, these examples from SUD research highlight the importance of accounting for subjective experiences into assessments of behavior and treatment in psychiatric disease.

Conclusion

Here we have argued that an increased focus on the contribution of subjective experience will help to elucidate the behavioral and neural mechanisms at play in value-based decision-making. Contrary to most laboratory tasks, natural decision-making typically occurs in a fully open world, and may evolve and interact over a continuous and richly varied contextual and temporal space. It therefore seems likely that the brain has adapted for this more open, continuous type of value-based decision-making. By taking into account the diverse aspects of experience - above and beyond mere reward or binary choice - there is tremendous potential to reveal fundamental mechanisms of decision-making. Here we have highlighted just a few of these potential benefits, including the ability to better understand what behavioral factors are driving decision-making, how those same experiential factors drive neural activity (perhaps revealing nuanced differences in seemingly distributed representations), and important mechanisms such as the patterning of intake that contribute to development and persistence of SUDs. This approach would also allow for testing the generalizability of existing theoretical accounts of value-based decision-making. This needed insight could pave the way for advances in understanding disease-induced disruption to value-based decision-making and further treatment potential.

Highlights.

• -Animals use varied and not necessarily task-dependent experiential information to guide their decisions and/or actions.

• -Incorporating naturalistic and continuous approaches to evaluating subjective experience has the potential to elucidate ethologically-relevant novel behavioral and neural contributions to value-based decision-making.

• -Substance use disorders are strongly impacted by subjective experience, highlighting the necessity of investigating how continuous experiential information contributes to value-based decision-making and its disruption.