Computational theory-driven studies of reinforcement learning and decision-making in addiction: What have we learned?

Abstract

Computational psychiatry provides a powerful new approach for linking the behavioral manifestations of addiction to their precise cognitive and neurobiological substrates. However, this emerging area of research is still limited in important ways. While research has identified features of reinforcement learning and decision-making in substance users that differ from health, less emphasis has been placed on capturing addiction cycles/states dynamically, within-person. In addition, the focus on few behavioral variables at a time has precluded more detailed consideration of related processes and heterogeneous clinical profiles. We propose that a longitudinal and multidimensional examination of value-based processes, a type of dynamic “computational fingerprint”, will provide a more complete understanding of addiction as well as aid in developing better tailored and timed interventions.

Introduction

Reinforcement learning and decision-making—collectively, “value-based decision-making” [1]—are integral to adaptive behavior in everyday life. Value-based decision-making comprises a feedback loop whereby the values of candidate actions are learned and updated through experience, and used to guide behavior that maximizes utility (and minimizes disutility). Disruption in value-based decision-making is considered a key factor in the development and maintenance of addiction [2–4], across people with substance use disorders (SUD) [5] and laboratory animals exposed to drugs of abuse [6,7], but the specific contributing mechanisms remain unknown. Decision-making biases in addiction may be due to disruption in distinct components of learning, such as error encoding or value updating, or subjective preferences that are not readily observable in coarse behavioral performance measures. The nascent field of computational psychiatry applies formal models to understand the precise mechanisms (or “failure modes”) that give rise to pathological behavior in psychiatric conditions [8–10]. While there is no consensus on what qualifies as computational psychiatry, here we take this term to mean a mathematically rigorous understanding of the latent drivers of behavior. Findings from theory-driven computational psychiatry [11] suggest models that focus on algorithmic processes of value-based decision-making (Box 1) are well-suited to identify the specific components of reinforcement learning and decision-making that characterize SUD. This is exciting as such mechanistic research can bridge the behavioral manifestations of SUD with underlying neurobiology, providing fertile ground for cross-species translation [12–16]. Computational theoretical models thus hold promise as tools to provide additional mechanistic insight into SUD diagnosis and prognosis, and to help guide personalized treatments based on the latent variables governing individual behavior.

Box 1. Common models of reinforcement learning and decision-making in research on addiction and key model parameter definitions.

Simple Reinforcement Learning	Key estimated parameters α- Learning rate, rate at which past outcomes influence current choices
Standard learning model based on learning rate and prediction errors that are used to update action-outcome (or stimulus-outcome) associations	Q learning δt = Rt − Qt Qt+1 = Qt + α · δt
Model-Based/Model-Free Reinforcement Learning	Key estimated parameters α- Learning rate, rate at which past outcomes influence current choices ω- Weight parameter to determine relative influence of MB vs. MF
Based on learning that is updated using a balance of previous prediction error from past choices and knowledge of the task structure with the available actions (a) at each state (s), and typically tested with “2-stage” tasks.	Model-Free (MF) QMF (si,t+1, ai,t+1) = QMF (si,t, ai,t) + αi ⋅ δi,t QMF (s1,t, a1,t)= QMF (s1,t, a1,t) + α1 ⋅ λδ2,t (where λ is an eligibility trace allowing outcome at 2nd stage to influence 1st stage choice) Model-Based (MB) QMB (sA, aj) = P (sB|sA, aj)⋅max QMF (sB, a) + P (sC|sA, aj)⋅max QMF (sC, a) MB-MF balance Qnet (sA, aj)= ⍵ ⋅ QMB (sA, aj) + (1 − ⍵) ⋅ QMF (sA, aj)
Economic Choice and Valuation	Key estimated parameters κ- Discount rate, measure of attitude towards delayed rewards α- Risk tolerance, measure of attitude towards risky rewards β- Ambiguity tolerance, measure of attitude towards ambiguous rewards λ- Loss aversion, measure of avoidance of potential loss B- Sensitivity to losses and gains
Discounting. how temporal factors depreciate value when reward/gratification is delayed
Risk preference. how individual attitudes about known risk and ambiguity influence the value of choice options Loss aversion. the balance between individual gain and loss sensitivities	Hyperbolic discounting $U_{\mathit{\text{option}}}=\frac{v}{1+\kappa D}$ Expected utility theory with only risk Uoption = p ⋅ vα Expected utility theory with risk and ambiguity $U_{\mathit{\text{option}}}=\left(p-\beta \frac{A}{2}\right)\cdot v^{\alpha }$ Prospect theory Uoption = π (pi) ⋅ v(xi) Loss aversion $\lambda =\frac{\left|B_{\mathit{\text{loss}}}\right|}{B_{\mathit{\text{gain}}}}$

Here, we review recent theory-driven computational psychiatry studies of SUD primarily conducted with human subjects, highlighting the ways in which these studies have extended and refined our understanding of value-based decision-making processes in addiction. We focus on two key objectives of this work: to identify deviations from health (via case-control comparisons), and to map specific SUD symptoms and clinically-relevant states onto specific model variables—the latter aimed at moving closer to understanding the most defining yet most elusive aspect of the disorder: its dynamic, cyclical course. We conclude by outlining two directions for future research. We propose that a holistic approach that expands the typical parameter space examined within the same individual, and the duration of observation, may better serve these critical objectives and significantly enhance the clinical impact of computational psychiatry for addiction applications.

Deviation from health as indication of psychopathology: diagnostic differences between addicted and healthy individuals

SUD is a chronic, relapsing disorder characterized by repeated periods of drug craving, intoxication, bingeing, and withdrawal [17]. Drug use is maintained despite harmful consequences. The reinforcing and addictive effects of drugs center on the brain’s reward (or “valuation” [18]) circuit. At the core of this circuit lie the dopaminergic pathways originating from the midbrain (ventral tegmental area and substantia nigra) and projecting onto the striatum and prefrontal cortex (orbitofrontal and ventromedial prefrontal cortex in particular). Dopaminergic circuits are intrinsic to reinforcement learning [19,20]. Decades of work in animal models suggests excessive stimulation of these circuits by drugs of abuse leads to an over-selection of drug-related actions at the expense of other adaptive behavior [2–4,6,7]. Early functional and molecular imaging work in humans also suggested abnormalities in dopaminergic function [21–23], potentially underlying abnormal value-based processes in SUD [5,24,25], but only recently have reinforcement learning (RL) mechanisms been dissected using formal modeling approaches.

Simple reinforcement learning

Combining functional brain imaging with computational modeling of choice behavior on simple (“model-free”) RL paradigms, initial studies tested the theoretical assumption chronic drug users have deficits in value updating (Box 1), impeding learning from (non-drug) reward and punishment outcomes. Contrary to theory, this research revealed minimal differences in fitted learning rates ([26–31], cf. [32]), and mixed evidence for reduced reward prediction error encoding in dopaminergic targets [28,33,34], with many finding no differences at all [26,27,29,31], in people with SUD across drug classes (nicotine, alcohol, stimulants, opioids) compared to healthy individuals. Further, under certain conditions, some users actually showed increased learning from punishment [32] and punishment prediction errors [34], while drugs with effects on dopamine administered acutely either normalized (rather than exacerbated) deviant learning phenotypes [32] or had no measurable impact on error encoding [33,34]. These data, together with subtle differences in quantitative measures of e.g., choice “stickiness” [30,32], strategic exploration [35], decision policy [36], and “transfer” of learning signals within frontostriatal circuits [27], hint that adaptations in other subprocesses of decision-making, or in the interplay between them, are involved in SUD.

Model-based/model-free reinforcement learning

To address these questions, recent work has leveraged additional tasks/models. Compulsive behavior in addiction is long thought to arise from a shift toward habitual and away from goal-directed behavioral control [37], a hypothesis that has found empirical support in some [38,39], though not all [40–42], studies in humans using outcome devaluation tests that can arbitrate between these controllers. Computationally, habitual and goal-directed control can be mapped onto distinct mechanisms: a “model-free” system that reflexively learns action-outcome contingencies and a “model-based” system requiring knowledge of the task structure, respectively (Box 1). Research examining these competing algorithms using sequential “two-stage” decision-making tasks has found evidence consistent with an imbalance in model-based vs. model-free learning in SUD across drug classes [43–46] that emerges only with chronic use [47]. Rather than overreliance on model-free RL, however, as might be expected from a habit account of addiction, this imbalance appears to stem from reduced model-based RL [43,45]. More directly, a computational re-analysis of the data in Ersche et al. [39] on a classic devaluation paradigm indicated that the tendency toward forming habits in stimulant users cannot be explained by model-free RL processes [48] (instead, increased ‘reinforcement sensitivity’—a.k.a. inverse temperature—better accounted for users’ behavior).

Taken together, computational approaches have permitted formal testing of theories of addiction. So far, the work on RL mechanisms reviewed here, and previously for alcohol [49], does not suggest the type of abnormality found in studies using coarser measures bears out in model-derived measures, as people with SUD do not appear to have reduced learning rates or reduced prediction error signaling relative to their healthy counterparts. This work is also beginning to shed light on related theories of habit learning, further showing that while SUD is associated with an imbalance in goal-directed vs. habitual control, this imbalance may not stem from differences in iterative learning from prediction errors as in model-free RL. This raises intriguing questions about what is at the core of observed behavioral biases in addiction. One possibility is that there is a complex interaction between internal drivers of this behavior, which may be missed by focusing on a single task/parameter and timepoint.

Economic choice and valuation

In support of this idea, parallel computational neuroeconomic studies find vast differences in people’s “preferences”, e.g., for delayed and risky—probabilistic—reward on paradigms that do not entail an explicit learning component (Box 1). Increased discount rates, or the rate at which the value of delayed reward diminishes with time to its delivery, have been reliably observed across SUD [50,51], and may stem from a similar latent decision process as model-based RL [52]. However, these variables are seldom measured together in the same individual. Similarly, risk preferences interact with RL processes [53], nonlinearly scaling prediction errors, and with discounting behavior [54]. Loss aversion, the idiosyncratic sensitivity to gains vs. losses, may further modulate learning, possibly accounting for some of the known asymmetry in positive and negative reward prediction error on choice. Importantly, studies applying formal economic models to quantify these preferences in addiction have found that, in aggregate, people with SUD have differential probability weighting [55], and are more risk tolerant [56] and less loss averse [57], than healthy individuals. To capture separable dimensions of value-based processes, and to more precisely map the resultant latent factors to SUD, we propose that a multidimensional examination of decision-making will be required, as illustrated in Figure 1.

Figure 1.

Computational “fingerprinting” and dynamic characterization of addiction trajectories and transitions.

(A) The computational parameter space (a type of “computational fingerprint”) of a healthy individual showing select value-based decision-making parameters reported in the reviewed studies as being altered in addiction. The green shaded area represents a “healthy norm”.

(B) Fluctuations over time of the computational fingerprints for prognosis-based addiction classification, i.e. recovering, cycling (abstinence and relapse stages), and sustained use cases as compared to health (green).

(C) Evolution of the parameter space over time shown here for three components of the fingerprint for illustrative purposes (the full space may contain additional components). The example cases represent realistic trajectories/states: 1) healthy, shown to stay at the same multidimensional space over time; 2) sustained use, also shown to stay in the same space over time but to occupy a different one from health; 3) cycling use, shown to move away from an initial starting point and then start to return back to it. Here we also highlight at what time points tailored treatment might be most efficacious (i.e. when individuals might be most susceptible to intervention strategies) designated by the solid arrows (→); and 4) recovering, also shown to move but in a single direction approaching health. Note that here “component” could be a single estimated parameter (as shown in the 3D plot), a single estimated parameter accounting for the influence of another parameter (e.g., risk-preference adjusted learning rate), or a principal component (dimension comprised of a combination of parameters).

Initial efforts to quantify multidimensional drivers of behavior in SUD took advantage of the Iowa Gambling Task. This complex decision-making task, widely used in the SUD literature, taps into both learning mechanisms and preferences (though these are not completely separately identifiable in the task). Computationally-informed analyses revealed poor learning on the IGT (captured by reduced average choice probabilities of higher reward-yielding options) was explained by reduced loss aversion in opioid users, increased risk tolerance in stimulant users, and by both alongside increased recency bias and reduced choice consistency in marijuana users [58,59]. In addition to providing initial support for interactions between value-based processes in SUD, this research also highlights previously unappreciated heterogeneity within SUD. Broadening the space of model parameters/tasks examined in the same individual, using the type of computational fingerprint approach we advocate, could provide a more detailed assessment of drug-specific effects [60] and a clearer mapping to clinical subtypes based on individual-level biological and clinical characteristics [14,61].

Such computational fingerprinting could take the form of a factor analysis to find lower-order dimensions, or principal components, in a space of model parameters, or via joint modeling of these parameters within an individual. This “fingerprint” could also be monitored across time to capture addiction-relevant transitions as discussed below and illustrated in Figure 1B and C, though we note this will require combining dimension reduction methods with complex trajectory analyses such as multidimensional scaling latent class/growth curve modeling.

Capturing addiction dynamics: using computational models to understand within-person variability, symptom expression, prognosis, and treatment

Addiction is not static, and indeed, it can be said that understanding addiction’s longitudinal course is to understand addiction itself. The “addiction cycle” has been described as having three stages: preoccupation-anticipation, bingeing-intoxication, and withdrawal-negative affect [22,62–64]. These stages are likely associated with distinct value-based processes. Although no research to date has identified the algorithmic mechanisms that underlie the transition between each stage, initial work has advanced our understanding of the computational correlates of abstinence and withdrawal as well as features of preoccupation, namely craving. This more dynamic way of conceptualizing SUD holds great promise for realizing the clinical utility of computational psychiatry for addiction applications. In our proposed framework (Figure 1), the evolution of the computational fingerprint (multidimensional parameter space) can be used to identify critical periods when relapse vulnerability or treatment need is highest.

Abstinence and withdrawal

The most basic clinically-relevant transition is that between abstinence and use. Short-term abstinence, typically associated with aversive withdrawal states, has been associated with different RL mechanisms. Unsated vs. sated smokers were found to have reduced learning rates in the context of positive outcomes but enhanced learning from punishment [65], paralleling earlier observations of reduced prediction error encoding in striatum in this group [26]. However, others have observed more diffuse effects of nicotine abstinence [66]. Similarly, recently abstinent stimulant users, relative those with recent use, were found to have both selectively increased positive learning rates and heightened neural positive reward prediction errors [67], but reduced electrocortical signatures of both positive and negative reward prediction errors [68]. Finally, alcohol users with shorter abstinence durations exhibited more model-based/model-free imbalance [46]. Though mostly cross-sectional, these studies suggest computational measures may be used to dissect the specific mechanisms associated with abstinence/withdrawal states.

Craving

The preoccupation-anticipation stage of the addiction cycle is defined by intense subjective desire for the drug. Though it remains an open question what exactly craving “is”, its importance in the maintenance of addiction cannot be overstated. In RL studies, craving has been associated with heightened frontostriatal encoding of prediction errors in drug deprived users [26,67]. Similarly, prediction error encoding in striatum was higher in smokers told there was nicotine in a smoked cigarette prior to an RL task (vs. when told there was no nicotine, and despite both cigarettes having nicotine) [69], an effect of drug expectation mediated by changes in insular activity and subjective craving [70]. No consistent relationship has been observed between craving and economic choice [57]. Indeed, craving appears to be an independent time-varying predictor of drug reuse when assessed alongside such measures [56]. More recently, computationally-informed conceptualizations of craving itself have been proposed, in which craving is defined as a time- and attribute similarity-dependent multiplicative weight on value [71] or as a Bayesian process of hyper-precise prior estimates of interoceptive experience [72, 73]. However, key predictions of these models remain untested in SUD.

One important limitation is that almost all of the reviewed studies focus on non-drug reward. Particularly in the context of assessing craving (and arguably any theory of addiction), it is critical to test model predictions distinguishing between reward types (drug vs. non-drug) [42, 74]. Such studies may help answer questions about whether SUD is characterized by disrupted value-based processes broadly, or whether behavioral phenotypes are drug-stimulus specific and shifting across time as craving emerges.

Clinically-relevant transitions and treatment tailoring

Although addiction is defined by its longitudinal course, there has been a dearth of computationally-informed longitudinal research. At the chronic stages of SUD, the goal is to predict and hopefully prevent transitions within the addiction cycle (sustained abstinence or craving/withdrawal→drug use). People motivated to abstain, such as those initiating treatment, represent a clinically-important subgroup as well as one in which transitions are likely to occur on relatively short timescales (weeks to months). This provides an opportunity to address key questions about dynamic value-based processes. For example, such prior studies have found reduced model-based RL after detoxification predicted prospective 12-month alcohol relapse, though in combination with positive expectations about the reinforcing effects of alcohol [75].

Using a more temporally dense data collection protocol, we recently sought to identify proximal predictors of reuse events in treatment-engaged opioid users [56]. We measured two types of economic risk preferences (risk tolerance and ambiguity tolerance) repeatedly over 7 months and up to 15 times per person. We found that only ambiguity tolerance was associated with increased odds of prospective opioid use week-to-week. However, in aggregate, no significant differences in ambiguity tolerance were observed between opioid users and healthy controls, while opioid users were more risk tolerant regardless of reuse risk status. This suggests that even conceptually related value-based parameters may have distinct timecourses that convey distinct clinical information, further arguing for multidimensional assessment of this behavior.

There is strong theoretical impetus for treatments targeting value-based decision-making in addiction [76,77]. The identification and continuous monitoring of multidimensional computational fingerprints will be key for tailoring such interventions to the particular set of value-based processes at play for a given individual, at a given timepoint. Notably, elements of this proposed approach are already being tested in a new landmark study of computationally-informed behavior monitoring in SUD [12,78].

Of note, this initiative includes plans for cross-species work. This is critical as, in addition to permitting more precise investigation of neural circuits, animal models will prove particularly useful in addressing longer-timescale (developmental and lifespan) trajectories of value-based contributions to addiction that are impractical if not impossible to capture in humans. Emerging findings already support the utility of computational approaches for interrogating RL mechanisms that are differentially altered preceding [79] vs. following [16,80,81] initiation of drug self-administration in rats, though as eluded to earlier these efforts may also be bolstered by re-analysis of existing data.

Conclusion and future directions

Computational psychiatry has garnered considerable attention in recent years but enthusiasm for its presumed clinical utility is rightly tempered [82]. Here, we review the promise of this approach for addiction applications. While computationally-informed studies have produced novel explanatory insights about value-based processes in addiction that help to refine long-held theoretical accounts, we also identified two directions for future research that could significantly enhance the clinical translational potential of this approach.

First, we emphasize the importance of multidimensional assessment. Currently, multiple computational mechanisms are rarely assessed within the same individual, precluding identification of shared and distinct latent constructs underpinned by distinct neural substrates. SUD is extremely heterogeneous, including differences in the pharmacological actions of different drugs, patterns of use, symptom phenomenology, and availability of adequate treatments. A multidimensional assessment of value-based decision-making could provide the needed precision for mapping computational mechanisms to heterogeneous clinical profiles. Ultimately, this might allow for identification of person-specific combinations of model parameters underlying different disease mechanisms.

Critically, such quantifiable computational fingerprints should be examined longitudinally. The most defining feature of addiction is its cyclic course. This has not been adequately captured in prior work. Most research remains cross-sectional, emphasizing between-person differences. We propose that a “holistic”, longitudinal and multidimensional examination of value-based processes within-person, a type of dynamic computational fingerprint, will provide a more complete understanding of addiction as well as aid in the development of better tailored and timed interventions.

Highlights.

• Computational psychiatry holds promise for mechanistic discovery in addiction

• This approach captures latent factors driving behavioral differences from health

• Emerging support also for capturing variation defining addiction cycles and states

• Research needs to better account for the heterogeneous, dynamic nature of addiction

• Expanding the parameter space examined and duration of observation will be key