Computational models of behavioral addictions: state of the art and future directions

Abstract

Non-pharmacological behavioral addictions, such as pathological gambling, videogaming, social networking, or internet use, are becoming major public health concerns. It is not yet clear how behavioral addictions could share many major neurobiological and behavioral characteristics with substance use disorders, despite the absence of direct pharmacological influences. A deeper understanding of the neurocognitive mechanisms of addictive behavior is needed, and computational modeling could be one promising approach to explain intricately entwined cognitive and neural dynamics. This review describes computational models of addiction based on reinforcement learning algorithms, Bayesian inference, and biophysical neural simulations. We discuss whether computational frameworks originally conceived to explain maladaptive behavior in substance use disorders can be effectively extended to non-substance-related behavioral addictions. Moreover, we introduce recent studies on behavioral addictions that exemplify the possibility of such extension and propose future directions.

Introduction

Psychobiological and neurocomputational investigations in addictive disorders have largely focused on the effects of substances of abuse on neural dynamics, cognitive processes and behavior (cf. reviews: Everitt & Robbins, 2016; Koob & Volkow, 2016; Mollick & Kober, 2020; Redish, Jensen, & Johnson, 2008; Smith, Taylor, & Bilek, 2021). However, recent studies strongly suggest that non-pharmacological behavioral addictions share with substance use disorders key neurobiological (Antons, Brand, & Potenza, 2020; Potenza, 2013), computational (Lindstrom et al., 2021; Ognibene, Fiore, & Gu, 2019; Redish, Jensen, Johnson, & Kurth-Nelson, 2007; Shimomura, Kato, & Morita, 2021), and behavioral features (Grant & Chamberlain, 2014; Grant, Potenza, Weinstein, & Gorelick, 2010). These include widely accepted behavioral addictions such as pathological gambling (el-Guebaly, Mudry, Zohar, Tavares, & Potenza, 2012), as well as others on which the consensus is still forming, such as videogaming (Petry & O'Brien, 2013; Yao, Potenza, & Zhang, 2017), social network or internet addiction (Jorgenson, Hsiao, & Yen, 2016; Veisani, Jalilian, & Mohamadian, 2020), compulsive buying (Granero et al., 2016; Grant et al., 2010), compulsive sexual behavior or pornography addiction (Griffiths, 2016; Love, Laier, Brand, Hatch, & Hajela, 2015) and finally, more controversial, disordered eating behaviors such as binge eating (Wiss, Avena, & Gold, 2020; Wiss, Avena, & Rada, 2018; Wiss, Criscitelli, Gold, & Avena, 2017).

In this review, we cast a wide net relying on an inclusive definition of addictions: a relapsing, chronic disorder characterized by an initial pursuit of a desired outcome that leads to the inflexible repetition of maladaptive behaviors, despite the harmful consequences (Everitt & Robbins, 2016; Koob & Volkow, 2016). This definition highlights two complementary elements of behavioral and cognitive control in addictions. First, it emphasizes a transition from reinforcing action-outcome associations to compulsive stimulus-responses, i.e., from goal-oriented to habitual behavior (Ersche et al., 2016; Everitt & Robbins, 2013; Volkow & Morales, 2015). In other words, an ‘urge’ to respond to a reinforced cue is triggered irrespective of an actual desire for the outcome (cf. 'need' vs 'want', Berridge & Robinson, 2016) or any assessment about desired future environment or body states (cf. 'model-free control', Dolan & Dayan, 2013). Second, the chronic and relapsing elements of the definition assign an important role to an underperforming goal-oriented behavior and forward planning (or 'model-based control', cf. Dolan & Dayan, 2013), possibly due to an incomplete, incorrect, or otherwise impaired belief structure or internal model of both environment and body states. For instance, incorrect representations of future positive and negative interoceptive outcomes can lead to craving (Grimm, Hope, Wise, & Shaham, 2001; Gu & Filbey, 2017), often followed by the reinstatement of the addictive behavior (relapse), even after prolonged periods of abstinence.

Here we consider computational models of addiction based on reinforcement learning algorithms, Bayesian inference and biophysical neural simulations, with a focus on ‘model-free’ and ‘model-based’ aberrant control. We discuss whether computational models originally conceived to describe substance use disorders could be validly extended to behavioral addictions and we present computational models that have been specifically developed to describe maladaptive behaviors in behavioral addictions.

Reinforcement learning models

Reinforcement learning (RL, Sutton & Barto, 1998) is the dominant approach for modeling addictive behaviors (CPSYMAP, Kato, Kunisato, Katahira, Okimura, & Yamashita, 2020). In RL, a behavioral policy determines one’s actions at each state, resulting in state transitions that can yield positive or negative outcomes, so allowing an agent (e.g., a person or an animal) to pursue total reward maximization with temporal discounting. In model-based RL, agents use an internal model of the environment (i.e., representations of transitions and rewards) to estimate the values of behavioral policies and plan a course of action-state transitions. By contrast, in model-free RL, agents estimate immediately available state/action values, typically through updating them by using reward prediction errors (RPEs, Watkins & Dayan, 1992). As a result, model-based RL is computationally costly yet flexible to changes in the environment, once these are represented in the internal model, whereas model-free RL is computationally parsimonious yet characterized by low flexibility (Strehl, Li, & Littman, 2009). Within this perspective, addictions are described as a dysfunction of value-based behavior that affects both model-free and model-based control modalities (Redish et al., 2008).

Based on the understanding that substances of abuse interfere with dopamine signals, which have been suggested to encode RPEs in biological agents (Schultz, Dayan, & Montague, 1997; see section: Neural models), a seminal model (Redish, 2004) proposed that drugs of abuse could act as fictitious RPEs. Due to their exogenous cause, these RPEs cannot be canceled out by reward predictions, so that the estimated value of drug indefinitely increases. Other models have proposed that enhanced RPEs may cause a decrement in the reward system sensitivity (Dezfouli et al., 2009), resistance to habituation (Di Chiara, 1999), or sensitization to reinforced cues (Bernheim & Rangel, 2004), and might accumulate through hierarchical decision-making processes (Keramati & Gutkin, 2013). Any of these dysfunctions would promote overwhelming biases towards the model-free control, driving the compulsive and inflexible selection of addictive behaviors, irrespective of the negative outcomes (e.g., detrimental effects for one’s health or social relations). These models assume that over-reliance on model-free control is caused by dopamine-related, drug-induced, alterations in healthy neural circuit dynamics (Koob & Volkow, 2016; Korpi et al., 2015; Luscher & Malenka, 2011). However, aberrant functioning in the neural regions involved in reward processing have been also described in association with compulsive use of pornography (Hilton, 2013), compulsive sexual behavior (Gola et al., 2017; Golec, Draps, Stark, Pluta, & Gola, 2021), compulsive buying behavior (Granero et al., 2016), eating disorders (Baik, 2013; G. K. W. Frank, Shott, Stoddard, Swindle, & Pryor, 2021; Wiss et al., 2018), problematic videogaming (Palaus, Marron, Viejo-Sobera, & Redolar-Ripoll, 2017), and internet addiction (Love et al., 2015). These findings indicate that aberrant RPE signals putatively responsible for the over-reliance on model-free, at the expenses of model-based, control can be elicited in the absence of pharmacological manipulation, e.g., due to predisposing factors (Antons et al., 2020), suggesting the computational models based on these mechanisms can be used to describe behavioral addictions, as well. For instance, a bias towards model-free control has been reported in binge eating disorder using a task designed to highlight model-free vs model-based arbitration (Voon et al., 2015). In another RL model tackling social network use, participants showed high sensitivity to social rewards and reliance on RPE updates to determine their post sharing policies (Lindstrom et al., 2021). However, formal testing with computational modeling is still very sparce across behavioral addictions.

Interestingly, RL models focusing only on aberrant model-free control underperform when trying to account for those behaviors in addiction that are not cue-induced, e.g., because they are novel, complex, or context-dependent. To fill this gap, several models have proposed to include addiction-related dysfunctions in the model-based control component, so focusing on the generation, update, and recall of state-action-state transitions. Crucially, although dopamine is suggested to be also involved in model-based control (Deserno et al., 2015; Wunderlich, Smittenaar, & Dolan, 2012), aberrant RPEs are not directly considered in the computational models focusing on the dysfunctions of model-based control, rendering irrelevant the issue of whether behavioral addictions can develop in the absence of the drug-based manipulation of the brain reward system. Model-based control dysfunctions fall into a few categories: forward planning malfunctions (Redish & Johnson, 2007), incomplete representations in the internal model (Redish et al., 2007), or incomplete access to the internal model during recall (Simon & Daw, 2012) . In other words, this new class of RL models focuses on one’s internal representation of the environment, showing that incomplete or incorrect representations of state transitions (or mental forward explorations of these transitions) can drive addiction-like suboptimal goal selections, planning and ultimately behaviors. This approach, which changes the focus of investigation from the generation of a habitual response to an impaired ability to plan and select goals, was used in several models to account for behavioral addictions.

One study (Redish et al., 2007) simulated state misrepresentation in pathological gambling. While gambling, one can experience big wins and subsequent losses, developing a (mis)recognition that there is a state associated with wins and a different state associated with losses, thus misrepresenting the same state as two different states. Then, negative RPEs caused by losses would not attenuate a large positive value of the state associated with wins, and this misassignment of credit due to the discrepancy between the actual environment and its internal representation could lead to pathological gambling and relapse. Another proposal addresses the issue of forward planning malfunctions by focusing on the relation between environment complexity and cognitive resources available to the agent (Fiore, Ognibene, Adinoff, & Gu, 2018; Ognibene et al., 2019). In this case, a mismatch between resources available and those required results in repetitive suboptimal behavioral policies, reducing the sampling of contingencies in the environment and escalating the exploration cost across phenotypes characterized by different model-based and model-free control balance. Thus, addiction-like behaviors can emerge in agents with bounded model-based resources, also inducing an inadequate representation of the environment and irrespective of RPE malfunctions. Another study (Shimomura et al., 2021) developed a model that relied on the "successor representation" (SR) of states (Dayan, 1993), a process suggested to be used by humans (Momennejad et al., 2017; Russek, Momennejad, Botvinick, Gershman, & Daw, 2017). In SR encoding for a given policy, states are similar if they give access to similar sets of states, e.g., two doors leading to the same reward room would have similar SR representations. Formally, a state is represented by a matrix of expected cumulative discounted future state occupancies under a certain policy. The matrix that describes the relationship among states enables partially model-based behavior through model-free RL-like RPE-based value update. Shimomura et al. (2021) proposed that: 1) through a long-standing reward-obtaining behavior, one potentially establishes “dimension-reduced SR”, and (2) the reduced SR can become rigid. Under such a rigid and dimension-reduced SR, a sustained large positive RPE is generated at the state with reward due to the inaccurate value approximations caused by inadequate state representations, irrespective of any pharmacological manipulation, potentially enhancing reward-obtaining behavior. Moreover, negative outcomes occurring after the addictive positive reward cannot induce changes in behaviors, under the rigid reduced SR.

All these models (Ognibene et al., 2019; Redish et al., 2007; Shimomura et al., 2021) are theoretically applicable across behavioral addictions, irrespective of the type of reward. Potentially related to the environment exploration and representation dysfunctions, studies have indicated that a key factor in the development of behavioral addictions such as problematic gambling, videogaming, shopping, or social network use can be found in the complex (i.e., difficult to compute and predict) organization of rewards experienced on a variable ratio reinforcement schedule (Cash, Rae, Steel, & Winkler, 2012; Greenberg, Zhai, Hoff, Krishnan-Sarin, & Potenza, 2022; Young & Abreu, 2011). Another study has shown that gamblers are characterized by reduction in directed (uncertainty-based) exploration and not in random exploration compared with healthy controls (Wiehler, Chakroun, & Peters, 2021). Finally, a deficit in the exploration-exploitation balance has been also suggested for binge-eating disorder (Reiter, Heinze, Schlagenhauf, & Deserno, 2017). However, the mechanisms proposed in these models have not yet been directly tested in ad hoc experiments.

Bayesian and active inference models

Computational models based on Bayesian inference suggest that the brain computes probability distributions associating states, actions and events or outcomes (whether value-based or not). These probability distributions, termed as prior beliefs or priors, are updated into posterior beliefs or posteriors, relying on a signal of precision in prediction error, i.e., the dopamine-encoded discrepancy between one’s priors and actual state-action-outcome observations (Friston et al., 2012). This is a relatively new approach in comparison with RL, with a comparably smaller literature in terms of models of addictive behaviors. Current analyses carried out in relation with substance use disorders have highlighted slower belief updating and related behavioral adaptation, as in perseverative habitual responses (Ide, Hu, Zhang, Yu, & Li, 2015) and reduced ability to use environment representations to guide choice selections, as in over-reliance on model-free control (Harle, Zhang, et al., 2015). Other studies have highlighted the relation between neural responses evoked by non-value based prediction errors and likelihood to relapse (Harle, Stewart, et al., 2015; Harle, Yu, & Paulus, 2019), suggesting that the aberrant RPEs described within the RL framework might be part of a more generalized dysfunction across all prediction errors. One further study (Schwöbel, Markovic, Smolka, & Kiebel, 2021) has proposed that context inference may play a key role in substance use disoders, in a mechanism analogous to context-based RL (Redish et al., 2007). An advantage that Bayesian inference models have on RL algorithms is that they estimate the computational processes underlying belief updates, irrespective of rewards, therefore allowing their seamless use across addictive behaviors. One example is provided by a recent investigation into instrumental learning in bulimia nervosa (Berner et al., in press), in which a Bayesian observer model highlighted slow belief updates and associated behavioral rigidity, consistent with similar investigations in substance use disorders (Ide et al., 2015). Further investigations using Bayesian inference models are required to reveal whether these mechanisms can be found across behavioral addictions and highlight shared computational mechanisms with substance use disorders.

Among the theories based on Bayesian inference, active inference (Friston, 2013; Friston et al., 2015) has emerged to describe behaviors as the result of the minimization of dopamine-encoded prediction error (Friston et al., 2012). This theory has been successful in accounting for a wide range of physiological and behavioral phenomena, including substance use disorders (Smith, Taylor, et al., 2021). Similar to the described effects of over-reinforcement of addiction-related cues, resulting in an over-reliance on model-free control, active inference explains compulsive behavior in addiction in terms of excessive prediction error signaling. This in turn, results in excessive belief confidence (precision), characterized as narrow distributions for the priors (Kinley, Amlung, & Becker, 2022). Such distributions make it more likely to repeat the choice associated with the addictive behaviors (Miller, Kiverstein, & Rietveld, 2020; Schwartenbeck et al., 2015) and at the same time prevent further updates, e.g., to include negative outcomes or interoceptive signals, contributing to craving (Gu, 2018; Gu & Filbey, 2017). Consistent with this hypothesis, increased belief updating in association with drug related positive values, and a reduced sensitivity to negative outcomes has been described across substance use disorders (Smith, Kirlic, Stewart, Touthang, Kuplicki, Khalsa, et al., 2021; Smith et al., 2020). This hypothesis has not been formally tested in behavioral addictions, yet. However, the already discussed ubiquitous presence of both reward processing dysfunctions and aberrant rewards in association with behavioral addictions (Baik, 2013; G. K. W. Frank et al., 2021; Gola et al., 2017; Golec et al., 2021; Granero et al., 2016; Hilton, 2013; Love et al., 2015; Palaus et al., 2017; Wiss et al., 2018) once again suggests that a process triggered by excessive prediction error signaling will be found also across compulsive behaviors associated with behavioral addictions.

Active inference also describes model-based dysfunctions as affecting the formation or recall of the structure of priors characterizing an internal model. Different models have investigated aberrant forward planning in terms of low confidence in future outcomes (Schwartenbeck et al., 2015), reduced precision in the state transition matrix (Fradkin, Adams, Parr, Roiser, & Huppert, 2020), reduced confidence in the generated internal model of the environment (Smith, Kirlic, Stewart, Touthang, Kuplicki, Khalsa, et al., 2021), or a reduced ability to generate deep policies (Mirza, Adams, Parr, & Friston, 2019). These deficits in turn generate the belief that events projected in the future, when computable, are characterized by uncertainty and unpredictability, and this bias seems to remain stable in the long term (Smith, Kirlic, Stewart, Touthang, Kuplicki, McDermott, et al., 2021). Therefore, immediate, precise, and easy to compute (and to predict) rewards are, once again, preferred, in a process analogous to the dysfunctions associated with model-based control for the RL framework. We previously discussed that a common feature across several behavioral addictions is the presence of environments characterized by complex reward schedules that are difficult to compute (Cash et al., 2012; Greenberg et al., 2022; Young & Abreu, 2011). The active inference framework generalizes this principle beyond the need to focus on rewards, entailing that behavioral addictions can emerge in the intersection between model-based dysfunctions and any sufficiently complex environment (e.g., due to ramified or variable state-action-outcome contingencies). This would include behavioral addictions characterized by complex reward schedules (such as gambling or videogaming), as well as others characterized by difficult to compute and variable health related, economic, or social outcomes (e.g., social network use, eating disorders, compulsive buying, compulsive sexual behavior, or pornography addiction).

Neural models

Both RL and active inference perspectives rely on dopamine signals to trigger behavioral plasticity. The RL paradigm interprets dopamine burst firings as encoding RPEs, responsible for value-based updates (Montague, Dayan, & Sejnowski, 1996; Schultz et al., 1997; Watabe-Uchida, Eshel, & Uchida, 2017). Instead, the active inference approach postulates that dopamine signals represent precision in event distribution predictions and trigger the update of beliefs (Friston et al., 2016; Friston et al., 2015). Whether in terms of rewards or precision, the ubiquity and robustness of the neurocomputational mechanisms underlying dopamine signals (Fiore, Dolan, Strausfeld, & Hirth, 2015) has led to the early belief that dopamine release would be significantly affected only by extreme events, such as pharmacological manipulations. Indeed, substances of abuse interfere with dopaminergic signals, albeit neither homogenously nor linearly (Nutt, Lingford-Hughes, Erritzoe, & Stokes, 2015), triggering significant and long-lasting synaptic alterations across several brain regions (Korpi et al., 2015; Luscher & Malenka, 2011). However, as mentioned above, several investigations have now revealed that a similar, prediction error-based (cf. Shimomura et al., 2021), multifaceted role is played by dopamine in behavioral addictions (Antons et al., 2020; Baik, 2013), including pathological gambling (Clark, Boileau, & Zack, 2019; Potenza, 2013) , videogaming (Liu et al., 2017; Palaus et al., 2017; Weinstein, 2010), compulsive sexual behavior (Kraus, Voon, & Potenza, 2016; Voon et al., 2014), compulsive use of pornography (Gola et al., 2017; Hilton, 2013), and binge eating (Bello & Hajnal, 2010; Volkow et al., 2002; Wang et al., 2011). These investigations further suggest that the neural plasticity triggered by drug-induced dopamine signals is comparable with the same dopamine-mediated process triggered by the consumption of palatable food, variable reward schedules characterizing gambling or videogaming, use of pornography and so forth.

A key target of dopamine-mediated neuroplasticity are the cortico-striatal synapses (Everitt & Robbins, 2016; Koob & Volkow, 2016; Luscher, Robbins, & Everitt, 2020). In biophysical neural models simulating cortico-striatal circuit dynamics, mesolimbic dopamine bursts trigger cortico-striatal long-term synaptic potentiation (Montague et al., 1996; Nelson & Kreitzer, 2014; Redgrave, Prescott, & Gurney, 1999). These alterations in turn bias future choice selections, favoring the repetition of the stimulus-response combination that led to dopamine signals, resulting in instrumental conditioning, thus providing the neural mechanisms underlying the described increased reliance on model-free control (Barto, 1995; M. J. Frank, Seeberger, & O'Reilly R, 2004; Gurney, Prescott, & Redgrave, 2001a, 2001b). Furthermore, the presence of multiple parallel cortico-striatal circuits characterized by different functions, but similar architectures (Haber, 2016; Jahanshahi, Obeso, Rothwell, & Obeso, 2015; Obeso, Rodriguez-Oroz, Stamelou, Bhatia, & Burn, 2014) led to the hypothesis that dopamine signals could affect the neural dynamics of multiple circuits at the same time (Fiore et al., 2018). In particular, the neural dynamics of dorsal (sensorimotor selections) and ventral (value processing and goal selections) cortico-striatal circuits are usually associated with model-free and model-based control (Dolan & Dayan, 2013; O'Doherty, Cockburn, & Pauli, 2017). Therefore, in the dorsal circuit dopamine signals are assumed to bias sensorimotor selections and model-free behavior. Conversely, in the prefrontal circuit, these signals are hypothesized to bias the selections of goals or future values, affecting forward planning and goal selection plasticity, or the model-based control system (Fiore et al., 2018).

In terms of neural circuit transient dynamics (Durstewitz, Huys, & Koppe, 2021) cortico-striatal long-term potentiation triggered by mesolimbic dopamine signals results in increased circuit stability (Fiore et al., 2018). Neural models indicated that drug-induced mesocortical dopamine signals have a similar effect on prefrontal cortico-cortical connectivity and dynamics (Lapish, Balaguer-Ballester, Seamans, Phillips, & Durstewitz, 2015), further deteriorating state-transition flexibility, and strengthening attractor-like dynamics. Although the effects of mesocortical dopamine release in substance use disorders are multifaceted (Ceceli, Bradberry, & Goldstein, 2022), attractor-like dynamics in the prefrontal cortex are consistent with the reported rigid representation of future state-action values (cf. 'incentive salience', Ceceli et al., 2022) and interoceptive states (Gu et al., 2015; Naqvi & Bechara, 2009), both key elements in the phenomena of craving and relapse. As discussed for the mesolimbic dopaminergic signals, these phenomena associated with mesocortical dopamine-induced alterations are not restricted to pharmacological manipulations, since behavioral addictions are based on the same dopamine dynamics (Antons et al., 2020; Baik, 2013). Further studies into the specific neural mechanisms underlying behavioral addictions will be needed to confirm or disprove this hypothesized similarity.

Conclusions and future directions

The objective of computational psychiatry (Huys, Maia, & Frank, 2016; Montague, Dolan, Friston, & Dayan, 2012) is to develop neurocomputational measures of disease- and subject-specific neural and cognitive mechanisms underlying decision-making, with the ultimate goal to inform precision diagnosis and treatment. Models relying on RL algorithms, Bayesian inference and neural dynamics that focus on vulnerabilities related to model-free and model-based control can explain the emergence and rigidity of maladaptive choices, despite the adverse consequences. Although most of the models here discussed were developed to describe substance use disorders, they can explain hallmark features of a wide range of behavioral addictions, with RL and active inference models more suitable to investigate (aberrant) structures of rewards and beliefs, respectively, and neural models dedicated to the investigation of (aberrant) attractor dynamics in neural activity. A key challenge in relation with behavioral addictions is to determine which of these compulsive behaviors qualifies as a legitimate form of addiction. We propose that the multifaceted neurocomputational representations of substance use disorders here described across modelling frameworks can be used as a benchmark to formally define a cluster of alterations that characterizes addictions. Those behavioral addictions found to meet these criteria -e.g., behavioral rigidity due to over-reliance on model-free control, impaired forward planning due to inadequate representations in model-based control, or increased stability in transient neural dynamics- should be included as a form of addiction. Some data suggested that behavioral and brain activity measures estimated relying on computational models can outperform traditional clinical measures in predicting clinical status, likelihood of relapse or vulnerability in substance use disorders (e.g., see: Harle, Stewart, et al., 2015; Yu et al., 2020). Thus, we expect the discussed computational models could provide a guide for behavioral addiction classification, in the near future.

Finally, it has been suggested (Heilig, Epstein, Nader, & Shaham, 2016; van den Ende et al., 2022) that these models have so far neglected the complex interplay of social (e.g., peer influence or isolation, societal stigma or tolerance etc.) and psychobiological factors. As many behavioral addictions are clearly affected by social interactions, it will be crucial for future investigations to include these social components (cf. Frolichs, Rosenblau, & Korn, 2022), using a new generation of tasks and neuro-computational models.