A Salience Misattribution Model for Addictive-Like Behaviors

Abstract

Adapting to the changing environment is a key component of optimal decision-making. Internal-models that accurately represent and selectively update from behaviorally relevant/salient stimuli may facilitate adaptive behaviors. Anterior cingulate cortex (ACC) and dopaminergic systems may produce these adaptive internal-models through selective updates from behaviorally relevant stimuli. Dysfunction of ACC and dopaminergic systems could therefore produce misaligned internal-models where updates are disproportionate to the salience of the cues. An aspect of addictive-like behaviors is reduced adaptation and, ACC and dopaminergic systems typically exhibit dysfunction in drug-dependents. We argue that ACC and dopaminergic dysfunction in dependents may produce misaligned internal-models such that drug-related stimuli are misattributed with a higher salience compared to non-drug related stimuli. Hence, drug-related rewarding stimuli generate over-weighted updates to the internal-model, while negative feedback and non-drug related rewarding stimuli generate down-weighted updates. This misaligned internal-model may therefore incorrectly reinforce maladaptive drug-related behaviors. We use the proposed framework to discuss ways behavior may be made more adaptive and how the framework may be supported or falsified experimentally.

1. Introduction

The Greek philosopher Heraclitus is famous for the quote “one cannot step in the same river twice” (Sedley, 2003). This captures an essential property of our world in that it is non-stationary where at no time again will the weather, social relationships, resources and other environmental factors ever be exactly the same again. Consequently, a critical element of survival and therefore optimal decision-making, is the ability to adapt to ever-changing environments (Bindra, 1976; Darwin, 1859). An enabler of this are generative internal models or representations of our environments which integrate incoming sensory input, including feedback, with past experience to dynamically update and enrich our representations of the changing environmental contingences (Friston, 2010; Kersten et al., 2004; Knill and Pouget, 2004). These generative and dynamic internal models are encoded in the brain where even the most abstract cognitive processes are thought to be represented as neurophysiological states (Posner et al., 1988). Hence, an important part of brain function is to facilitate and update these internal models such that they represent environmental contingences that are most relevant and salient in optimizing future decisions.

Internal models are defined here as an agent’s (animal or simulation) representation of environmental contingencies or state-space which allows them to predict future events based on past observations and adapt behaviors accordingly. The notion that an agent forms abstract cognitive task-space or internal models was posited by Tolman (1948). This notion has been used to investigate fundamental aspects of learning and decision-making (Cochran and Cisler, 2019; Gershman and Niv, 2012; Redish et al., 2007). One component of internal models are cue-outcome relationships. Here, the strength of an associative relationship between cue and outcome is contingent upon the predictive value of the cue, which may change over time and therefore the internal model needs to be updated accordingly. In addition to cue-outcome relationships, internal models will also need to be updated when there is a detected change in environmental contingences without a given cue (e.g. when reward outcome probabilities change). Aberrant updating of environmental contingencies may produce misaligned internal models that drive maladaptive behaviors, reflected in the failure to adapt to the changing contingences. At any given moment, we are faced with a great abundance of information which is continuously changing. Therefore, the brain must filter and update this information as weighted by its relevance/salience and predictive values such that future behavior is guided by accurate and adaptive internal models. Hence, uncovering the neurophysiological basis of internal model updates appear critical towards our understanding of the adaptive decision-making processes.

In the present paper, we review evidence for the involvement of anterior cingulate cortex (ACC) and dopamine in selectively updating from relevant environmental stimuli such that the changing environmental contingences are accurately represented. We then review evidence for dysfunction in the ACC and dopaminergic systems that may produce maladaptive internal models, driving maladaptive decision-making. We review this ACC and dopaminergic dysfunction found in those with a drug-dependence. Accordingly, we propose that misaligned internal models, generated as a result of ACC and dopaminergic dysfunction, may be drivers of addictive-like behaviors¹ where there may be a reduced ability to adapt. Specifically, we argue that these maladaptive internal models misattribute an abnormally high salience to drug-related cues relative to non-drug-related cues, which generates over-weighted updates from any drug related rewarding outcomes while downweighing any updates from non-drug related rewarding outcomes and negative feedback signals. These maladaptive internal models may then drive addictive-like behaviors where there is a strong persistence in drug-related behaviors. Overall, we hope to use this salience misattribution framework to clarify how fundamental brain processes and computations can drive some aspects of addictive-like decision-making deficits. We also aim to explore ways to quantitively influence and modulate these processes to reduce aspects of maladaptive decision-making which may be found in those with a drug dependence.

2. Processes involved in the enrichment of internal models

According to the predictive-coding hypothesis of the brain, inaccurate internal models produce a prediction-error or surprise signal, and the brain tries to minimize this by having accurate predictions about the future state of the environment (Rao and Ballard, 1999). To reduce prediction errors the brain must dynamically update and enrich its internal model such that there are fewer violations of the predicted environmental state. It is important to consider that not all surprising observations necessarily indicate a change in the state of the environment (e.g. unreliable cues) and a surprise is not necessary to generate an internal model update (e.g. when entering a new environment, knowing a priori that the previous internal model is not relevant here) (Dayan, 2012). Hence, following an observation that elicits a surprise, it can be maladaptive to update when the observation is not predictive of a change in the state of the environment, likewise it can be maladaptive to fail to update when it does. The dissociable roles of surprising observations (i.e. surprises that elicits an update vs surprises that do not) are captured mathematically in Information Theory (Ghahramani, 2006; Shannon, 1948). Shannon information (I_S) captures the magnitude of entropy or the pure unexpectedness of an observation given the current internal model. In contrast, the Kullback-Leibler divergence (D_KL) captures an update of the internal model given an observation, and importantly, this is when there is a net information gain. This is determined by calculating the difference in the prior and posterior belief distributions, where if there is a difference between the two, there is an information gain and the D_KL is greater than 0, which indicates an update to the internal model. Critically, the observer needs to dissociate between observations and surprises that require an update of the internal model and those that do not, which is what ultimately allows for enriched internal models. These enriched internal models then guide optimal decision-making, which are based on updating from environmental stimuli with a high predictive value and accurately represented environmental contingencies. Dysfunction of this system may therefore produce maladaptive addictive-like behaviors, driven by misrepresented internal models of the environmental contingencies.

3. The role of anterior cingulate cortex in internal model update

O’Reilly et al. (2013) experimentally provided evidence for the involvement of ACC in selectively updating from cues with a high predictive-value. In their study, participants performed a saccadic eye-movement task while their brain activity was recorded using functional magnetic resonance imaging (fMRI). Participants were tasked to fixate on a dot that would appear in a different location during each trial. The experiment had two distinct surprise trials; one that required an update of the internal model, as it was predictive of where future dots would appear (termed “update” trials) and second, surprises that did not require an update due to its unpredictability on the location of the future dots (termed “one-off” trials). Both these cues were signaled by different colors so that participants knew which dot required an update and which did not. The key finding was that the ACC was activated during surprises that required an update, and this was associated with a greater D_KL of the difference between the prior and posterior distributions – which is a measure of net information gain (or an update). This was not the case when there was low D_KL, but high I_S (a measure of surprise, but not necessarily an update). This is suggesting that ACC is not just encoding surprise but more specifically, an internal model update from cues with high predictive values. Further support for this comes from Behrens et al., (2007) who investigated the weight given to observations in stable and volatile environments that had changing reward probabilities. Here, the authors reported greater blood oxygenation level-dependent (BOLD) ACC activity when the participant’s estimated volatility was high - when more weight was given to observations due to their greater predictive value in their task. The ACC’s responsiveness to volatility in this Behrens et al., (2007) task is consistent with it contributing to the adaptive updating of internal models based on stimuli with high predictive values.

For a neurophysiological account, Karlsson et al. (2012) recorded from neural ensembles in the medial PFC of rats that also encompassed the ACC. Karlsson and colleagues (2012) found abrupt and coordinated changes in the activity of the medial PFC neural ensembles when there was a detected change in the environment – a mismatch between predicted state and actual state (a prediction-error). In the Karlsson et al. task, the reward-outcome probabilities switched without a cue; when this was detected by the rats, they resampled the previously preferred choice, which was temporally correlated with the changes in the neural ensembles’ dynamics. These neuronal changes may be interpreted as there being a prediction-error signal following the detected change in the reward-outcome contingencies, which is associated with the rats having greater uncertainty in their current internal model. This is likely to generate an update to the internal model, as reflected in the change in activity of the medial PFC neural ensembles. Similar recordings in the rat medial PFC were also made by Powell and Redish (2016). In the Powell & Redish (2016) task, when there was a forced change in the reward-outcome contingencies, there were changes observed in the neural dynamics of the medial PFC which encoded a change in the representation of task-contingences. Critically, this medial PFC’s neural dynamic activity occurred after the animals detected the change in reward contingences, and before they adapted their behavior. A possible interpretation here may be that the medial PFC is encoding an update to the task-contingencies when there is a detected change in the environment. This updated internal model may then be used to drive future behavioral adaptation. Given that a reduced ability to adapt is a component of addictive-like behaviors, ACC dysfunction during the update processes would suggest misaligned updates to the internal model, which may then be used to drive the maladaptive addictive-like behaviors.

Another critical component of enriching internal models is utilizing and updating from negative feedback or negative prediction-errors to optimize future behavior (Walsh and Anderson, 2012). Negative prediction errors result from the actual state of the environment being worse than predicted and there is evidence to suggest that ACC plays an active role in updating from these negative prediction errors. Ruchsow et al. (2002) demonstrated that greater error-related negativity (ERN) signals are generated particularly when participants were presented with negative feedback, based on their errors in a card sorting task. These ERN signals are generated as negative deflections in the electroencephalograms (EEG) waveforms and source localization analysis revealed these signals to be generated in the ACC (Holroyd and Coles, 2002). Further studies utilizing EEG’s high temporal resolution in humans have also provided support for this negative feedback signal to be generated and processed in the ACC (Bellebaum and Daum, 2008; Gruendler et al., 2011). Therefore, the ACC may be involved in updating negative feedback, when the actual state of the environment is worse than expected and the internal model needs to be updated to optimize future behavior. In addition to this, Jocham et al. (2009) found that in a probabilistic reversal learning task, there was greater BOLD ACC activity for reversal errors, which generated negative feedback signals. They also found that in an environment where more weight is placed on negative feedback and where behavior is adjusted in fewer trials (with increased learning rates), there was greater ACC activity when contrasted with conditions that placed lower weight on negative feedback and required fewer behavioral adjustments. Critically, ACC’s greater activity during negative feedback was correlated with greater behavioral adaptations (switch to the option with higher reward probability) during their task. Additionally, ACC activity increased following each preceding error until there was a behavioral adaptation. This finding may initially seem counterintuitive as repeated errors should be more expected, and therefore produce a smaller negative prediction error signal. However, this could be interpreted as ACC activity updating and encoding not just the negative prediction-error but also the value of taking an action (which is dependent on both learning rate and prediction error magnitudes). Collectively, there is evidence to suggest that ACC may be integrating negative feedback as well as action-values over several trials, through continual updating of the internal model, until the behavior is adapted to suit the changing environment. ACC dysfunction may therefore lead to maladaptive/reduced updates from negative feedback, producing internal models which may drastically discount or dismiss the negative consequences of a decision. This misaligned internal model may then possibly be used to produce maladaptive addictive-like behaviors.

In addition to the ACC actively updating from observations with high predictive values and negative feedback to enrich the internal model, it might also encode updates from positive reward prediction errors, which generate positive feedback signals (Hayden et al., 2011). This occurs when the outcome is better than expected, hence, it is critical for the internal model to update and reinforce behaviors that maximize rewards. Collectively, there is evidence to suggest that ACC encodes unsigned prediction-errors where activity is increased during both positive and negative prediction errors, as discussed by Hayden et al. (2011) and Hyman et al. (2017). Additionally, Redish et al. (2007) suggested (using simulations) that both relief (lack of expected negative outcome) and disappointment (lack of expected positive outcome) are useful times to update internal models. Given these updates form enriched internal models, it is likely that ACC is also involved in facilitating updates from these disappointment and relief processes. Overall, ACC plays an integral role in enriching internal models though updates of information that is most relevant for optimizing behavior. This information includes cues that have high predictive values, as well as positive and negative feedback signals. As a result, adaptive internal models are produced, based on the changing environmental contingences. ACC dysfunction during update processes would therefore produce misaligned internal models that may drive maladaptive addictive-like behaviors.

4. The role of dopaminergic systems in internal model updating

To gain a more fundamental understanding of the mechanisms driving these internal model updates, it is critical to also account for the dopaminergic system. It is in the dopaminergic neurons of the ventral tegmental area (VTA) where the neural substrate of signed prediction-errors were experimentally demonstrated in the seminal paper by Schultz et al. (1997). Here, Schultz et al. (1997) demonstrated that positive reward prediction-errors are signaled by increases in VTA neuronal activity and conversely, negative reward prediction-errors by VTA decreases. More recently, a study by Steinberg et al. (2013) used optogenetics in rats to demonstrate that the VTA dopaminergic activity plays an integral role in forming enriched cue-outcome internal models through prediction-error signals. Steinberg et al. (2013) experimentally demonstrated that stimulating dopaminergic neurons can counteract the phenomenon of ‘blocking’ (as predicted in Redish (2004) and Waelti et al. (2001)). Blocking is where another stimulus (‘B’) is paired and simultaneously presented with the already conditioned stimulus (‘A’) where now stimuli ‘AB’ predicts a reward. After this pairing, when the animal is only presented the stimulus ‘B’, there is no expectation formed for a reward, as the associative relationship for ‘B => reward’ is blocked. But the ‘A => reward’ relationship is still intact, which suggests that no new learning took place as a result of ‘AB’ => reward pairing. This is theoretically explained by the Rescorla-Wagner model (Rescorla and Wagner, 1972) where learning is driven by prediction-errors and because A was reliably predicting a reward, there would be no prediction-error signals generated during ‘AB’ => reward pairing and thus, no new learning association between B and reward was formed. Given stimulation of VTA can counteract this ‘blocking’ phenomenon, the animal will expect a reward from stimulus B when it otherwise wouldn’t. This suggests that VTA dopaminergic neurons can be modulated to selectively form associative relationships, and potentially even enrich internal models when they go awry, through manipulating these prediction-error or update learning signals. Therefore, one consequence of dopaminergic dysfunction may be that misaligned internal models are produced, which may produce addictive-like maladaptive behaviors.

Interestingly, there is also evidence to suggest that the human VTA and substantia nigra complex (VTA/SN), which is rich in dopamine, plays an active role in selectively updating from surprises that have a greater predictive value, as is also the case for ACC. In this study, Nour et al. (2018) used fMRI and positron emissions technology (PET) to investigate the relationship between dopamine and BOLD fMRI activity while participants performed a behavioral task. The task involved making inferences on which of the two cues (presented simultaneously) predicted the outcome, and this changed after every few trials so that participants had to regularly update their internal model of the task-related contingencies. The Nour et al. task had two dissociable surprise trial types; one which required an update (from the informative cues that had high predictive values) and another which did not require an update (from the uninformative cues with low predictive values). Participants were not explicitly informed on the validity of the relevant cues or reversal probabilities, therefore this had to be learnt through training on the task. Nour et al. found that greater D_KL (net information gain) was associated with informative surprises, calculated based on differences in participants’ prior and posterior belief distributions (belief of which cue predicted outcome), where a difference between the two distributions indicates an update of the internal model. This result makes sense given that it is adaptive to update from informative surprises and not from the uninformative surprises. Critically, however these increased updates and D_KL were associated with greater BOLD VTA/SN activity when contrasted with I_S which is a measure of surprise given a prior belief, and does not necessarily implicate an update (this finding replicated a previous study’s finding that used the same task (Schwartenbeck et al., 2016)). The VTA/SN activity is suggestive of selectively encoding updates and not just surprise, hence, dissociatively enriching the internal model based on the most behaviorally relevant cues. To directly link dopamine’s involvement, they also found, using PET, that greater baseline VTA/SN dopamine D2/3 receptor availability was negatively correlated with VTA/SN activity effect size that encoded internal model updates from the D_KL > I_S contrast. Dopamine D2/3 receptors are auto-receptors which act to inhibit phasic dopamine release (Grace, 2000). Thus, less D2/3 receptor availability predicts less inhibition and therefore more phasic dopamine firing during the BOLD VTA/SN activity. Therefore, greater BOLD VTA/SN activity during encoding updates is suggestive of greater dopamine release during the internal model updates. This implication of greater dopamine release based on greater BOLD VTA/SN activity and reduced D2/3 receptor availability is also supported by studies showing that human D3 receptor antagonists increase BOLD VTA/SN activity and D3 receptor knockout mice have greater extracellular dopamine levels. Similarly, BOLD midbrain activity in rats is correlated with dopamine neuronal firing, collectively suggesting lower D2/3R availability is implicated in higher dopaminergic neuronal activity (Ferenczi et al., 2016; Koeltzow et al., 1998; Murphy et al., 2017). Overall these findings suggest a possible role of human VTA/SN dopaminergic neurons in dissociatively updating from surprises that are informative (i.e. have a greater predictive value) and not uninformative surprises. Hence, a balanced functioning of this system appears critical to form enriched internal models and imbalance to the system may produce addictive-like maladaptive behaviors.

Subregions of the striatum, that are also dopamine rich, have also been implicated in internal model updates and adaptive behaviors in humans and rodents (Daw et al., 2011; Diederen et al., 2016; Howard and Kahnt, 2018; Takahashi et al., 2016). Gershman and Uchida (2019) sought to unify the role of dopamine as playing a central role in encoding and updating of Bayesian beliefs. Their hypothesis linked levels of uncertainties on the environmental states, with values and actions, and dopaminergic neuronal activity in a reinforcement learning framework. They argued that prediction-errors drive learning, updates and therefore future behaviors.

5. Conclusion on ACC and dopaminergic system’s role in enriching internal models

Overall, it appears that both the ACC and the dopaminergic systems play a critical role in enriching internal models through dissociative updates from observations that have a greater predictive value, allowing for an accurate representation of environmental contingencies. The selective updates allow for adaptive behavior based on the dynamic environmental contingencies. Further, dopaminergic activity also enables adaptive stimuli-outcome associations to be formed via prediction-error learning and update signals such that future behavior is optimized in maximizing future rewards. ACC is also critically involved in updating from negative feedback signals that are generated when the actual outcome is worse than expected, until the suboptimal behavior is adapted to suit the changing environmental contingencies. Collectively, these systems generate internal model updates based on the most relevant environmental stimuli which is then used to drive future behaviors (Figure 1). The critical message is therefore that dysfunction and abnormal activity in these brain systems may generate internal models which are not selectively updated from the most behaviorally relevant stimuli or are updated from non-informative surprises. Hence, misaligned internal models may be what drives maladaptive addictive-like decision making, based on updates that do not fully integrate these relevant environmental stimuli.

Figure 1. Updating from reliable cues that have high predictive values.

Reliable cues have a greater predictive value, hence, ACC and VTA are activated which may facilitate a dissociative internal model update from the reliable cue. This updated internal model may then the future decision, optimized based on what the reliable cue is predicting about the future state of the environment. Following the decision, there is a feedback signal which also updates the internal model. Green activation maps in the ACC and VTA depicts typical activity.

Addiction is a disorder that often includes abnormal functioning of the ACC and the dopaminergic system, both of which are critical in updating selectively from behaviorally relevant environmental stimuli. Hence, these processes may actively be involved in explaining a possible aspect of addictive-like behaviors; the reduced ability to adapt to the changing environmental contingencies. Considering this, the following section will discuss the specific consequences of aberrations in these brain processes and how these may produce aberrant internal models that may drive aspects of addictive-like behaviors.

6. Over-updating from drug-predicting cues in addiction

Formerly, the prominent view to explain the strong persistence of drug-abuse and drug-seeking behavior of people with a dependence was attributed to an increase in dopamine in the brain’s reward circuitries, caused by the drug administration, which then enhanced the motivation towards the continued use of the drug. While the initial evidence to support this model was compelling, subsequent evidence suggested otherwise. The seminal paper by Volkow et al. (1997) experimentally demonstrated that, relative to non-dependent individuals, there was actually a smaller dopamine release as a result of drug administration in cocaine-dependent individuals (this was further supported by Martinez et al. (2007)). Instead, the greater levels of dopamine release was found to be in response to the conditioned stimulus (drug predicting cues) and not the drug itself (Phillips et al., 2003; Volkow et al., 2006). As pointed out by Redish (2004), these dopamine release findings fits with findings from Schultz and colleagues’ (1997) implicating dopamine firing in encoding the reward-predictors (i.e. conditioned stimuli) and with Di Chiara’s (1999) hypothesis that addiction is a disorder of associative learning with abnormally strong drug-related cue associations. Therefore, these dopamine release findings also directly relate to the role of dopaminergic systems in updating and enriching internal models – driving subsequent decision-making via forming cue-outcome associations. Further, Redish (2004) demonstrated with simulations, that one factor that could explain part of the dependent individual’s persistence to drug taking is the non-compensable dopamine release in response to the drug administration, continually causes a positive prediction-error learning or update signal and so the actions and the cues involved with taking the drug are continuously and strongly reinforced. Taken together, these seminal findings strongly suggest the role of dopamine in forming strong associative relationships between the drug-predictors and rewards in those with addictions, by facilitating selective updates from these drug-related cues.

In addition to the dopaminergic system in addiction being hyperresponsive to drug-predicting cues, there is also hyperactivity in the ACC in response to these cues, relative to non-addicted individuals (Kalivas and Volkow, 2005). Given ACC’s critical role in updating from behaviorally relevant stimuli, hyperactivity here suggests that there may be selective updates from drug-related cues and not from other potentially behaviorally relevant cues. Interestingly, ACC activity is greater when people with a dependence to cocaine are subjected to cocaine-related cues compared to other primary rewarding stimuli (e.g. sex drive) (Garavan et al., 2000). This suggests that there is an abnormally high salience and reward predictive value associated with drug-predicting cues in those with a dependence and the associated maladaptive updates in the internal model may further drive the drug-seeking behavior due to its misattribution as being the main reward predictors. Additionally, it demonstrates that this drug cue association, which is formed as a result of maladaptive updates, is powerful enough to override even survival critical primary rewards. More recent evidence demonstrates that this hyperactive ACC during drug-related cue presentations is predictive of a higher chance of relapse in those with a cocaine dependence (Li et al., 2015). Similarly, this ACC drug-cue hyperactivity in those with a nicotine dependence is predictive of a higher severity of dependence (McClernon et al., 2008; Smolka et al., 2006). Collectively, this suggests that the stronger the drug-related cue reward associations, the more difficult it may be to abstain from the drug, with ACC hyperactivity having a critical role in attributing high salience and associations to such cues.

Overall, these findings in the addiction literature strongly relate to the brain processes involved in updating from cues with the greatest predictive value (i.e. the conditioned stimuli). To someone with a drug dependence, this conditioned stimuli may be anything that is spatially or temporally related to the action of taking the drug (Di Chiara, 1999). One interpretation to bring together the increased activity of dopaminergic systems and ACC during presentation of drug cues, is by considering the critical role of these brain processes in updating from information in our environments that has the greatest predictive value. These updates then aid in adapting and optimizing future behavior. However, in the addicted brain, hyperactivity during drug-predicting cues relates to overly high salience and reward predictive value (relative to other information in the environment including primary rewards). This therefore generates an aberrant internal model that is very strongly biased towards updates from drug-related cues (misattributed as the main reward-predictors), while a much lower salience is placed in other reward predicting cues unrelated to drugs. We propose that this misaligned internal model may be one driver of strongly persistent drug-seeking behaviors.

This aberrant internal model may also generate maladaptive predictions based on attributing a very low salience on non-drug related rewarding stimuli and overly high salience on drug-related stimuli. According to the biased competition theory of attention (Desimone, 1996; Desimone and Duncan, 1995; Duncan et al., 1997), stimuli that are most salient biases the individual’s attention towards those stimuli. As a consequence, highly weighted prediction-errors are generated due to higher precision (i.e., heightened by attention) when predictions on these salient drug-related stimuli are violated (Desimone, 1998). Since in addiction there is an extremely high salience attributed to drug-related stimuli and extremely low salience to non-drug related rewarding stimuli, any positive prediction error from drug related stimuli will be highly weighted, therefore generating a much stronger update under this maladaptive internal model in addiction. This may partially explain why dependent individuals find it so difficult to avoid drugs and be reinforced by non-drug related rewards. Under this maladaptive internal model, very low weight is placed in non-drug related positive reward prediction-errors, leading to very little reinforcement of non-drug related behaviors (Figure 2). Critically, this framework not only suggests an aberrant internal model based on maladaptive updates in addiction but also maladaptive prediction-error signals that strongly bias learning and updating from drug-predictive cues. We propose this as one potential factor in the continual persistence of drug-seeking behaviors.

Figure 2. Aberrantly selective reward updates in addiction.

The drug-related cues such as the matches and cigarettes as in this case, are misattributed as having an abnormally high reward predictive value and salience compared to other reward predictive cues such as barbells predicting exercise and better health outcomes as in this case. These drug-related cues increase ACC and VTA activity which generates an internal model update that drives the decision to smoke the cigarettes. This decision is positively reinforced through positive feedback signals, also associated with increased ACC and VTA activity. Non-drug related reward predictors are associated with hypoactive ACC and VTA and thus do not cause considerable updates and gets down-weighted positive reinforcements from the positive feedback. R = reward. Red brain activity = hyperactive, Blue = hypoactive.

7. Under-updating from errors and negative feedback in addiction

Besides a strong persistence in drug seeking behavior, there are also cognitive decision-making deficits found in those with addictions. For example, impulsive choices that come with an immediate reward are favored, with much steeper discounting rates for temporally delayed rewards (Bickel et al., 2014; Crews and Boettiger, 2009; Jentsch and Taylor, 1999; Kirby et al., 1999; Verdejo-Garcia et al., 2018). This has partially been attributed to reduced cognitive control (among other factors; Madden and Bickel, 2010) that would otherwise enable the individual to monitor their behaviors such as learning from errors and engaging in more adaptive behaviors (Botvinick et al., 2001; Ridderinkhof et al., 2004). Several studies have reported a reduced error awareness in people with cocaine, cannabis and nicotine dependence (Hester et al., 2009; Luijten et al., 2011), as well as an inability to adapt future behaviors that rely on learning from these errors and punishments (Carey et al., 2015; Duehlmeyer et al., 2018; Duehlmeyer and Hester, 2019; Franken et al., 2007; Hester et al., 2007). Attenuated ACC responses during error processing in those with a dependence is thought to underly these behaviors. For example, Hester et al. (2009) reported that individuals with a dependence to cannabis have an impaired error-awareness in a go/no-go task, based on reduced accuracy in determining when they made an error, compared to controls. Moreover, this was correlated with ACC hypoactivation in dependent individuals, but not in non-dependent controls. More recently, Carey et al. (2015) conducted a paired associative learning task in cannabis users who had to learn numbers associated with spatial locations and were subsequently tested on this task twice, once before and once afterward feedback. The authors reported a significantly poorer error-correction rate in the cannabis group compared to controls that was associated with ACC attenuation, particularly when the negative feedback was presented. Because ACC function is critical for updating from errors and negative feedback, hypoactivity suggests a failure to update the internal model based on negative feedback signals. Subsequently this is reflected as the inability to adapt future responses based on errors.

In contrast, Goldstein et al. (2010) found that in cocaine users the administration of methylphenidate reduced errors in a drug-related words version of the Stroop-task. Moreover, greater ACC activity was predictive of a greater reduction in errors and therefore decreased response impulsivity. Hester et al. (2012) also showed similar findings where methylphenidate administration improved error-awareness in non-dependents, also associated with increased ACC activation for errors made with verses without awareness. Although Goldstein et al. (2010) and Hester et al. (2012) demonstrate the role of ACC in adapting future behaviors based on errors, it is important to note that methylphenidate does not selectively increase ACC activity. Hence, it cannot directly be concluded from these experiments whether ACC’s increased activity is the cause or the consequence of the reported behavioral effects. In any case, there is strong evidence to suggest that ACC hypoactivity contributes to reduced error-monitoring and error-dependent adaptation in those with a dependence. Therefore, restoring ACC activity may improve these processes – by updating the internal model to also account for the negative feedback from errors.

Reductions in error processing by nicotine smokers when subjected to smoking-related stimuli are reflected in an attenuation of the error-related negativity and error-positivity. Both these event-related potentials (ERPs) are components of error-processing recorded with EEG (Luijten et al., 2011), which have been shown to be generated in the ACC (Bellebaum and Daum, 2008; Gruendler et al., 2011; Miltner et al., 1997). This suggests that not only is there a dysfunction in error awareness but exposure to drug-related stimuli is likely to further reduce error-awareness, possibly reflected in ACC hypoactivity. This experiment also showed that smokers exhibit an attenuated post-error slowing of reaction time, which normally aids in correcting behavior after an error (Laming, 1968). Collectively, these results can be interpreted under our proposed framework, which hypothesizes that in addiction maladaptive internal models misattribute more salience to drug-related cues such that other non-drug, yet task relevant cues, have a reduced salience. This means that there is reduced weight in any prediction-error signals, or negative feedback, related to non-drug stimuli, hence leading to inappropriate updating and thereby maladaptive future behaviors that fail to take these errors into account.

In sum, these collective findings can be interpreted under the proposed framework of maladaptive internal models in addiction where a failure to update in light of negative feedback is reflected in a hypoactive ACC. Further, this maladaptive internal model with downweighed updates from errors may explain a failure to optimize future behaviors. With fewer updates from errors and a reduced awareness of these errors, lesser salience is placed in negative feedback signals. Thus, this maladaptive internal model reduces the ability to adequately monitor and compute error related-performance which may therefore partially reduce cognitive control and inhibition over maladaptive decisions. This may also partially explain the strong preference for impulsive decisions where the positive value of immediate outcomes erroneously outweighs negative future consequences, which may either be dismissed or drastically discounted. Hence, maladaptive high rewarding behaviors may instead be overly reinforced.

8. Addiction and maladaptive internal models driving maladaptive decision-making

The ACC and the dopaminergic system are critical for updating internal models that enable adaptive future behavior. This is accomplished by forming strong associative relationships with the cues that have the greatest predictive values for positive future outcomes. We constantly face a great abundance of ever-changing information, hence it is critical that the brain selectively updates from the most relevant cues, so that associative relationships are strengthened in proportion to the salience of the cue. Ultimately, these updates construct and fine tune our internal models which are used to drive future decisions, with respect to what is most relevant to these internal models. If these updates are biased in being less sensitive to negative consequences and more sensitive to cues that reliably predict drug administration, the internal model is then also biased, attributing greater relevance to drug-predictors while downweighing negative consequences. Hence, with respect to this internal model, it is “adaptive” to continuously update from drug-related cues as these are misattributed with the greatest predictive values and are most reliably predictive of future rewards (Figure 3). In those with a dependence, it is in response to these positive cues where we observe hyperactivity in the ACC and dopaminergic systems, whereas ACC hypoactivity is seen in response to negative consequences. These indicate that the internal models of those with a dependence are maladaptive due to an unbalanced salience attribution that overweights and downweights positive and negative feedback, respectively (Figure 4). These saliency misattributions bias model updating that drive the strong persistence in drug-seeking behavior as well as impulsive decisions.

Figure 3. Misattribution of rewards and updates leading to drug-related behaviors in addiction vs non-addiction.

Both the dependent and non-dependent individuals process reward predicting cues in a similar process, however, the critical difference is that in the internal model of the dependent individual, the cue that has the highest reward predictive value is drug related, which is what selectively causes internal model updates and further positive feedbacks, attributed to the ACC and VTA activity. This drives the continual cycle of drug-related behaviors. In contrast, similar is true for non-dependent individual with non-drug related reward predictor (barbell). The bottom panel are probability distributions with the probability of reward given either the non-drug related positive cue (barbell) or the drug related cue (matches and cigarettes). The left panel is a non-dependent individual associating more reward with the barbell, in contrast, the right panel depicts an individual with a dependence attributed more reward to drug-related cues than other positive cues, which also have a lower precision. This indicates smaller updates and positive feedbacks from non-drug related positive cues. Green activation maps in the ACC and VTA depicts typical activity, relative to the individual.

Figure 4. Unbalanced internal model updates from negative and positive feedback signals in addiction.

Drug-related cues (matches and cigarettes) are associated with hyperactive ACC and VTA due to their highly attributed reward predictive values, which drives updates to the internal model, leading to the drug-related behaviors. These behaviors’ negative consequences/feedback (financial loss) are severely downweighted, attributed to a hypoactive ACC. The positive consequences (reward from drug) are much highly weighted, as attributed to hyperactivity in ACC and VTA, which further drives updates and positive feedbacks for drug-seeking behaviors. Red brain activity = hyperactive, Blue = hypoactive.

9. Other Models of Addiction

9.1. Incentive Sensitization

The incentive sensitization theory of addiction proposes that the neural circuits normally mediating salience attribution to stimuli are hyper-sensitized towards drugs and drug-related cues following the continual use of drugs (Berridge and Robinson, 2016; Robinson and Berridge, 2008, 1993). Sensitization processes are proposed to occur via neural adaptations within the dopaminergic mesolimbic pathway. These processes include physical alterations in the shape and number of tiny spines on dendrites of the neurons within the mesolimbic pathway; which consequently renders the pathway hyper-reactive to drugs and drug related cues (Robinson and Kolb, 2004; Singer et al., 2009; Steketee and Kalivas, 2011). The incentive sensitization theory was one of the first to propose that drug addiction is not entirely a disorder of pleasure or hedonics (i.e. “liking”), which may actually decrease with continual drug use. But instead addiction is a disorder of the motivational “wanting” which increases with continual drug-use where dopaminergic neural circuits attributing motivational salience to stimuli are hyper-sensitized towards the drug-related cues and behaviors.

9.2. Impaired response inhibition and salience attribution (iRISA) theory

The impaired response inhibition and salience attribution (iRISA) theory emphasizes the critical role of the PFC and impairments within the PFC-related cognitive processes that contribute to addictive behaviors (Goldstein and Volkow, 2011, 2002). Critically, the theory proposes that impairments in PFC functioning significantly contributes to addictive behaviors such that there is high salience attributed to drug-related cues compared to non-drug related cues, a decreased ability to inhibit maladaptive drug-related behaviors and a decreased ability to be reinforced by non-drug related cues/reinforcers. A central component of this theory is discussing these impairments within a cyclic process of four clinical symptoms/states of drug addiction including; intoxication, binging, withdrawal and craving. A key concept proposed by the iRISA theory is that in a healthy state, there is greater functioning of the ventral and dorsal PFC regions towards inhibiting the drug-related functions (i.e. incentive salience, motivational “wanting”, attentional bias) of these same ventral and dorsal PFC. However, in the craving and withdrawal states inhibition towards drug-related functions are downregulated and consequently, the drug-related functional state is upregulated. In the intoxication and bingeing state, the inhibition towards these drug-related functions are downregulated even further and therefore the drug-related functional state of the ventral and dorsal PFC predominates and there is an increase in drug-related behaviors and functions. Overall, a critical proposal of the iRISA theory is that addictive behaviors manifest in impaired cognitive inhibition and in attributing high salience to drugs and drug-related cues/contexts – largely owing to PFC dysfunction.

9.3. Reward Deficiency Syndrome (RDS)

The RDS model characterizes addiction and other disorders which also manifest in high levels of impulsivity and compulsivity, as a biogenic disease with aberrations in the dopaminergic reward circuitry (Blum et al., 2012, 2000, 1996). One aberration is proposed in those carrying the A1 allele of the dopamine receptor D2 (DRD2) gene, which is associated with a reduced DA receptor concentration and reduced responses to rewards. This A1 allele in the DRD2 gene was found to be a strong predictor (74.4%) of a compulsive/impulsive disorder (including alcoholism, pathological gambling, smoking and cocaine dependence and, several others including also attention-deficit hyperactive disorder) (Blum et al., 1996). These behavioral disorders with high levels of impulsivity and compulsivity were collectively termed as RDS (Blum et al., 2000, 1996). Overall, the RDS theory highlights the importance of dopaminergic receptors and genetic traits which may predispose individuals to addiction-like symptoms and produce aberrations in reward processing within the dopaminergic reward neural pathways.

9.4. Dual process models of addiction

At the core of the dual-process models of addiction is an imbalance between the two separate but interacting systems; the habitual/impulsive system and the cognitive control/reflective system; with the former eventually predominating such that drug-related behaviors are drastically increased. The dual-process model put forth by Bechara (2005) suggests that the amygdala is a critical component of the habitual/impulsive system and the vmPFC a component of the reflective system. A core idea from Bechara (2005) is that those vulnerable to developing addictions are less able to inhibit the impulsive system and consequently, the smaller-sooner reward (i.e. drugs) are over-valued. Bottom-up signaling, through neurotransmitter release (i.e. dopamine, serotonin), was proposed to influence the cortical system such that the top-down reflective system is “hijacked” and its capacity to inhibit the impulsive system is weakened. Consequently, attentional bias towards drug-related cues and rewards are increased.

Another dual-process model (Everitt and Robbins, 2016, 2013, 2005) proposes that drugs act as instrumental reinforcers and therefore, learnt behaviors that lead to the drug (instrumental responses) are strongly reinforced and strong stimulus-response associations are formed. Instrumental responses are thought to be involuntary and habitual, and therefore are less sensitive to consequences of actions or to the devaluation of the reinforcer (i.e. drugs) (see Table 1 in Everitt and Robbins, 2013). Similar to Bechara (2005), Everitt and Robbins (2005) proposed that impairments in PFC related executive functioning contributes to a balance shift towards subcortical regions predominating where habitual drug-related behaviors are increased. However, Everitt and Robbins (2005) go on to propose that the transition may be explained by the Pavlovian Instrument Transfer effect where stimuli associated with taking drugs act as strong conditioned stimuli and elicit the automatic drug-related instrumental response, which is not sensitive to devaluation of the drug; explaining the strong persistence in drug-related behaviors in those with a drug dependence.

A separate dual-process model focuses specifically on implicit cognition as a critical driver of drug-related behaviors (Stacy and Wiers, 2010; Wiers and Stacy, 2006). The model proposes that several learnt associations, memories or implicit attitudes (collectively implicit cognitions) are spontaneously activated and influences the decision-output of an individual with a dependence towards drug-related behaviors; especially if there is weaker cognitive control over these implicit cognitions. Therefore, a prediction by this model is that people with a weaker cognitive control will have implicit cognitions as a stronger predictor of an increase in of drug-related behaviors; this prediction was supported by experimental data in humans (Grenard et al., 2008; Houben et al., 2009; Thush et al., 2008).

Two recent dual-process models integrate and add to the previous models. These models include the tripartite model (Wei et al., 2017) and the interaction of person-affect-cognition-execution (I-PACE) model (Brand et al., 2016). The models were developed for internet-use (Brand et al., 2016) and internet gaming (Wei et al., 2017) addictions, though the I-PACE model was recently updated and generalized to also include other addictions such as gambling, gaming and buying-shopping addictions (Brand et al., 2019). The tripartite model by Wei et al., (2017) model integrates a third interoceptive insular system with the impulsive (striatum and amygdala) and reflective systems (PFC). The general idea proposed is that increased interoceptive insula activity further increases addictive-like behaviors by upregulating the subcortical impulsive system and downregulating the PFC reflective system; this is done by maintaining a high craving state.

In contrast, the I-PACE model by Brand et al., (2019) aimed to integrate several factors that contribute to addictive behaviors, starting with genetic, environmental and personal features (e.g. coping styles, childhood experiences and temperamental features) as potential vulnerabilities to developing addictive behaviors. The general idea proposed is that in early stages of addiction, feelings of gratification are sought. In the later stages, gratification decrease due to an increase in negative consequences (e.g. loneliness, conflicts and feelings of emptiness). Therefore, the addictive behavior now persists to compensate for this lack of gratification and the negative consequences.

9.5. A unified framework for addiction: vulnerabilities in the decision process

The key idea in the framework proposed by Redish et al. (2008) is that addiction arises from vulnerabilities or “failure-modes” within the decision-making systems of the brain. Vulnerabilities are found within three separate decision-making systems; the planning system which is flexible and able to deliberate the consequences of a decision-output. The second is the habit system which is computationally less expensive to execute but is inflexible and depends on the learnt automatic situation-action responses. The third is a Pavlovian action system that learns to release pre-wired actions in response to stimuli and situations. All three systems require an accurate classification of a situation in a given environment and decisions are executed based on this classification. Critically, Redish et al. (2008) propose that addictive behavioral symptoms such as impulsivity, incentive sensitization and overvaluations of actions can be attributed to a maladaptation within one or more of these systems (see Table 4 in Redish et al. (2008) for a full list and see Walters and Redish (2018) for the updated version). Overall, the framework brought together mechanisms of normal learning and decision-making processes to link them with other key theories of addiction. This was to propose that each addiction theory targets separate vulnerabilities within the decision-making systems and each of these vulnerabilities may give rise to addictive behaviors.

10. Salience misattribution model relative to the previous models of addiction

The previous models of addiction have in common the concept or principal that some aspects of addictive behaviors arise (at least partly) because of dysfunction in the dopaminergic and PFC systems. They argue that because of these dysfunctions, there is 1) high salience in drug-related cues, 2) weaker cognitive control, 3) reduced processing of negative consequences (i.e. less sensitivity to devaluation of the drug) and, 4) reduced ability to be reinforced by non-drug related reinforcers. In contrast to the previous models, the proposed salience misattribution model specifically predicts the role of ACC and dopamine dysfunctions as they relate to updating internal models and produce (mal)adaptive behaviors. The concept that addictive behaviors arise from dysfunctions in dopamine and PFC, including ACC, has been proposed by previous models of addiction. However, the critical difference is grounded in the mechanism through which the maladaptive addictive-like behaviors emerge. We propose one possible computational mechanism specifically to do with updates to the internal model. Dysfunction to updating processes produces a misaligned internal model and this internal model may be used to give rise to the maladaptive addictive-like behaviors. Therefore, while we are consistent with other models that there is a greater salience attributed to drug-related cues, we suggest that one possible mechanism of this high salience attribution may be though overweighted updates to the internal model due to an increased ACC and dopaminergic activity. Second, we are also consistent with previous models proposing a reduced cognitive control and reduced sensitivity from negative outcomes. However, we propose a mechanism whereby reduced updating of the internal model is caused by reduced activity of the ACC during these processes. Lastly, our model is also consistent with previous models in a reduced ability to be reinforced by non-drug related reinforcers. However, we propose that one mechanism or process of this may be through downweighted updates to the internal model from non-drug related reinforcers due to a reduced ACC and dopaminergic activity. It is important to emphasize that our model is not in any way, the first model to propose that ACC and dopaminergic systems are dysfunctional in addiction, leading to addictive-like maladaptive behaviors. However, it is (to the best of our knowledge) novel in its attempt to link maladaptive updating of the internal model to ACC and dopaminergic dysfunctions, in producing misaligned internal models and addictive-like behaviors. We propose that maladaptive internal model updates may be one (of many) possible mechanism that cause maladaptive decision-making in people with a dependence. The aberrant internal model updating is not an exclusive alteration in addiction and it may well synergistically interact with the other mechanisms proposed by previous models of addiction. For example, maladaptive updates may be one-way incentive sensitization (Robinson and Berridge, 1993) towards drug-related cues may be enhanced. Further, maladaptive updates may also produce aberrations in the situation-classification decision-making system (Redish et al., 2008). One advantage of the proposed mechanism is grounded in the quantifiable treatment implications (see section 11) and directly testable methods and experiments to support or falsify the framework (see section 13).

11. Implications of the proposed framework

The salience misattribution framework proposed here brings together findings in the addiction literature that specifically implicates dopaminergic and ACC dysfunction as it relates to aberrant internal models that lead to maladaptive decision-making. This implies that an individual with a dependence may actually be making “adaptive” decisions, but only with respect to their misaligned internal model. If the internal model that drives future decisions is aberrant, the decisions that arise as a result are also likely to be maladaptive. This proposed framework may therefore be a step towards better understanding the underlying causes of some aspects of decision-making deficits in addiction.

Additionally, it may suggest a future treatment-outcome measure by quantifying internal model updates from differential cues and stimuli in those with a dependence. For example, treatment options that reduces salience and associative relationships between drug-related cues and rewards while increasing them between non-drug related cues and rewards may help in reducing the maladaptive drug-seeking behaviors by enriching the internal model. One way this may be achieved is by selectively modulating ACC activity (through brain stimulation; see De Ridder et al. (2011) or drugs; see Udo De Haes et al. (2007)). Stimulating the ACC during error processing may facilitate greater updates from errors and negative feedback signals by attributing more weight to them. This may then better enrich the internal model where the negative consequences of a decision are more precisely computed. As a result, impulsive decisions may occur more sparsely and also take into account the negative caveat of an immediately positive reward but negative future outcome (e.g. misuse of drugs now with a negative health outcome in the future). Conversely, disrupting ACC activity during the processing of drug related cues may reduce updates from these, where the internal model now has weaker predictive value or salience attributed to drug related cues and may then place lower weights in any positive prediction errors that arise as a result. Overall, downweighing a drug-predictive cue’s association with a reward and upweighting a non-drug related cue’s association with a reward may help in reducing drug seeking and impulsive decisions in those with a dependence.

12. Proposed framework’s generalizability to other phenomena

Addiction is a complex, multifaceted phenomenon, which manifests in multiple symptoms rather than as single unitary disease (Redish et al., 2008). Its underlying causes are not just in the complex neurobiology but also related to complex environmental interactions (Bickel et al., 2010; Li and Burmeister, 2009; Perron and Bright, 2008; Venniro et al., 2018). A comprehensive discussion of all these processes is beyond the scope of the present paper. Instead, we focus on better understanding aspects of the decision-making deficits in addiction through cognitive processes involving ACC and dopaminergic systems; namely the reduced ability to adapt to ever changing environmental contingencies. However, this reduced ability to adapt is not unique to addiction, hence, our proposed framework may also speak to other disorders where aberrations in learning and decision-making processes manifest. These include state-dependent disorders such as obsessive-compulsive disorder (OCD) and post-traumatic stress disorder (PTSD). For example, ACC hyperactivity is found in those with an OCD during error-processing and is positively correlated with symptom severity (Fitzgerald et al., 2005). This finding may be interpreted in light of our proposed framework where selective updating from errors leads to an internal model that overweights these errors and drives further maladaptive behaviors. Similarly, those with PTSD show a hyperactive ACC in response to fear-related stimuli (see Hughes and Shin (2011) for a review). Further, ACC activity in response to combat-related photos in war veterans is positively correlated with PTSD symptom severity (Morey et al., 2008). This may also be interpreted under our framework where stress cues and stimuli more related and salient with respect to the individual’s PTSD may have over-weighted updates that generates an internal model that is biased towards updates from such stress cues. Collectively, these suggest that misaligned internal models may partially be causative accounts that drive maladaptive decision-making (Cochran and Cisler, 2019; Gershman and Niv, 2012; Redish et al., 2007; Tolman, 1948).

13. How the proposed framework may be tested/falsified experimentally

The present framework makes an experimental prediction that those with a dependence or a susceptibility to dependence misattribute drug-related cues as having an abnormally high predictive value associated with a positive outcome, even if the positive outcome is unrelated to drugs (e.g. monetary gain). This could be tested experimentally with a probability reversal task where a drug-related cue and a non-drug related cue is presented simultaneously with one of them being more predictive of the outcome (positive or negative, e.g. monetary loss or gain). A prediction of our framework is that the group with a dependence compared to non-dependents, would have a bias for attributing positive outcomes to a drug-related cue (than the non-drug related cue). In other words, the aberrant internal model in those with a dependence is such that drug-related cues are misattributed as more strongly predictive of positive outcomes and are therefore incorrectly reinforced (Figure 5). A further falsifiable prediction of our framework is that it would take a greater number of trials for a person with a dependence to identify a cue reversal (i.e., that the non-drug related cue is predictive of the positive outcome). If true, this could reveal the lower weights placed in updates from prediction-error signals to non-drug related cues. The converse may be true with non-drug related cues being attributed with a higher probability of predicting negative outcomes, even when in fact the drug-related cues are more highly or equally predictive. In the case where both drug and non-drug related cues are not predictive of an outcome, the dependent group may have a bias which misattributes the drug-related cue to be more predictive of the positive outcome, and neutral cue to be more predictive of negative (even though neither are predicting the outcome). Overall, experiments and simulated models of such nature may allow to experimentally support or falsify the proposed framework. Hence, the validity of the present framework is not in that it explains the many complex aspects of addiction and associated symptoms, but in that the specific predictions that arise as a result are subsequently supported with future experiments.

Figure 5. Experimental prediction of misattributing outcomes to differential cues.

The top panel represents the actual probability distributions of positive and negative outcomes given either a neutral or a drug-related cue, where all probabilities are equal. The bottom panel is our framework’s prediction of the estimated probabilities by someone with a dependence or a susceptibility to dependence. Here, the positive outcomes are more often attributed to be predicted by drug-related cues and less so by neutral cues (even though they have equal probabilities). The converse may be the case for negative outcomes, which are less often attributed to be predicted by the drug-related cues and more often by neutral cues.