Computational Dysfunctions In Anxiety: Failure To Differentiate Signal From Noise

Abstract

Background

Differentiating whether an action leads to an outcome by chance or by an underlying statistical regularity that signals environmental change profoundly affects adaptive behavior. Prior studies have shown that anxious individuals may not appropriately differentiate between these situations. This investigation aims to precisely quantify the process deficit in anxious individuals and determine the degree to which these process dysfunctions are specific to anxiety.

Methods

122 subjects recruited as part of an ongoing large clinical population study completed a change point detection task. Reinforcement learning models were used to explicate observed behavioral differences in Low Anxiety (OASIS<=8) and High Anxiety (OASIS>=9) groups.

Results

High anxious individuals used a sub-optimal decision strategy characterized by a higher lose-shift rate. Computational models and simulations revealed that this difference was due to a higher base learning rate. These findings are better explained in a context-dependent reinforcement learning model.

Conclusions

Anxious subjects’ exaggerated response to uncertainty leads to a suboptimal decision strategy that makes it difficult for these individuals to determine whether an action is associated with an outcome by chance or by some statistical regularity. These findings have important implications for developing new behavioral intervention strategies utilizing learning models.

Introduction

Anxiety disorders are the most common mental health problem [1] with a lifetime prevalence of approximately 33% [2]. Anxiety disorders are the sixth leading cause of disability world-wide and show no signs of reduced burden over recent years [3]. As with MDD, extant treatments are only partially effective (e.g., [4]). Several heuristic models have been put forth to explain the process dysfunctions observed in anxious individuals, including individuals with Generalized Anxiety Disorder. These have been focused on avoidance of internal affective experiences (i.e., thoughts, beliefs, and emotions) and cluster into three types: cognitive models (i.e., Intolerance of Uncertainty Model [5], Metacognitive Model [6]), emotional/experiential (i.e., Emotion Dysregulation Model [7], Acceptance-based Model [8]), and an integrated model (Avoidance Model of Worry [9]). While these models are useful to develop a heuristic understanding of the processing dysfunctions in anxiety and may help to develop new treatment strategies [10], these models do not offer a precise quantitative characterization of these dysfunctions that could enable one to examine the degree of process dysfunction or the degree to which interventions remediate this process. The computational approach assumes that the brain uses an algorithm that takes inputs (i.e. the history of events and behaviors) and forms an output (i.e. a particular behavior) in order to adapt to the rules given by a particular context. A dysfunction can be defined as a set of parameters in this input/output relationship that is related to the individual’s characteristic, i.e. the level of anxiety, and leads to a behavior that is either nonadaptive or non-optimal. Therefore, computational models may help to delineate quantitatively anxiety-related process dysfunctions and can help to integrate these dysfunctions across different levels of analyses, i.e. relate behavior to physiology, circuits and molecules.

Intolerance of Uncertainty in Anxiety

From choosing the route of driving to work, selecting which restaurant to go, to deciding whether to bring an umbrella or not, we make decisions based on our past experience and the degree of which it can inform us about future outcome [11]. Because most real-world problems have inherent uncertainty, the ability to efficiently use past experience to infer future outcomes is an essential skill in daily life. In particular, being able to differentiate signal from noise is crucial in effectively learning and utilizing the statistical regularity in decision-making [12]. Thus, being able to differentiate noise and ‘signal’ from the observed information is important in developing an optimal decision strategy under uncertainty. Several studies have shown that individuals with anxiety may become overwhelmed with such uncertainty or noise in the environment and focus on solving minor problems [13–15] thus fail to use the useful information that signals the true statistical regularity in the big picture. In fact, intolerance of uncertainty is a critical component and a broad risk factor of anxiety disorder [5]. These findings are consistent with our proposition of an exaggerated processing of statistically random events.

Computational approaches in studying learning under uncertainty

Browning and colleagues [16] showed that related to intolerance of uncertainty, individuals with anxiety may not appropriately adjust their outcome expectations when they are in stable versus volatile environments, i.e. when updating action-outcome estimates in a learning process described by a Reinforcement Learning model [17]. Anxious individuals have difficulty with adjusting the trade-off between long-term statistics and recent information. For example, in a fast changing environment individuals are expected to rely more on most recent information, i.e. have a higher learning rate. Thus, choosing an appropriate learning rate is important for human behavior and enables one to effectively learn the underlying statistical regularity and quickly adapt to changes in the environment. Specifically, as a high learning rate may lead to over-learning and exhibit “Win-stay-Lose-shift” like behavior, a low learning rate may lead to insufficient updating and exhibit slow adaption to the change in the environment.

This investigation addressed the question whether individuals with high levels of anxiety have difficulty in differentiating signal from noise when they have to adjust their decision-making to changing contingencies. Given prior findings by Browning and colleagues, we hypothesized that high anxious would overreact to changes in the associations between choice options and outcomes and it would worsen over time. To that end, individuals completed a change-point detection task with changing contingencies. Two reinforcement learning models were obtained from the data to determine the learning rate of high anxious individuals over time.

Method

Subjects

This experiment was part of a larger study termed the “Tulsa 1000” study, or T-1000, which aims to examine the underlying latent structure of individuals with mood and anxiety problems using a dimensional psychopathology (RDoC) approach. Subjects were recruited from mental health providers or via general advertisement, and were excluded if diagnosed with any of the following DSM-5 disorders: Schizophrenia Spectrum and Other Psychotic Disorders, Bipolar and Related Disorders, Obsessive-Compulsive and Related Disorders. Detailed exclusion criteria are listed in supplementary material. 122 T-1000 subjects participated in this experiment (age: 35.03 ± 11.08 yrs; gender: 38 male and 84 female; medication status: 53 on medication: antidepressant 45, anxiolytic 17, antipsychotic 1, mood stabilizer 10). Subjects’ diagnoses are summarized in Supplementary Table S1. We grouped subjects based on their OASIS score (Overall Anxiety Severity and Impairment Scale, Norman et al. 2006) as: low anxiety (LA) group (OASIS<=8, n = 45, mean OASIS 5.51±.37) and high anxiety (HA) group (OASIS>=9, n= 77, mean OASIS 11.51±.29). The cutoff is based on [18] where they reported a cut-score of 8 correctly classified 87% of the sample with anxiety diagnosis or not.

Experiment

Subjects were instructed to identify the location of the target, which is one of the three random-dot stimulus patches [19] with a pre-specified coherent motion direction (left or right, counterbalanced across subjects, Figure 1A), the other two being distractor with the opposite direction [20]. Each trial started with three dots indicating the locations that the target may appear. Subjects searched each location sequentially by pressing corresponding keys on the keyboard and made the final decision by pressing the Up arrow key to indicate the last viewed stimulus was the target. At the end of each trial, the true target location was indicated with a green (correct) or red (incorrect) dot, and subjects received points based on time used, number of switches, and the correctness of the final response. The points were assigned as follows:

$$\text{Points}=100-12.5\times \text{RT}-25\times \text{Number}\text{of}\text{switches}\pm 50\left(\text{correct}/\text{incorrect}\text{response}\right)$$ [1]

The experiment comprised of two parts: training session (30 trials with random dot coherence = 50%, i.e. the percentage of dots moving in the same direction) and main experiment (90 trials/block x 3 blocks with random dot coherence = 30%). All subjects achieved at least 75% accuracy in the training session to proceed to the main experiment. In the main experiment, the target appeared in the three locations with a relative frequency of 1:3:9, and the reward contingency at the three locations changed based on a Gaussian distribution (N(30,1)) that was used to determine the length of runs within each block (Figure 1B). Note that subjects were informed about how points were calculated based on their reaction time, number of switches and correct/incorrect decision, but were not told about the reward probability or change points.

Figure 1.

Experimental Paradigm. A. Visual-search task with random-dot motion stimuli. Each trial starts with a central-cross followed by three location cues. Subjects were instructed to use keyboard to select where to search for the target. At any time, they could decide 1) continue searching by switching to another location if they decided the current patch was not the target, or 2) respond by pressing ‘Up’ arrow when they decided the current patch was the target. Feedback with points earned was shown in the end of the trial. B. Volatility condition. There are 3 blocks with 90 trials per block in the experiment. Reward rate at three locations is fixed at 1/13, 3/13, 9/13, but the associated location of the three probability changes within each block based on a Gaussian Distribution (mean = 30, std = 1).

Data Analysis

Two main behavioral measures were obtained to determine if and how anxiety affects performance and decision strategy under uncertainty: 1) points earned, which is defined by formula [1]; 2) Win-stay-Lose-shift rate, in which Win-stay rate is defined as the percentage of trials subjects started searching from the same location as in the last trial if their 1^st choice was the target in previous trial, and Lose-shift rate is defined as the percentage of trials subjects started searching from the location where last target appeared if their 1^st choice was not the target in previous trial.

Modeling

We compared two learning models: 1) Naïve RL: a standard temporal difference (or Rescorla-Wagner) learning model with a constant learning rate [16, 17, 21, 22]. 2) Vmax RL: the Naïve RL model with a constant learning rate does not allow learning rate adjustment to adapt to changes in the environment. To account for this, we modified the Naïve RL with a sensitivity adjustment rate based on change of the option with the maximal value, where it assumes subjects may adjust their learning rate to a constant base learning rate, but only when the choice with the maximum value changes. Softmax function was used as the action-selection model and to translate values in RL models to subject’s choices. Model descriptions are shown in Table 1. Demonstration of value estimation using RL is shown in Supplementary Figure S1.

Table 1.

Model description of three learning models and the choice model used in this paper. It includes the value updating equation (RL) for each model, and the description of the parameters in the equation.

Models		Value/belief updating equation
Learning Model	Naïve RL	$V_i^{t+1}=V_i^t+\eta (R_t-V_i^t)$
		η: Fixed learning rate of chosen option, in which Rt is reward (0/1) in trial t
	Vmax RL	$V_i^{t+1}=\{\begin{array}{l}{V}_i^t+{\eta }_0(R_t-V_i^t),\mathit{argmax}\left\{V^t\right\}=\mathit{argmax}\left\{V^{t-1}\right\}\\ V_i^t+({\eta }_0+{\eta }_d)(R_t-V_i^t),\mathit{argmax}\left\{V^t\right\}\ne \mathit{argmax}\left\{V^{t-1}\right\}\end{array}$
		η0: Fixed base learning rate of chosen option ηd: Adjustment to η0 when the option with the maximal value changes
Choice Model	Softmax function	$q(i)=\frac{e^{\beta V_i}}{{\sum }_j{e}^{\beta V_j}}$
		β: Inverse decision parameter

Results

1. HA individuals differ in decision-strategy

The overall learning curve across subjects shows that both groups learned the task (Figure 2A). There was no significant difference in points earned between HA and LA groups (p = .89), but HA group had a significantly higher Lose-shift rate (Figure 2B, LA: .32 +/− .04, HA:.42 +/.02, t(120) = 2.35, p = .02, d = .43) while no significant difference in Win-stay rate was observed (p = .87). Additional analyses showed no significant main effect or interaction of medication status or depression level for points earned or lose-shift rate was observed (p> .1). Examining the difference in Lose-shift rate over time (Figure 2C), there was a significant main effect of group (F(1,120) = 5.51, p = .02, d = .43) and block (F(1,242) = 18.90, p = .000, d = .56), and a trend of interaction between group and block (F(1,242) = 2.84, p = .09). A significant block effect was only found within HA group (t(153) = 4.25, p = .000, d = .69), while it was not observed in LA group (t(89) = 1.45, p = .15). In addition, a significant group difference in Lose-shift rate was only observed in the 3^rd block (block 3: t(120) = 2.67, p = .009, d = .49; block 1: p = .06, block 2: p = .08. Dimensional analyses using OASIS scores yielded trend-level significant effects (see supplementary text)

Figure 2.

A. Learning in both groups over 3 blocks. It shows both groups had increasing points earned within a run until the change points. B. Win-stay-Lose-shift rate in both groups. No significant difference of Win-stay rate was observed, while HA group had significantly higher lose-shift rate. C. Lose-shit rate over time in 3 blocks. It shows that HA group had increasing lose-shift rate over time while it was not observed in LA group.

2. The HA decision strategy is not optimal with respect to the underlying probability

In a volatile environment one would expect a high Lose-shift rate due to the reliance on more recent experiences, i.e. individuals should show a shift to the last target when experiencing a loss in the current trial. In comparison, selecting the option with a higher reward rate should be preferred in a stable environment. That is, to get better performance (reward points defined by formula [1]), the optimal strategy would be to start searching from the most likely rewarded location (i.e. the location with the highest reward rate 9/13, see simulation in Figure 3A). Thus we examined how LA/HA group’s Lose-shift rates differ based on the underlying reward probability and found that HA group had significantly higher Lose-shift rate regardless of the underlying reward structure (Figure 3B). In particular, HA group had a significantly higher Lose-shift rate at the most likely location (t(120) = 2.26, p = .03, d = .41). Similar to the group difference shown in overall lose-shift rate above (Figure 2C), only HA individuals but not LA subjects (t(128) = .69, p = .49) showed a significant block effect for Lose-shift rate at the most likely rewarded location (t(227) = 2.40, p = .02, d = .69, Figure 3C), which is consistent with the notion that these individuals develop a sub-optimal strategy over time. In addition, a significant group difference was only observed in the 3^rd block (t(120) = 2.52, p = .01, d = .46).

Figure 3.

A. Simulation of points earned conditioned on lose-shift rate the most likely rewarded location, generated from 500 simulated behavioral sequences with lose-shift rate ranges from 0 to 1 with an increment of 0.1, assuming the same reaction time per location searched. B. Lose-shift rate conditioned on reward probability. It indicates HA group had higher lose-shift rate than low-anxiety group in all choices with different reward-probabilities. C. Lose-shift rate at the most likely rewarded location in 3 blocks. It shows that HA group has increasing lose-shift rate over time while it was not observed in LA group.

3. Computational Models

Two computational models were used to explain the observed behavioral data. Whereas a simple RL model assumes that individuals have a constant learning rate, the Vmax RL model assumes that individuals adjust their learning rate in response to changing contingencies. Maximum Likelihood Estimation (MLE) was used to fit model parameters. Model fitting details can be found in the Supplement. We examined group difference in parameters estimated in each model and determined whether this could provide explanations of the observed behavioral difference across groups.

Among all parameters (η, β in Naïve RL, η₀, η_d, β in Vmax), group difference was only observed in base learning rate in the η₀ block estimated from Vmax (t(120) = 2.46, p = .02, p = .45, Figure 4A). η₀ also has a significant positive correlation with Lose-shift rate (correlation coefficient = .80, p < .0001), as shown in Figure 4B.

Figure 4.

A. Group difference in base learning rate from Vmax RL model in Block 3. B. Lose-shift rate as a function of base learning rate.

We compared AICs of the two models across all subjects (Naïve RL: 111 .85 (+/− 3.67), Vmax: 114.19 (+/− 3.74)). Though with a slightly lower AIC, there was a moderate correlation between parameters η, β in Naïve RL (correlation coefficient −.47), which may point toward unstable parameter estimation. In comparison, the correlations between parameters in Vmax were weaker (η₀, η_d, correlation coefficient= .15; η₀, β, correlation coefficient = −.34), which provides some evidence for higher parameter stability. Subject intra-class correlation (ICC) across all three blocks for Naïve RL was 0.37 for η and 0.27 for β, whereas for Vmax ICC was 0.41 for η₀ and 0.32 for β. Taken together these findings indicate that the higher lose-shift rate in HA group may be due to a higher base-learning rate, and the behavioral difference is better captured in Vmax model. At the same time, these models capture very similar aspects of the individual’s behavior, e.g. the constant learning rate in the Naïve RL correlated strongly with the base learning rate in Vmax (correlation coefficient = .90, p <.0001), and also correlated with the lose-shift rate (correlation coefficient = .74, p < .0001).)

Discussion

Based on previous studies [16], this investigation examined the processing differences between HA and LA individuals in a decision-making situation with uncertainty and varying preferences and yielded two main results. First, HA individuals showed comparable task performance by using a decision-strategy that systematically differs from LA subjects. Second, this decision strategy by HA individuals is not optimal, because they used a decision strategy that requires a higher attention to short-term statistics that was not optimal based on the environmental regularity. Taken together, both behavioral and model evidence support the hypothesis that individuals with anxiety show specific, uncertainty-related processing deficits characterized by difficulty inferring the underlying statistical regularity.

We examined several possible models to disambiguate the possible computational principles that lead to behavioral differences in individuals with HA. Compared to the Naïve model, the RL model with an adjustment rate to base learning rate provides a better explanation based on the parameter difference that is consistent with observed behavioral difference as well as its lower risk of correlated parameters for unstable parameter estimation. However, there is a high correlation between the constant learning rate in the Naïve model and the base learning rate in Vmax model, thus further model selection will be necessary to arbitrate between those models. Notably, Browning et al. 2015 reported that anxious individuals have difficulty learning the causal statistics of aversive environment by showing there was no significant difference in their learning rate across different volatility conditions. Our finding is consistent with their conclusion because anxious individuals were not able to adjust their learning rate in a way that was optimal based on the change in the environment (i.e. a lower learning rate in stable period and a higher learning rate during change points). Instead HA used a higher learning rate that led to a sub-optimal decision strategy, which was not optimal based on the underlying reward structure. However it should be noted that Browning et al. 2015 fitted the RL model parameters by comparing different volatility blocks, whereas we fitted the model parameters within the same volatility block. Moreover, these authors used an aversive task while we used a reward-based task. In future studies, modulating the volatility and the degree of reward or aversion may help to further disambiguate the learning dysfunctions in HA individuals.

The high anxiety group subjects had significantly higher lose-shift rate than low anxiety group over time. This indicates they may have a sub-optimal decision strategy, because behaviorally lose-shift requires subjects pay attention to every trial’s feedback, and computationally a higher learning rate indicates more ‘attention’ needed to allocate to the recent/short-term statistics in the environment (while discounting long-term statistics). Optimally if subjects learned the underlying probability structure and were aware of the volatility then behaviorally they can ignore ‘losses’ at the most-likely rewarded condition and computationally only need a delta-like increase in learning rate during change points while keep it low during a stable period. Future studies can address this issue by adding an ‘energy-based’ term that measures the cost of the decision-strategy.

This framework has the potential to contribute to the understanding of learning deficits and develop the relationship between behavioral changes to behavioral therapy. The visual-search experimental paradigm provides a platform to examine learning inability under different levels of uncertainty. For example, one can reduce the uncertainty in the task by controlling the noise in the reward distribution by increasing or decreasing the relative frequency of target location or adjusting the frequency of change points. The computational models provide possible explanations of the underlying learning dynamic that lead to observed behavioral differences, which will give important insight in designing individual-based treatment plan.

Limitations

First, this is a cross-sectional sample, which only provides a ‘snapshot’ of the situation thus may not fully reflect how the change of anxiety within each individual will affect their decision-making process. To fully understand how anxiety may affect individuals at different time points. Second, we did not further investigate anxiety subjects based on specific anxiety disorders (e.g. generalized anxiety disorder vs. social anxiety disorder). It is likely that decision-process for different anxiety disorders is context-dependent and it will be important to explore with a larger sample size in the future. It should be noted that healthy controls were not included in this study, thus results shown here cannot speak to causal factors of anxiety. Third, though medication status was examined as a covariate in the overall behavioral analysis, we did not further examine if and how medicated subjects’ behavior could be affected by their medications. Fourth, in some cases MLE based parameter estimation may lead to unstable estimation. Therefore, future investigations may need to compare different parameter estimation approaches [23, 24]. Fifth, reinforcement models were the primary focus of this investigation, however other approaches based on precision-based models of decision-making, which are based on Bayesian formulation, have been used to examine choice-selection under uncertainty. However, given our primary behavioral finding of increased lose-shift behavior it is important to point out that these models would also make similar predictions about the lose-shift rate [25]. In the future, we will further examine behavioral difference in different volatility conditions, and improve current models by using other hierarchical models, such as Hierarchical Gaussian Filtering [26, 27]. Lastly, current study is at behavioral level and it will be important to examine the neural substrates in the future.

Conclusion

Using a computational approach to examine reward-related and uncertainty related processing in a decision-making situation we provide evidence for anxiety-related processing deficits. The increased sensitivity to randomness in the environment among HA individuals can interfere with their ability to learn the true action-outcome relationships in the environment for optimal decision-making. These results may help to explain why anxious individuals have difficulty in forming a stable representation of an uncertain environment and show resistance to behavioral change.