Social anxiety and dynamic social reinforcement learning in a volatile environment

Abstract

Adaptive social behavior requires learning probabilities of social reward and punishment, and updating these probabilities when they change. Given prior research on aberrant reinforcement learning in affective disorders, this study examines how social anxiety affects probabilistic social reinforcement learning and dynamic updating of learned probabilities in a volatile environment. N=222 online participants completed questionnaires and a computerized ball-catching game with changing probabilities of reward and punishment. Dynamic learning rates were estimated to assess the relative importance ascribed to new information in response to volatility. Mixed-effects regression was used to analyze throw patterns as a function of social anxiety symptoms. Higher social anxiety predicted fewer throws to the previously punishing avatar and different learning rates after certain role changes, suggesting that social anxiety may be characterized by difficulty updating learned social probabilities. Socially anxious individuals may miss the chance to learn that a once-punishing situation no longer poses a threat.

Social interaction is both uncertain and volatile; few people give exclusively positive or negative social feedback, and the probability of a person reacting positively or negatively towards you changes over time. Even your closest friend will sometimes be angry at you, and it is unlikely that you are still close with all of your childhood friends. Navigating this uncertain, volatile social environment requires learning the probabilities of social reward and punishment, and then updating these mental representations when those probabilities change. Despite research suggesting that socially anxious individuals process social information differently than non-anxious individuals, including information about social rewards and punishments, little research to date has addressed how socially anxious individuals update probabilities about volatile social environments. The present study aims to address this question, here defining volatility as probabilities of reward and punishment that change over time.

Reinforcement Learning and Social Anxiety

On a basic level, human behavior is driven by seeking rewards and avoiding punishments. Reinforcement learning (RL) is the fundamental process by which an agent learns to predict and optimize their behavior in an environment where taking actions leads to rewards and punishments (Sutton & Barto, 1998). The agent chooses actions based on what they anticipate will optimize their outcomes over time, and optimizing performance requires adjusting their predictions of what the outcomes of their actions will be as they notice errors in their predictions. A growing body of neural findings provide support and validation for these RL models as descriptive of human decision-making processes (Chase, Kumar, Eickhoff, & Dombrovski, 2015; Cohen, 2007; Ruff & Fehr, 2014). Increasingly, researchers are applying RL algorithms to pinpoint and quantify dysfunctions in human learning processes that may underlie psychopathology, providing a more granular understanding of the mechanisms of dysfunction than may be achieved by examining behavior or neural activity without applying these models (Montague et al., 2012; Whitton, Treadway, & Pizzagalli, 2016).

Problems with RL, which may include slow learning of rewards and punishments and/or decision-making that misaligns with learned probabilities of rewards and punishments, can have enormous downstream effects in terms of maladaptive behavior patterns. Aberrant RL has been implicated in many psychological disorders, including major depression, schizophrenia, social anxiety disorder, and addiction, and at many levels of analysis, including neural function, structure, and connectivity, and behavior (for a review, see Dayan, 2009). For instance, studies of major depressive disorder suggest that anhedonia, or loss of pleasure, is related to a reduced ability to modulate behavior as a function of rewards, and this may be due to blunted phasic dopaminergic signaling and reduced reward anticipation (Whitton et al., 2016). Given the high comorbidity of depression and social anxiety disorder (the 12-month correlation is 0.43; Kessler, Chiu, Demler, & Walters, 2005), similar processes may underlie both disorders, including blunted reward sensitivity.

Anxiety disorders are typically associated with hypervigilance for threat, but there is reason to also consider aberrant responses to reward, particularly in the case of social anxiety disorder. The integrated hierarchical model of anxiety and depression (Brown, Chorpita, & Barlow, 1998) suggests that although high negative affect is implicated in many mood, anxiety, and obsessive-compulsive disorders, low positive affect is only associated with mood disorders and social anxiety disorder, and at similar magnitudes (Brown et al., 1998). Further, a meta-analysis found that social anxiety is negatively related to positive affect beyond what can be explained by co-occurring depression (Kashdan, 2007). Ecological momentary assessment studies have also found that highly socially anxious people experience less intense positive emotions both when with other people and when alone (Kashdan, Weeks, & Savostyanova, 2011). Moreover, individuals with social anxiety disorder tend to respond negatively to both negative and positive social evaluation (Weeks, Heimberg, Rodebaugh, & Norton, 2008; Weeks & Howell, 2014), suggesting that social events that are rewarding for most people may be processed differently by socially anxious individuals. Further, neuroimaging studies have found that, relative to healthy control participants, highly socially anxious individuals show blunted activation in the nucleus accumbens (part of the brain’s reward system) during social reward anticipation, but either intact or increased activation during monetary reward anticipation (Gossen et al., 2014; Richey et al., 2014). These studies suggest that non-social reward learning may be intact for highly socially anxious individuals, but they may have aberrant social reward learning processes.

Sensitivity to reward and/or punishment.

Notably, aberrant social RL in social anxiety may be driven by hyposensitivity to social reward, hypersensitivity to social punishment, or some combination of both. As evidence of hypersensitivity to social punishment, highly socially anxious individuals have been found to avoid stimuli with high probabilities of social punishment (scowling faces) more than less socially anxious individuals (Abraham & Hermann, 2015). Further, Koban and colleagues (2017) applied RL algorithms to responses to feedback on a social evaluative stressor, and found that people with social anxiety disorder more heavily weight negative feedback when updating their feelings about themselves as compared to healthy control participants (i.e., socially anxious individuals have a higher learning rate for social punishment, a concept discussed in detail in the next section). Individuals with social anxiety disorder also show facilitated fear conditioning to critical faces, evidenced by potentiation of the startle-blink reflex, versus conditioning to neutral or happy faces, an effect not seen in individuals without social anxiety disorder (Lissek et al., 2008). This learned fear response to angry faces is also more resistant to extinction (Olsson, Carmona, Downey, Bolger, & Ochsner, 2013). These studies all suggest that social anxiety disorder may be characterized by an exaggerated learned response to social punishment, and this may be seen in differences in learning rate.

Consistent with the idea that both hyposensitivity to social reward and hypersensitivity to social punishment are present in social anxiety disorder, one study found that during social evaluative threat, higher trait punishment sensitivity predicted better punishment learning and poorer reward learning (Cavanagh, Frank, & Allen, 2011). Although this study did not select for social anxiety and used a non-social RL task, the results may also apply to socially anxious individuals (given socially anxious individuals are often higher in punishment sensitivity: Kimbrel, Mitchell, & Nelson-Gray, 2010; Kimbrel, Nelson-Gray, & Mitchell, 2012), and suggest likely social anxiety-based differences in sensitivity to both reward and punishment.

Adapting to Volatility

Beyond learning static probabilities of reward and punishment, our volatile social environment requires individuals to update their mental representations when circumstances change. Consider a new coworker who initially seems standoffish and rude, but after a few weeks, starts asking you about your weekend and seems more interested in getting to know you. You might have been correct to associate this person with a low probability of social reward and a high probability of social punishment at first, but if you are too slow to update this mental representation of them when they become friendlier, you may miss out on an opportunity to build a collegial working relationship or even a friendship. The social environment is volatile; as people and circumstances change over time, so too do relationships. Tracking these changes dynamically and updating how you think of other people is essential for optimal social behavior (Ronay & von Hippel, 2014). To do this, people need to assess how volatile the environment is (i.e., how stable or unstable probabilities of reward and punishment are) and adjust their decision-making accordingly. Computationally, this process is reflected in the learning rate, or how heavily recent information is weighted relative to previously learned information in predicting the outcomes of future actions (Behrens, Woolrich, Walton, & Rushworth, 2007). To continue the previous example, if you think the social environment is more volatile, you would pay more attention to how your coworker treated you this morning than last month when considering whether he would likely accept or reject an invitation to join you for lunch today.

How might anxiety predict learning in volatile environments? To optimize decision-making, a person’s learning rate should increase in more volatile environments because information learned further in the past is less predictive of current probabilities of reward and punishment. Research suggests, however, that individuals high in trait anxiety adjust their learning rate less in response to the volatility of an aversive environment than less anxious individuals do (Browning, Behrens, Jocham, O’Reilly, & Bishop, 2015). One study found no significant difference in learning rate between participants high and low in trait anxiety in either stable (i.e., probability of punishment does not change) or volatile (i.e., probability of punishment changes several times) aversive environments, but did find that highly anxious participants showed a smaller increase in the learning rate between the stable and volatile environments, reflecting an insensitivity of the learning rate to volatility (Browning et al., 2015). Supporting the hypothesis that anxiety is characterized by difficulty learning the underlying statistics in volatile environments, Huang and colleagues (2017) found that highly anxious individuals have a higher base learning rate and tend to rely more on a suboptimal strategy in a volatile environment, shifting choices after a loss regardless of the true probability of reward. Further, Raio and colleagues (2017) found that acute stress is associated with a lower learning rate in an aversive reversal learning task, demonstrating less adaptability to changing probabilities of punishment. However, Cavanagh and colleagues (2011) found that more (vs. less) punishment-sensitive individuals showed a lower learning rate for reward when not under social evaluative threat (suggesting state anxiety cannot fully explain the pattern of results; Cavanagh et al., 2011). Taken together, these studies suggest that social anxiety may be associated with some differences in learning rate, but the relationship may be dependent on whether the individual is updating probabilities of reward or punishment, the volatility of the environment, and the individual’s state anxiety. Based on this growing evidence base about learning rate differences in volatile environments as a function of anxiety, further investigation is warranted to understand how these learning processes affect behavior of socially anxious individuals in social situations, which are inherently volatile. Such an analysis is particularly called for given the relationship between social anxiety disorder and diminished positive affect.

Overview and Hypotheses

To better understand how aberrant social RL characterizes social anxiety disorder, the present study investigates how individual differences in social anxiety symptoms predict how people learn and update probabilities of social reward and punishment in a volatile social environment. The environment was created through a novel modification of Cyberball (Williams, Cheung, & Choi, 2000), a virtual game of catch that has been used in over 200 studies and induces strong feelings of ostracism by making computerized avatars exclude the participant (Hartgerink, van Beest, Wicherts, & Williams, 2015). Our modified version varies the probability of each avatar throwing the ball to the participant, such that one avatar is highly rewarding (includes the participant), one is neutral (throws the ball equally to all players), and one is highly punishing (tends to exclude the participant), and these probabilities change at several time points. We analyzed the participants’ throwing and catching behavior in raw form (i.e., better performance is indicated by more throws to the rewarder than the neutral and punishing avatar, and by receiving the ball more often). We also applied a RL algorithm to model learning rate in response to environmental volatility (i.e., higher learning rate indicates more heavily weighting new information about the probabilities of reward and punishment for each avatar). This study builds on previous studies of RL in social anxiety that computationally model affective responses specific to social performance anxiety (e.g., fear of public speaking; Koban et al., 2017) to model a type of learning relevant to social anxiety more generally - learning about the probabilities of social inclusion and exclusion. Further, this study extends the extant literature by assessing social RL in a volatile environment.

Given past research on hypersensitivity to social punishment and hyposensitivity to social reward, we hypothesized that participants higher (relative to lower) in social anxiety would be less likely to throw to the punishing avatar, but also less likely to throw to the rewarding avatar, resulting in relatively more throws to the neutral avatar. Because avoiding the punisher would improve performance on the task (receiving the ball more), but not choosing the rewarder would impair performance, it was unclear whether there would be a main effect of social anxiety on overall task performance based on number of catches. We also hypothesized that differences as a function of social anxiety might emerge in the learning rate analyses; that participants higher (vs. lower) in social anxiety symptoms would be slower to adjust their learning rate when the punishing avatar became more rewarding, and faster to adjust when the rewarder became more punishing. These hypotheses were based on the emerging literature about anxiety’s effects on learning rate and on more established findings that socially anxious individuals show blunted social reward anticipation, but facilitated learning and impaired extinction of fear responses to socially punishing stimuli.

Method

Participants

N=292 participants aged 18 and above were recruited through Amazon’s Mechanical Turk to complete an online study approved by the University of Virginia Institutional Review Board. Only high reputation (90% approval rate or higher) workers were recruited to improve data quality. Because of potential cultural differences in expression of social anxiety (Heinrichs et al., 2006; Hofmann, Asnaani, & Hinton, 2010), 51 participants from outside of Pacific, Mountain, Central, and Eastern Time Zones were excluded prior to conducting analyses. To remove participants who were not paying attention to the dynamic social reinforcement task or who were not adhering to instructions, participants who almost exclusively targeted only one avatar throughout the game (despite changing roles) were removed. This was done by summing each participant’s throws to their two least-targeted avatars, performing a mixture distribution analysis on this sum (using the Mclust package in R), and removing the cluster of participants with low counts on this sum. This analysis yielded three clusters, and the 19 participants in the cluster with fewest throws to the two least-targeted avatars were removed from subsequent analyses, resulting in a final N=222 participants (M_age = 34.89 years, SD_age = 10.95; 107 female, 114 male, 1 prefer not to answer).¹

Measures

Volatile Social Learning Task (VSLT). Social RL was measured with a modified version of a popular psychological task, Cyberball (Williams et al., 2000), versions of which have been used in hundreds of research studies (Hartgerink et al., 2015). Previous research has found that this simple online game of catch can elicit surprisingly strong emotional and behavioral responses to social inclusion and exclusion (Eisenberger, Lieberman, & Williams, 2003; Hartgerink et al., 2015). In our novel modified version, the stated goal is to maximize the number of times you receive the ball. The game is played with three computerized players who differ in the probability of reward (throwing the ball to the participant) and punishment (excluding the participant). Specifically, one avatar throws the ball to the participant with 0.7 probability (the “rewarder”), one with 0.33 probability (the “neutral player”), and one with 0.1 probability (the “punisher”). These roles change at three time points in the game (after each block of 100 throws, with a total of 4 blocks, resulting in 400 throws), such that all three avatar roles are always represented, but which avatar is in which role switches. Order of these role changes and the starting locations of each role were counterbalanced.

Similar modifications of Cyberball have been employed to study social cognition in social anxiety (Fang, Hoge, Heinrichs, & Hofmann, 2014), autism spectrum disorder (Andari et al., 2010), and after traumatic brain injury (Kelly, McDonald, & Kellett, 2014). The VSLT is novel because it incorporates a true neutral player (.33 probability) and includes more role switches and more trials than other versions, providing more data on different types of role switches than other versions, and the VSLT includes solely social rewards (other versions also include monetary incentives). For more information about the development of the VSLT and its instructions, see Appendix. VSLT files are available for use in other studies: https://osf.io/9g56x/

From this task, we examined the throwing behavior of each participant. The main variables of interest were each participant’s number of throws to each avatar during each block and during segments of each block (to capture learning processes over the course of a block), and the number of times each participant received the ball (task performance). Each segment was a sliding window over the throws. The width of the sliding window was selected to be 25 throws with a 5-throw overlap. These parameters were tuned using several initial runs of the algorithm. The final values for the parameters (window size = 25 and overlap = 5) were chosen to balance the tradeoff between having enough data to adequately estimate participants’ learning rates and having enough segments to capture the dynamics of learning rate.

Slightly adapting a method used in numerous other studies (see Frank, Moustafa, Haughey, Curran, & Hutchison, 2007), we also used a Q-learning algorithm (Sutton & Barto, 1998) to computationally model learning rate, or the weight given to new information relative to what a participant had learned from all previous trials. Each participant’s trial-by-trial behavior was fit by this model, in which the expected value (Q) of any state-action pair (where state is the location of the ball, and action is where the ball is thrown) was computed after each time the ball was thrown, either by the participant or any computerized avatar. Let s indicate a particular state, and let a represent a particular action. At each time point, the Q-value of the state-action pair was updated by adding a prediction error multiplied by a learning rate (α) to their previous Q-value, following the equation:

$$Q(s,a)=Q(s,a)+\alpha \ast \left(R+\gamma \ast \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })-Q(s,a)\right),$$

where R is the immediate reward received from taking action a in state s, s′ is the state after transitioning from state s, a′ is the action that maximizes the Q-value in state s′, and γ is the discount factor.

In this algorithm, the prediction error is the difference between the observed reward (R = 1 when the participant receives the ball, and R = 0 in all other states) plus the expected value of their next state (temporally discounted) and their previous Q-value. Here, the discount rate was set to one, indicating no temporal discounting, which is common practice in problems with short time horizons. Higher learning rates indicate more heavily weighting prediction errors to update the expected value of a state-action pair, and lower learning rates indicate relying more on the previous expected value with less influence of recent prediction errors. In line with previous literature (e.g., Frank et al., 2007; Koban et al., 2017; Strauss et al., 2011), we compared the goodness-of-fit of a model with separate learning rates for positive and negative prediction errors and a model with a single learning rate.

The best-fitting parameters were found by entering Q-values into a softmax equation to determine the probability of each action at each time, using the following equation:

$$P(s,a)=\frac{e^{Q(s,A)∕\delta }}{e^{Q(s,A)∕\delta }+e^{Q(s,B)∕\delta }+e^{Q(s,C)∕\delta }},$$

where throwing to each avatar was modeled as a separate action (A, B, or C) and the inverse gain parameter (δ) was fixed to 1 so that Q-values directly corresponded to the probabilities of throwing to each avatar. From these probabilities, the log-likelihood for each participant taking their particular action sequence given a state sequence is calculated using:

$$LL=\log \left(\prod _{t=1}^{T}P(s_t,a_t)\right)=\sum _{t=1}^T\log P(s_t,a_t),$$

where T is the length of the action sequence, and P(s_t, a_t) is the probability of taking action a_t when in state s_t at time t. A brute-force search was then used to maximize this log likelihood, searching over the space from α=0.01 to 1 with a step size of 0.01. Learning rate was calculated in MATLAB.

Questionnaires²

The Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) is a widely-used 20-item self-report scale measuring social anxiety in dyads and groups (e.g., “I find myself worrying that I won’t know what to say in social situations.”). Participants rate how much they endorse each item on a 5-point Likert-type scale ranging from “not at all” to “extremely.” Three of these items are reverse scored, and Rodebaugh, Woods, and Heimberg (2007) have demonstrated that these items do not load onto the same factor as the straightforwardly worded items, and in fact seem to reflect extraversion more than neuroticism. Further, removing these reverse scored items generally improves the psychometric properties of the scale. Following Rodebaugh et al.’s recommendations, only ratings on the straightforwardly worded items were included.

The Depression, Anxiety and Stress Scale – Depression subscale (DASS-21; Lovibond & Lovibond, 1995) is a seven-item self-report scale that assess severity of depression in the past week. Participants rate their endorsement of each item on a 4-point Likert-type scale ranging from “did not apply to me at all” to “applied to me very much or most of the time.”

Procedures

After informed consent, participants were directed to Qualtries to complete a brief demographic questionnaire, the VSLT, questionnaires, and debriefing, in that order. For the VSLT, participants were randomly assigned to one of 36 versions of the task (six different orders of role switches, crossed with six different starting locations/avatars for each role). Participants were compensated $2.50 for their participation in the study, which took less than an hour to complete.

Statistical Analyses

Mixed-effects modeling was used to model the number of throws from the participant to each avatar, and the number of catches the participant made. Mixed-effects modeling was chosen because our outcome measures were taken at multiple time points for each participant, and mixed-effects modeling allows for modeling individual differences with a separate random intercept for each participant. Across all models, random effects included random intercepts for each participant and each location (sequences of role changes were counterbalanced across locations of each avatar; i.e., left, top, and right). Due to possible effects of depression on our outcomes, a fixed effect for depression severity score (as measured by the depression subscale of the DASS-21) was also included, but the depression measure was not included in interaction terms. For all models, block number was modeled as an ordered categorical variable with polynomial contrasts, and orthogonal contrasts were used for all other categorical variables.

First, we aimed to validate the task by determining whether participants were able to discriminate the avatars by their roles and throw the ball accordingly: most to the rewarder, least to the punisher, with the neutral avatar in between. Second, we sought to determine whether social anxiety symptoms affected task performance, defined as succeeding at the task of receiving the ball as much as possible. Third, we tested how social anxiety, avatar role, and previous avatar role affected participants’ frequency of throws to each avatar over the course of each block for the whole game. Finally, we analyzed how social anxiety may relate to changes in learning rate as a function of environmental volatility and certain types of role shifts.

All analyses were also performed with social anxiety split categorically into low and high groups. Participants with SIAS scores less than or equal to three quarters of a standard deviation (10 or under) below the mean of a previous community sample (M=18.8, SD=11.8; Mattick & Clarke, 1998) were included in the low social anxiety group, and those scoring greater than or equal to three quarters of a standard deviation (28 or greater) above the mean were included in the high social anxiety group. Results were largely similar to those described below and are not included to reduce redundancy but are available from the first author.

Results

Descriptive Statistics

In our final sample, SIAS (M = 24.93, SD = 18.38) and depression severity, as measured with the depression subscale of the DASS-21 (DASS-D: M = 4.65, SD = 5.66) exhibited a moderate to strong positive correlation, r = 0.59.

Validating the Task

To validate the task, a mixed-effects model was run predicting the number of throws from the participant to each avatar in each block from fixed effects of block number and each avatar’s role, the interaction of block number and avatar role, and random intercepts for each participant and location. The interaction between block number and avatar role could not be included due to rank deficiency.

As expected, avatar role significantly predicted how frequently participants threw to that avatar: participants threw to the rewarder significantly more frequently than to the neutral avatar (β = 7.97, p < 0.001), and participants threw to the punisher significantly less frequently than to the neutral avatar (β = −5.08, p < 0.001); see Figure 1. For other model effects, including interactions with block number, see Supplementary Online Material

Figure 1.

Learning rate and throwing behavior over the Volatile Social Learning Task. Dotted vertical lines represent role shifts between blocks. A) Throws to each avatar role were grouped into 25-trial bins and averaged across all participants. Error bars represent the 95% confidence interval. B) Learning rate was estimated over 25-trial windows with a 5-trial overlap. Although analyses reported were performed on social anxiety as a continuous variable, social anxiety is split into extreme groups for this figure: participants with SIAS scores less than or equal to three quarters of a standard deviation (10 or under) below the mean of a previous community sample (M= 18.8, SD=11.8; Mattick & Clarke, 1998) were included in the low social anxiety group, and those scoring greater than or equal to three quarters of a standard deviation (28 or greater) above the mean were included in the high social anxiety group.

Task Performance

To assess the effect of social anxiety on task performance, a model was run predicting the number of times the participant received the ball (catches) in each 25-trial segment of the game from fixed effects of the participant’s social anxiety score (sum on the SIAS, standardized), depression score (sum on the DASS-D, standardized), the block number containing that segment, and the position of that segment within its block; and the aforementioned random effects. All two- and three-way interaction terms were included for social anxiety score, block number, and position of segment within block.

Neither SIAS (β = −0.03, p = 0.69) nor DASS-D (β = 0.05, p = 0.42) significantly predicted the number of catches. There was a significant main effect of position of segment within block (β = 0.14, p < 0.001) such that, as a block progressed, participants tended to receive the ball more. There was also a significant cubic effect of block number (β = −0.32, p < 0.001). Visual inspection revealed that catches were highest in block 1, lowest in block 2, increased in block 3, but not to the level of block 1, and then slightly decreased again in block 4. No two- or three-way interactions of SIAS, position of segment within block, and block number were significant. Task performance did not vary as a function of social anxiety symptoms, but it did improve over the course of each block, with some differences in task performance across blocks.

Throws to Each Avatar as a Function of Social Anxiety

To assess whether social anxiety affected the frequency of throws to each avatar, a model was run predicting number of throws from the participant to each avatar during each 25-trial segment of the game from fixed effects of the participant’s social anxiety score, the participant’s depression score, the avatar’s role during that segment, the role of that avatar in the previous block, the position of that segment within its block, and the block number containing that segment; and a random intercept for location but not for each participant, given the model with a random intercept for each participant was not structurally sound (rank deficient). All two-, three-, and four-way interactions were included among fixed effects, excluding depression score and block number, due to rank deficiency, and interactions between current and previous avatar role; see Table 1. This analysis excluded the first block of 100 trials, because avatars did not have previous roles during this block.

Table 1.

Model effects for role by social anxiety analysis, predicting participant’s number of throws to an avatar during a 25-trial bin.

Predictor	Fixed Effect	t	p
Intercept	2.46	38.07	<0.001*
SIAS	7.18 e−5	0.001	0.999
DASS-D	−9.28 e−3	−0.28	0.78
Role (P vs N)	−1.03	−10.61	<0.001*
Role (R vs N)	1.21	12.51	<0.001*
Prev role (P vs N)	−0.75	−7.73	<0.001*
Prev role (R vs N)	0.95	9.79	<0.001*
Position (of bin in block)	0.04	1.84	0.07
Block number (linear)	0.02	0.45	0.65
Block number (quadratic)	−0.06	−1.31	0.19
Role (P vs N) × position	−0.15	−4.23	<0.001*
Role (R vs N) × position	0.33	9.21	<0.001*
Prev role (P vs N) × position	0.09	2.65	0.01*
Prev role (R vs N) × position	−0.16	−4.47	<0.001*
Role (P vs N) × SIAS	−0.06	−0.62	0.53
Role (R vs N) × SIAS	−0.10	−1.02	0.31
Prev role (P vs N) × SIAS	−0.25	−2.56	0.01*
Prev role (R vs N) × SIAS	0.11	1.10	0.27
Position × SIAS	−2.69 e−3	0.11	0.91
Role (P vs N) × position × SIAS	−2.43 e−4	−0.01	0.99
Role (R vs N) × position × SIAS	0.04	1.16	0.25
Prev role (P vs N) × position × SIAS	0.03	0.87	0.39
Prev role (R vs N) × position × SIAS	−0.03	−1.00	0.32
	Random Effect Variance
Location intercept	5.59 e−15

Note: P = punisher, N = neutral, R = rewarder, Prev role = role of that avatar in the previous block, SIAS = Social Interaction Anxiety Scale, DASS-D = depression subscale of Depression Anxiety Stress Scale-21.

The avatar’s current role significantly predicted how frequently the participant threw to that avatar: participants threw to the rewarder significantly more frequently than to the neutral avatar (β = 1.21, p < 0.001), and participants threw to the punisher significantly less frequently than to the neutral avatar (β = −1.03, p < 0.001). The avatar’s previous role was also a significant predictor. Participants threw significantly more to avatars who, in the previous block, had been the rewarder compared to neutral (β = 0.95, p < 0.001). They threw significantly less to avatars who, in the previous block, had been the punisher compared to neutral. (β = −0.75, p < 0.001). We found no main effects of SIAS (β < 0.001, p = 0.999), DASS-D (β = −0.01, p = 0.78) segment position within its block (β = 0.04, p = 0.07), or block number.

Several interactions were also significant predictors of a participant’s throws. The interaction between an avatar’s current role and the position of the segment within the block was significant: as the block progressed, participants threw more to the rewarder compared to the neutral avatar (β = 0.33, p < 0.001) and less to the punisher compared to the neutral avatar (β = −0.15, p < 0.001). As the block progressed, participants threw less to the avatar who had been the rewarder (as compared to neutral) in the previous block (β = −0.16, p < 0.001) and more to the avatar who had been the punisher (as compared to neutral) in the previous block (β = 0.09, p = 0.01). This suggests that the influence of previously learned reward values lessens with greater experience with the current reward values. There was also a significant interaction between a participant’s SIAS score and an avatar’s previous role, as anticipated: participants with higher social anxiety threw less to the avatar who had just been the punisher versus neutral (β = −0.25, p = 0.01), in line with the expectation of greater sensitivity to social punishment (see Figure 2). The interaction between SIAS and previous avatar role was not significant for the contrast of previous rewarder and previous neutral avatar (β = 0.11, p = 0.27). The interactions of SIAS and an avatar’s current role, as well as SIAS and segment position within a block, were not significant. None of the three-way interactions were significant. In summary, the differentiation of throws by avatar role became more extreme over the course of a block, and higher social anxiety symptoms predicted fewer throws to the avatar who had been the punisher in the previous block, suggesting less updating of expectations of social punishment in a more neutral or rewarding direction.

Figure 2.

A) In each 25-trial bin, participants higher in social anxiety were less likely to throw to the avatar that had been the punisher in the prior block, independent of that avatar’s current role. For this figure, participants with SIAS scores less than or equal to three quarters of a standard deviation (≤10) below the mean of a previous community sample (M=18.8, SD=11.8; Mattick & Clarke, 1998) were included in the low social anxiety group, and those scoring greater than or equal to three quarters of a standard deviation (≥28) above the mean were included in the high social anxiety group. Error bars represent 95% confidence intervals. B) Relative to participants with high social anxiety, those with low social anxiety had a lower learning rate over the course of the block after the punishing avatar did not change roles (as compared to when the rewarder became the punisher). “Low SA” lines represent learning rates predicted by our model for participants with SIAS 2 SD below the mean, and “High SA” lines represent model-predicted learning rates for participants with SIAS 2 SD above the mean. Gray shading represents 95% confidence intervals.

Learning Rate

To assess whether participant behavior was better modeled with separate learning rates for positive and negative prediction errors versus a single learning rate, we fitted these two models to each participant’s data (all 25-trial sliding windows over the course of the VSLT, including transitions between blocks) and compared model fit using −2 log likelihood (−2LL) and Bayesian information criterion (BIC). When the optimal learning rate parameters were found using maximum likelihood estimation, the −2LL of each model was identical for each person. Because the single learning rate model had fewer parameters and a lower BIC for all participants, the simpler model was chosen, and a single learning rate was used in all models incorporating learning rate.

To assess whether learning rate differed as a function of social anxiety and environmental volatility, a model was run predicting learning rate estimated over blocks in which the roles of the avatars were volatile (blocks two through four) versus the first block when the roles were stable, depression score, social anxiety score, and the interaction of volatility and social anxiety score. As expected, learning rate was significantly higher in the volatile portion of the task (μ = 0.78, σ = 0.11) than the stable block (μ = 0.72, σ = 0.17, β = 0.063, p < 0.001). Neither DASS, SIAS, nor the interaction of SIAS and volatility significantly predicted learning rate, indicating that base learning rates did not differ for more socially anxious individuals, and the adaptation of learning rate between stable and volatile environments was not significantly different for more socially anxious individuals. To investigate how individuals updated their mental representations of the social environment after a role switch as a function of social anxiety symptoms and the type of role shifts occurring, a mixed-effects model was run predicting each participant’s learning rate in each 25-trial sliding window. Fixed effects included their depression symptom score, social anxiety symptom score, the position of that window within the block (an indicator of time after role switch; the first “position” after a role switch was defined as when the majority of trials in a sliding window occurred after the switch), and the previous roles of the rewarder and punisher; and the model included the aforementioned random effects. Because the previous role of the neutral avatar is dependent on the previous roles of the rewarder and punisher, we opted to exclude the previous role of the neutral avatar from this model to avoid rank deficiency. We also excluded block number to avoid rank deficiency, as certain role switches only occurred between certain blocks. This analysis excluded the first block of 100 trials, because avatars did not have previous roles during this block. All two- and three-way interactions between position in block, social anxiety symptom score, and previous role of an avatar were included, though interactions between previous role of the rewarder and previous role of the punisher were not included due to rank deficiency. Contrasts for the previous role variables were set to compare each role change to no role change (e.g., comparing when the neutral avatar becomes the rewarder to when the rewarder stays the rewarder, and when the punisher becomes the rewarder to when the rewarder stays the rewarder).³

In line with our hypothesis about social anxiety being characterized by faster adjustment of learning rate after the rewarder becomes more punishing, results revealed a significant three-way interaction between SIAS, position of segment in the block, and the previous role of the punisher. After the rewarder became the punisher (as compared to when the punisher did not change), learning rate was lower as the block proceeded for participants higher (vs. lower) in social anxiety (β = −1.7 e⁻⁴, p = 0.01). However, visual inspection of the effects revealed that this interaction was driven only in small part by the expected effect of more socially anxious participants more quickly settling on a rule (decreasing learning rate) after the rewarder became the punisher; it was also driven by a much larger decrease in learning rate for less socially anxious participants over the course of block when the punisher did not change (see Figure 2). This suggests that more socially anxious participants may remain hypervigilant for changing probabilities when the role of the punisher is stable, not showing the possibly adaptive settling on a rule that less socially anxious participants show in this scenario. There were also significant effects subsumed in the interaction, and so not interpreted here, of the position of the segment in that block (β = −7.4 e⁻⁴, p < 0.001) and the interaction of SIAS and position (β = 1.9 e⁻⁴, p = 0.04).

In addition to the contrast between rewarder becoming punisher versus punisher staying punisher, the interaction of position and previous role of the punisher was significant for the contrast of the neutral avatar becoming the punisher versus punisher staying punisher (β = 4.4 e⁻⁴, p = 0.03). When the neutral avatar became the punisher, learning rate was higher as the block proceeded, relative to more of a decrease in learning rate over the course of the block after the punisher stayed in that role.

Discussion

This study used a novel, modified version of Cyberball to assess the effect of social anxiety symptoms on updating mental representations of volatile probabilities of social reward and punishment. The results partially supported hypotheses. Participants were able to distinguish the avatars by their probabilities of social reward and punishment, throwing more frequently to the rewarder than the neutral avatar and more frequently to the neutral avatar than the punisher, suggesting the task was a valid way to demonstrate RL. Moreover, throws to the rewarder increased and throws to the punisher decreased relative to the neutral avatar over the course of a block, as participants had more time to learn these probabilities, providing further evidence that participants learned the probabilities of reward and punishment. Results also supported the hypothesis that social anxiety is characterized by hypersensitivity to social punishment, as participants higher in social anxiety symptoms were less likely to throw to the avatar who had been the punisher (versus neutral) in the prior block, suggesting that they were slower to update their expectations of reward after experiencing social punishment. We did not find support for the hypothesis of hyposensitivity to social reward in social anxiety. Thus, while past probabilities of social punishment interacted with social anxiety to predict throw patterns, current probabilities did not.

Our results also supported the hypothesis that learning rate would differ as a function of social anxiety and the type of role switch occurring. Specifically, we did not find differences in learning rate as a function of social anxiety for either static or volatile portions of the task, and social anxiety did not interact with volatility to predict learning rate. However, estimating learning rates more dynamically over shorter, sliding windows revealed a different pattern of change in learning rate as a function of social anxiety and the type of role shift: when the punisher did not change, more socially anxious individuals showed less of a decrease in learning rate over time than did less anxious individuals, and when the rewarder became the punisher, more socially anxious individuals showed a larger decrease in learning rate over time. Adding to a growing literature on RL of anxious individuals in volatile environments (Browning et al., 2015; Huang et al., 2017), these results suggest that socially anxious individuals may not lower their learning rate appropriately in response to certain, less volatile contexts (i.e., the probability of punishment does not change), but may be quick to settle on a policy that a previously rewarding avatar is now more punishing.

For an example of how this might play out in real life, imagine two people, one socially anxious and one non-anxious, who have taken the same bus to work each morning for a few months. The bus driver tends to be grumpy in the mornings, and although she occasionally offers a smile when they get on the bus, she usually makes rude comments. The non-anxious person remains unfazed by the bus driver’s rudeness and barely notices changes in her demeanor because he has come to think that the driver is unfriendly, and he is not concerned what the driver thinks of him. On the other hand, the socially anxious person, who cares much more about social evaluation, remains vigilant for each smile and rude remark, updating his representation of the driver daily. Despite low environmental volatility, minor deviations matter more; the socially anxious person’s learning rate remains high.

Although most studies that have compared goodness of fit of models with separate learning rates for positive and negative prediction errors versus a single learning rate have found that the two-learning rate model fits better, the current study found a single learning rate model to have better model fit. This is likely due to a key difference in the design of the VSLT, which allows the computerized avatars to take actions, as compared to bandit-style probabilistic learning tasks, in which only the participant can take actions. In our two-learning rate model, the optimal value for the negative learning rate was estimated to be 0.01 (the minimum value possible) for all blocks and participants. Because participants can receive delayed rewards in the VSLT (e.g., if the participant throws the ball to an avatar, who throws to another avatar, who then returns the ball to the participant), our model includes a notion of state, or the location of the ball, as well as an estimation of future rewards (the maximum Q-value for the next state). Including this in the model leads to a very low occurrence of negative prediction errors because the maximum Q-value for the next state is typically greater than the Q-value for the current state (because the best possible action is for the avatar to throw the ball back to the player who just threw the ball to the avatar). As such, a single learning rate model is more appropriate for this task due to its sequential nature.

Our results suggest that socially anxious individuals may have difficulty adapting after someone has rejected them (or they perceive that they have been rejected) and then later becomes more rewarding. This extends previous research on negative biases in social anxiety disorder. Our finding that participants higher in social anxiety were less likely to throw to a previously punishing avatar is in line with theorized and observed attentional and memory biases for threat in social anxiety disorder (Hirsch & Clark, 2004; Mogg & Bradley, 2002). Hirsch and Clark’s (2004) influential information-processing account of social anxiety disorder posits that one factor maintaining the disorder is the tendency to overestimate the probability of future social threat, which our findings suggest might occur due to lack of updating these probabilities after a rejection experience. Attentional bias for (Mogg & Bradley, 2002) and difficulty disengaging from socially threatening stimuli (Buckner, Maner, & Schmidt, 2010) may contribute to this update bias, as preferential allocation of attentional resources may render socially punishing experiences more salient in memory. Similarly, the tendency for socially anxious (compared to non-anxious) individuals to show greater fear conditioning to social stimuli (Lissek et al., 2008; Pejic, Hermann, Vaitl, & Stark, 2013) and impaired extinction of conditioned responses to socially threatening (i.e., angry) faces (Olsson et al., 2013) may also contribute to the difficulties “letting go” of a prior punishment experience.

The findings of hypersensitivity to social punishment may also be relevant to the growing literature on post-event processing in social anxiety, which generally finds evidence of negatively biased recall for social feedback over time (Cody & Teachman, 2010; Koban et al., 2017). The post-event processing literature typically refers to the persistence of negative explicit self-relevant thoughts, and our results suggest that negative implicitly learned rules (e.g., that person does not like me because they barely throw me the ball) may also persist over time. Probing participants’ conscious awareness of the changes in throw patterns would be interesting in future research to determine in what ways the rule learning is implicit versus explicit.

While the hypothesis tied to hypersensitivity to social punishment was supported, we did not find support for the expectations of hyposensitivity to reward. An intriguing possibility to consider in future research is that social anxiety might affect how individuals experience the magnitudes of reward and punishment within the paradigm. Reward and punishment were modeled consistently across all participants (e.g., it was assumed that a probability of 33% reward was experientially equivalent to a probability of 33% punishment), but it may be that more socially anxious individuals experience receiving the ball as less rewarding and not receiving the ball as more punishing than do less socially anxious participants. For socially anxious participants, the cost of being excluded might be much higher than the reward of receiving the ball, so receiving the ball might be better modeled as a lack of punishment (really, a relief) than receiving a reward. Future studies using inverse RL (Ng & Russell, 2000), the process of estimating an agent’s reward structure from observed behavior, may investigate potential social anxiety-related differences in reward function. Another possibly fruitful model to test would be one with a dynamic learning rate parameter estimated on a trial-by-trial basis, rather than over sliding windows of trials (Hauser et al., 2014; Krugel, Biele, Mohr, Li, & Heekeren, 2009; Reiter, Heinze, Schlagenhauf, & Deserno, 2017). Also, qualitative interviews with anxious participants after they have completed the task would be informative.

Clinical Implications

Punishment in the modified Cyberball task (VSLT) is a relatively minimal rejection experience, yet it was sufficient to elicit decreased updating in participants higher in social anxiety. Real social interactions are much richer, with information being communicated through many channels: facial expressions, body language, vocal tone, words, and timing of responses, to name a few. As such, updating expectations may be a potentially fruitful therapeutic target. In a one-on-one therapeutic setting, this might involve exploring negative social expectations about peers or other people in the client’s life and encouraging the client to draw on more recent versus historical interactions when anticipating how these people will treat the client in the future. Computerized cognitive trainings, such as cognitive bias modification, may also consider targeting this update bias by rewarding socially anxious users for more quickly trusting previously threatening others whose behavior has become more rewarding. Determining when to encourage this trust is not simple, however. It is an interesting challenge for both socially anxious and non-anxious individuals to determine when it is healthy to move beyond a prior actual or perceived slight and give someone another chance. The line between adaptive self-protection and maladaptive avoidance is not always obvious, and the literature on forgiveness may have some interesting lessons for how to overcome an updating bias.

Limitations and Future Directions

There are several limitations to this study that should be addressed in future research, including the design of the modified Cyberball task (VSLT) and the analytic approach. The VSLT was designed so that all roles would be represented at all times and all possible role shifts would occur, but this design could be improved (see Supplementary Online Material for more information about task development and design). Between blocks two and three (but not between any of the other blocks), all role shifts held one avatar role constant and switched the roles of the other two avatars, which would arguably create less volatility between blocks two and three. It is possible that this design necessitated less learning and behavior change between these blocks; as shown in Figure 1, there was no large decrease and spike in learning rate between blocks two and three, and throws changed relatively linearly over blocks two and three. This design might also be responsible for the block number effects found in some models (e.g., task performance may have dropped between blocks one and two and between blocks three and four, but increased between blocks two and three, because of less volatility between blocks two and three). This was unintentional and could be changed in future versions of the task.

Because the role shifts occurred at fixed intervals (every 100 trials), it is possible that some participants learned to anticipate when roles would shift, limiting perceptions of volatility (or “unexpected uncertainty”; Yu & Dayan, 2005). However, given the small number of role shifts (three), it seems unlikely that this would have a large effect on our data, as role shifts were not signaled and participants had few opportunities to learn about the intervals between them.

An important limitation of the VSLT is that it is not possible to distinguish computationally between effects on reward sensitivity versus learning. Given the unresolved debate in the RL literature on whether reward-related dysfunctions in depression are driven by aberrant reward learning or reward sensitivity (Robinson & Chase, 2017; Rutledge et al., 2017), it is worth noting that we cannot conclude whether the observed effects are due to biased valuation of social rewards and punishments or biased updating of learned values. Our evidence for biased updating is based on our finding of social anxiety-based differences in learning rate after certain role switches, and our finding that more socially anxious participants threw less to the avatar who had previously been the punisher, regardless of their current role. It could be argued, however, that if the socially anxious participants undervalued social reward, rather than showing a negative bias in updating, the same effect on throws might be observed.

There are limitations in our calculation of learning rate that may be addressed in future research. We selected a 25-trial window for calculating each learning rate, but 25 trials may be insufficient to estimate a valid learning rate. Further, to reduce noise, the temporal discounting parameter in the Q-learning algorithm was fixed, eliminating our ability to detect differences in this parameter. Additionally, while the brute-force search used to find the most likely learning rate underlying participants’ behavior used a very small step size, it did not test all possible parameter values.

Another limitation is that while we measured trait social anxiety with the SIAS, we did not collect any markers of state social anxiety or perceived exclusion during the task (ethnicity and socioeconomic status were also not assessed). Given alterations in learning rate as a function of state positive affect (Bakic, Jepma, De Raedt, & Pourtois, 2014) and other model-based decision-making parameters as a function of acute psychosocial stress (Radenbach et al., 2015), these state markers might differ between groups and could be important in modeling RL. Inclusion of psychophysiological measures throughout the VSLT and/or subjective distress ratings before, during, and after the VSLT would improve our ability to detect state effects of social anxiety on learning. Further, we recruited a non-clinical sample, so it is possible that using a sample seeking treatment for social anxiety disorder might find different results. However, our analysis of social anxiety symptoms as a continuous variable allows us to investigate a fuller range of functioning, from healthy to disordered, in line with the National Institute of Mental Health’s Research Domain Criteria initiative (Insel et al., 2010).

Collecting data online is another possible limitation of our research, given the setting in which participants complete the study cannot be controlled and participants may become distracted during the study. In order to mitigate these risks, comprehension questions were added following VSLT instructions and only highly rated mTurk workers were recruited. Further, participants whose throwing patterns on the VSLT indicated that they were likely not paying attention or were not following instructions were identified and excluded from analyses. Benefits of online data collection for clinical research include fast, inexpensive data collection from large samples, and higher prevalence of social anxiety symptoms than the general population, which was particularly helpful for recruitment for this study (Shapiro, Chandler, & Mueller, 2013).

Conclusion

In summary, a novel paradigm, the Volatile Social Learning Task, was used to assess how updating volatile probabilistic information about social reward and punishment varies as a function of trait social anxiety. Evidence of biased updating in social anxiety was found; more socially anxious participants were less likely to update their negative expectations of players who had previously been punishing. This update bias may contribute to avoidance behavior and maintenance of social anxiety disorder (though that cannot be established from this first study’s correlational design). Computational methods examining individualized learning parameters may improve our understanding of learning processes contributing to psychopathology, and incorporating volatility into learning environments allows us to examine dynamic processes relevant to real-world social interactions.

Table 2.

Model effects for predicting learning rate in the block after a role switch.

Predictor	Fixed Effect	t	p
Intercept	0.96	334.32	<0.001*
SIAS	−4.86 e−4	−0.23	0.82
DASS-D	2.10 e−5	0.01	0.99
Prev role of R (P vs R)	1.05 e−3	0.69	0.49
Prev role of R (N vs R)	2.43 e−3	1.02	0.31
Prev role of P (R vs P)	−2.19 e−4	−0.14	0.89
Prev role of P (N vs P)	−7.32 e−4	−0.31	0.76
Position	−7.41 e−4	−5.63	<0.001*
SIAS × position	1.92 e−4	2.07	0.04*
Position × prev role of R (P vs R)	1.62 e−4	1.30	0.19
Position × prev role of R (N vs R)	2.95 e−4	1.49	0.14
Position × prev role of P (R vs P)	1.36 e−4	1.07	0.29
Position × prev role of P (N vs P)	4.42 e−4	2.21	0.03*
SIAS × position × prev role of R (P vs R)	−4.51 e−5	−0.68	0.50
SIAS × position × prev role of R (N vs R)	−3.56 e−6	−0.04	0.97
SIAS × position × prev role of P (R vs P)	−1.70 e−4	−2.46	0.01*
SIAS × position × prev role of P (N vs P)	−7.15 e−5	−0.70	0.48
	Random Effect Variance
Participant intercept	4.92 e−4
Location intercept	2.08 e−5

Note: P = punisher, N = neutral, R = rewarder, Prev role = role of that avatar in the previous block, SIAS = Social Interaction Anxiety Scale, DASS-D = depression subscale of Depression Anxiety Stress Scale-21.