Dimensionality, Reliability, Invariance, and Validity of the Multidimensional Social Anxiety Response Inventory-21 (MSARI-21)

Abstract

Research shows that social anxiety disorder (SAD) is prevalent in the United States, and may interfere with many aspects of a person’s life. Although numerous psychological instruments have been developed to measure presence and intensity levels of social anxiety, these instruments fail to capture the range of responses individuals utilize to mitigate the negative affect associated with the anxiety, namely alcohol use and anger distress. Recently, the Multidimensional Social Anxiety Response Inventory – 21 (MSARI-21) was developed to address this limitation and increase our understanding of the complexity of social anxiety. We expand on this work by evaluating the psychometric properties of the instrument, using a combination of exploratory structural equation and bi-factor modeling, and item response techniques. Across two studies, data indicated the presence of a strong, 3-factor structure (i.e., Anger Distress, Alcohol Reliance, and Social Avoidance), strong internal consistency, and evidence of both convergent and discriminant validity. In addition, results showed that the MSARI-21 multidimensional structure was invariant across gender. We conclude that the MSARI-21 is a valid and valuable tool for assessing individuals’ responses to social anxiety, and that future research should evaluate the instrument within other samples to ensure its utility across clinical and subclinical populations.

Social anxiety disorder (SAD) is a psychological disorder in which individuals experience feelings of intense fear or anxiety in social situations, largely due to their expectations of being (negatively) evaluated by other people.¹ In addition to intense negative emotional states, SAD can also interfere with an individual by adversely influencing their social, professional, and recreational activities (e.g., Buckner & Heimberg, 2010; Lochner et al., 2003). Unfortunately, SAD is one of the most common psychological disorders within the U.S. (Kessler et al., 2005; Kessler, Chiu, Demler, & Walters, 2005). Given the widespread nature of the disorder and the negative impacts on functioning, SAD has been the focus of attention for clinicians and researchers alike. For example, the fifth Edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5; American Psychiatric Association [APA], 2013) made a number of definitional changes to SAD (Bögels et al., 2010) to aid in clinical research and diagnoses. In addition, several instruments have been developed to better assess the symptoms individuals experience, and identify the social situations or states that may exacerbate the disorder. Examples include the Brief Fear of Negative Evaluation Scale (BFNE; Leary, 1983), the Liebowitz Social Anxiety Scale, (LSAS; Liebowitz, 1987), and the Social Phobia and Anxiety Inventory (SPAI; Turner, Beidel, Dancu, & Stanley, 1989).

Although existing self-report measures of social anxiety provide important screening or assessment information, it is important to note that these instruments often assess different aspects of social anxiety (Modini, Abbott, & Hunt, 2015). For example, many emphasize the degree of distress felt from social anxiety (e.g., the Social Phobia and Anxiety Inventory; SPAI-23, Roberson-Nay, Strong, Nay, Beidel, & Turner, 2007) rather than the responses to that distress. Those that measure behavioral responses to social anxiety (e.g., the Subtle Avoidance Frequency Examination; SAFE, Cuming et al., 2009) typically capture only a single dimension of social anxiety (i.e., frequency of avoidance or safety behaviors). To understand fully the nature of the social anxiety construct, researchers and clinicians must investigate a wider range of emotional and behavioral responses linked to social anxiety across empirically supported dimensions of the construct. That is, individuals with social anxiety may respond in ways other than these safety behaviors or social avoidance, such as reacting with anger or turning to alcohol to cope. Although such behaviors have been linked to social anxiety within the broader literature, they have largely been ignored by current social anxiety measures. To address this gap, Osman, Freedenthal, Acosta, and Pirani (2015) recently developed the Multidimensional Social Anxiety Response Inventory-21 (MSARI-21) to better assess and understand the reasons individuals with social anxiety behave in such ways in response to being observed, criticized, or judged in various social situations. In this project, we build upon this research through an examination of the psychometric properties of the MSARI-21 in two new non-clinical samples. In the next section, we will briefly review the literature on the common affective and behavioral responses to social anxiety that informed the development of the instrument and provide a summary of steps in the construction of the instrument (more detailed analyses are reported in the unpublished manual; Osman et al., 2015).

Social Anxiety and Social Avoidance

One of the most common ways people with social anxiety may respond to potentially distressing social situations is to avoid them altogether. For example, people who experience marked social anxiety may refuse to attend a social event or only make a brief appearance and leaving early to manage their social anxiety symptoms. Indeed, social avoidance has been so closely linked to SAD that it is used in the diagnoses and assessment of the disorder per the DSM-5 (APA, 2013). Several existing measures examining social avoidance, such as the LSAS (Liebowitz, 1987) and the Social Avoidance and Distress Scale (SADS; Watson & Friend, 1969) include measures of social avoidance to determine severity of social anxiety.

Although avoiding potentially stressful social situations may help prevent experiencing SAD symptoms, such behaviors are maladaptive and can have significant negative consequences. For example, many career paths require extensive social interaction situations (e.g., making presentations at formal or informal meetings at a workplace). People who experience social anxiety and consequently avoid these events or situations could be limiting their job opportunities and/or their career advancements (Wittchen & Beloch, 1996). Additionally, such behaviors may lead to other mental health difficulties. In support of this, Moitra, Herbert, and Forman (2008) found that avoidance behavior was a significant mediating factor between SAD and the development of major depressive disorder.

Social Anxiety and Alcohol Use

Individuals who present with social anxiety may also try to cope with their fear or anxiety by turning to the use of illicit or licit substances (in particular, alcohol). Among researchers, the notion that alcohol is used to reduce tension associated with anxiety and serve as a “social lubricant” is not new (see Critchlow, 1986). For those who experience social anxiety, using alcohol to cope with social situations may be a desirable strategy for a number of reasons. As suggested by a recent model of social anxiety and substance use (Buckner, Heimberg, Ecker, & Vinci, 2013), alcohol may increase positive affect when confronted with a social situation, attenuate an individual’s fear of negative appraisals from others, or help facilitate social interactions. Note however, that research does not necessarily suggest that increased alcohol use is a hallmark of increased social anxiety (e.g., Ham, Bacon, Carrigan, Zamboanga, & Casner, 2016; Ham, Zamboanga, Olthuis, Casner, & Bui, 2010). Rather, research indicates social anxiety may be related to alcohol use through both positive and negative alcohol expectancies (Ham et al., 2016).

Ham and colleagues (2010) reported that students who endorsed greater amounts of social anxiety did not engage in as much drinking as students who reported less social anxiety. However, students with social anxiety had stronger expectations that alcohol use would reduce tension compared to students who were not as socially anxious (Ham et al., 2010). Similarly, Buckner and Heimberg (2010) found people who endorsed high levels of social anxiety did not differ in their reports of drinking (in either quantity or frequency) compared to individuals with lower levels of social anxiety, but were more likely to report using alcohol as an aid for social situations. It is important to note that although people who present with social anxiety may not necessarily be drinking more, SAD is related to an increased prevalence of alcohol use disorder (AUD; Morris, Stewart, & Ham, 2005). Indeed, over a 14-year longitudinal study, individuals with SAD were more likely to develop an alcohol dependence condition compared to those without SAD (Buckner et al., 2008).

Social Anxiety and Anger Distress

Research indicates people with anxiety disorders may also be more prone to reacting with feelings of anger or aggression. Moscovitch, McCabe, Antony, Rocca, and Swinson (2008) found that individuals with panic disorders and those with social phobia disorders were more likely to experience feelings of anger compared to control groups. Similarly, another study showed that individuals who met the criteria for generalized anxiety disorder (GAD) reported higher levels of trait anger and hostility compared to those who did not (Deschênes, Dugas, Fracalanza, & Koerner, 2012). Deschênes and colleagues also found that the measures of anger and hostility were significant predictors of the overall severity of the anxiety symptoms. Regarding social anxiety specifically, individuals with social anxiety have been found to spend greater portions of their day feeling angry compared to those without social anxiety (Kashdan & Collins, 2010). In addition to having higher levels of state and trait anger, individuals with social anxiety also display an increased tendency to react with anger to negative events (e.g., being criticized) as well as without direct provocation (Erwin, Heimberg, Schneier, & Liebowitz, 2003).

The link between anger and social anxiety may be due to individuals who present with social anxiety using less effective strategies to regulate their emotions. Research shows that individuals with social anxiety tend to engage in rumination and suppression strategies to regulate their emotional states (e.g., Blalock, Kashdan, & Farmer, 2016; Kocovski, Endler, Rector, & Flett, 2005). Both of these regulation strategies have been found to be ineffective at regulating anger, typically resulting in the anger being maintained or decreasing overall positive affect (e.g., Gross, 2002). This is consistent with Erwin et al. (2003) who found that people with SAD were more likely to engage in suppression strategies to control their anger and were more likely to display higher state and trait anger compared to non-clinical controls.

Brief Overview of the Development of the MSARI-21

Osman and colleagues (2015) noted that one major limitation of most existing self-report instruments is the use of the essential symptoms of social anxiety symptoms (i.e., intensity of fear or anxiety about being criticized or judged). In addition, other instruments tend to be composed of specific situations (e.g., participating in a small group event and asking an attractive person of the opposite sex for a date) that elicit anxiety-related responses (e.g., avoidance) or assess a relatively narrow range of responses (e.g., safety behaviors to minimize or conceal symptoms). Accordingly, Osman and colleagues (2015) noted the need for a measure that would assess several content-specific responses and the thoughts behind these responses within individuals with SAD. Although a battery of existing psychological instruments could theoretically be developed to investigate such responses and their underlying causes, Osman and colleagues (2015) sought to develop a more concise and specific measure in order to minimize participant burden and fatigue.

Osman and colleagues (2015) conceptualized several dimensions (i.e., social avoidance, anger distress, and alcohol reliance) for enhancing our understanding of the functioning of individuals who present with SAD. Specifically, social avoidance refers to a general tendency to avoid social situations. Anger distress refers to a general tendency to experience internalized anger within social situations. Finally, alcohol reliance refers to an individual’s tendency to use alcohol as a means to alter their behaviors and control their anxiety in social situations. The authors highlighted the importance of assessing these affective and behavioral responses of social anxiety and the underlying thoughts behind these responses, asserting such a measure would be invaluable to the understanding of the nature of SAD. Since maladaptive responses to SAD can exacerbate the disorder or contribute to the development of other psychological problems (e.g., substance abuse, loneliness, depression), a measure that can quickly identify and assess content-specific responses and underlying thoughts could be instrumental in SAD treatment.

Following the conventional qualitative steps (i.e., generating items from multiple sources, reviewing items for content specificity, clarity, and relevancy), Osman et al. (2015) developed the MSARI-21 across two studies, examining the factor structure of the instrument as well as convergent and discriminant validity.² The analyses for each of these studies are available in detail in an unpublished manual (Osman et al., 2015). Results from these studies provided strong empirical support for a 3-factor model (i.e., anger distress, alcohol reliance, and social avoidance), and each factor was comprised of seven items.

Overview of Objectives and Planned Analyses

The goal of the current project was to use modern statistical techniques to further examine the psychometric properties of the MSARI-21 instrument. Specifically, we identified the following objectives to guide this project:

1. Examine confirmatory data for the multidimensional structure of the MSARI-21 items (Study 1; N = 629).

2. Re-examine evidence of internal consistency reliability for scores on the MSARI-21 scales by calculating internal consistency reliability estimates using alternative estimation methods (Study 1; N = 629).

3. Examine measurement invariance across male and female study participants for the items within each MSARI-21 scale (Study 1).

4. Establish evidence of convergent and discriminant validity estimates for the MSARI-21 scale scores using empirically grounded measures available within the literature (Study 2).

For the first objective, we adopted exploratory structural equation modeling (ESEM) and bifactor modeling techniques. An advantage of ESEM is that it incorporates aspects from both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) to overcome inherent limitations in either technique. That is, unlike the conventional CFA strategy, ESEM allows items to load freely across factors and thus is less restrictive and allows for interpretable item-factor loadings or factor intercorrelations (Marsh, Morin, Parker, & Kaur, 2014).

We conducted ESEM with Mplus version 7.4 (Muthén & Muthén, 1998–2011), comparing the fit of two models (a unidimensional model and the proposed three-factor oblique model) using a robust maximum likelihood estimator (MLR) and oblique geomin rotation. In addition to χ² values, we also assessed model fit using the following indices and suggested cutoffs: Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI), values ≥ .95 (Schermelleh-Engel, Moosbrugger, & Müller, 2003); Root Mean Square Error of Approximation (RMSEA), values ≤ .08 (Browne & Cudeck, 1993); and Standardized Root Mean Square Residual (SRMR), values ≤ .08 (Hu & Bentler 1999).

We also estimated an ESEM bifactor model, in which items are specified to load on their respective factors (i.e., specified subscales) as well as on a “general” factor (i.e., an underlying factor of “general social anxiety”). This procedure allowed us to calculate two additional indices to interpret the dimensionality of the instrument: the explained common variance (ECV) and the percentage of uncontaminated correlations (PUC). The ECV refers to the proportion of common variance that is due to the underlying general factor relative to the common variance explained by the group factors (Reise, 2012). PUC refers to the ratio of the number of “uncontaminated” matrix correlations (i.e., correlations due solely to the general factor) to total number of unique correlations. High ECV and PUC values suggest stronger influence of the general factor, and thus suggest unidimensionality, whereas lower values suggest multidimensionality. Reise, Scheines, Widaman, and Haviland (2013) suggest using an ECV benchmark of .60 when PUC values are less than .80.

The second objective served to extend previous work with the MSARI-21 by calculating internal consistency reliability estimates for the scale scores. Although Cronbach’s coefficient-α (Cronbach, 1951) is typically used for this purpose, it may often be an inaccurate estimate due to unrealistic assumptions. That is, coefficient-α can underestimate reliability when the assumption of tau-equivalence is violated, and overestimate when the assumption of uncorrelated error variances is violated (Raykov & Marcoulides, 2015). Due to these potential issues, we calculated point and interval estimates for coefficient-ω (McDonald, 1999), which does not hold these assumptions. This measure of internal consistency reliability was calculated using the R package ‘coefficient alpha’ (Zhang & Yuan, 2016).

For the third objective, we assessed measurement invariance across male and female participants. To ensure items on the MSARI-21 scales have the same interpretation across these groups, we conducted differential item functioning (DIF) analyses using Item Response Theory for Patient Reported Outcomes (IRTPRO) Version 4.2 for each of the three scales (Cai, Thissen, & du Toit, 2011). This approach allowed us to calculate the DIF (χ ² c|a) and fit statistics (S – χ²) for each item for both males and females.

In Study 2, we examined evidence for convergent and discriminant validity estimates of the MSARI-21 scale scores by assessing the associations between the MSARI-21 individual scale scores and scores on well-established self-report measures of drinking motives, anger expression, social phobia and anxiety, and other psychiatric symptoms found within the broader literature. We predicted that the MSARI-21 scale scores would be moderately-to-highly correlated with scores on measures of similar constructs. To this end, we used SPSS version 22 and R software (version 3.3.3) to examine the pattern of associations between scores on the Anger Distress, Alcohol Reliance, and Social Avoidance scales of the MSARI-21 to scores on other empirically established measures of expressions of anger, drinking behaviors, and social anxiety.

Study 1

Method

Participants

Following institutional review board (IRB) approval, the SONA-Qualtrics web platform was used to administer the questionnaires to undergraduate students at a large Southwestern state university. The questionnaire could be completed within 30–35 minutes. Accordingly, of the initial pool of 640 participants, data for individuals (n = 11) completing the study within five minutes or less were not included in the current analyses. The final sample with complete item-level data included 629 participants (408 females, 221 males, M = 19.78 years, SD = 3.44 years). The self-reported ethnic background of this sample was: 47.7% Hispanic, 28.1% Caucasian/White, 7.6% African American, 9.4% Asian, 4.9% biracial/multiethnic, and 2.2% Middle Eastern.

Procedure

Measures

Multidimensional Social Anxiety Response Inventory (MSARI-21; Osman et al., 2015).

The MSARI-21 is a 21-item measure that assesses the affective and behavior responses and underlying reasoning for these responses within individuals with SAD. Participants read over several reactions to social evaluations/interactions and rate how well each describes their own thoughts, feelings or behaviors, using a 5-point scale from 1 (strongly disagree) to 5 (strongly agree). The measure consists of three 7-item scales: Anger Distress, Alcohol Reliance, and Social Avoidance. The Anger Distress scale assesses the extent to which the individual responds to socially anxious situations by experiencing internalized anger in response to being evaluated (negatively or positively) and specific reasons for this behavior (e.g., feeling judged by others). The Alcohol Reliance scale assesses the extent to which the individual uses alcohol to cope with feelings of social anxiety and underlying reasons (e.g., alcohol improves self-presentation). The Social Avoidance scale assesses avoidance and specific reasons for avoiding social situations or interactions (e.g., uncertainty about being called upon to say or do something). For copyright reasons, permission was granted for reproducing only abbreviated items (see Table 5).

Table 5.

Standardized factor loadings for Study 2 (N = 250)

	MSARI-21
Abbreviated Item	Anger Distress	Alcohol Reliance	Social Avoidance
1. I get mad easily…in front of other people.	.73
2. I get intensely angry…when I say or do something.	.80
6. I feel like telling people off…comments about me.	.61
9. I feel intensely annoyed…comments about what I say or do.	.68
14. My anger toward people…be quite intense.	.74
16. I experience intense feelings…failings.	.74
19. I feel the urge…in a social situation.	.77
5. Drinking alcohol helps…talking with people.		.76
8. Drinking alcohol before… comfortably with other people.		.88
10. It is helpful to me…before a social event.		.92
12. I have better control…contains alcohol.		.78
17. I am only able…a drink containing alcohol.		.63
20. Having a drink…my presentation or performance.		.88
21. It usually works best…social gathering.		.89
3. I avoid social events because…be expected to do or say.			.74
4. I avoid social get-togethers because…myself to other people.			.78
7. I often find ways…about my true feelings.			.80
11. I avoid social encounters because…in front of other people.			.69
13. I avoid social…when I am around people.			.86
15. I avoid social events…other people.			.79
18. I often wiggle my way…other people.			.84

Note. All factor loadings significant at p < .001; χ ² [186] = 91.37, p < 0.001; CFI = 0.92; TLI = 0.91; RMSEA = 0.08, 90%CI: .07 .09; SRMR = 0.05.

Results

ESEM and Bifactor Models

Fit indices for the tested models are shown in Table 1. The one-factor model showed very poor fit for the sample data (χ ² [189] = 4,678.71, p < .001; CFI = 0.38; TLI = 0.31; RMSEA = 0.19, LLCI: .19 ULCI: .20; SRMR = .21). However, the three-factor oblique model showed good fit (χ ² [150] = 391.46, p < .001; CFI = 0.97; TLI = 0.95; RMSEA = 0.05, 90%CI: .04.06; SRMR = .02). In addition, as shown in Table 2, scale items in this model showed strong loadings on their predicted factors, with minimal cross-loadings. The bifactor model (χ ² [132] = 272.57, p < .001; CFI = 0.98; TLI = 0.97; RMSEA = 0.04, 90%CI: .03 .05; SRMR = .02) also demonstrated good fit to the sample data. As noted previously, the bifactor model allows the researcher to calculate bifactor-specific indices (ECV = .38, PUC = .70). Taken as a whole, these results provide further evidence for the multidimensionality of the MSARI-21, and that each of the three scales should be scored separately rather than combined into a total score.

Table 1.

Model Fit Statistics for Study 1 and Study 2

							90% CI for RMSEA
	χ2	df	CFI	TLI	SRMR	RMSEA	LL	UL
Study 1
One-Factor	4678.71*	189	.38	.31	.21	.19	.19	.20
Three-Factor	391.46*	150	.97	.95	.02	.05	.04	.06
Bifactor	272.57*	132	.98	.97	.02	.04	.03	.05
Study 2
Three-Factor	491.37*	186	.92	.91	.05	.08	.07	.09

^*Note. p < .001;

CFI = Comparative Fit Index; TLI = Tucker Lewis Index; SRMR = Standardized Root Mean Square Residual; RMSEA = Root Mean Square Error of Approximation.

Table 2.

Standardized Factor Loadings for Study 1(N = 629).

Item	One-factor model	Three-factor model			Bifactor model
		AngD	AlcR	SocA	General	AngD	AlcR	SocA
1	.41	.67	.04	−.04	.22	.64
2	.44	.72	−.05	.01	.17	.69
6	.39	.74	.06	−.12	.24	.70
9	.51	.78	.05	−.01	.28	.75
14	.52	.80	−.05	.05	.21	.76
16	.53	.83	−.01	.01	.25	.78
19	.49	.79	−.03	.01	.22	.74
5	.42	.01	.82	.05	.82		.30
8	.37	.01	.89	−.04	.85		.35
10	.39	−.01	.92	−.02	.90		.13
12	.44	−.00	.85	.06	.88		−.08
17	.47	.01	.64	.18	.72		−.18
20	.38	−.01	.91	−.03	.89		−.01
21	.46	−.00	.91	.05	.93		−.02
3	.73	.01	−.03	.78	.24			.76
4	.72	−.12	.04	.82	.28			.79
7	.81	.04	−.03	.85	.28			.79
11	.76	.22	.01	.64	.31			.59
13	.81	−.00	.01	.88	.32			.82
15	.77	.04	−.02	.80	.28			.74
18	.80	−.01	.00	.86	.30			.79

Note. AngD = Anger Distress; AlcR = Alcohol Reliance; SocA = Social Avoidance. Factor loadings > .40 denoted in bold.

Internal Consistency Reliability

Given the evidence for multidimensionality, we calculated the internal consistency reliability estimates and related 95% confidence intervals for each scale score of the MSARI-21. As shown in Table 3, all three-scale scores demonstrated excellent estimates of internal consistency reliability.

Table 3.

Descriptive Statistics and Reliability Estimates for Study 1 and Study 2

Scale	M	SD	ω [95% CI]
Study 1 (N = 629)
Alcohol Reliance	12.91	6.95	.95 [.94-.96]
Social Avoidance	14.03	6.69	.93 [.92-.94]
Anger Distress	16.53	6.53	.90 [.89-.92]
Study 2 (N = 250)
Alcohol Reliance	10.78	5.98	.94 [.92-.96]
Social Avoidance	13.18	6.67	.92 [.90-.94]
Anger Distress	15.56	6.58	.90 [.89-.91]

Note. M = mean; SD = standard deviation; ω = coefficient-ω; CI = confidence interval.

Measurement Invariance

To determine if items within each scale would be useful across important demographic groups, we examined item functioning across males and females for each scale separately. The complete DIF and fit statistics are displayed in Table 4. To assess adequacy of the discrimination parameters, we used the range of values recommended by Baker and Kim (2004). Within the Anger Distress scale, we found that the discrimination parameters were very high for males, ranging from 2.01 (Item 1) to 3.79 (Item 14). Similarly, the discrimination parameters were high to very high for females, ranging from 1.63 (Item 1) to 3.55 (Item 16). Furthermore, the comparative analysis found no evidence of DIF between the groups for the Anger Distress scale items.

Table 4.

Fit statistics for differential item functioning analysis

		Fit Statistics
		Malesa				Femalesb				DIF Statistics
Scale	Item	a(SE)	S – X2	df	p	a(SE)	S – X2	df	p	X2c|a	p
Anger Distress	1	2.01 (0.24)	50.74	39	0.10	1.63 (0.30)	69.67	60	0.18	5.7	0.22
	2	2.42 (0.28)	50.83	37	0.06	1.96 (0.34)	68.95	57	0.13	4.8	0.32
	6	2.06 (0.25)	66.29	46	0.03	1.85 (0.33)	69.53	61	0.21	4.2	0.38
	9	3.05 (0.36)	42.08	34	0.16	2.53 (0.44)	58.73	50	0.19	6.9	0.14
	14	3.79 (0.50)	38.39	30	0.14	2.80 (0.50)	39.65	46	0.73	2.7	0.62
	16	2.98 (0.36)	43.24	35	0.16	3.55 (0.60)	44.25	42	0.38	2.3	0.69
	19	3.24 (0.43)	32.99	29	0.28	2.71 (0.48)	54.28	42	0.10	1.8	0.78
Alcohol Reliance	5	3.14 (0.37)	39.27	38	0.41	3.80 (0.36)	48.43	43	0.26	2.5	0.65
	8	4.75 (0.60)	43.93	33	0.09	4.37 (0.42)	56.25	33	0.01	3.3	0.50
	10	5.53 (0.76)	38.50	28	0.09	6.38 (0.74)	43.47	27	0.02	11.3	0.02
	12	3.93 (0.48)	54.02	32	0.01	5.02 (0.54)	49.14	31	0.02	1.8	0.78
	17	2.59 (0.35)	44.12	30	0.05	2.77 (0.31)	57.34	34	0.01	0.5	0.97
	20	5.34 (0.72)	39.87	24	0.02	5.37 (0.59)	38.48	30	0.14	0.5	0.97
	21	5.73 (0.83)	36.73	25	0.06	8.86 (1.40)	20.80	22	0.53	1.0	0.91
Social Avoidance	3	2.27 (0.27)	37.80	36	0.39	2.57 (0.22)	41.96	44	0.56	2.3	0.68
	4	2.20 (0.27)	48.94	38	0.11	2.62 (0.23)	52.64	47	0.26	1.0	0.91
	7	3.44 (0.45)	29.06	27	0.36	3.45 (0.33)	52.85	40	0.08	2.0	0.73
	11	2.75 (0.37)	62.62	35	0.01	2.21 (0.20)	104.72	48	0.00	1.3	0.87
	13	3.81 (0.52)	24.09	26	0.57	3.59 (0.36)	35.00	35	0.47	1.5	0.83
	15	3.02 (0.39)	34.99	29	0.20	2.85 (0.26)	53.74	41	0.09	0.8	0.94
	18	2.96 (0.37)	40.35	33	0.18	3.34 (0.32)	44.67	40	0.28	7.5	0.11

Note. DIF = differential item functioning; a = discrimination parameter; SE = standard error; S – X² = item fit statistics; df = degrees of freedom. Values shown in bold represent items where significant DIF was potentially detected.

^an = 221

^bn = 408

Within the Alcohol Reliance scale, the discrimination parameters for the items were very high for both males and females, ranging from 2.59 (Item 17) to 5.73 (Item 21) and 2.77 (Item 17) to 6.38 (Item 10), respectively. The analysis revealed mixed evidence for Item 10 (“It is helpful to me to have a drink containing alcohol before a social event”), as the DIF statistic approached significance for this single item. Further, the item-level diagnostic statistics for this item was significant for females, suggesting that this item may be performing differently between genders. However, we decided to retain this item for three reasons. First, the DIF only approached significance; second, retaining/dropping a single item would not substantially impact the overall score of the Alcohol Reliance scale; lastly, we opted to retain the item to maintain the content validity of the scale.

Within the Social Avoidance scale, the discrimination parameters for the items were very high for both males, ranging from 2.20 (Item 4) to 3.81 (Item 13) and females, ranging from 2.21 (Item 17) to 3.45 (Item 7). Similar to the Anger Distress scale, there was no evidence of DIF for any of the Social Avoidance scale items.

Study 2

Method

Participants & Procedure

The final sample for Study 2 was 250 undergraduate participants (161 females, 89 males, M = 19.74 years, SD = 4.29 years), after removing data for eight participants who completed the questionnaires within 5 minutes or less. The self-reported ethnic background information for this sample was: 54.4% Hispanic, 18.8% Caucasian/White, 12% African American, 8.4% Asian, 3.2% biracial/multiethnic, 1.2% Middle Eastern, 1.2% other ethnicities, and .8% American Indian/Indigenous. All participants provided informed consent and completed several questionnaires including the MSARI-21, a demographic questionnaire, and several concurrent instruments discussed in the measures subsection.

Measures

Drinking Motives Questionnaire, Revised (DMQ-R; Cooper, 1994).

The DMQ-R is a 20-item instrument that assesses different motives individuals may have towards using alcohol. Participants read over the different reasons people may drink, and indicate how often each is a factor in their own drinking behavior, using a 5-point scale ranging from 1 (almost never) to 5 (almost always). The instrument is composed of four 5-item subscales, tapping into different drinking motives (social, coping, enhancement, and conformity). Social motives include using alcohol as a way to celebrate, or improve special occasions or social events. Coping motives include using drinking as a way to ignore or reduce one’s negative thoughts and feelings. Enhancement motives include drinking to enhance a positive mood. Conformity motives include using alcohol as a means to “fit in” with others as well as to avoid negative social repercussions such as ridicule and rejection.

State-Trait Anger Expression Inventory (STAXI-2; Spielberger, 1999).

The 32-item Anger Expression scale of the STAXI-2 assessed how participants experience, express and control their anger. This scale is composed of four 8-item subscales: the Expression-Out (AX-O), Anger Expression-In (AX-I), Anger Control-Out (AC-O), and Anger Control-In (AC-I). For each, participants indicate how often they engage in specific reactions or behaviors when they feel angry, using a 4-point scale ranging from 1 (almost never) to 4 (almost always). The AX-O subscale captures the extent individuals express their anger through outward behaviors, such as slamming doors or yelling at those around them. The AX-I subscale measures the extent individuals try to “bottle up” their anger. The AC-O subscale measures how often individuals attempt to control their anger by actively monitoring and controlling their behaviors. The AC-I subscale captures the extent that individuals attempt to control their anger by making themselves calm down or “cool off.” Scores on these subscales are also used to calculate an overall anger expression index (AX index) for each individual.

Social Phobia and Anxiety Inventory (SPAI-23; Roberson-Nay et al., 2007).

The SPAI-23 is a 23-item measure of social (16 items) and agoraphobic anxiety (7 items). Items on the agoraphobic scale assess anxiety that arises from perceptions of the setting (e.g., open spaces, public transportation) rather than the interactions with others. Participants report how often they experience various thoughts or feelings when responding to the questionnaire items, using a 5-point scale ranging from 0 (never) to 4 (always). Scores on the agoraphobia subscale can be subtracted from scores on the social anxiety scale to provide a more accurate measure of SAD, (Turner, Beidel, & Dancu, 1996).

Symptom Assessment 45 Inventory (SA-45; Maruish, Bershadsky, & Goldstein, 1998).

The SA-45 is a multidimensional checklist of psychiatric symptoms. The instrument asks individuals to rate how much they were bothered by each symptom over the last week, using a 5-point scale from 1 (not at all) to 5 (extremely). It is composed of nine 5-item subscales: anxiety (e.g., “Feeling tense or keyed up”), depression (e.g., “Feeling no interest in things”), hostility (e.g., “Shouting or throwing things”), interpersonal sensitivity (e.g., “Feeling that people are unfriendly or dislike you”), obsessive compulsive disorder (e.g., “Having to check and double-check what you do”), paranoid ideation (e.g., “Feeling that most people cannot be trusted”), phobic anxiety (e.g., “Feeling afraid in open spaces or on the streets”), psychoticism (e.g., “Hearing voices that other people do not hear”), and somatization (e.g., “Soreness of your muscles”). Scores on each subscale are summed to create a total score. The subscales of the SA-45 were used to examine evidence of convergent and discriminant validity for the MSARI-21 scale scores.

Results

Item factor loadings and fit indices of the 3-factor solution for the MSARI-21 in Study 2 are reported on Table 5

Convergent and Discriminant Validity

The final aim was to examine convergent and discriminant validity of the MSARI-21. Specifically, we examined the correlations between scores on each MSARI-21 scale and scores on established, widely used instruments that assess the same or similar constructs. We expected moderate-to-high (r ≥ .40) correlations between participants’ scores on: 1) the Anger Distress scale with the STAXI-2 anger expression (AX-I, AX-O, and AX Index) and the SA-45 hostility scale scores; 2) the Alcohol Reliance scale with the DMQ-R subscale scores related to improving mood or social events (Enhancement and Social) and reducing negative thoughts (Coping); and 3) the Social Avoidance scale with scores on the SPAI-23 (social phobia and difference scores) and SA-45 phobic anxiety scale scores. For discriminant validity, we predicted weak or low (r < .40) correlations between scores on the MSARI-21 scales and scores on measures of dissimilar constructs. In addition, we assessed discriminant validity using dependent correlational analyses. The complete correlations are presented in Table 6.

Table 6.

Correlations between MSARI-21 scales and Concurrent Measures

		MSARI-21
Measures	Scales	Anger Distress	Alcohol Reliance	Social Avoidance
STAXI-2
	AX-O	.34**	.08	.04
	AX-I	.44**	.18**	.43**
	AC-O	−.34**	−.04	−.09
	AC-I	−.27**	.03	−.09
	AX Index	.49**	.09	.24**
SPAI-23
	Social Phobia	.39**	.24**	.72**
	Agoraphobia	.24**	.14	.39**
	Difference Score	.36**	.22*	.70**
SA-45
	Hostility	.50**	.10	.17**
	Phobic Anxiety	.32**	.14*	.49**
DMQ-R
	Social	.06	.62**	−.03
	Coping	.27**	.64**	.12
	Enhancement	.08	.56**	.02
	Conformity	.14*	.37**	.12

Note. STAXI-2 = State-Trait Anger Expression Inventory; AX-O = Anger Expression-Out; AX-I = Anger Expression-In; AC-O = Anger Control-Out; AC-I = Anger Control-In; AX Index = Anger Expression Index; SPAI-23 = Social Phobia and Anxiety Inventory; SA-45 = Symptom Assessment. Correlations ≥ .40 denoted in bold.

^*p < 0.05

^**p < .01

Anger Distress.

We first examined the correlations between scores on the MSARI-21 Anger Distress scale with scores on the STAXI-2 measure. As expected, the Anger Distress scale had moderate positive correlations with scores on the Anger Index (r = .49) and the AX-I (r = .44) scale and a weak negative relationships with the AC-O (r = −.34) and AC-I (r = −.27). However, the Anger Distress scale had a weak positive relationship with scores on the AX-O scale (r = .34). In addition, we also calculated dependent correlation tests (Steiger, 1980) to better examine evidence of discriminant validity for the Anger Distress scale. The Anger Distress scale was found to be more strongly related to the AX-I scale score than the AC-I scale score, t(249) = 8.95, p < .001. Scores on the Anger Distress scale were also found to be more strongly associated with AX-O scale scores than the AC-I scale scores, t(249) = 6.49, p < .001, and more related to Anger Index than AC-I scale scores, t(249) = 7.04, p < .001. Lastly, the SA-45 features a hostility scale score that was also found to be more strongly related to the Anger Distress scale score when compared to the AC-in scale score, t(249) = 8.84, p < .001.

Alcohol Reliance.

Next, we examined the correlations between scores on the MSARI-21 Alcohol Reliance scale with scores on the DMQ-R. The MSARI-21 Alcohol Reliance scale score was highly correlated with the DMQ-R Social Motives scale scores (r = 0.62), Coping Motives scale scores (r = 0.64), and Enhancement Motives scale scores (r = 0.56), all p’s < .05. The Conformity Motives scale scores of the DMQ-R were weakly correlated with scores on the Alcohol Reliance scale (r = 0.37). These findings support our hypotheses and provide evidence for convergent validity. Again, the dependent correlation test was used to help establish evidence for discriminant validity. This test showed that the MSARI-21 Alcohol Reliance scale score was more related to the DMQ-R Social Motives scale score, t(249) = 4.85, p < .001, the Coping Motives scale score, t(249) = 5.5, p < .001, and the Enhancement Motives scale score, t(249) = 3.29, p = .001, when compared to the Conformity Motives scale score.

Social Avoidance.

We also examined the correlations between scores on the MSARI-21 Social Avoidance scale with scores on the SPAI-45. As predicted, the Social Avoidance scale score was found to be highly correlated with the SPAI-45 Social Phobia scale score (r = 0.72), the SA-45 phobic anxiety scale score (r = 0.49) and weakly correlated with the Agoraphobia scale score (r = 0.39). Additionally, the difference scale score (a more accurate assessment of social phobia) was found to be highly correlated (r = 0.70) with the MSARI-21 Social Avoidance scale score. To establish discriminant validity, Steiger’s (1980) dependent correlation test showed that the Social Avoidance scale score was more related to the Social Phobia scale score than to the Agoraphobia scale score, t(249) = 7.79, p < .001.

Lastly, we examined the correlations between the Anger Distress and Alcohol Reliance scales with scores on the SPAI-23. Because these dimensions were conceptualized as responses to social anxiety, we expected the scores to be positively correlated. Scores on the Anger Distress scale were positively correlated with Social Phobia scale scores (r = 0.39). Similarly, scores on the Alcohol Reliance scale were positively correlated with Social Phobia scores, (r = 0.24).

General Discussion

Although individuals with social anxiety tend to avoid social situations, they may respond in other ways, namely reacting with internalized expressions of anger or relying on alcohol. However, current instruments have largely ignored these potential response tendencies. Osman and colleagues (2015) developed the MSARI-21 to better examine these behavioral responses and increase our understanding of social anxiety as a construct. The current project built upon this development by assessing the multidimensional nature of this new instrument, calculating estimates of internal reliability, assessing measurement invariance across genders, and assessing convergent and discriminant validity. In Study 1, we examined the multidimensional nature of the instrument using an SEM and bifactor modeling approaches. Results from the SEM and bifactor model revealed strong evidence for the multidimensionality of the instrument, suggesting that scores on each of the three scales should be calculated independently rather than summed to create an overall total inventory score. The analyses in Study 1 also revealed that each of the three MSARI-21 scale scores had adequate estimates of internal reliability estimates for use in research and clinical settings (all estimates ≥ .90).

The third objective of the study was to investigate measurement invariance of the MSARI-21 instrument. Because of the reported gender differences in the presentation of social anxiety (both SAD and endorsement of social anxiety symptoms; Ranta et al., 2007; Xu et al., 2012), Study 1 assessed invariance across males and females using a DIF technique. These analyses revealed a potential issue for only a single item (Item 10 from the Alcohol Reliance scale) across all three scales. That is, the DIF statistics approached significance for only one of the 21 items of the MSARI-21. Given that Item 10 merely approached statistical significance, and dropping this item would not substantially alter the scores on the Alcohol Reliance scale, we recommend retaining this item in future investigations. The results of Study 1 provide strong evidence for the utility of the MSARI-21 across gender. However, research suggests that experiences and displays of social anxiety may be influenced by factors associated with race and/or ethnicity (e.g., Hoffmann, Asnaani, & Hinton, 2010; Okazaki, Liu, Longworth, & Minn, 2002). To provide evidence that the MSARI-21 is a robust measure for all racial, ethnic, and cultural groups, future validation analyses (e.g., factorial invariance via CFA; DIF) should be conducted across these different groups as well.

In Study 2, we examined the structure of the MSARI-21 within a new sample and assessed the relationships between scores on the MSARI-21 scales and scores on established self-report measures of related and unrelated constructs. In addition to support for the three-factor structure of the measure, we found evidence for the convergent validity of the MSARI-21 scales scores. That is, scores on the Anger Distress scale were moderately-to-strongly positively related to other measures of anger and hostility (and negatively related to measures of anger control). Similarly, scores on the Alcohol Reliance scale (which assesses the extent to which individuals use alcohol to better cope with social situations) were positively related to drinking alcohol in order to improve mood or reduce negative thoughts and feelings. Scores on the Social Avoidance scale (which assesses the extent to which individuals attempt to avoid negative social situations altogether) were strongly related to scores on measures of social phobia.

Study 2 also found a positive relationship (r = .39) between Anger Distress scores and Social Phobia scores on the SPAI-23. This is in line with Breen and Kashdan (2011) who found a positive relationship between social anxiety and state anger (r = .25). Although the current project found a stronger relationship between anger and social anxiety, this is to be expected given that the Anger Distress scale items assess responses specific to social anxiety. Our findings of a positive correlation (r = .24) between Alcohol Reliance scale scores and Social Phobia are also in line with Schry and White (2013) who found a positive relationship between social anxiety and both positive and negative alcohol expectancies (r = .15 and r = .16 respectively).

Limitations and Future Research Directions

Several limitations must be considered when interpreting the findings of the current project. For example, it is important to note that some characteristics of the present sample may limit the generalizability of these findings to other groups. For Studies 1 and 2, we had data from relatively young (mean age < 21), non-clinical, predominantly Hispanic college student samples. Although symptomatic experiences of social anxiety are generally common in this age group, clinical diagnoses of SAD are far less so (see Grant et al., 2005; Purdon, Antony, Monteiro, & Swinson, 2001). Thus, the need to extend the methodologies adopted here to data obtained from clinical inpatient and outpatient samples. In addition, there is still a need to examine invariance testing of the MSARI-21 at the scale-level. Future studies could investigate this using other measurement invariance techniques, such as multiple-groups CFA.

Another potential limitation relates to the nature of the MSARI-21 instrument. As the instrument is designed to assess both the responses to social anxiety and the underlying reasons behind these responses, the items may be fairly complex. For example, it is possible that participants may endorse the response (“I avoid social situations”) without endorsing the underlying reason (“because I am uncertain how to present myself”) and may experience some confusion on how best to respond. This could impact the results and reliability of the instrument. However, such complexity may be necessary given the instrument’s purpose of assessing the responses and underlying reasoning for these responses within individuals. In addition, the high reliability estimates found in Study 1 (all estimates ≥ .90) may suggest that participants are not experiencing a high degree of uncertainty.

Previous research has linked endorsement of social anxiety to adverse alcohol related outcomes within non-clinical samples of college students (Gilles, Turk, & Fresco, 2006). Furthermore, research demonstrates that subclinical levels of anxiety can adversely influence the performance of an individual such as on working memory tasks (see Moran, 2016). Given the extensive cognitive demands placed upon college-age students, future investigations might examine the extent to which scores on this scale differentiate the responses of students who present with moderate to high level of social anxiety symptoms and those less influenced by the process of being evaluated in social situations (such as giving a presentation or working in a group).

Future research should also expand upon validating scores on this measure in a number of ways. First, as the current project collected cross-sectional data, we did not examine evidence for test-retest reliability or stability of the MSARI-21 scale scores over time. Data collected at multiple time points to more thoroughly assess for these measures of consistency would be beneficial to the further validation of the measure. Additionally, although the current study found evidence of convergent validity with the SPAI-23, future research should also compare scores on the MSARI-21 with other measures of social anxiety, including behavioral assessments from observers.