The Social Anxiety and Depression Life Interference—24 Inventory: Classical and modern psychometric evaluations

Abstract

Two instrument validation studies broadened the research literature exploring the factor structure, internal consistency reliability, and concurrent validity of scores on the Social Anxiety and Depression Life Interference—24 Inventory (SADLI-24; Osman, Bagge, Freedenthal, Guiterrez, & Emmerich, 2011). Study 1 (N = 1065) was undertaken to concurrently appraise three competing factor models for the instrument: a unidimensional model, a two-factor oblique model and a bifactor model. The bifactor model provided the best fit to the study sample data. Study 2 (N = 220) extended the results from Study 1 with an investigation of the convergent and discriminant validity for the bifactor model of the SADLI-24 with multiple regression analyses and scale-level exploratory structural equation modeling. This project yields data that augments the initial instrument development investigations for the target measure.

1. Introduction

Social anxiety disorder (SAD) is generally indicated by persistent and extreme fear or anxiety of one or more evaluative social situations, which provokes at least six consecutive months of related anxiety and avoidant behavior (Diagnostic and Statistical Manual of Mental Disorders [DSM-5], American Psychiatric Association [APA], 2013). The DSM-5 (APA, 2013) also requires that persons seeking treatment for SAD demonstrate considerable life interference across several events prior to being assigned a formal diagnosis. Clinical research shows that individuals present with SAD as early as mid-adolescence (Sumter, Bokhorst, & Westenberg, 2009; Klein, 2009; Van Oort, Greaves-Lord, Verhulst, Ormel, & Huizink, 2009; Weeks, Coplan, & Kingsbury, 2009) and many continue to endure the negative consequences of the disorder into adulthood (Kessler, Berglund, Demler, Jin, & Walters, 2005; Magee, Eaton, Wittchen, McGonagle, & Kessler, 1996). In many cases, individuals who present with SAD also report low life satisfaction (Hambrick, Turk, Heimberg, Schneier, & Liebowitz, 2003) and severe depressive symptomatology (Beesdo et al., 2007). Moreover, the lifetime prevalence rate for SAD may be as high as 14.3% in the general population (Essau, Conradt, & Petermann, 1999; Kessler et al., 2005; Wittchen & Fehm, 2001).

An important aspect in assigning formal diagnosis of social anxiety disorder is confirmation that the essential symptoms of the disorder interfere significantly with the functioning of the individual. To date, however, existing self-report measures of the social anxiety disorder construct tend to focus entirely on the essential symptoms of the disorder. Unfortunately, self-report measures targeting daily interference from SAD are not readily available for use by clinicians or researchers. In fact, in a recent review of the psychometric properties of over 70 widely used trait measures of social anxiety, Modini, Abbott, and Hunt (2015) concluded that not a single existing instrument met basic psychometric standards. In particular, limitations were noted with respect to inadequate content validity, internal consistency, criterion validity, construct validity, reliability (defined as the ability to consistently distinguish between patients with high vs. low levels of social anxiety), reproducibility of scores, responsiveness (defined as the ability to detect clinically important changes in social anxiety over time), mitigation of floor and ceiling effects, and ease of interpretability (Modini et al., 2015). Measures of life interference will enhance our understanding of the impact of the social anxiety disorder symptoms on the individual (APA, 2013).

Upon identifying the former need, Osman et al. (2011) developed the Social Anxiety and Depression Life Interference—24 (SADLI-24) Inventory to remove two critical barriers preventing the widespread use of psychometric tools for detecting SAD interference. First, the SADLI-24 includes two subscales that separate depressive symptomology from symptoms specific to SAD. Moreover, SAD often co-occurs with depression (Brown, Chorpita, & Barlow, 1998; Cho & Telch, 2005; Kessler, Stang, Wittchen, Stein, & Walters, 1999; Ranta, Kaltiala, Pelkonen, & Marttunen, 2009; Watson, Gamez, & Simms, 2005) and consequently, the SADLI-24 aids clinical diagnostic accuracy with items differentiating the two internalizing disorders. Second, unlike traditional self-report measures, the SADLI-24 includes items assessing the level of life interference caused by SAD, instead of measuring the severity or intensity of the disorder. Accordingly, the SADLI-24 is a valuable assessment tool that compliments existing self-report severity measures. For the current project, we conducted two studies to extend findings of the original instrument development investigations reported by Osman et al. (2011), which are briefly reviewed below.

1.1. Instrument development studies for the SADLI-24

In the initial instrument development study, Osman et al. (2011) generated an initial pool of 58 potential items that captured internalizing behaviors relevant to the constructs of social anxiety and depression. This initial pool was reduced to 30 items after the authors inspected the item contents for specificity, clarity, relevancy, and construct representativeness. For the second study, the authors recruited 438 high school and undergraduate students to examine the factor structure and internal consistency of the scale scores. Six additional items were removed after repeated exploratory factor analyses, resulting in a two-factor inventory with 24 items. All items are scaled from 1 (strongly disagree) to 5 (strongly agree). Each factor (Social Anxiety Life Interference [SALI-12] and Depression Life Interference [DLI-12]) is composed of 12 items (inter-factor correlation, r = .61; range of primary factor loadings: .51 to .94). A sample item from the SALI-12 subscale reads, “In general, how much of the time has fear or concerns about what other people think of you in social situations interfered with your normal daily life?” A sample item from the DLI-12 subscale reads, “In general, how much of the time have feelings of being depressed, down, or sad interfered with your normal daily life?” In addition, the related subscale scores demonstrated adequate internal consistency reliability.

For the third instrument development study, Osman et al. (2011) recruited a confirmatory sample of 430 undergraduates and replicated the two-factor structure of the SADLI-24 using confirmatory factor analysis. The authors also used receiver operating curve analysis to derive an empirically supported diagnostic cut point for the SALI-12 and DLI-12 subscale scales. Results showed that raw scores of the 24 items were able to differentiate between clinical-level social anxiety (SALI-12) and depressive symptomology (DLI-12) for the majority of the study participants. Study Three also incorporated several relevant concurrent measures and showed evidence for the discriminant and convergent validity of the SADLI-24 subscale scores using the validation self-report measures.

For the fourth instrument development study, Osman and colleagues administered the SADLI-24 items to two samples of adolescents (M_age = 15.62 years), who were inpatients at a state psychiatric hospital. One group of participants exhibited internalizing symptoms (n = 56) whereas the other group exhibited externalizing symptoms (n = 56). The authors used participants’ scores on the SADLI-24 to examine evidence for known-groups validity and reliability for the specific subscale scores. The final instrument development study demonstrated adequate test–retest validity with an independent clinical inpatient sample of adolescents.

1.2. Total study objectives

The current project extends the initial instrument development studies in several critical ways. Given the strong correlation between the SALI-12 and DLI-12 subscale scores (r = .61; Osman et al., 2011), in Study 1 we utilized a confirmatory bifactor model strategy to investigate the extent to which the subscale scores represent a single latent construct. Study 1 also included empirical evaluations of the convergent validity, discriminant validity, and internal consistency reliability of the SADLI-24 subscale scores. The analyses in Study 2 broadened findings from Study 1, with central aims of delineating the nomological network of the instrument and of assessing the concurrent and discriminant validity estimates of the SADLI-24 subscale scores.

2. Objectives for Study 1

Study 1 addresses three specific objectives. The first objective was to strengthen evidence of validity for the SADLI-24. We performed multiple regression analyses with scores on the SADLI-24 subscales serving as predictors of scores on two criterion measures: The Positive and Negative Affect Scale (PANAS; Watson, Clark, & Tellegen, 1988) and The Social Phobia and Anxiety Inventory—23 (SPAI-23; Roberson-Nay, Strong, Nay, Beidel, & Turner, 2007). Each concurrent measure assesses constructs that are theoretically related to SAD (i.e., scores on the SALI-12) and depression (i.e., scores on the DLI-12). More specific predictions regarding the relationships between the indicators and criterion measures are outlined in the subsequent sections.

The second objective of Study 1 was to compute coefficient-ω (McDonald, 1999; Zinbarg, Revelle, Yovel, & Li, 2005) for the SADLI-24 subscale scores and the concurrent measures. The instrument development studies reported coefficient-α (Cronbach, 1951; Raykov & Marcoulides, 2015) and composite reliability (Raykov, 1997) to show evidence of internal consistency reliability with a different estimation method. For comparative purposes, we also provided estimates of coefficient-α.

The third and final objective of Study 1 was to explore the instrument’s latent factor structure using confirmatory factor analysis (CFA). More specifically, we used CFA to compare the fit of three possible factor structures for the SADLI-24: a one-factor model, a two-correlated factors model (as specified in the original studies), and a bifactor model. Due to the strong association between the SALI-12 and DLI-12 recorded in the previous studies, we reasoned that the variance captured by the SADLI-24 may actually reflect a shared latent construct tapped by both subscale scores, and not necessarily two separate latent constructs. That is to say, perhaps the items measuring life interference reflect a respondents’ perception of impairment from general psychological distress, instead of intrusions resultant from either social anxiety or depression.

We tested this prediction within a bifactor framework, which allowed us to specify a model with a central, latent “general distress” dimension. Each bifactor-inventory item contributes variance to two factors where the first factor represents a specific factor (or subscale) and the second more general factor comprises variance from all of the inventory items. Any specific factors that remain stable after controlling for the contributions of its items to the general factor are considered discrete dimensions of a multidimensional instrument, whereas specific factors that are completely represented by the general factor are no longer considered distinct dimensions (Reise, Moore, & Haviland, 2010). Notably, the bifactor model also necessitates that the specific and general factors be orthogonal to each other.

3. Method for Study 1

3.1. Participants and procedure

We combined data from research settings in the Southeast and Midwest to maximize parameter estimates and to increase the generalizability of the study results. Specifically, using secure online survey format, participants ages 18 to 29 years who self-identified as college-age students (n = 855) were included in the initial pool. As noted above, we recruited an additional 210 participants to increase the heterogeneity and size of the study sample; the second sample included participants ages 30 and older who self-identified as non-students.

The full sample with complete item-level data included 328 women and 737 men, ranging in age from 18 to 49 years (M_age = 22.85, SD = 7.50). Three hundred and sixty-eight (368; 34.6%) participants self-identified as being Caucasian/White (368; 34.6%), 402 (37.7%) as being Hispanic/Latino American, 104 (9.8%) as being African American, 98 (9.2%) as being Asian American, 66 (6.2%) as being biracial, and 25 (2.3%) as being Middle Eastern. Additionally, of the sample, 828 (77.7%) were single, never married, 125 (11.7) were married, 43 (1.0%) had “live-in” partners, 41 (3.8%) were engaged, 15 (1.4%) were divorced/annulled, 8 (0.8%) were separated, and 5 (0.5%) were widowed. All participants completed the SADLI-24, a brief background information questionnaire, and self-report measures of affect and social anxiety. Institutional review boards at both institutions approved the study protocol.

3.1.1. PANAS

The PANAS is a 20-item, multidimensional instrument that includes two subscales: one subscale captures positive affect (PANAS-PA) and one subscale captures negative affect (PANAS-NA). Respondents rate the extent to which they endorse each affective indicator on a 5-point Likert-type scale ranging from 1 (very slightly) to 5 (extremely). Example positive affect items include, interested, excited, active, and strong. Example negative affect items include upset, nervous, guilty, and irritable.

3.1.2. SPAI-23

The Social Phobia and Anxiety Inventory—23 (SPAI-23; Roberson-Nay et al., 2007) is made up of three subscales, and is often used as a measure of social anxiety and agoraphobia in clinical and nonclinical settings. The social anxiety subscale (SPAI-SA) is composed of 16 items (e.g., “I feel anxious when speaking in a small informal meeting”), and the agoraphobia subscale is composed of seven items (e.g., “I feel anxious when riding in a car”). Items are scored on a 5-point scale ranging from 0 (never) to 4 (always). In obtaining the SPAI-23 Difference score, the Agoraphobia subscale score is subtracted from the Social Phobia subscale score. Taken together, the Difference score is used as a “pure” form of the SAD construct.

4. Results for Study 1

4.1. Data analytic plan

To complete our first objective of Study 1, we performed multiple regression analyses in SPSS 22 to test the relative contributions of the SALI-12 and DLI-12 to the variance in the PANAS and SPAI-23 subscale scores.¹ Mean scores on the subscales of the concurrent measures and the subscales of the SADLI-24 (rated on a frequency scale with endpoints where 1 is Rarely and 5 is Every day) represented each variable in the regression analyses. Predictions about the relationships between the concurrent and target measures are outlined below.

Proceeding from a hypothesis that life intrusions caused by symptoms of depression and social anxiety would exert additive (negative) effects on affect, we predicted that scores on the SALI-12 and DLI-12 subscale scores should each be a unique, positive predictor of the negative affect subscale score of the PANAS and a unique, negative predictor of the positive affect subscale score of the PANAS. Given that the DLI-12 is a measure of depression and the SALI-12 is a specific measure of social anxiety life interference, we also expected that the DLI-12 subscale score would be a more powerful indicator of the PANAS subscale scores than would the SALI-12. With respect to the SPAI-23, we anticipated that each SADLI-24 subscale score would positively predict the three subscale scores of the SPAI-23. However, the SALI-12 subscale score should be a stronger predictor of scores on the SPAI-SP and SPAI-DIFF subscales than the DLI-12. Because the SPAI-AG is not a specific measure of social anxiety, we expected its associations with scores on the SALI-12 and DLI-12 subscales to be undifferentiated.

To address objective two, we used a structural equation modeling approach in the Mplus 7.2 program (Muthén & Muthén, 2012) to derive point estimates for internal consistency reliability for the SADLI-24 subscale scores and the concurrent measures. We also computed bootstrapped confidence intervals (95% confidence interval [CI]) with 2,000 sample iterations. All estimates were computed on the individual (unidimensional) scale scales with uncorrelated error terms. We also calculated omega hierarchical (ω_h; McDonald, 1999) to estimate the proportion of scale variance attributable to the general factor.

To address objective three, we examined the fit of the sample data to the specified CFA models using the maximum likelihood estimation with robust standard errors (MLR) in Mplus (Muthén & Muthén, 2012). Because we expected a high correlation between the factors, we selected an oblique rotation for the two-factor analysis. We set the following criteria to identify acceptable factor models: robust comparative fit index (R-CFI) values >.90; root mean square error of approximation (RMSEA) values and standardized root mean square residual (SRMR) values ≤.06. After evaluating the absolute fit of the three models, we used chi-square differences tests to determine which of the three nested models most precisely reproduced the observed data matrix.

Within the bifactor modeling, we used bifactor-specific statistics to evaluate the performances of the item and subscale scores (see Reise, Bonifay, & Haviland, 2013; Reise et al., 2010). In particular, we also reported coefficient-omega hierarchical (ω_h), or the proportion of the variance in total scale scores that are attributable to a general distress factor. We also calculated the explained common variance (ECV) for the bifactor model to define the proportion of variance in the factor loadings encompassed by general distress. Regarding factor loadings, for the unidimensional and two-factor models, items with factor loadings λ ≥ .40 on one factor and λ ≤ .30 on any secondary factors, were considered acceptable indicators. For the bifactor model, items could cross-load as important specific and/or general indicators.

4.2. Reliability and validity analyses

Descriptive statistics for the study measures and results of the internal consistency reliability analyses are included in Table 1; the results from the regression analyses are presented in Table 2. Each of the SADLI-24 subscale scores (the SALI-12 and the DLI-12) demonstrated strong internal consistency reliability according to coefficient-α and coefficient-ω. The SALI-12 had point coefficients of .95 with 95% CIs between .94 and .95 for both reliability estimates. For the DLI-12, the point estimates and associated 95% CIs for coefficient-α (α = .92, [95% CI: .91, .93]) were slightly higher than the equivalent values for coefficient-ω (ω = .91 [95% CI: .90, .92]).

Table 1.

Descriptive statistics and internal consistency reliability estimates of scale scores: Studies 1 and 2.

Scale	M	SD	Min	Max	α [95% CI]	ω [95% CI]
Study 1 (N = 1065)
SALI-12	25.31	10.39	12	60	.95 [.94, .95]	.95 [.94, .95]
DLI-12	25.05	8.98	12	59	.92 [.91, .93]	.91 [.90, .92]
PANAS-PA	33.21	7.82	11	50	.88 [.86, .89]	.90 [.89, .91]
PANAS-NA	21.26	7.67	10	45	.87 [.86, .88]	.88 [.87, .89]
SALI-12	25.31	10.39	12	60	.95 [.94, .95]	.95 [.94, .95]
DLI-12	25.05	8.98	12	59	.92 [.91, .93]	.91 [.90, .92]
SPAI-SP	22.91	12.48	0	64	.94 [.94, .95]	.95 [.94, .95]
SPAI-AG	4.77	4.75	0	21	.84 [.82, .85]	.85 [.84, .86]
Study 2 (N = 220)
SALI-12	27.01	10.34	12	56	.94 [.92, .95]	.94 [.93, .95]
DLI-12	26.99	9.65	12	56	.90 [.87, .91]	.91 [.89, .93]
APPQ-SP	10.74	7.32	0	34	.87 [.84, .90]	.90 [.88, .92]
APPQ-AG	6.3	5.82	0	27	.83 [.79, .86]	.89 [.86, .91]
APPQ-INT	1.77	2.92	0	16	.80 [.73, .85]	.84 [.79, .88]
BFNE	34.82	9.67	14	60	.88 [.85, 90]	.88 [.86, .90]
SA45-ANX	8.88	3.74	5	21	.71 [.65, .77]	.76 [.70, 80]
SA45-DEP	10.6	5.13	5	25	.88 [.85, .91]	.89 [.85, .91]
SA45-HOS	7.38	3.41	5	25	.78 [.69, .85]	.83 [.78, .88]
SA45-INT	10.05	4.66	5	24	.85 [.81, .88]	.86 [.83, .88]
SA45-OCD	12.65	4.97	5	24	.81 [.77, .85]	.83 [.80, .86]
SA45-PAR	9.93	4.35	5	22	.75 [.69, .79]	.80 [.76, .84]
SA45-PHO	6.84	2.82	5	20	.71 [.61, .80]	.81 [.75, .86]
SA45-PSY	7.13	2.92	5	22	.65 [.52, .75]	.74 [.64, .80]
SA45-SOM	8.62	4.02	5	23	.80 [.74, .85]	.84 [.80, .87]
SAD-Q-5	13.83	4.01	8	28	.84 [.80, .87]	.87 [.85, .90]

Notes. M = Mean; SD = Standard Deviation; Min = Minimum Score; Max = Maximum Score; α = coefficient-a; ω = coefficient ω; CI = Confidence Interval; PANAS-PA = Positive and Negative Affect Scale — Positive Affect; PANAS-NA = Positive and Negative Affect Scale — Negative Affect; SALI-12/11 = Social Anxiety Life Interference; DLI-12 = Depression Life Interference; DLI-12D = Interference from General Distress; SPAI-SP = Social Phobia and Anxiety Inventory—Social Phobia; SPAI-AG = Social Phobia and Anxiety Inventory—Agoraphobia; APPQ-SP = Albany Panic and Phobia Questionnaire—Social Phobia; APPQ-AG = Albany Panic and Phobia Questionnaire—Agoraphobia; APPQ-INT = Albany Panic and Phobia Questionnaire—Interoceptive Fear; BFNE = Brief Fear of Negative Evaluation Scale; SA45-ANX = Symptom Assessment 45—Anxiety; SA45-DEP = Symptom Assessment 45—Depression; SA45-HOS = Symptom Assessment 45—Hostility; SA45-INT = Symptom Assessment 45—Interpersonal Sensitivity; SA45-OCD = Symptom Assessment 45—Obsessive Compulsive Disorder; SA45-PAR = Symptom Assessment 45—Paranoia; SA45-PHO = Symptom Assessment 45—Phobic Symptoms; SA45-PSY = Symptom Assessment 45—Psychosis; SA45-SOM = Symptom Assessment 45—Somatization; SAD-Q-5 = Social Anxiety Distress Questionnaire—5.

Table 2.

Results from multiple regression analyses predicting scores on concurrent measures, Study 1 (N = 1065).

Scale	F	R2 [95% CI]	Predictor	t	B	β	Zero-order r	Partial r
PANAS-PA	147.07	.22 [.17, .26]	SALI-12	−9.91	−.09	−.09	−.35	−.08
			DLI-12	−11.42	−.40	−.40	−.46	.33
PANAS-NA	430.5	.45 [.40, .49]	SALI-12	3.00	.09	.09	.48	.09
			DLI-12	20.54	.61	.61	.67	.53
SPAI-SP	603.12	.53 [.49, .57]	SALI-12	26.12	.71	.71	.73	.63
			DLI-12	0.92*	.03	.03	.48	.03
SPAI-AG	243.04	.31 [.50, .57]	SALI-12	9.42	.31	.31	.51	.28
			DLI-12	9.32	.31	.31	.51	.28
SPAI-DIFF	339.71	.39 [.34, .44]	SALI-12	22.07	.68	.69	.62	.56
			DLI-12	−3.43	−.11	−.11	.33	−.11

Notes. All F tests were significant, p ≤ .001; all correlations were significant, p ≤ .01; all t values were statistically significant at p < .01 except those indicated by *; see Table 1 for full scale names.

For the multiple regression (validity) analyses, the observed correlation between scores on the SADLI-24 scales was r = .64, p < .001. R² values ranged from .22 to .53 (average R² = .38), suggesting that the SADLI-24 subscale scores accounted for a substantial proportion of variance in scores on the selected concurrent measures. With the exception of the non-significant relationship between the SPAI-SP and DLI-12 subscale scores (controlling for SALI-12 subscale score), scores on the SALI-12 and DLI-12 subscale scores were each significant and unique predictors of the criterion variables.

The relationships between the predictors and dependent variables were also in the expected directions and the incremental validity of the SALI-12 and DLI-12 subscale scores aligned with our former predictions. Additionally, as expected, the scores on the SPAI-SP and SPAI-DIFF subscales evidenced stronger associations with the SALI-12 subscale score than with the DLI-12 subscale score. Conversely, and as predicted, the DLI-12 accounted for significantly more of the variance in the PANAS-PA than did the SALI-12 (t = 6.93, p < .001). A similar pattern emerged for the PANAS-NA (t = 13.63, p < .001). Finally, the respective variance accounted for in the SPAI-AG subscale score by the SALI-12 and the DLI-12 subscale scores did not differ significantly (t = 0.93, p = .354). Taken together, these results support the convergent and discriminant validity of the SADLI-24 subscale scores as well as the specific validity of scores on the SALI-12 as an indicator of social anxiety life interference.

4.3. CFA

Tables 3 (fit indices) and 4 (factor loadings) contain the results of the CFAs. Notably, according to the two-correlated factor model, the latent inter-factor correlation between the SALI-12 and DLI-12 factors (r = .67) was higher than the association reported in Osman et al., 2011. Compared to the unidimensional (one-factor) and two-correlated factor models, the bifactor model demonstrated the strongest absolute and relative matches to the observed data. Though the unidimensional and two-factor model did not meet the criteria we set to define acceptable models, the bifactor model did. We note that the bifactor model yielded a non-zero test of model fit, which may be due the established sensitivity of χ² to sample size. Accordingly, the study data indicate that the SADLI-24 is better represented and interpreted with a bifactor structure, and not a two-factor structure.

Table 3.

CFA fit statistics for competing factor models for SADLI-24.

	Uni.	Two-factor	Bifactor
χ2	3242.11	1509	1226.60
df	252	251*	228*
R-CFI	0.75	0.89	0.92
SRMR	0.09	0.05	0.04
RMSEA [90% CI]	.11 [.10, .11]	.07 [.06, .07]	.06 [.06, .07]

Notes. Uni. = one-factor model; two-factor = two factor model with correlated factors; Bifactor = bifactor model with three orthogonal factors; explained common variance = .64; χ² = chi-square test of model fit; df = degrees of freedom; R-CFI = robust comparative model fit; SRMR = standardized root mean square residual; RMSEA = root mean square error of approximation; CI = confidence interval; for bifactor model: ω_h = .77 [95% CI: .74, .80]; models noted by * are smaller than the preceding model at p < .001.

In addition to the favorable model fit estimates for the bifactor model, the data produced tenable reliability and item slope parameters for the common latent factor. In particular, the general distress factor explained 64% of the variance common among the 24 scale items (ECV = .64) and also accounted for a high proportion of the variance in the total SADLI-24 scale scores (ω_h = .77 [95% CI: .74, .80]). A closer inspection of the factor loading patterns for the bifactor model suggested inclusion of the general distress factor as well as a specific social anxiety (SALI-12) factor, but did not support a specific depression (DLI-12) factor, and may be due in part to our imposition of a general distress factor. All 24 of the SADLI-24 items demonstrated a high capacity for detecting general distress (common factor loading range: .43 to .78). Moreover, 11 of the 12 SALI subscale items were capable of capturing the extent to which individuals experience routine life intrusions as a direct consequence of social anxiety (λ range: .44 to .67). Only Item 7 (“fear or concerns about your heart beating too hard or fast when you are in most new or social situations”) fell below the cut-point of .40 (λ = .29) for indicating high content specificity, though this item performed adequately at discriminating high and low levels of interference from general distress (λ = .50). Finally, each of the 12 DLI-12 subscale items became complete indicators of the general factor when the common dimension was included in the structural model (DLI-12 λ range: −.26 to .34). This finding is not particularly surprising, as depressive symptom items tend to show high overlap with indices of general distress and life interference.

5. Discussion of Study 1

The CFA results highlight the exceptional quality of the SADLI-24 items. Several factor loadings were well above the established cut-point of .40, with most items (k = 11) exhibiting the capacity to precisely detect the presence of not one but two latent traits (e.g., social anxiety and general depressive life interference). Results from the CFAs also show that most of the variance in depression is allocated to a general distress factor whenever a general factor is specified in the model. Nonetheless, a latent social anxiety specific factor also emerged from variance in 11 of the original SALI-12 subscale items.

Given the high overlap frequently reported between depression and anxiety, it is notable that only one SALI-12 subscale item [Item 7] had high loadings on the general factor with the Depression subscale items. The focal item targets physiological arousal, which is traditionally a specific indicator of anxiety. Based on these results, we recruited an independent sample and performed analyses of reliability and validity for two original SADLI-24 subscales. Though beyond the scope of the current study, researchers may consider performing additional empirical psychometric evaluations to examine the viability of a new 11-item SALI subscale that excludes item 7. As an alternative scoring method, scale users intent on identifying interference from general psychological dysfunction may consider computing and interpreting a total inventory score for the SADLI-24.

6. Objective for Study 2

The purpose of Study 2 was to further evaluate the convergent and discriminant validity of the SADLI-24 subscales. We investigated the potential for shared variance of the subscale scores with a number of conceptually relevant concurrent measures and also assessed the extent to which the DLI-12 and SALI-12 are differential predictors of these measures.

7. Method

7.1. Participants and procedure

Participants were 220 community adults (M age 33.26, SD 9.50, range 18–49 years) recruited from the Midwest. Men (n = 80; 36.4%) and women (n = 140; 63.6%) did not differ significantly in age, t (218) = 0.91, p = .36. A range of ethnic groups was represented in the sample, with 121 (55.0%) participants self-identifying as Caucasian, 25 (11.4%) self-identifying as African American, 41 (18.6%) as Hispanic American, 20 (9.1%) as Asian American, 11 (5.0%) as bi-racial, and 2 (0.9%) as Native American/Alaskan. In terms of marital status, 112 (50.9%) participants indicated that they were married, 57 (25.9%) were single, 16 (7.3%) indicated “live-in partner”, 14 (6.4%) were separated, 9 (4.1%) were engaged, 7 (3.2%) were widowed, and 5 (2.3%) were divorced. In terms of education, 108 participants (49.1%) indicated high school diploma or general equivalency diploma as highest level of education attained, while 55 (25%) had attained an associate’s degree, 33 (15%) a vocational trade diploma, and 24 (10.9%) had attained a bachelor of arts or advanced post-graduate degree.

After providing their informed consent, participants completed a packet of questionnaires including the SADLI-24, a brief demographic questionnaire, and several concurrent self-report measures. All study measures were arranged randomly in each questionnaire packet. Senior undergraduates, who were enrolled in advanced research experience courses, carried out data collection.

7.2. Measures

7.2.1. Albany Panic and Phobia Questionnaire (APPQ; Rapee, Craske, & Barlow, 1994)

The original APPQ is composed of 27 items designed to assess fear of sensation-producing activities and fear of agoraphobic and social phobic situations. We used a modified 24-item version (APPQ-24), retaining the same dimensions of the instrument (APPQ-24 Agoraphobia; APPQ-24 Interoceptive Fear; and APPQ-24 Social Phobia).² The APPQ-24 items are scored using a 5-point Likert-type scale (0 = no fear at all to 4 = extreme fear). We used the APPQ-24 subscale scores to examine evidence for the convergent and discriminant validity estimates of the SADLI-24 subscale scores.

7.2.2. Brief Fear of Negative Evaluation Scale (BFNE; Leary, 1983)

The BFNE comprises 12 items that are modified to assess a social anxiety-related construct, fear of negative evaluation in social situations. The BFNE items are scored on a 5-point scale, ranging from 1 (not at all true of me) to 5 (extremely true of me). This instrument has strong reliability and validity estimates, and the total scale scores have been used widely in research in the area of social anxiety for clinical and nonclinical samples (Collins, Westra, Dozois, & Stewart, 2005; Duke, Krishnan, Faith, & Storch, 2006; Rodebaugh et al., 2004). Four of the 12 items were reverse-scored, as recommended by the developer of the instrument. An example item is “I am afraid that others will find fault with me.” We included the BFNE as a measure of fear of negative evaluation in validating scores on the SADLI-24 scales.

7.2.3. Symptom Assessment—45 (SA45; Davison et al., 1997; Maruish, Bershadsky, & Goldstein, 1998)

The SA45 is a 45-item self-report questionnaire that we included in this study to assess several types of psychopathology. This instrument is composed of nine 5-item scales that were used to tap separate dimensions of psychopathology, including anxiety (SA45-ANX), depression (SA45-DEP), hostility (SA45-HOS), interpersonal sensitivity (SA45-INT), obsessive–compulsive symptoms (SA45-OCD), paranoia (SA45-PAR), phobic symptoms (SA45-PHO), psychosis (SA45-PSY), and somatization (SA45-SOM). Each item is rated using a 5-point scale, with responses ranging from 1 (not at all) to 5 (extremely). Scores within each scale are summed to obtain a total scale score. Studies have reported estimates of reliability (coefficient alpha estimates typically range between .71 and .87 in nonclinical samples) and validity (e.g., associations with measures of depression and related constructs) for scores on the SA45 (e.g., Davison et al., 1997; Maruish et al., 1998). In this study, we used scores from all nine scales as validation measures to identify potential correlates of the SADLI-24 scales.

7.2.4. SAD-Q-V (Osman, 2013–2015)³

The Social Anxiety Distress Questionnaire — V (SAD-Q-V) is an unpublished self-report instrument that assesses social anxiety as described in the DSM-5 (APA, 2013). The first four items tap the intensity of essential symptoms, such as irrational and severe anxiety about social or performance situations. These items are scaled from 1 (i.e., “Never”) to 4 (i.e., “Very Often” or “Almost Always”). Items five, six, and seven measure the level of interference, distress, and cultural aspects associated with social anxiety, respectively. The scaling for these three items ranges from 1 (i.e., “Never” or “No Distress”) to 4 (i.e., “Almost Always” or “Severe Distress”). The last two items are multiple-choice questions to which the respondent indicates the social situation that evokes the greatest anxiety and when the social anxiety became problematic, both of which provide clinical utility. The sum of ratings from items 1 through 7 provides an index of social anxiety. A sample item from the measure reads, “To what extent did you experience fear or anxiety on entering or about to enter social situations (in the last six months)?” We used the SAD-Q-V index to examine differential correlates with the SADLI-24 subscale scores.

8. Results and discussion for study two

8.1. Data analytic plan

To explore the associations of the two identified SADLI-24 scale scores with other psychological constructs, we used multiple linear regression. Each concurrent scale was entered as a separate criterion variable, and the SALI-12 and the DLI-12 were entered as predictors. Inspection of univariate distributions revealed that scores on APPQ-INT, SA45-HOS, SA45- PHO, and SA45-PSY had values for skewness ≥2. Thus, consistent with the multivariate normality assumption of ordinary least squares (OLS) multiple regression, a logarithmic transformation was applied to these variables prior to multivariate analysis. Next, we used Mplus 7.2 program to perform exploratory structural equation modeling (ESEM) on the scale scores of the SALI-12, the DLI-12, and the 14 concurrent measures in order to provide an alternate method for assessing the convergent and discriminant validity of the SADLI-24 subscale scores. Specifically, we examined the pattern of rotated factor loadings to determine which concurrent measures were most closely related to the SADLI-24 scales.

Concurrent measures were chosen on the basis of their conceptual and clinical relevance. The SA-45 scales were chosen as measures of general psychopathology, whereas the APPQ, the SAD-Q-5 and the B-FNE were chosen as measures of discomfort or distress related to fear of negative social appraisal (i.e., social anxiety). As such, we expected scores on the DLI-12 subscale to be more closely associated with most of the scores on the SA-45, whereas we expected scores on the SALI-12 subscale to be more closely associated with scores on the SAD-Q-5, the B-FNE, the APPQ (specifically, the Social Phobia scale), and the SA-45 Phobic Anxiety scale.

8.2. Multiple regression analyses

Results of the multiple regression analyses are summarized in Table 5⁴. For Study 2, the observed correlation between scores on the SADLI-24 scales was r = .68, p < .001. Coefficients of determination and bivariate correlations were substantial across all 14 regression models. In addition, R² ranged from .15 to .61 (average R² = .36), suggesting that the SADLI-24 scale scores accounted for a sizeable proportion of variance in scores on the selected concurrent measures. Furthermore, the pattern of regression coefficients strongly supported the convergent and discriminant validity of the SADLI-24 scales. As expected, scores on the APPQ-SP, the BFNE, and the SAD-Q-5, which are specific measures of social anxiety, evidenced significantly stronger associations with the SALI-12 (.66 ≤ β ≤ .70, p < .001) than with the DLI-12 (.01 ≤ β ≤ .10, p ≥ .082). For all three comparisons, the associated t-statistics exceeded 6.24 (p < .001). The SALI-12 also evidenced a stronger association with the APPQ-SP than with either the APPQ-AG (t = 4.50, p < .001) or the APPQ-INT (t = 7.86, p < .001). In addition, scores on the SA-45 Phobic Anxiety scale were more closely associated with the SALI-12 than with the DLI-12 (t = 2.67, p = .008). We note the sizeable associations of scores on the SALI-12 with scores on the specific measures of social anxiety and social phobia, lending support to the validity of the SALI-12 as an indicator of social anxiety life interference.

Table 5.

Results from multiple regression analyses predicting scores on concurrent measures, Study 1 (N = 1,065).

Scale	F	R2	Predictor	t	B	β	Zero-order r	Partial r
APPQ-SP	101.62	.48 [.91, .58]	SALI-12	9.91	.04	.66	.69	.59
			DLI-12	0.78*	.17	.05	.50	.04
APPQ-AG	24.96	.19 [.10, .28]	SALI-12	3.57	.10	.30	.41	.24
			DLI-12	2.05	.01	.17	.37	.14
APPQ-INT	18.98	.15 [.06, .24]	SALI-12	2.62	.01	.22	.36	.18
			DLI-12	2.31	.62	.20	.35	.16
B-FNE	85.38	.44 [.34, .54]	SALI-12	9.52	.01	.66	.66	.54
			DLI-12	0.09*	.27	.01	.46	.01
SAD-Q-5	160.32	.60 [.52, .68]	SALI-12	11.88	.04	.70	.77	.63
			DLI-12	1.75*	.06	.10	.58	.12
SA45-ANX	95.32	.46 [.37, .56]	SALI-12	2.30	.22	.16	.54	.15
			DLI-12	8.42	.05	.57	.67	.50
SA45-DEP	171.61	.61 [.53, .69]	SALI-12	1.76*	.38	.10	.58	.12
			DLI-12	12.33	.00	.71	.78	.64
SA45-HOS	23.62	.18 [.09, .27]	SALI-12	−1.43*	.01	−.12	.22	−.10
			DLI-12	5.86	−.05	.49	.41	.37
SA45-INT	114.28	.51 [.42, .60]	SALI-12	5.64	.20	.35	.64	.35
			DLI-12	6.62	.07	.43	.67	.41
SA45-OCD	71.83	.40 [.30, .50]	SALI-12	2.04	.27	.15	.50	.11
			DLI-12	7.27	.07	.52	.62	.38
SA45-PAR	54.33	.33 [.23, .43]	SALI-12	2.20	.20	.17	.47	.15
			DLI-12	5.97	.01	.45	.56	.38
SA45-PHO	39.22	.27 [.17, .36]	SALI-12	5.45	.00	.43	.51	.35
			DLI-12	1.42*	.00	.11	.41	.10
SA45-PSY	32.76	.23 [.14, .33]	SALI-12	1.41*	.01	.11	.38	.10
			DLI-12	4.89	−.05	.40	.47	.32
SA45-SOM	21.13	.16 [.08, .25]	SALI-12	−1.45*	.20	−.12	.20	−.10
			DLI-12	5.63	.04	.48	.39	.36

Notes. All F tests were significant, p ≤ .001; all correlations were significant, p ≤ .01; all t values were statistically significant at p < .01 except those indicated by *; see Table 1 for full scale names.

When examining the associations of scores on those concurrent measures that indicated a broad range of psychopathology/psychological distress, the opposite pattern emerged. As expected, scores from all remaining subscales of the SA-45 were more strongly associated with the DLI-12 scores (.40 ≤ β ≤ .71, p < .001) than with those on the SALI-12 (−.12 ≤ β ≤ .43, p ≤ .16). In particular, scores on the SA45-DEP were much more strongly associated with those on the DLI-12 than with those on the SALI-12 (t = 7.70, p < .001). A potentially surprising finding was the strong association between scores on the SA-45 Anxiety scale and scores on the DLI-12, with a lower association between scores on the SALI-12 and SA45-ANX. Given the overall pattern of associations, we find it tenable that, in the current sample, the latter scale taps a facet of anxiety that is more closely related to general psychological distress than to any specific form of phobia, such as social anxiety.

8.3. Scale level factor analysis

In order to further investigate the specificity of scores on the SALI-12 as an indicator of social anxiety, we conducted ESEM in Mplus 7.2 with robust maximum likelihood estimation. Given the expectation of substantial correlations among the factors, we selected an oblique (direct oblimin) rotation. Of the three models tests (one-, two-, and three-factor solutions), the three-factor solution provided the best relative approximation to the data matrix (χ² = 186.84, p < .001, R-CFI = .94, RMESA = .08, 90% CI [.09, .11]). Table 5 shows the rotated factor loadings for the three-factor solution (Table 6).

Table 6.

Rotated factor loadings from conjoint analysis: Study 2 (N = 220).

Scale	Factor 1	Factor 2	Factor 3
SA45-ANX	.76	.06	.07
SA45-OCD	.75	.05	.04
SA45-SOM	.71	.03	−.25
SA45-PAR	.70	.05	.03
SA45-PSY	.67	−.04	−.15
SA45-HOS	.64	.21	−.13
SA45-DEP	.64	−.02	.26
DLI-12	.62	−.05	.36
SA45-INT	.62	.00	.34
APPQ-AG	.05	.84	.06
APPQ-INT	.08	.63	.01
APPQ-SP	−.05	.61	.43
SA45-PHO	.27	.43	.10
SALI-12	.12	.11	.80
BFNE	.09	−.05	.71
SAD-Q-5	.08	.23	.65

Notes. Oblimin rotation with maximum likelihood estimation. See Table 1 for full scale names. Items with factor loadings >.40 are in boldface.

The solution yielded three interpretable factors, with only one cross-loading >.40. Factor intercorrelations were moderate (.46 to .52). We interpreted F1 as a General Distress Factor, F2, as a Phobia–Panic factor and F3 as a Social Anxiety factor. Scores on the DLI-12 loaded on the General Distress factor, along with all SA-45 subscales, except for SA-45 Phobic Anxiety scale, providing further evidence of the DLI-12 as a relatively undifferentiated indicator of general psychological distress. The APPQ scores loaded on the Phobia–Panic factor along with the SA45-PHO. The SALI-12 scores loaded on the Social Anxiety factor along with scores on the BFNE and the SAD-Q-5. Scores on the APPQ Social Phobia scale cross-loaded on factors two and three, indicating roughly comparable associations with the Phobia–Panic and Social Anxiety dimensions. We note in particular, the close association between scores on the SALI-12 on those on the BFNE and the SAD-Q-5. This pattern of factor loadings is consistent with the findings of the multiple regression analyses and further supports the validity of the SALI-12 as a specific measure of social anxiety life interference.

9. General discussion

Social anxiety entails not only psychological distress and avoidance, but also substantial disruptions to normal social functioning and activities of daily living (Modini et al., 2015). A number of studies have shown that clinically significant levels of social anxiety are associated with diminished life satisfaction (Hambrick et al., 2003), difficulty maintaining gainful employment, and higher risk for substance dependence (Patel, Knapp, Henderson, & Baldwin, 2002). Moreover, the link between social anxiety and depression is well established (e.g., Beesdo et al., 2007). As research efforts examining etiological and treatment aspects of social anxiety have grown, a variety of self-report instruments have been developed to meet the needs of researchers and clinicians. As articulated by Modini et al. (2015), the effectiveness and utility of these instruments for research and clinical work largely depend on the degree to which they can demonstrate adequate psychometric properties.

In particular, effective and useful self-report measures of trait social anxiety must demonstrate evidence of adequate content validity, internal consistency, criterion validity, construct validity, reproducibility/agreement of scores, reliability, responsiveness of items, mitigation of floor/ceiling effects, and ease of interpretation. Modini et al.’s systematic review of the measurement properties of existing social anxiety measures revealed a surprising lack of psychometrically sound instrumentation for social anxiety research and clinical applications. The current studies offer a unique contribution to the measurement and clinical assessment of social anxiety by using both classical factor analytic techniques to provide further support for the construct and content validity, internal consistency, and reliability as defined by Modini et al. (i.e., the extent to which patients with high vs. low levels of social anxiety can be distinguished from each other) of the Osman et al.’s (2011) SADLI-24 instrument.

Results of the internal consistency analyses from Studies 1 and 2 revealed that scores on the SADLI-24 subscales demonstrated strong internal consistency using coefficient-α and coefficient-ω estimates. A confirmatory analysis of competing structural models suggested that a bifactor model best detects the latent factor structure of the SADLI-24.

According to the bifactor model, all SADLI-24 items demonstrated a high ability to detect general distress. Moreover, 11 of the original 12 SALI-12 items were capable of discriminating individuals with high social anxiety from individuals with low levels of trait social anxiety. However, the items of the DLI-12 failed to produce a depression-specific dimension after the general factor was specified in the model. Notably, most of the variance in the SADLI-24 scores was also attributable to the general factor.

A practical application of the former results would be the use of a total inventory score for the SADLI-24, should scale users wish to determine the extent of overall life disruption accompanying SAD. In addition, due to the highly specific SALI-12 dimension, we direct users seeking to differentiate intrusions from social anxiety from intrusions from general distress to utilize the original scoring method. Because of the strong slope parameters for the SALI-12 and DLI-12 subscales for the two-factor model (see Table 4), and the detection of a single nonspecific SALI-12 item (Item 7), we determined that the instrument should continue to be scored as a two-factor instrument for Study 2 of the current project. Although beyond the scope of the current study, future projects aimed at evaluating a short-form of the SALI-12 may direct users to modify the scoring guidelines of the target instrument.

Table 4.

Standardized factor loadings, CFA fit statistics of competing models for SADLI-24, and variance partitioning based on bifactor model.

	One-factor model		Two-factor model		Bifactor model
Item			SALI-12	DLI-12	General	SALI-12	DLI-12	h2	u2
10	.81		.84	.	.54	.65		.72	.28
12	.81		.85		.53	.67		.72	.28
20	.81		.81		.63	.51		.66	.34
11	.79		.83		.51	.66		.70	.30
23	.78		.81		.52	.62		.66	.34
3	.78		.79		.58	.55		.63	.37
4	.78		.81		.53	.61		.66	.34
21	.72		.72		.52	.50		.52	.48
15	.75		.77		.50	.59		.60	.40
14	.69		.72		.43	.59		.53	.47
24	.64		.65		.47	.44		.42	.58
2	.64			.74	.75		.34	.68	.32
9	.62			.70	.69		−.12	.49	.51
17	.63			.78	.78		.20	.64	.36
13	.61			.66	.65		−.06	.43	.57
19	.59			.74	.74		−.25	.61	.39
7	.59		.55		.50	.29		.34	.66
1	.57			.75	.76		.24	.64	.36
18	.55			.72	.72		−.08	.53	.47
22	.53	.52		.61	.61		−.11	.38	.62
5	.52			.68	.68		−.26	.53	.47
8	.51			.67	.67		−.15	.47	.53
6	.45			.52	.53		−.24	.34	.66
16	.39			.51	.52		.30	.35	.65

Notes. N = 1065. Factor loadings > .40 are in boldface; Two-Factor model specifies correlated factors; Social Anxiety Life Interference; DLI-12 = Depression Life Interference; Uni. = one factor (univariate) model; h² = communality; u² = uniqueness.

Taken together, our CFA results are consistent with the broader scientific literature reporting the recurrent comorbidity between depression-related interference and interference from physiological arousal (i.e., anxiety). In Study 1, all 24 general factor slope values exceeded established benchmarks. This empirical pattern hints at the depth of daily intrusions that may result from any type of psychological impairment (e.g., social anxiety, depressive symptomology, etc.). Perhaps any disruption to a person’s typical routine is upsetting, regardless of the actual form psychological intrusions assume. On the other hand, our results also imply that social anxiety in particular appears to generate a secondary form of distress, beyond the typical interference suffered by individuals presenting with depressive symptoms alone. Consequently, comorbid presentation of social anxiety and depression may additively detract from an individual’s capacity to perform even seemingly mundane behaviors. Our data parallel the extant research literature and indicate that the association between these two disorders is extremely high (r ranges .64 to .68 in the current studies).

Finally, results of the validity analyses strongly supported the convergent and discriminant validity of the SALI-12 and the DLI-12 in Study 1 and the SALI-12 and DLI-12 in Study 2. Of particular note was the sharply differentiated pattern of correlations, with the SALI items demonstrating strong associations with all specific measures of social anxiety and social phobia included in the study, while the depression and general distress dimensions demonstrated significantly weaker associations with these measures. Conversely, the DLI-12 was more strongly associated with many of the measures of general psychological distress and psychopathology, than was the SALI subscales. Of course, one unexpected finding was the stronger association between the DLI-12 and the SA45-ANX subscale scores relative to the association between the SALI-12 and SA45-ANX subscale scores. In the Study 2 sample, the SA45-ANX was a better indicator of general psychological disturbances than of any specific form of anxiety. Future studies may involve a more systematic investigation of this pattern. Results of principal axis factoring also confirmed that the SALI-12 and the DLI-12 had clearly differentiated associations with the other constructs operation-alized in our study.

9.1. Conclusion

Limitations of our studies include those generally applicable to cross-sectional designs. In particular, a longitudinal design would allow analysis of responsiveness as defined by Modini et al., (i.e., ability to detect clinically important changes over time), as well as assessment of test–retest reliability (i.e., reproducibility/agreement of scores). In addition, our study samples consisted of healthy community adults, potentially limiting the generalizability of our findings to clinical populations. Further research is needed examining the psychometric properties of the SADLI-24 for use with highly symptomatic populations. Other important psychometric properties that remain to be addressed in future research are interpretability of scores and criterion validity. Finally, additional research would also be needed to examine the psychometric properties of a shortened form of the SALI-12.

Despite the aforementioned limitations, the current study is one of only two projects that has aimed to evaluate the psychometric properties of the SADLI-24, which is a self-report measure of social anxiety and general distress that specifically targets daily interference. The DSM-5 necessitates that individuals with a SAD diagnosis report that social anxiety has caused a significant reduction in their ability to perform routine tasks. The SADLI-24 would undoubtedly aid therapists and researchers in detecting the extent to which life intrusions occur as a result of depression, general distress, or social anxiety.

Supplementary data to this article can be found online at http://dx.doi.org/10.1016/j.paid.2016.03.048.