Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Apr 24.
Published in final edited form as: Am J Drug Alcohol Abuse. 2018 Jun 5;45(2):151–160. doi: 10.1080/00952990.2018.1476520

Reliability and Validity of an Internalizing Symptom Scale Based on the Adolescent and Adult Semi-Structured Assessment for the Genetics of Alcoholism (SSAGA)

Laura Acion a,b,c,*, John Kramer a,*, Xiangtao Liu a, Grace Chan d, Douglas Langbehn a, Kathleen Bucholz e, Vivia McCutcheon e, Victor Hesselbrock d, Marc Schuckit f, Danielle Dick g, Michie Hesselbrock d, Samuel Kuperman a
PMCID: PMC6481182  NIHMSID: NIHMS1014071  PMID: 29870277

Abstract

Background:

The Semi-Structured Assessment for the Genetics of Alcoholism (SSAGA) is an interview that assesses psychiatric symptoms and diagnoses, including substance use disorders and anxiety and mood (i.e., internalizing) disorders. Although the SSAGA is widely used, there exists no overall internalizing characteristics scale based on items drawn from SSAGA’s mood and anxiety disorder sections.

Objectives:

To design and assess a SSAGA-based measurement instrument capturing the overall internalizing dimension that underlies more specific internalizing conditions.

Methods:

We developed, assessed, and characterized a new scale for measuring internalizing problematic characteristics derived from the SSAGA interview. All samples were drawn from the Collaborative Studies on the Genetics of Alcoholism, a prospective multi-site genetic study of families at high risk for alcohol use disorders. All participants taking part in the study between September 2005 and September 2017 were eligible (n=904, 52.2% female).

Results:

The scale had adequate internal consistency (ordinal α=0.85, 95% CI = [0.81, 0.89]). Construct validity was supported by its association with other measures of internalizing characteristics (Internalizing Scale from Achenbach Self Reports; Neuroticism Scale from the Neuroticism-Extraversion-Openness Five-Factor Personality Inventory). Several indices of alcohol, marijuana, and nicotine misuse were also positively associated with Internalizing Scale scores.

Conclusions:

The Internalizing Scale has very good psychometric properties and can be used in studies that incorporate the SSAGA interview to study the association between internalizing characteristics and problematic alcohol and other substance use. These associations can potentially be utilized to identify individuals at risk for substance problems and to design treatments targeting such individuals.

INTRODUCTION

Child and adult psychopathology is sometimes broadly dichotomized into externalizing problems and disorders, involving characteristics such as impulsivity, sensation seeking, and aggression, and internalizing problems and disorders, involving characteristics such as anxiety and depression.1 The robust association between the externalizing domain and substance problems is well documented.2 For example, young adolescents with conduct disorder are at elevated risk to develop alcohol use disorders in adulthood,3 and the two diagnoses share some genetic4 and electrophysiological5,6 risk factors.

The internalizing domain has also been linked to substance problems, although these relationships are less frequently studied, particularly their longitudinal links, and they are often smaller in magnitude than those found for externalizing traits.7 Nevertheless, internalizing conditions such as anxiety and mood disorders are well known to co-occur with substance use disorders at high rates.8,9 For example, persons with internalizing disorders have two to eleven times the odds of having alcohol dependence when compared to subjects without such psychiatric co-morbidity.10 In addition, internalizing disorders are associated with an increased likelihood of relapse after substance treatment,11 and there is some evidence for connections between very early internalizing characteristics and adult substance initiation and/or problems many years later.12

Internalizing characteristics are thought to involve pathways of risk for alcohol and other substance use disorders partially through the mechanism of negative reinforcement,13 that is, the reduction of unpleasant emotional states through drinking, characterized behaviorally as “self-medication.” At a neurophysiological level, this process is considered central to the second stage of the brain disease model of addiction,14 which Koob and colleagues refer to as withdrawal/negative affect.15 Despite resurgent interest in this topic, the association between internalizing characteristics and the initiation, developmental course, and severity of alcohol and substance use disorders is in need of considerable further investigation. There are several methodological and historic reasons for our relatively rudimentary knowledge. Hussong and colleagues2 note that internalizing traits are more difficult than externalizing characteristics to measure reliably during adolescence (the most common onset period of substance use, including alcohol) and that extended timeframes (such as early childhood to adulthood) are under-studied. Also, a longstanding overrepresentation of males in addiction research may have downplayed the potentially greater risk implications of internalizing characteristics among female substance users,2,16–18 for whom internalizing characteristics are more prevalent.19,20

From a methodological point of view, the aggregation of several measurements reflecting the same construct is a common strategy to maximize the signal-to-noise ratio of the behaviors or characteristics of interest.21 For example, Kushner and colleagues22 obtained a positive association between alcohol dependence and an overall internalizing construct (based on both mood and anxiety symptoms). This trans-diagnostic measure, which drew items from several specific anxiety and mood diagnoses, was not improved by the addition of these diagnoses to their model. An additional advantage of aggregating across diagnoses is the decreased number of statistical comparisons needed.

The Semi-Structured Assessment for the Genetics of Alcoholism (SSAGA) was developed by the Collaborative Study on the Genetics of Alcoholism for use in its large scale, multi-site study of alcohol use disorders. The SSAGA is a comprehensive interview that obtains information about physical, psychological, and social manifestations of alcohol and other substance use disorders in adolescents and adults23–25 as part of its diagnostic assessment of all major Diagnostic and Statistical Manual of Mental Disorders (DSM5) psychiatric disorders.26 The SSAGA also exhibits very good diagnostic reliability25 and validity (as measured by comparisons with diagnoses obtained using best-estimate procedures27 and with the Schedule for Clinical Assessment in Neuropsychiatry).28 In sum, the SSAGA is an excellent instrument for assessing current and past psychiatric problems in clinical and general population samples. It has been used in over 250 studies in the US and has been translated into 9 foreign languages (Hesselbrock, personal communication). The SSAGA was previously used to study the association between particular internalizing characteristics and diagnoses (e.g., suicidality, major depressive disorder) and substance use disorders,29–35 but to our knowledge there exists no published report of an overall trans-diagnostic internalizing symptom scale based on this interview. Although the SSAGA has been shown to be reliable and valid at the diagnostic level, it is also necessary to establish the psychometric validity of scales derived from it, such as the Internalizing Scale proposed in this manuscript, for the purposes of conducting high-quality research.

The primary aim of this work was to provide researchers who use the SSAGA with a psychometrically valid and well-characterized Internalizing Scale that can be derived directly from the SSAGA without the need for an additional instrument. Because the SSAGA also contains items that can be used to measure externalizing traits, it is hoped that our scale will enable investigators to examine the relationship between internalizing and externalizing characteristics over time, particularly in relation to substance use problems. This paper describes the development, assessment, and characteristics of a SSAGA-based Internalizing Scale for measuring internalizing problematic characteristics that underlie more specific internalizing conditions such as major depressive episode.22,36 We assessed the Internalizing Scale for internal consistency and construct validity. We hypothesized that the Internalizing Scale scores would be higher in females than in males and would be positively associated with alcohol as well as other substance (i.e., nicotine, marijuana) use disorder symptoms and characteristics.

METHODS

Participants

The samples used for the development and validation of the Internalizing Scale were drawn from the Collaborative Study on the Genetics of Alcoholism, a prospective multi-site genetic study of families at high risk for alcohol use disorders (AUD).37,38 Institutional Review Boards at all six sites reviewed and approved the study. Parents provided consent for all offspring below 18; individuals 13 and older also provided consent, and children aged 12 provided assent.

All adolescents and adults who participated in the Collaborative Study on the Genetics of Alcoholism between September 2005 and September 2017 were eligible for this study (n = 3,834 in 904 families). Only one subject was randomly drawn from each family to prevent associations of observations within families. Correlation within each site was accounted for using bootstrap resampling at the site level to estimate standard errors. The sample encompassed 904 participants and was randomly divided in two subsamples (20% for development and 80% for validation). Sex, age, household income, and self-reported race for these samples are included in Table 1.

Table 1:

Participant characteristics for development, content validity and validation samples by age (adolescents: 12 – 17 years, adults: 18+ years).

Development and Content Validity Sample
n = 181		 Validation Sample
n = 723
Adolescents
n = 51		 Adults
n = 130		 Adolescents
n = 220		 Adults
n = 503
Interview age in years
(mean, SD)		 14.9 (1.7) 23.8 (6.7) 14.4 (1.9) 23.3 (5.6)
Sex (n, % female) 26 (51.0) 82 (63.1) 105 (47.7) 259 (51.5)
Self-reported ethnicity
 European American (n, %) 38 (75.0) 90 (69.2) 154 (70.0) 352 (70.0)
 African American (n, %) 11 (21.6) 32 (24.6) 53 (24.1) 108 (21.5)
Household income
(median range/year)		 NA $20,000 - $29,9991 NA $30,000 - $39,9992
YSR Internalizing Problems Raw Score
(mean, median, Q1–Q3)		 10.8, 9.0
(6.0 – 13.0)3 		NA 11.8, 10.0,
(6.0 – 16.0)4 		NA
ASR Internalizing Problems Raw Score
(mean, median, Q1–Q3)		 NA 13.8, 10.0
(5.0 – 20.0)5 		NA 13.8, 12.0,
(4.0 – 20.0)6
NEO-FFI Neuroticism Scale (mean, SD) NA 50.7 (11.6)7 NA 50.1 (11.7)8
Lifetime Internalizing Scale total score
(mean, median, Q1–Q3)		 0.68, 0
(0 – 0)		 1.61, 0
(0 – 3)		 0.80, 0
(0 – 1)		 1.55, 0
(0 – 2)
Open in a new tab
1

Excluded 9 (6.9%) participants whose responses were missing (either refused or unknown).

2

Excluded 28 (5.6%) participants whose responses were missing (either refused or unknown).

3

Available only for 31 (60.8%) participants.

4

Available only for 130 (59.1%) participants.

5

Available only for 37 (28.4%) participants.

6

Available only for 153 (30.4%) participants.

7

Available only for 32 (24.6%) participants.

8

Available only for 134 (26.6%) participants.

NA: Not available. YSR: Youth Achenbach Self Report. ASR: Adult Achenbach Self Report. NEO-FFI: Neuroticism-Extraversion-Openness Five-Factor Personality Inventory. SD: Standard deviation. Q1: First quartile. Q3: Third quartile.

Scale Development and Content Validity

Scale development and validation followed all relevant consensus-based standards for the selection of health measurement instruments (also known as COSMIN guidelines).39 We developed a 7-item scale by extracting core diagnostic items from 6 lifetime DSM5-compatible diagnoses: social phobia, panic disorder, agoraphobia, obsessive compulsive disorder, post-traumatic stress disorder, major depressive episode, and symptoms of suicidality from the SSAGA. Suicidality was added because it can reflect depression and anxiety not captured by other items. Because suicide attempts can also contain impulsive, externalizing elements, we only included attempts that: a) took place in the context of depression; b) did not take place in the context of alcohol or drug use, psychosis, or mania; and c) were preceded by at least 7 days of premeditation. Only symptoms and diagnoses occurring free of the influence of alcohol, drugs, changes in medication, co-occurring illnesses, or post-partum conditions contributed to the Internalizing Scale total score. Higher scores indicated more internalizing problematic characteristics.

The Internalizing Scale was constructed by drawing questions from several sections of the SSAGA interview that address internalizing characteristics. Since the SSAGA is a long and detailed assessment, several iterations were considered prior to the version of the scale presented in this manuscript. Earlier versions of the Internalizing Scale had up to 32 items with scores of 0 or 1, reflecting diagnostic criteria for the 6 disorders contained within it. However, because these disorders contain differing numbers of criteria, they were unequally represented in the scale (e.g., agoraphobia contributed 2 items, whereas obsessive-compulsive disorder contributed 8). Later iterations balanced the input of each disorder (see below) and added suicidal thoughts and behaviors from the SSAGA. The final version of the Internalizing Scale, reported in this manuscript, provides superior item balance. The current version also exhibits slight improvements in terms of higher internal consistency, larger effect sizes with scales used for validation, and larger effect sizes with demographic and alcohol and other substance characteristics. It also performed slightly better when heritability was measured in a genetic framework (further details about this will be presented in a publication in preparation). Although none of these improvements reached statistical significance on its own, we consider the sum of these improvements supports the decision to improve item balance. More details about these results are omitted for the sake of brevity.

Because internalizing disorders contain differing numbers of criteria, we wished to construct a scale that incorporated equal weighting or input from the disorders. To accomplish this, the scale includes 7 items reflecting the 6 diagnoses and suicidality (Table 2). Each of the 7 items is scored between 0 and 3. A score of 0 indicates no symptoms for the disorder or symptoms of suicidality; a score of 1 indicates the person has fewer than half the core symptoms for the disorder and therefore (by DSM requirements) cannot meet diagnostic criteria for the disorder (or, for suicidality, the person has only thoughts about killing self); a score of 2 is given when the person has half or more of the symptoms for a disorder but does not meet full diagnostic criteria (because symptom clustering and/or functional impairment requirements are not met; or, for suicidality, the person has both thoughts and plans of killing self but no attempts); and a score of 3 is assigned when a person meets diagnostic criteria for the disorder (or has made at least one suicide attempt). The resulting theoretical total range of the 7-item scale is 0 to 21. The algorithm used to calculate the total Internalizing Scale score using SSAGA is provided as Supplementary Material.

Table 2:

Description and scoring of the Internalizing Scale.

Disorder Item Description Item Score
Agoraphobia (AG)

  1. Anxiety about being in places or situations from which escape might be difficult

  2. Situations are avoided or endured with marked distress

 0: No symptoms
1: a. or b. present, but not both, and does not meet AG dx1
2: a. and b. present, but does not meet AG dx
3: AG dx met
Panic Disalpha-lower (PD)

  1. Recurrent unexpected panic attacks

  2. Discrete period of intense fear with 4+ symptoms (e.g., palpitation, sweating, trembling) developed abruptly

  3. At least one attack followed by 1+ month of 1+ of the following: i) concern of having additional attacks, ii) worry about attack implications, iii) significant change of behavior

 0: No symptoms
1: One of a. to c. present and does not meet PD dx
2: More than one of a. to c. but does not meet PD dx
3: PD dx met
Social Phobia
(SP)

  1. Marked and persistent fear of social or performance situations

  2. Exposure to the feared social situation almost invariably provokes anxiety

  3. Feared social situations avoided/endured with intense anxiety

 0: No symptoms
1: One of a. to c. present and does not meet SP dx
2: More than one of a. to c. but does not meet SP dx
3: SP dx met
Obsessive Compulsive Disorder (OCD)

  1. Recurrent and persistent thoughts, impulses, or images

  2. Thoughts, impulses, or images are not excessive worries about real-life problems

  3. Attempts to ignore or suppress such thoughts, impulses, or images

  4. Recognizes that obsessional thoughts/impulses/images are product of own mind

  5. Repetitive behaviors/mental acts the person feels driven to perform

  6. Behaviors/mental acts aimed at preventing distress/dreaded event or situation

  7. Person recognizes obsessions/compulsions are excessive

 0: No symptoms
1: Any combination of at most 3 of a. to g. present and does not meet OCD dx
2: Any combination of more than 3 of a. to g. but does not meet OCD dx
3: OCD dx met
Post-Traumatic Stress Disorder (PTSD)

  1. Traumatic event is persistently re-experienced

  2. Persistent avoidance of stimuli associated with the trauma and numbing of general responsiveness

  3. Persistent symptoms of increased arousal (e.g., hypervigilance)

 0: No symptoms
1: One of a. to c. present and does not meet PTSD dx
2: More than one of a. to c. but does not meet PTSD dx
3: PTSD dx met
Major Depression Episode (MDE)

  1. Depressed most of the day or markedly diminished interest or pleasure

  2. Weight loss/gain or decrease/increase in appetite

  3. Insomnia/hypersomnia

  4. Psychomotor agitation/retardation

  5. Fatigue/loss of energy

  6. Worthlessness/Guilt

  7. Difficulty concentrating

  8. Recurrent thoughts of death

 0: No symptoms
1: Any combination of at most 4 of a. to h. present and does not meet MDE dx
2: Any combination of more than 4 of a. to h. present but does not meet MDE dx
3: MDE dx met
Suicidality

  1. Thoughts about killing self for at least 7 days in a row

  2. Plans of killing self

  3. Suicidal attempt while feeling depressed but not under other circumstances (e.g., drinking, psychosis) that may or may not require medical assistance

 0: No symptoms
1: a. present but b. and c. absent
2: a. and b. present but c. absent
3: c. present
Open in a new tab
1

All diagnoses are compatible with DSM5.25

Factor Analyses and Internal Consistency

Following the recommendations in Peters,21 we used exploratory factor analysis (EFA) in the development sample to determine the scale’s dimensionality based on the polychoric correlation matrix of the scales’ items.40 The factor structure in the development sample was then confirmed through confirmatory factor analysis (CFA) in the validation sample. Adequacy of fit was evaluated using the Root Mean Square Error of Approximation (RMSEA) and the Comparative Fit Index (CFI). We used minimum rank factor analysis to estimate the percentage of common variance explained by the EFA and CFA models,41,42 and parallel analysis43 and the Bayesian Information Criterion (BIC) for factor retention. Closeness to unidimensionality was assessed using the Explained Common Variance (ECV).44

Since the unidimensionality assumption was met, we calculated ordinal standardized α,45 a statistic more appropriate than Cronbach’s α for assessing internal consistency of scales in which individual item scores are ordinal.21,46,47

Construct Validity

Construct validity was assessed in three ways, using the validation sample. First, we tested the associations between the Internalizing Scale under development and the Internalizing Scale from the Youth and Adult Achenbach Self Reports. The Internalizing Scale from the Achenbach Self Reports is a compilation of three syndrome scales that tap psychological, emotional and behavioral problems associated with internalizing tendencies: Withdrawn (e.g., won’t talk), Somatic Complaints (e.g., headaches), and Anxious/Depressed (e.g., nervous).48,49 Second, we tested the association between the Internalizing Scale under development and the Neuroticism Scale from the Neuroticism-Extraversion-Openness Five-Factor Personality Inventory.50 We targeted the Neuroticism Scale because it includes many internalizing characteristics, such as anxiety and depression, and is strongly associated with both mood and anxiety disorders.51 Because the Achenbach Self Reports and Five-Factor Personality Inventory comparisons were conducted between an interview-based scale and self-administered questionnaires (rather than between two interview-based scales or between two questionnaires), we hypothesized that validity correlations would be modest to moderate at best. Third, we examined the association of the SSAGA-based Internalizing Scale with sex to address our expectation that females would have higher scores than males.

Associations with Alcohol-Related Behavioral Characteristics and Use of Other Substances

We also explored the association between the Internalizing Scale with several alcohol-related variables, which included: a) lifetime DSM5 AUD diagnosis, b) number of AUD symptoms, and c) the total score of the Desires for Alcohol Questionnaire, a measure of alcohol craving.52,53 Associations between the scale and lifetime diagnoses and number of substance use disorder symptoms for nicotine and marijuana were also examined. All associations were explored using the validation sample. Because the literature suggests that internalizing characteristics are not strongly associated with substance use and problems (e.g.,54,55), it was anticipated that the Internalizing Scale would exhibit a modest positive relationship with the alcohol, nicotine, and marijuana variables.

We used nonparametric Mann-Whitney tests for group comparisons (i.e., male/female, having/not having current AUD, having/not having current nicotine dependence, having/not having marijuana dependence) of the Internalizing Scale total score and report the generalized Mann-Whitney (θ) effect size measure, because the scales’ total scores were skewed to the right. Theta expresses the degree of overlap of the distributions of the two groups compared (θ’s theoretical range is 0 to 1). The range extremes (0 and 1) indicate no overlap between the distributions, while 0.5 indicates complete overlap. Under the homoscedastic and normal assumptions for each group, θ compares to Cohen’s d in the following way: θ = 0.65 is equivalent to d = 0.55, θ = 0.80 results in d = 1.19, and θ = 0.95 results in d = 2.33.56 We also used Spearman’s correlations (r) to characterize bivariate associations of continuous variables. All statistical tests were performed using all data available in each case, utilising SAS version 9.4,57 R,58 and FACTOR Version 10.7.01 ×64bits.59

RESULTS

In the validation sample (n = 723), the mean Internalizing Scale score was 1.32 (SD = 2.36, Q1 = 0, median = 0, Q3 = 2, range = [0, 19], Figure 1). Descriptive statistics were similar in the development sample. Age-group-specific descriptive statistics of scale scores are presented in Table 1. We examined all the results that follow in adolescents and adults, unless otherwise stated. Results in each age group were similar to the pooled ones presented.

Figure 1:

Figure 1:

Open in a new tab

Distribution of the Internalizing Scale score in the validation sample. SD: standard deviation. Q1: First quartile. Q3: Third quartile.

Factor Analyses and Internal Consistency

Parallel analysis based on minimum rank EFA performed in the development sample recommended the extraction of a single factor for the Internalizing Scale, suggesting unidimensionality. Factor loadings for the EFA are reported in Table 3. CFA in the validation sample confirmed these findings. Single-factor models had good fit, as indicated by RMSEA smaller than 0.05 and CFI bigger than 0.95. ECV was bigger than 0.81 both in the development and the validation samples; that is, more than 81% of the common variance in its items was explained by a single general factor. A two-factor model showed goodness of fit (by means of the RMSEA and CFI statistics) similar to the single-factor model, but with a considerably larger BIC (i.e., single-factor BIC = 143.29 vs two-factor BIC = 156.90). Parallel analysis did not support the two-factor model either.

Table 3:

Factor loadings for the exploratory factor analysis performed in the development sample.

Item Factor Loading
Agoraphobia 0.829
Panic Disorder 0.858
Social Phobia 0.709
Obsessive Compulsive Disorder 0.869
Post-Traumatic Stress Disorder 0.596
Major Depression Episode 0.542
Suicidality 0.701
Open in a new tab

The Internalizing Scale exhibited adequate internal consistency in the validation sample (ordinal α = 0.85, 95% CI = [0.81, 0.89]).

Construct Validity

Table 1 displays characteristics of the samples used for evaluating the associations between the SSAGA-based Internalizing Scale and the Internalizing Scale from the Youth and Adult Achenbach Self Reports, and the Neuroticism Subscale of the Neuroticism-Extraversion-Openness Five-Factor Personality Inventory. There was a moderate association between the Internalizing Scale from the Adult Achenbach Self Report and the Neuroticism Subscale of the Five-Factor Personality Inventory (r = 0.39; 95% CI = [0.18, 0.56]; p < 0.0001). Positive associations between the Internalizing Scale and the Internalizing Problems raw score of: a) the Adult Achenbach Self Report (r = 0.40; 95% CI = [0.25, 0.53]; p < 0.0001) and b) the Youth Achenbach Self Report (r = 0.35; 95% CI = [0.18, 0.50]; p < 0.0001) supported construct validity. The Internalizing Scale was also positively associated with the Neuroticism Subscale (r = 0.20; 95% CI = [0.03, 0.36]; p = 0.02), providing further support for construct validity.

Females (n = 364, 50.3 %) had higher INT Scale scores than males (n = 359, 49.7 %) (mean, median, Q1–Q3: [1.55, 0, 0 – 2.5] vs [1.08, 0, 0–1]; θ = 0.94; 95% CI = [0.92, 0.96]; p = 0.0049), consistent with expectations.

Associations with Alcohol-Related Behavioral Characteristics and Use of Other Substances

Table 4 includes alcohol- and other-substance-related characteristics for each age group. Table 5 describes the associations between AUD, nicotine dependence, and marijuana use disorders with the Internalizing Scale. Lifetime total scores for the scale were moderately higher for subjects with these diagnoses than subjects without the diagnoses. The number of AUD, nicotine dependence, and marijuana use disorder symptoms, and the Desires for Alcohol Questionnaire total score were also modestly but positively associated with the Internalizing Scale scores (see Table 5).

Table 4:

Alcohol- (AUD), nicotine- (ND), and marijuana-related (MJUD) characteristics by age category in the validation sample

Characteristic Adolescents
(12 – 17 years)
n = 220		 Adults
(18+ years)
n = 503
Current AUD diagnosis
(n, %)		 14 (6.3) 174 (34.6)
Number of AUD symptoms
(mean, median, Q1–Q3)		 0.3, 0, (0–0) 1.7, 1, (0–3)
Desires for alcohol total score
(mean, median, Q1–Q3)		 26.7, 24.0, (21–32)1 29.4, 28, (23–36)2
Current ND diagnosis
(n, %)		 12 (5.5) 87 (17.3)
Number of ND symptoms
(mean, median, Q1–Q3)		 0.3, 0, (0–0) 1.1, 0, (0–2)
Current MJUD diagnosis
(n, %)		 15 (6.8) 131 (26.0)
Number of MJUD symptoms
(mean, median, Q1–Q3)		 0.3, 0, (0–0) 1.3, 0, (0–2)
Open in a new tab
1

Available only for 30 adolescents of the 60 (27.3%) who ever had a full drink of alcohol.

2

Available only for 291 adults of the 455 (90.5%) who ever had a full drink of alcohol.

Q1: First quartile. Q3: Third quartile.

Table 5:

Association between the Internalizing Scale and Alcohol- (AUD), nicotine- (ND), and marijuana-related (MJUD) behavioral in the validation sample

Internalizing Scale total score
Mean, median, Q1–Q3 Theta 95% CI p
Current AUD diagnosis
 Yes (n = 188) 2.15, 1, 0 – 3 0.79 0.75–0.83 < 0.0001
 No (n = 535) 1.02, 0, 0 – 1
Current ND diagnosis
 Yes (n = 99) 2.22, 1, 0 – 3 0.68 0.63–0.74 0.0002
 No (n = 624) 1.17, 0, 0 – 2
Current MJUD diagnosis
 Yes (n = 577) 1.74, 1, 0 – 3 0.71 0.66–0.75 0.001
 No (n = 146) 1.20, 0, 0 – 2
n = 723 Spearman’s r 95% CI p
Number of AUD symptoms 0.19 0.12–0.26 < 0.0001
Desires for alcohol total score1 0.20 0.09–0.31 0.0003
Number of ND symptoms 0.19 0.12–0.26 < 0.0001
Number of MJUD symptoms 0.12 0.05–0.19 0.001
Open in a new tab
1

Available only for 321 (44.4%) individuals.

CI: Confidence interval. Q1: First quartile. Q3: Third quartile.

DISCUSSION

The Internalizing Scale has very good psychometric properties and can be used, in studies that incorporate the SSAGA interview, to study the association between lifetime internalizing characteristics and problematic alcohol and substance use. The scale is unidimensional and has good internal consistency. Its validity is supported by associations with two other dimensional measures of internalizing characteristics found in the Youth and Adult Achenbach Self Report Internalizing Scales and the Neuroticism-Extraversion-Openness Five-Factor Personality Inventory Neuroticism Scale. Although just moderate, the magnitude of the correlation between the proposed Internalizing Scale and the Achenbach Internalizing Scale was similar to the correlation between the Neuroticism Five-Factor Personality Inventory Subscale and the Achenbach Internalizing Scale. Our scale was more correlated with the Achenbach Internalizing Scale than with the Neuroticism Five-Factor Personality Inventory Subscale, perhaps because the Achenbach Internalizing Scale is an amalgam of disorder-based symptoms, most similar in construction to the SSAGA-based Internalizing Scale. In contrast, the lower strength of the correlation between the SSAGA-based Internalizing Scale and the Neuroticism Scale may reflect the fact that the Neuroticism Scale does not derive from specific disorders or diagnoses and includes some non-internalizing traits, such as hostility. In sum, these data suggest that our results are within range of reasonable expectations.

The summary statistics for the Internalizing Scale scores reveal that 75% of the sample scored a 2 or lower despite the instrument’s theoretical range of 0 to 21. However, Figure 1 shows that, even though the distribution is skewed to the right, the right tail of the distribution is not driven by outliers, but rather by the upper quartile of the sample where total scale scores steadily decrease. We believe that the range of this scale will accommodate populations with more internalizing characteristics; predominantly female individuals or patients ascertained for anxiety or mood disorders, for example, might be reasonably expected to exhibit higher mean and median statistics and to more fully exploit the range of this instrument.

Also, as hypothesized, several indices of alcohol and other drug misuse were positively associated with higher scales’ scores. Furthermore, females had higher internalizing scores than males, consistent with both clinical observation and research (e.g.,19,20). In sum, the Internalizing Scale echoes SSAGA’s diagnostic consistency and validity. It is important to acknowledge the modest to moderate magnitude of relationships between the scales and substance use variables. The validation sample was young, with mean ages of 14.4 and 23.3 for the adolescent and young adult members, respectively (Table 1). Although 34.6% of the adult members had an AUD, only 6.3% of their adolescent counterparts did (Table 4). For this younger group, the full flowering of drug and alcohol involvement has not yet occurred, and the relationship between internalizing characteristics and substance misuse may be less robust than at later stages of life.2,54 In addition, as mentioned earlier, internalizing characteristics are not as strongly associated with substance misuse as are externalizing characteristics, particularly among relatively young samples (e.g.,54). The purpose of this work was not to demonstrate a strong relationship between internalizing characteristics and substance use but rather to see if there was evidence for a positive link, as suggested by the literature. Had there been no relationship, there would have been concerns about the utility of this scale for studying addiction.

Strengths of this work include the high-quality and large data set available to develop and characterize the Internalizing Scale; for example, our sample size allowed us to examine internalizing characteristics in relation to sex. Also, we characterized the scale from multiple perspectives: construction, internal consistency, and external validity.

There is a lack of test-retest reliability data for the scale. However, Bucholz and colleagues25 demonstrated good one-week test-retest diagnostic reliability for the SSAGA. Since the Internalizing Scale is, like diagnosis, based on aggregations of single SSAGA questions, it is reasonable to expect that its test-retest reliability might also be acceptable. However, further work is needed to support this hypothesis. Another limitation of this study was that it includes data obtained across 12 years, where an accumulation of differences in the administration or coding of the SSAGA (interviewer drift) may have occurred. However, interviewer- and site-specific drift in the Collaborative Studies on the Genetics of Alcoholism have been minimized by intensive interviewer training sessions and monthly conference calls to review coding and problematic subject responses.23 In addition, scales used for assessing convergent validity were available only for 25% to 60% of the participants (see Table 1). However, all results reported in this manuscript were similar when different random samples of all data available were used and also when a sample with complete data for all variables was examined.

The scores reported in this work are true to diagnostic guidelines; hence, any internalizing symptoms that occurred in the context of alcohol and drug use, medication changes, co-occurring illness or birth are scored as absent. Depending on the intended use of the scores, this could be an asset or an additional limitation. Our approach makes most sense for scoring internalizing as a precursor to the initiation, exacerbation, or recurrence of substance problems, but might be problematic if scores are examined as mediators or moderators between life stressors or other forms of mental or physical illness and substance use. This potential limitation can be easily addressed by a minimal change in the code provided in the Supplementary Materials. These code changes permit the inclusion of symptoms occurring in the context of substance use, illness, or other circumstances. The psychometric properties of the Internalizing Scale based on these alterations are similar to the results reported above (results not shown).

Data in the current analyses were cross-sectional, as is much of the current knowledge about internalizing traits and substance problems. However, data collected in the ongoing Collaborative Studies on the Genetics of Alcoholism prospective study, which include SSAGA and, consequently, the Internalizing Scale, will permit longitudinal investigations of multivariable associations between internalizing trans-diagnostic characteristics, problematic substance use, and the temporal relationship between them. It is hoped that the examination of genetic and neurophysiological underpinnings of internalizing characteristics will also be well served by this scale.

Supplementary Material

Supplemental

ACKNOWLEDGMENTS

The Collaborative Study on the Genetics of Alcoholism (COGA), Principal Investigators B. Porjesz, V. Hesselbrock, H. Edenberg, L. Bierut, includes eleven different centers: University of Connecticut (V. Hesselbrock); Indiana University (H.J. Edenberg, J. Nurnberger Jr., T. Foroud); University of Iowa (S. Kuperman, J. Kramer); SUNY Downstate (B. Porjesz); Washington University in St. Louis (L. Bierut, J. Rice, K. Bucholz, A. Agrawal); University of California at San Diego (M. Schuckit); Rutgers University (J. Tischfield, A. Brooks); Department of Biomedical and Health Informatics, The Children’s Hospital of Philadelphia; Department of Genetics, Perelman School of Medicine, University of Pennsylvania, Philadelphia PA (L. Almasy), Virginia Commonwealth University (D. Dick), Icahn School of Medicine at Mount Sinai (A. Goate), and Howard University (R. Taylor). Other COGA collaborators include: L. Bauer (University of Connecticut); J. McClintick, L. Wetherill, X. Xuei, Y. Liu, D. Lai, S. O’Connor, M. Plawecki, S. Lourens (Indiana University); G. Chan (University of Iowa; University of Connecticut); J. Meyers, D. Chorlian, C. Kamarajan, A. Pandey, J. Zhang (SUNY Downstate); J.-C. Wang, M. Kapoor, S. Bertelsen (Icahn School of Medicine at Mount Sinai); A. Anokhin, V. McCutcheon, S. Saccone (Washington University); J. Salvatore, F. Aliev, B. Cho (Virginia Commonwealth University); and Mark Kos (University of Texas Rio Grande Valley). A. Parsian and M. Reilly are the NIAAA Staff Collaborators.

We continue to be inspired by our memories of Henri Begleiter and Theodore Reich, founding PI and Co-PI of COGA, and also owe a debt of gratitude to other past organizers of COGA, including Ting-Kai Li, P. Michael Conneally, Raymond Crowe, and Wendy Reich, for their critical contributions. This national collaborative study is supported by NIH Grant U10AA008401 from the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and the National Institute on Drug Abuse (NIDA).

Funding Source: This study is supported by NIH grant U10AA008401 from the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and grant AA05524 from the National Institute on Drug Abuse (NIDA).

Footnotes

Financial Disclosures: The authors report no relevant financial conflicts.

REFERENCES

  • 1.Krueger RF. The structure of common mental disorders. Archives of General Psychiatry. 1999;56(10):921–926. [DOI] [PubMed] [Google Scholar]
  • 2.Hussong AM, Jones DJ, Stein GL, Baucom DH, Boeding S. An internalizing pathway to alcohol use and disorder. Psychology of Addictive Behaviors. 2011;25(3):390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Pardini D, White HR, Stouthamer-Loeber M. Early adolescent psychopathology as a predictor of alcohol use disorders by young adulthood. Drug & Alcohol Dependence. 2007;88:S38–S49. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Slutske WS, Heath AC, Dinwiddie SH, et al. Common genetic risk factors for conduct disorder and alcohol dependence. Journal of Abnormal Psychology. 1998;107(3):363. [DOI] [PubMed] [Google Scholar]
  • 5.Bauer LO. Predicting relapse to alcohol and drug abuse via quantitative electroencephalography. Neuropsychopharmacology. 2001;25(3):332–340. [DOI] [PubMed] [Google Scholar]
  • 6.Costa L, Bauer L, Kuperman S, et al. Frontal P300 decrements, alcohol dependence, and antisocial personality disorder. Biological Psychiatry. 2000;47(12):1064–1071. [DOI] [PubMed] [Google Scholar]
  • 7.ChassinL, Ritter J. Vulnerability to substance use disorders in childhood and adolescence In Ingram RE, & Price JM (Eds.) Vulnerability to psychopathology: Risk across the lifespan. Guilford Press; 2010; pp. 107–134. [Google Scholar]
  • 8.Grant BF, Goldstein RB, Saha TD, et al. Epidemiology of DSM-5 alcohol use disorder: results from the National Epidemiologic Survey on Alcohol and Related Conditions III. JAMA psychiatry. 2015;72(8):757–766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Grant BF, Stinson FS, Dawson DA, et al. Prevalence and co-occurrence of substance use disorders and independentmood and anxiety disorders: Results from the national epidemiologic survey on alcohol and relatedconditions. Archives of general psychiatry. 2004;61(8):807–816. [DOI] [PubMed] [Google Scholar]
  • 10.Dawson DA, Goldstein RB, Moss HB, Li T- K, Grant BF. Gender differences in the relationship of internalizing and externalizing psychopathology to alcohol dependence: Likelihood, expression and course. Drug and alcohol dependence. 2010;112(1):9–17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Kushner MG, Abrams K, Thuras P, Hanson KL, Brekke M, Sletten S. Follow‐up study of anxiety disorder and alcohol dependence in comorbid alcoholism treatment patients. Alcoholism: Clinical and Experimental Research. 2005;29(8):1432–1443. [DOI] [PubMed] [Google Scholar]
  • 12.Wu P, Bird HR, Liu X, et al. Childhood depressive symptoms and early onset of alcohol use. Pediatrics. 2006;118(5):1907–1915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Koob GF, Ahmed SH, Boutrel B, et al. Neurobiological mechanisms in the transition from drug use to drug dependence. Neuroscience & Biobehavioral Reviews. 2004;27(8):739–749. [DOI] [PubMed] [Google Scholar]
  • 14.Volkow ND, Koob GF, McLellan AT. Neurobiologic advances from the brain disease model of addiction. New England Journal of Medicine. 2016;374(4):363–371. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Koob GF, Le Moal M. Plasticity of reward neurocircuitry and the’dark side’of drug addiction. Nature neuroscience. 2005;8(11):1442–1444. [DOI] [PubMed] [Google Scholar]
  • 16.Del Boca FK, Hesselbrock MN. Gender and alcoholic subtypes. Alcohol Research and Health. 1996;20(1):56. [PMC free article] [PubMed] [Google Scholar]
  • 17.Harford TC, Yi Hy, Chen CM, Grant BF. Psychiatric symptom clusters as risk factors for alcohol use disorders in adolescence: a national study. Alcoholism: Clinical and Experimental Research. 2015;39(7):1174–1185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Patrick ME, Schulenberg JE, O’Malley PM, et al. Age-related changes in reasons for using alcohol and marijuana from ages 18 to 30 in a national sample. Psychology of addictive behaviors. 2011;25(2):330. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.McLean CP, Asnaani A, Litz BT, Hofmann SG. Gender differences in anxiety disorders: prevalence, course of illness, comorbidity and burden of illness. Journal of psychiatric research. 2011;45(8):1027–1035. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Nolen-Hoeksema S Gender differences in depression. Current directions in psychological science. 2001;10(5):173–176. [Google Scholar]
  • 21.Peters G- JY. The alpha and the omega of scale reliability and validity: why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale quality. European Health Psychologist. 2014;16(2):56–69. [Google Scholar]
  • 22.Kushner MG, Wall MM, Krueger RF, et al. Alcohol dependence is related to overall internalizing psychopathology load rather than to particular internalizing disorders: evidence from a national sample. Alcoholism: Clinical and Experimental Research. 2012;36(2):325–331. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Kuperman S, Schlosser SS, Kramer JR, et al. Developmental sequence from disruptive behavior diagnosis to adolescent alcohol dependence. American Journal of Psychiatry. 2001. [DOI] [PubMed] [Google Scholar]
  • 24.Hesselbrock M, Easton C, Bucholz KK, Schuckit M, Hesselbrock V. A validity study of the SSAGA-a comparison with the SCAN. Addiction. 1999;94(9):1361–1370. [DOI] [PubMed] [Google Scholar]
  • 25.Bucholz KK, Cadoret R, Cloninger CR, et al. A new, semi-structured psychiatric interview for use in genetic linkage studies: a report on the reliability of the SSAGA. Journal of studies on alcohol. 1994;55(2):149–158. [DOI] [PubMed] [Google Scholar]
  • 26.American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5). Washington, DC: American Psychiatric Association; 2013. [Google Scholar]
  • 27.Bucholz KK, Nurnberger JI Jr, Kramer JR, Hesselbrock VM, Schuckit MA, Bierut LJ. Comparison of psychiatric diagnoses from interview reports with those from best-estimate procedures. Journal of studies on alcohol. 2006;67(1):157–168. [DOI] [PubMed] [Google Scholar]
  • 28.Hesselbrock M, Easton C, Bucholz KK, Schuckit M, Hesselbrock V. A validity study of the SSAGA-a comparison with the SCAN. Addiction. 1999;94(9):1361–1370. [DOI] [PubMed] [Google Scholar]
  • 29.Brown SA, Brumback T, Tomlinson K, et al. The National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA): a multisite study of adolescent development and substance use. Journal of studies on alcohol and drugs. 2015;76(6):895–908. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Kramer J, Chan G, Kuperman S, Wetherill L, Bucholz K, Acion L. The combined role of internalizing and externalizing symptoms in alcohol use and problems. Alcoholism: Clinical and experimental research. 2014;38:262A. [Google Scholar]
  • 31.Kramer J, Chan G, Kuperman S, et al. The role of internalizing and externalizing symptoms in alcohol, nicotine, and marijuana involvement. Alcoholism: Clinical and experimental research. 2015;39:152A. [Google Scholar]
  • 32.Kuperman S, Chan G, Kramer JR, et al. A model to determine the likely age of an adolescent’s first drink of alcohol. Pediatrics. 2013;131(2):242–248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Nurnberger JI, Foroud T, Flury L, Meyer ET, Wiegand R. Is there a genetic relationship between alcoholism and depression? Alcohol research and health. 2002;26(3):233–240. [PMC free article] [PubMed] [Google Scholar]
  • 34.Preuss U, Schuckit M, Smith T, Barnow S, Danko G. Mood and anxiety symptoms among 140 children from alcoholic and control families. Drug and alcohol dependence. 2002;67(3):235–242. [DOI] [PubMed] [Google Scholar]
  • 35.Schuckit MA, Smith TL, Barnow S, Preuss U, Luczak S, Radziminski S. Correlates of externalizing symptoms in children from families of alcoholics and controls. Alcohol and Alcoholism. 2003;38(6):559–567. [DOI] [PubMed] [Google Scholar]
  • 36.Krueger RF, Markon KE. Understanding psychopathology melding behavior genetics, personality, and quantitative psychology to develop an empirically based model. Current Directions in Psychological Science. 2006;15(3):113–117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Begleiter H, Reich T, Hesselbrock V, et al. The collaborative study on the genetics of alcoholism. Alcohol Health and Research World. 1995;19:228–228. [PMC free article] [PubMed] [Google Scholar]
  • 38.Edenberg HJ, Foroud T. Review: The genetics of alcoholism: identifying specific genes through family studies. Addiction biology. 2006;11(3–4):386–396. [DOI] [PubMed] [Google Scholar]
  • 39.Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Quality of Life Research. 2010;19(4):539–549. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Baglin J Improving your exploratory factor analysis for ordinal data: a demonstration using FACTOR. Practical Assessment, Research & Evaluation. 2014;19(5):2. [Google Scholar]
  • 41.Shapiro A, Ten Berge JM. Statistical inference of minimum rank factor analysis. Psychometrika. 2002;67(1):79–94. [Google Scholar]
  • 42.Ten Berge JM, Kiers HA. A numerical approach to the approximate and the exact minimum rank of a covariance matrix. Psychometrika. 1991;56(2):309–315. [Google Scholar]
  • 43.Horn JL. A rationale and test for the number of factors in factor analysis. Psychometrika. 1965;30(2):179–185. [DOI] [PubMed] [Google Scholar]
  • 44.Ferrando PJ, Lorenzo-Seva U. Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educational and Psychological Measurement. 2017:0013164417719308. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Gadermann AM, Guhn M, Zumbo BD. Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation. 2012;17(3):1–13. [Google Scholar]
  • 46.Crutzen R, Peters G- JY. Scale quality: alpha is an inadequate estimate and factor-analytic evidence is needed first of all. Health psychology review. 2015:1–6. [DOI] [PubMed] [Google Scholar]
  • 47.Dunn TJ, Baguley T, Brunsden V. From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology. 2014;105(3):399–412. [DOI] [PubMed] [Google Scholar]
  • 48.Achenbach T, Rescorla L. Manual for the ASEBA adult forms & profiles. Burlington, VT: University of Vermont; 2003. [Google Scholar]
  • 49.Ivanova MY, Achenbach TM, Rescorla LA, et al. The generalizability of the Youth Self-Report syndrome structure in 23 societies. Journal of Consulting and Clinical Psychology. 2007;75(5):729. [DOI] [PubMed] [Google Scholar]
  • 50.Costa PT, MacCrae RR. Revised NEO personality inventory (NEO PI-R) and NEO five-factor inventory (NEO FFI): Professional manual. Psychological Assessment Resources; 1992. [Google Scholar]
  • 51.Griffith JW, Zinbarg RE, Craske MG, et al. Neuroticism as a common dimension in the internalizing disorders. Psychological medicine. 2010;40(7):1125–1136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Kramer JR, Chan G, Hesselbrock VM, et al. A Principal Components Analysis of the Abbreviated Desires for Alcohol Questionnaire (DAQ)*. Journal of studies on alcohol and drugs. 2010;71(1):150–155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Love A, James D, Willner P. A comparison of two alcohol craving questionnaires. Addiction. 1998;93(7):1091–1102. [DOI] [PubMed] [Google Scholar]
  • 54.King SM, Iacono WG, McGue M. Childhood externalizing and internalizing psychopathology in the prediction of early substance use. Addiction. 2004;99(12):1548–1559. [DOI] [PubMed] [Google Scholar]
  • 55.Kotov R, Gamez W, Schmidt F, Watson D. Linking “big” personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. Psychological bulletin. 2010;136(5):768. [DOI] [PubMed] [Google Scholar]
  • 56.Newcombe RG. Confidence intervals for proportions and related measures of effect size. CRC Press; 2012. [Google Scholar]
  • 57.Institute SAS. Base SAS 9.4 Procedures Guide. SAS Institute; 2014. [Google Scholar]
  • 58.R Core Team. R: A language and environment for statistical computing. In: Computing RFfS, ed. Vienna, Austria: ISBN 3-900051-07-0; 2016. [Google Scholar]
  • 59.Lorenzo-Seva U, Ferrando PJ. FACTOR: A computer program to fit the exploratory factor analysis model. Behavior research methods. 2006;38(1):88–91. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental
NIHMS1014071-supplement-Supplemental.docx (46.7KB, docx)

RESOURCES