Development and Validation of Short Versions of the Internal Mental Distress and Behavior Complexity Scales in the Global Appraisal of Individual Needs (GAIN)

Abstract

Co-occurring mental distress and behavior problems are the norm in substance abuse treatment but are often poorly assessed due to resource constraints. This paper describes the development and validation of scales measuring internalizing mental distress and externalizing behavior problems that are shorter versions of comorbidity scales found in the full Global Appraisal of Individual Needs (GAIN). GAIN data from two treatment outcome studies, one involving adolescents and the other on adults, were used in the creation and testing of the scales. Subsets of items from the full GAIN scales were selected for the short scales through the application of standard psychometric principles. The short comorbidity scales still have moderate to high reliability and are highly correlated with the full scales. Parallel tests of construct validity show no substantial loss when moving from the longer to shorter versions. The short scales maintain good sensitivity and specificity for predicting diagnostic impressions.

Introduction

Alcohol and substance use disorders among both adults and adolescents usually do not occur in isolation, but rather they co-occur with a wide variety of psychological and behavioral difficulties. In the general population, it has been estimated that approximately 53% of those with a lifetime diagnosis of an alcohol or drug use disorder also have a lifetime diagnosis of a mental health disorder.^1,2 Among those who enter treatment for alcohol and drug abuse, it is estimated that close to 80% also present with one or more co-occurring psychiatric disorders.^3,4

There is an extensive literature on adults and adolescents entering substance abuse treatment demonstrating that internalizing psychological disorders (e.g., depression and other mood disorders, suicide attempts, generalized and other anxiety disorders, including trauma related disorders) are prevalent and can interact with treatment effectiveness.^5–9 While certainly some of these conditions are substance induced, others result from childhood maltreatment and other forms of victimization that often lead to serious distress which interferes with functioning but does not necessarily fit neatly into diagnostic categories.^10,11

The literature also demonstrates a high prevalence of externalizing behavior problems among those entering substance abuse treatment.^7–9 These include childhood disorders [or continuing disorders for adults; e.g., attention deficit-hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), conduct disorder (CD)], other impulse control (e.g., pathological gambling), and cluster B [e.g., antisocial personality disorder (ASPD), borderline personality disorder (BPD)] personality disorders.^4,12,13 Crime and violence also fall under the rubric of externalizing behavior problems, and reductions in delinquency, crime, and their associated costs to society have historically been one of the major impetuses behind public funding for substance abuse treatment.¹⁴ Higher rates of externalizing disorders are associated with higher substance use severity and resistance to treatment.^4,15,16

Given that comorbidity is highly intertwined with the severity of substance use disorders and response to treatment, simultaneous screening for co-occurring mental health disorders and behavioral problems is essential. Although several longer scales exist, the need for a thorough yet brief assessment of co-occurring problems has been repeatedly identified as one of the key gaps in the field of substance abuse treatment.^17–19 Even when current assessments do contain a psychological distress measure, it often is limited in breadth. For instance, about a third of the programs in the USA use the Addiction Severity Index (ASI), making it the single most frequently used substance abuse treatment instrument.²⁰ However, this measure does not include symptoms related to childhood or other traumatic victimization, common adolescent disorders like ADHD or CD, and interpersonal violence or types of illegal activities beyond arrests. The need to develop more comprehensive but still efficient tools for use in community-based settings was a key recommendation made in several recent reports, including the Report to Congress on co-occurring disorders among substance abusers,²¹ the Center for Substance Abuse Treatment guidelines on treating co-occurring disorders,²² and the National Institute on Drug Abuse’s (NIDA) Blue Ribbon panel on the needs for health services research.²¹ The development of such tools are also central to several Federal, State, and local efforts to improve the continuum of care by adding brief multi-dimensional screening in a broad range of settings such as schools, community organizations, employee assistance programs, health care delivery systems, and law enforcement agencies.

The purpose of this paper is to describe the development and validation of a set of scales that efficiently measure co-occurring psychological and behavioral conditions that occur frequently among people with substance use disorders. The process for shortening the scales will be described, reliability and comparability with the longer scales from the parent assessment will be examined, and evidence in support of the scales’ construct validity will be presented.

Method

Intake data from two projects was used to create the short comorbidity scales: the Cannabis Youth Treatment (CYT) project^23,24 and the Early Re-Intervention (ERI) project.^8,25 These studies were chosen to represent heterogeneous subsamples (differing in age, race, gender, substance use pattern, comorbidity, and level of care) to evaluate the robustness of the relationship between the short comorbidity and full-length scales. Both studies measured participant characteristics, functioning, mental health symptoms and diagnostic impressions with the Global Appraisal of Individual Needs (GAIN).²⁶ Below are short descriptions of each project, the characteristics of the participants, and the measures used in both samples.

Data source

Cannabis Youth Treatment project

The CYT study was a multi-site randomized field experiment examining five manual-guided approaches for outpatient treatment of adolescents who abuse or are dependent on cannabis. The goals of CYT were to (a) learn more about the characteristics and needs of adolescents presenting to short-term (less than 90 days) outpatient treatment, (b) produce developmentally appropriate treatment manuals for promising approaches to short-term outpatient treatment that could be readily disseminated to the field, and (c) field test their clinical effectiveness, cost and cost effectiveness of the treatments. The study took place from 1997 to 2000.

The study’s sample consisted of 600 adolescents (83% males, 17% females) ages 12 to 18 who reported one or more DSM-IV criteria for cannabis abuse or dependence, had used cannabis in the past 90 days (or 90 days prior to being in a controlled environment), and met ASAM patient placement criteria for Level I (outpatient) or Level II (intensive outpatient) treatment. For safety and logistical reasons, participants were excluded if they had: (a) used alcohol 45 or more days of the 90 days before intake, (b) used other drugs 13 or more of the 90 days prior to intake, (c) an acute medical or psychological condition that was likely to prohibit full participation in treatment, (d) insufficient mental capacity to understand the consent and/or participate in treatment, (e) lived outside of the program’s catchment area, or (g) a history of repeated, violent behavior or severe conduct disorder that might put other participants at unacceptable risks.

Early Re-Intervention project

The ERI study was an experimental study testing outcome monitoring versus outcome monitoring plus early re-intervention for adult drug users who have completed treatment. The specific aims of the study were to determine the relative effectiveness of adding an early re-intervention protocol to outcome monitoring in terms of its ability to (1) reduce the time to treatment re-entry and increase the rates of treatment re-entry within 9 months, and consequently (2) improve long-term outcomes related to (a) substance use, HIV risk behaviors, and illegal activity, (b) training- and employment-related behaviors, and (c) utilization of expensive services (e.g., inpatient substance abuse treatment, inpatient mental health treatment, emergency room admissions, hospital nights, days in jail, days participants’ children required foster or institutional care).

The 448 ERI participants (41% males, 59% females) were between the ages of 18 and 59, with most (47%) in the 30–39 age range. To be included in the study, individuals needed to meet lifetime criteria of substance abuse or dependence, have used alcohol or other drugs during the past 90 days, complete a central intake unit assessment and receive a referral to substance abuse treatment at a collaborating treatment agency, and be 18 years of age or older. For logistical reasons, participants were excluded if they did not reside in the catchment area or did not plan to reside there during follow-up interviews, had been sentenced to jail or prison on a DUI program for most of the upcoming 12 months, were unable to use English or Spanish, or were too cognitively impaired to provide an informed consent.

Comparison of sample characteristics

As displayed in Table 1, the adolescent sample was composed mainly of Caucasian males while most of the adult sample consisted of African American females. The treatment careers of the two groups reflect the progression from adolescent to adult substance abusers, with most adolescents never having been in treatment and most adults having had at least two prior treatment episodes. Patterns of weekly drug use also differed in the expected directions. While both adolescents and adults had similar rates of weekly use of alcohol and drugs, adolescents favored marijuana followed only by alcohol. None reported use of the harder drugs (cocaine and opioids). In contrast, most adults favored cocaine, alcohol, and opioids, with relatively few reporting use of marijuana.

Table 1.

Demographics and clinical characteristics of the two samples

Characteristic	Adolescent (n=600)	Adult (n=448)
Demographics
Male	83%	41%
Caucasian	61%	8%
Prior substance abuse treatment episodes
None	74%	32%
One	16%	25%
2+ episodes	10%	44%
Drug use
Used alcohol or drugs weekly	75%	83%
Used alcohol weekly	17%	47%
Used marijuana weekly	71%	16%
Used cocaine weekly	0%	60%
Use opioids weekly	0%	26%
Used other drugs weekly	1%	2%
Met criteria for substance dependence	48%	91%
Mental health
Any internalizing disorder	33%	75%
Major depression disorder	18%	61%
Generalized anxiety disorder	23%	60%
High traumatic distress	14%	44%
Suicide risk	9%	5%
Any externalizing disorder	61%	45%
ADHD	38%	34%
Conduct Disorder	53%	37%
Pattern of co-occurring disorders
Neither internalizing nor externalizing	34%	23%
Internalizing only	5%	32%
Externalizing only	33%	2%
Both internalizing and externalizing	28%	43%
Crime and violence
Mod/high crime-violence	71%	50%
Moderate/high violence in handling disagreements	64%	45%
Moderate/high illegal activities	56%	34%
Current criminal justice involvement	62%	27%
Any prior arrests	72%	72%

Adolescents were less likely than adults to report internalizing mental problems overall (33 vs 75%). The gap between adolescents’ and adults’ reported symptoms of major depression (18 vs 61%), generalized anxiety (23 vs 60%), and traumatic distress (14 vs 44%) was quite wide. However, adolescents were more likely to report externalizing problems overall (61 vs 45%), with a moderately wide gap between adolescents’ and adults’ reported symptoms of conduct problems (53 vs 37%). The pattern of co-occurring problems also distinguished adolescents from adults, with adolescents more likely than adults to report an externalizing disorder in isolation (33 vs 2%) and adults more likely to report either an internalizing disorder in isolation (32% for adults, 5% for adolescents) or both internalizing and externalizing disorders (43% for adults, 28% for adolescents). Across all measures of moderate-to-high levels of crime and violence (overall, interpersonal violence, and illegal activities), adolescents reported engaging in these behaviors more often than adults. Although both adolescents and adults reported the same level of prior arrests (72% for each), twice as many adolescents were currently involved in the criminal justice system (62 vs 27%).

Measures

Global Appraisal of Individual Needs-Initial

The Global Appraisal of Individual Needs-Initial (GAIN-I)²⁶ is a comprehensive standardized biopsychosocial assessment battery of reliable and valid measures designed to support clinical decision making. The instrument provides diagnostic impressions based on the Diagnostic and Statistical Manual IV-TR²⁷ and the ASAM patient placement criteria 2-R,²⁸ and offers a variety of scoring, interpretation, and reporting options. With eight main sections (background, substance use, physical health, risk behaviors, mental health, environment, legal, and vocational), it requires about 90–120 min to complete. With both adolescents and adults, the GAIN-I’s main scales have good internal consistency (alpha over 0.90 on main scales, 0.70 on subscales) and test–retest reliability (rho over 0.70 on number of days and problem counts, kappa over 0.60 on categorical measures). The scales are also highly correlated with measures of use from timeline follow-back measures, urine tests, collateral reports, treatment records, and blind psychiatric diagnosis (rho of 0.70 or more; kappa of 0.60 or more).^{8,23,24,26,29,30} Confirmatory factor analyses suggest that the GAIN’s collection of psychiatric and behavioral problems items vary largely along four dimensions: (1) substance use problems (e.g., abuse, dependence, induced), (2) internalizing problems (e.g., depression, anxiety, trauma, suicide), (3) externalizing problems (e.g., attention deficit, hyperactivity/impulsivity, conduct disorders), and (4) crime and violence (e.g., interpersonal, oral, and physical violence, property crime, drug related crime, violent crime; CFI=0.92, RMSE=0.06).³¹ In this paper, we focus only on the non-substance use scales. Copies of the actual GAIN instruments and items, the syntax for creating scales and problem-specific group variables, and a comprehensive list of supporting studies are publicly available at http://www.chestnut.org/li/gain.

Three comorbidity scales from the GAIN-I were used to create the short comorbidity scales: the Internal Mental Distress Scale, the Behavior Complexity Scale, and the Crime and Violence Scale. Descriptive statistics on each scale and its associated subscales using data from the CYT and ERI studies are reported in Table 2.

Table 2.

Descriptive statistics on the GAIN-I full scales, criterion scales, and short comorbidity scales

	Adolescent (n=600)		Adult (n=448)
	Mean	Standard deviation	Mean	Standard deviation
GAIN-I Full Scales
Internal Mental Distress Scale	6.15	6.47	14.98	10.33
Somatic Symptom Index	0.93	1.15	2.04	1.59
Depression Symptom Scale	1.83	1.75	3.95	2.18
Homicidal-Suicidal Thought Scale	0.31	0.69	0.70	1.18
Anxiety-Fear Symptom Scale	1.59	1.92	3.83	2.97
Traumatic Stress Scale	1.49	2.60	4.45	4.92
Behavior Complexity Scale	9.70	6.95	9.40	9.14
Inattentive Disorder Scale	4.26	3.27	3.75	3.69
Hyperactivity-Impulsivity Scale	2.36	2.47	2.96	3.18
Conduct Disorder Scale	3.09	2.48	2.69	3.19
Crime and Violence Scale	5.77	4.68	4.24	4.98
General Conflict Tactic Scale	3.89	3.21	2.96	3.52
General Crime Scale	1.89	2.32	1.31	2.42
Criterion Variable Scales
Comorbidity Count	4.94	2.44	5.83	3.02
Emotional Problems Scale	0.21	0.17	0.26	0.24
Mental Health Treatment Index	0.01	0.04	0.01	0.06
Illegal Activities Scale	0.20	0.21	0.17	0.24
Substance Frequency Scale	0.16	0.13	0.34	0.22
Short Comorbidity Scales
Internal Behavior Scale-17	3.20	3.54	7.28	4.67
Depression Symptom Scale-5	1.39	1.54	3.09	1.79
Suicide Risk Scale-5	0.31	0.69	0.70	1.18
Anxiety-Trauma Scale-7	1.50	1.98	3.49	2.64
External Behavior Scale-16	5.83	3.92	4.48	4.12
Activity-Inattention Scale-6	2.08	1.91	2.06	2.22
Behavior Problem Scale-6	2.78	1.90	2.01	1.87
General Crime Scale-4	0.97	1.24	0.44	0.89

Internal Mental Distress Scale

The Internal Mental Distress Scale (IMDS) is a past-year measure of internalizing distress, consisting of 43 items answered on a yes/no (1/0) scale. Each item represents a symptom of internalizing difficulties, and scores are the sum of the endorsed “yes” items. Higher values indicate greater levels of internalizing mental distress. The IMDS is broken down into five subscales: Somatic Symptom Index (SSI, 4 items), Depression Symptom Scale (DSS, 9 items), Homicidal-Suicidal Thought Scale (HSTS, 5 items), Anxiety-Fear Symptom Scale (AFSS, 12 items), and the traumatic stress scale (TSS, 13 items). The SSI, DSS, and AFSS were based on factor analyses of the Hopkins’ Symptom Checklist (HSCL)³² and were supplemented with questions from DSM-IV²⁷ to better map onto criteria for depression and anxiety. The HSTS was based on Chestnut Health Systems’ suicide risk assessment, and the TSS was based on multiple IRT analyses of the combat and civilian Mississippi scale^33–36 for measuring PTSD and trauma.

Behavior Complexity Scale

The BCS consists of 33 items representing past year symptoms of externalizing difficulties that are answered on a yes/no scale and clustered into three subscales: the Inattentive Disorder Scale (IDS, 9 items), the Hyperactivity–Impulsivity Scale (HIS, 9 items), and the Conduct Disorder Scale (CDS, 15 items). Scores are the count of items endorsed “yes”, with higher scores representing greater levels of externalizing distress. Items were adapted from American Psychiatric Association DSM-IV criteria.²⁷

Crime and Violence Scale

The CVS consists of 31 items representing past-year symptoms of increasingly more violent responses to conflict and more diverse types of crime. They are answered on a yes/no scale and cluster into two subscales: the General Conflict Tactic Scale (GCTS; 12 items) and the General Crime Scale (GCS; 19 items). The 12-item GCTS is adapted from Strauss’³⁷ work on domestic violence and includes questions related to oral and physical violence. The 19-item GCS is a count of the number of different types of property, interpersonal, and drug-related illegal activities and is based on a lay version of the uniform crime reports³⁸ introduced in the 1995 National Household Survey on Drug Abuse.³⁹

Additional scales used for testing construct validity

To help address the question of the short comorbidity scales’ construct validity, five existing GAIN measures²⁶ that are logically related to domains covered by the short comorbidity scales were used. Descriptive statistics on these measures using data from the CYT and ERI samples are defined in Table 2.

The Comorbidity Count (alpha=0.71 adolescents, 0.78 adults) is a past-year measure of an individual’s number of comorbid problems as determined by the full scales. It has values of 0, 1, or 2, with 0 representing no reported internalizing or externalizing problems, 1 representing either an internalizing or an externalizing problem, and 2 representing both an internalizing and an externalizing problem. The remaining measures cover the past 90-day time period.

The Emotional Problems Scale (EPS; alpha=0.65 adolescents, 0.85 adults) is a measure of the severity of emotional and behavioral problems. It is created by taking the average of all proportional measures of each of the following: recency of, days bothered by, and days kept from responsibilities due to emotional problems; recency of and days disturbed by memories; and recency and days having problems paying attention or with self-control.

The Mental Health Treatment Index (MHTI; Spearman rho test–retest coefficient=0.67 adults, not available for adolescents) is a measure of service utilization. It is created by taking the average of all proportional (divided by 90) measures of each of the following: the number of nights or times visiting the emergency room, staying in the hospital, or visiting an outpatient facility formental health problems.

The Illegal Activity Scale (IAS; alpha=0.71 adolescents, 0.84 adults) is a measure of the frequency and severity of illegal activity. It is a proportional measure with higher scores indicating more recent and more days of illegal activity and increasingly more dependence on illegal activities for financial support.

The Substance Frequency Scale (SFS; alpha=0.76 adolescents and adults), a measure of the frequency of using alcohol and drugs, is created by taking the average of the percent of (a) days of any alcohol or drug use, (b) days of heavy use, (c) days of problems from use, (d) days of alcohol, marijuana, crack/cocaine, and heroin use, (e) days staying high most of the day, and (f) days where substance used kept the participant from meeting his/her responsibilities. Each of these last four scales’ scores range from 0 to 1, with greater values representing respectively more problems, service utilization, illegal activity, substance use, and substance-related problems.

This particular collection of scales was chosen to evaluate construct validity since it was anticipated that as scores on the short comorbidity scores increased, so too would the count of comorbid problems, severity of emotional problems, frequency of service utilization and substance use, and frequency and severity of illegal activity.

Development of the short comorbidity scales

Using the same process, the CYT and ERI datasets were each used to independently identify items for the short comorbidity scales. First, items from the ten full-length GAIN-I comorbidity subscales (five subscales from IMDS, three from BCS, and two from CVS) were each correlated with the total scores of the remaining items in their subscales. Subsets of three to five items per subscale with item-total correlations greater than 0.40 were selected for further analysis. Alpha reliabilities for each subset were computed with the goal of retaining the fewest number of items per subset combined with the highest reliabilities (preferably 0.70 or higher). The total score for each subset was then correlated with the total score on its associated full-length subscale. If the alpha or correlation criteria could not be met within five items, the subset was collapsed with other subsets in the same general domain (i.e., internalizing subsets could be collapsed with other internalizing subsets, externalizing subsets could be collapsed with other externalizing subsets). This led to a reduction from ten full-length subscales to six subsets (now deemed short subscales). The collapsed subscales contained items related to (a) somatic and depression symptoms, (b) anxiety and trauma, (c) impulsivity and hyperactivity symptoms, and (d) violence and conduct disorder. In sum, the process described above reduced the number of scales from 3 to 2 [i.e., Internal Behavior Scale (IBS-17) and External Behavior Scale (EBS-16), with the latter subsuming crime and violence], the number of subscales from 10 to 6, and the number of items from 107 to 33.

The full-length IMDS was shortened to the Internal Behavior Scale with 3 subscales: the Depression Symptom Scale with 5 items (DSS-5), the Suicide Risk Scale with 5 items (SRS-5), and the Anxiety-Trauma Scale with 7 items (ATS-7). The name of each internalizing short scale and subscale includes the number of items in order to distinguish it from the full GAIN-I scale and subscale with the same or a similar name. The range of scores for each scale and subscale is 0 to the total number of items.

The SRS-5 is the only short subscale in the analysis that is identical to its corresponding GAIN-I full-length subscale, the HSTS. In the original analysis, only 3 items were retained in the SRS subscale since the inclusion of the remaining 2 did not meet an acceptable level of psychometric integrity for both the adolescent and adult samples. However, in practice, counselors using the scale asked that the remaining 2 items (on homicide and suicide attempts) be retained. Although their inclusion slightly reduced the psychometric integrity of the subscale, its usefulness increased for those in the field.

The GAIN-I Behavior Complexity Scale (BCS) and Crime and Violence Scale (CVS) were combined into one External Behavior Scale with 3 subscales: the Activity-Inattention Scale with 6 items (AIS-6), the Behavior Problem Scale with 6 items (BPS-6), and the General Crime Scale with 4 items (GCS-4). To meet the specifications described above, the AIS-6 contains items associated with both the attention-deficit and hyperactivity dimensions of ADHD and the BPS-6 contains items on both conduct disorder and aggression. The name of each short externalizing scale and subscale includes the number of items in order to distinguish it from the full GAIN scale and subscale with a similar name. The range of scores for each scale and subscale is 0 to the total number of items.

Descriptive statistics on each short scale and its associated subscales using data from the CYT and ERI studies are reported in Table 2. The set of items composing the internal and external short comorbidity scales and subscales are described further at http://www.chestnut.org/LI/gain/GAIN_Q/index.html.

Results

Reliability analysis

Table 3 contains the internal consistency estimates for the short and full versions of the GAIN comorbidity scales and subscales for the adolescent and adult samples. The abbreviation and number of items for each measure are shown in parentheses. The first row in each set is the performance of the short comorbidity scale or subscale. Whenever the short scale items are drawn from 2 subscales in the full GAIN, the second row in the table shows the combination; a combination is the single most appropriate comparison for a short version since it represents the full version of that measure. If this combination has a name in the full GAIN, it is listed in Table 3. Where necessary, the next two rows in each set display the performance of the 1 to 2 subscales on the full GAIN from which the short subscales draw their items. To evaluate reliability, the last two columns display the Cronbach^40–42 alphas for each sample. Alpha is a measure of internal consistency and equivalent to the average correlation of all split half tests, the lower bound for an immediate test-retest reliability of precision, and the percent of total item variance explained by the first common factor. It is a function of both the average inter-item correlation and the number of items. Thus, while long scales (20 to 30 items) should have alphas near 0.90, it is common to accept alphas of approximately 0.70 for shorter scales (5 to 7 items).⁴³

Table 3.

Internal consistency estimates of short comorbidity scales^a

Version	Scale name (abbreviation or calculation—number of items)	Adolescent (n=600)	Adult (n=448)
Internal scales
Short	Depression Symptom Scale (DSS−5)	0.73*	0.84*
Fullb	Combined (SSI+DSS−13)	0.78*	0.88*
	Somatic Symptom Index (SSI−4)	0.63*	0.82*
	Depression Symptom Scale (DSS−9)	0.72*	0.86*
Shortc	Suicide Risk Scale (SRS−5)	0.54*	0.75*
Fullc,d	Homicidal-Suicidal Thought Scale (HSTS–5)	0.54*	0.75*
Short	Anxiety-Trauma Scale (ATS−7)	0.82*	0.88*
Fullb	Combined (AFSS+TSS−25)	0.88*	0.94*
	Anxiety–Fear Symptom Scale (AFSS−12)	0.72*	0.85*
	Traumatic Symptom Scale (TSS−13)	0.88*	0.96*
Short	Internal Behavior Scale (IBS−17)	0.86*	0.90*
Fulld	Internal Mental Distress Scale (IMDS: SSI+DSS+HSTS+AFSS+TSS−43)	0.91*	0.95*
External scales
Short	Activity-Inattention Scale (AIS−6)	0.77*	0.88*
Fulld	ADHD Scale (ADHDS: IDS+HIS−18)	0.90*	0.96*
	Inattentive Disorder Scale (IDS−9)	0.89*	0.95*
	Hyperactivity-Impulsivity Scale (HIS−9)	0.81*	0.91*
Short	Behavior Problem Scale (BPS−6)	0.74*	0.78*
Fullb	Combined (CDS+GCTS−27)	0.86*	0.92*
	Conduct Disorder Scale (CDS−15)	0.75*	0.86*
	General Conflict Tactic Scale (GCTS−12)	0.85*	0.91*
Short	General Crime Scale (GCS−4)	0.71*	0.69*
Fulld	General Crime Scale (GCS−19)	0.77*	0.88*
Short	External Behavior Scale (EBS−16)	0.83*	0.88*
Fullb	Combined (BCS+GCTS+GCS−64)	0.92*	0.95*
	Behavior Complexity Scale (IAS+HIS+CDS−33)	0.91*	0.96*
	Crime and Violence Scale (CVS: GCTS+GCS−31)	0.86*	0.90*

^aValues are Cronbach’s coefficient alpha.

^bNot a formal scale in the full GAIN, but the most appropriate combination for comparison with the short version.

^cThe SRS−5 and HSTS are identical.

^dA formal scale on the full GAIN.

^*p<0.05

A significance level of 0.05 was used for all statistical tests. All of the short and long measures are significantly different from 0, and 49 of 52 have alphas over 0.70. Though the short internal subscales are each typically only one third the length of their corresponding combined GAIN subscales, they have only slightly lower internal consistency. In each case the loss of alpha was 0.06 points or less. Although the adult data was more internally consistent than the adolescent data, the magnitude of the differences in alpha between the short and full combined subscales was nearly identical for both samples. The one short scale that dropped below 0.70 was the Suicide Risk Scale, which is also lower in the full GAIN for adolescents.

The shortened external subscales are typically one fourth the length of the full combined external subscales, but still have alphas over 0.80 on the short main scale (EBS-16), over 0.70 on 5 of the 6 subscales (a range of 0.71 to 0.88), and one subscale at 0.69. Here the loss of alpha was larger, ranging from 0.06 to 0.14 for most and 0.19 for adults on the General Crime Scale. The full length subscales – from where the short versions pull their items – have higher alphas than those for the short subscales, but this is in part because they have 2 to 4 times more items.

Correspondence analysis

Table 4 presents the correlations between the short (column 1) and full (column 2) versions of the comorbidity scales and subscales for adolescents and adults. As in Table 3, where subscales from the full GAIN comorbidity subscales are combined, the correlation between the short subscale and the corresponding combined GAIN-I subscales in the first row of a set is reported. The short depression symptom scale-5, which is designed to approximate the GAIN-I’s combined Somatic Symptom Index and Depression Symptom Scale, is very highly correlated with the full versions for both the adolescent (r=0.90) and adult (r=0.94) samples. It is also correlated with each of the individual subscales at 0.70 or higher. The short Anxiety-Trauma Scale-7 also correlates highly (r=0.91 and 0.95 respectively) with the combined full version of the Anxiety-Fear Scale and the Traumatic Stress Scale. It is also correlated with each of the individual subscales at 0.70 or higher. More importantly, the short (17-item) Internal Behavior Scale is correlated highly with the full IMDS (r=0.94 and 0.97, respectively).

Table 4.

Correlations between short and full versions of the GAIN comorbidity scales^a

Short comorbidity scales (abbreviation/ calculation— number of items)	Full comorbidity scales (abbreviation/calculation— number of items)	Adolescent (n=600)	Adult (n=448)
Internal scales
Depression Symptom Scale (DSS−5)	Combined (SSI+DSS−13)b	0.90*	0.94*
	Somatic Symptom Index (SSI−4)	0.75*	0.84*
	Depression Symptom Scale (DSS−9)	0.79*	0.83*
Suicide Risk Scalec (SRS−5)	Homicidal/Suicidal Thought Scale (HSTS−5)c,d	1.00*	1.00*
Anxiety-Trauma Scale (ATS−7)	Combined (AFSS+TSS−25)b	0.91*	0.95*
	Anxiety-Fear Symptom Scale (AFSS−12)	0.75*	0.78*
	Traumatic Symptom Scale (TSS−13)	0.82*	0.88*
Internal Behavior Scale (IBS: DSS+SRS+ATS−17)	Internal Mental Distress Scale (IMDS: SSI+DSS+HSTS+AFSS+TSS−43)d	0.94*	0.97*
External Scales
Activity-Inattention Scale (AIS−6)	ADHD Scale (ADHDS: IDS+HIS −18)d	0.93*	0.96*
	Inattentive Disorder Scale (IDS−9)	0.88*	0.94*
	Hyperactivity-Impulsivity Scale (HIS−9)	0.79*	0.91*
Behavior Problem Scale (BPS−6)	Combined (CDS+GCTS−27)b	0.92*	0.96*
	Conduct Disorder Scale (CDS−15)	0.74*	0.83*
	General Conflict Tactic Scale (GCTS−12)	0.82*	0.82*
General Crime Scale (GCS−4)	General Crime Scale (GCS−19)d	0.92*	0.86*
External Behavior Scale (EBS−16)	Combined (BCS+GCTS+GCS−64)b,e	0.96*	0.97*
	Behavior Complexity Scale (IDS+HIS+CDS−33)	0.86*	0.92*
	Crime/Violence Scale (GCTS+GCS - 31)e	0.75*	0.72*

^aValues are Spearman Rho’s.

^bNot a formal scale in the full GAIN, but the most appropriate combination for comparison with the short version.

^cThe SRS-5 and HSTS are identical.

^dA formal scale on the full GAIN.

^eOn the full GAIN, the GCTS and PCS contain some of the items from the Crime/Violence Scale.

^*p<−0.05

For the external scales and subscales, the activity-inattention scale-6 correlates highly with the combined full ADHD subscale among both adolescents (r=0.93) and adults (r=0.96). Moreover, it is correlated with the original subscales for inattentiveness and hyperactivity–impulsivity at 0.79 or higher. Similarly, the behavior problem scale-6 is highly correlated with the combined conduct disorder and conflict subscales (r=0.92 and 0.96, respectively) as well as for each full GAIN-I subscale (0.74 or higher). The short general crime scale (with four items) was highly correlated with the full version (with 19 items) for both adolescents (r=0.92) and adults (r=0.86). The short (16-item) External Behavior Scale was highly correlated (r=0.96 and 0.97, respectively) with the combined behavior complexity and crime/violence subscales from the full GAIN.

Construct validation

Tables 5 (adolescents) and 6 (adults) compare the correlations of the short and full GAIN comorbidity scales and subscales as predictors of several health outcomes with which they are expected to correlate. The logic of these construct validity tests lies in the expectation that a good approximation of a full scale or subscale should not only correlate highly with the original, but also produce parallel correlations with external criteria.⁴⁴ As shown in Tables 5 and 6, virtually all of the external criteria were correlated with the short and full comorbidity scales (p<0.05 shown in bold), with the strength of the correlations in descending order (from left to right columns) as follows: (a) the count of co-occurring conditions based on items in the full GAIN, (b) the emotional problems scale based on emotional and behavioral problems in the past 90 days, (c) the utilization of Mental Health Treatment Index (MHTI) measuring services in the past 90 days, (d) the Illegal Activity Scale for the past 90 days, and (e) the Substance Frequency Scale (SFS) measuring frequency of use in the past 90 days. Correlations with the Illegal Activity Scale were highest for the six-item behavior problem scale (and corresponding full subscales for conduct disorder and conflict) and both the short and long versions of the general crime scale, logical relationships given the similarity of the constructs.

Table 5.

Comparison of short and full comorbidity scales with correlations of health outcomes-adolescents

Version	Scale name (abbreviation or calculation— number of items)	Comorbidity Countd	Emotional Problems Scale	Mental Health Treatment Index	Illegal Activity Scale	Substance Frequency Scale
Internal scales
Short	Depression Symptom Scale (DSS−5)	0.62	0.53	0.26	0.13	0.18
Fulla	Combined (SCI+DSS−13)	0.65	0.57	0.24	0.16	0.20
	Somatic Symptom Scale (SSI−4)	0.48	0.42	0.14	0.12	0.15
	Depression Symptom Scale (DSS−9)	0.62	0.54	0.27	0.16	0.19
Shortb	Suicide Risk Scale (SRS−5)	0.37	0.37	0.12	0.22	0.15
Fullb,c	Homicidal-Suicidal Thought Scale (HSTS–5)	0.37	0.37	0.12	0.22	0.15
Short	Anxiety-Trauma Scale (ATS−7)	0.66	0.70	0.21	0.20	0.15
Fulla	Combined (AFSS+TSS – 25)	0.68	0.71	0.23	0.22	0.19
	Anxiety-Fear Symptom Scale (AFSS–12)	0.64	0.54	0.23	0.17	0.17
	Traumatic Symptom Scale (TSS−13)	0.52	0.72	0.19	0.20	0.16
Short	Internal Behavior Scale (IBS−17)	0.72	0.70	0.25	0.21	0.20
Fullc	Internal Mental Distress Scale (IMDS: SSS+DSS+HSTS+AFSS+TSS - 43)	0.73	0.71	0.25	0.23	0.22
External scales
Short	Activity-Inattention Scale (AIS−6)	0.70	0.58	0.26	0.21	0.18
Fullc	ADHD Scale (IAS+HIS −18)	0.75	0.63	0.28	0.25	0.20
	Inattentive Disorder Scale (IDS−9)	0.72	0.61	0.27	0.28	0.22
	Hyperactivity-Impulsivity Scale (HIS−9)	0.63	0.52	0.22	0.15	0.12
Short	Behavior Problem Scale (BPS−6)	0.54	0.46	0.19	0.35	0.22
Fulla	Combined (CDS+GCTS –27)	0.56	0.46	0.18	0.39	0.24
	Conduct Disorder Scale (CDS –15)	0.70	0.53	0.17	0.38	0.24
	General Conflict Tactic Scale (GCTS−12)	0.32	0.30	0.15	0.29	0.19
Short	General Crime Scale (GCS−4)	0.34	0.28	0.13	0.68	0.23
Fullc	General Crime Scale (GCS−19)	0.37	0.32	0.11	0.77	0.25
Short	External Behavior Scale (EBS−16)	0.72	0.60	0.26	0.47	0.27
Fulla	Combined (BCS+GCTS+GCS−64)	0.75	0.61	0.26	0.47	0.27
	Behavior Complexity Scale (BCS:IAI+HIS+CDS−33)	0.79	0.64	0.27	0.31	0.23
	Crime and Violence Scale (CVI: GCTS+GCS−31)	0.41	0.37	0.15	0.53	0.25

Correlations are Spearman Rho’s, with bold indicating significant difference (p<0.05) from 0

^aNot a formal scale in the full GAIN, but the most appropriate combination for comparison

^bSame items in both the short and full version

^cA formal scale on the full GAIN

^dCount of whether participants have 0, 1, or 2 co-occurring disorders

Table 6.

Comparison of short and full comorbidity scales with correlations of health outcomes-adults

Version	Scale name (abbreviation or calculation— number of items)	Comorbidity Countd	Emotional Problems Scale	Mental Health Treatment Index	Illegal Activity Scale	Substance Frequency Scale
Internal scales
Short	Depression Symptom Scale (DSS−5)	0.65	0.56	0.22	0.17	0.29
Fulla	Combined (SCI+DSS–13)	0.70	0.60	0.21	0.18	0.29
	Somatic Symptom Scale (SSI−4)	0.54	0.48	0.18	0.10	0.26
	Depression Symptom Scale (DSS−9)	0.70	0.58	0.19	0.22	0.27
Shortb	Suicide Risk Scale (SRS−5)	0.44	0.47	0.25	0.21	0.11
Fullb,c	Homicidal–Suicidal Thought Scale (HSTS−5)	0.44	0.47	0.25	0.21	0.11
Short	Anxiety–Trauma Scale (ATS−7)	0.72	0.81	0.24	0.20	0.16
Fulla	Combined (AFSS+TSS−25)	0.75	0.83	0.26	0.22	0.18
	Anxiety–Fear Symptom Scale (AFSS−12)	0.71	0.65	0.22	0.19	0.20
	Traumatic Symptom Scale (TSS−13)	0.57	0.79	0.24	0.20	0.12
Short	Internal Behavior Scale (IBS−17)	0.76	0.79	0.29	0.23	0.23
Fullc	Internal Mental Distress Scale (IMDI: SSS+DSS+HSTS+AFSS+TSS−43)	0.78	0.81	0.28	0.23	0.22
External scales
Short	Activity-Inattention Scale (AIS−6)	0.76	0.73	0.15	0.27	0.18
Fullc	ADHD Scale (ADHDS: IAS+HIS −18)	0.79	0.75	0.15	0.29	0.16
	Inattentive Disorder Scale (IDS−9)	0.76	0.73	0.16	0.27	0.14
	Hyperactivity–Impulsivity Scale (HIS−9)	0.78	0.71	0.11	0.30	0.17
Short	Behavior Problem Scale (BPS−6)	0.63	0.22	0.05	0.35	0.22
Fulla	Combined (CDS+GCTS−27)	0.62	0.22	0.02	0.35	0.22
	Conduct Disorder Scale (CDS−15)	0.76	0.18	0.08	0.34	0.18
	General Conflict Tactic Scale (GCTS−12)	0.35	0.18	−0.01	0.27	0.18
Short	General Crime Scale (GCS−4)	0.31	0.21	0.03	0.65	0.21
Fullc	General Crime Scale (GCS−19)	0.32	0.20	0.03	0.79	0.20
Short	External Behavior Scale (EBS−16)	0.76	0.24	0.11	0.42	0.24
Fulla	Combined (BCS+GCTS+GCS−64)	0.77	0.22	0.10	0.43	0.22
	Behavior Complexity Scale (BCS:IAI+HIS+CDS−33)	0.81	0.18	0.13	0.33	0.18
	Crime and Violence Scale (CVI: GCTS+GCS−31)	0.41	0.22	0.01	0.50	0.22

Correlations are Spearman Rho’s, with bold indicating significant difference (p<0.05) from 0.

^aNot a formal scale in the full GAIN, but the most appropriate combination for comparison.

^bSame items in both the short and full version.

^cA formal scale on the full GAIN

^dCount of whether participants have 0, 1, or 2 co-occurring disorders.

A key focus of this paper is to compare the pattern of correlations between the criterion variables with the short scales and subscales and the combined versions of the corresponding full subscales (the first and second rows of each set). For example, among adolescents (Table 5), the short (five items) Depression Symptom Scale-5 and full (13 items) combined SSI + DSS-13 have very similar correlations with the Comorbidity Count (r=0.62 vs 0.65), Emotional Problems (r=0.53 vs 0.57), Mental Health Treatment Index (r=0.26 vs 0.24), Illegal Activity Scale (r=.13 vs 0.16), and Substance Frequency Scale (r=0.18 vs 0.20). A review of the other internal short subscales and total combined subscales shows that the correlations with short and full versions do not differ by more than 0.04 points for adolescents (top of Table 5) or 0.05 points for adults (top of Table 6. In most cases, the difference is less than or equal to 0.02 and in a few cases the short version is actually 0.01 to 0.02 higher. Similarly, for the external scales and subscales, all but one of the adolescent and adult correlations differed by less than or equal to 0.05, with most less than 0.20 and a few in the opposite direction. The one exception was the correlation between the General Crime Scale and Illegal Activity Scale. In this case, the short comorbidity scale did significantly worse than the full scale for both adolescents (r=0.68 vs 0.77) and adults (r=0.65 vs 0.79).

Sensitivity and specificity of cut points for clinical interpretation

Interpretive cut-off scores for the full GAIN comorbidity scales vary by scale. Because the short comorbidity scales have fewer items, the clinical cut-offs were modified. To simplify interpretation, a common scheme was developed to triage the clinical severity of each short scale: a low severity level is assigned when 0–24% of the items are endorsed, moderate is assigned when 25–74% of the items are endorsed, and a high severity is designated by endorsement of 75–100% of the items.

Table 7 presents information on the ability of the short scales’ interpretive ranges to predict diagnostic impressions generated by the GAIN-I full comorbidity scales for internalizing and externalizing problems. The prevalence of each diagnostic impression is listed for adolescents and adults in the first column. The next four columns list the sensitivity (percent of cases as determined by the full GAIN-I comorbidity scales that are correctly identified by the short comorbidity scales) and specificity (percent of non-cases as determined by the full GAIN-I comorbidity scales that are correctly ruled out by the short comorbidity scales) for each of the proposed interpretive ranges when used to predict diagnostic impressions using the full GAIN-I scales.

Table 7.

Accuracy of short comorbidity scales to predict diagnostic impressions based on the full comorbidity scales

GAIN-Q measure (predictor) GAIN-I measure (criterion)	Prevalencea	Sensitivityb,c,e		Specificityb,d,e
	Overall	Mod/high	High only	Mod/high	High only
Adolescents (N=600)
Internal Behavior Scale	32%	98%	49%	73%	99%
(0–17) any internal problem
Depression	18%	84%	13%	85%	100%
Anxiety	20%	80%	12%	86%	100%
Suicidal ideation	9%	84%	20%	78%	99%
High traumatic stress	14%	95%	17%	84%	100%
External Behavior Scale	61%	99%	56%	42%	97%
(0–16) any external problem
AD, HD or both	38%	99%	23%	51%	100%
Conduct problems	53%	95%	17%	62%	100%
Adults (N=448)
Internal Behavior Scale	75%	99%	74%	65%	99%
(0–17) any internal problem
Depression	61%	95%	25%	74%	100%
Anxiety	60%	95%	25%	72%	100%
Suicidal ideation	22%	97%	61%	40%	98%
High traumatic stress	44%	97%	34%	54%	99%
External Behavior Scale	45%	100%	59%	62%	98%
(0–16) any external problem
AD, HD or both	34%	97%	15%	73%	100%
Conduct problems	37%	96%	14%	75%	100%

^aPrevalence of problem(s) using the GAIN-I full scale score.

^bMod/High score 5–17 for internal, 4–16 for external; High Only score 13–17 for internal, 12–16 for external

^cSensitivity - percent of actual cases correctly identified at this score or higher

^dSpecificity - percent of actual cases correctly excluded at this score or higher

^eMod/High is based on scores falling in the Moderate or High ranges (25–100% of symptoms) on any scale or subscale; high only (75–100%) is based on scores falling in the High range on any scale or subscale.

Using an interpretive range of moderate or above (Mod/High) had at least 98% sensitivity on any internalizing and any externalizing problems for both adults and adolescents (in bold), with sensitivity for many common specific problems in the 95 to 99% range. Using this same cut point to determine specificity resulted in correctly ruling out 42 to 73% of non-cases. Thus, the moderate/high cut point errs on the side of inclusion, with over identification of potential cases.

Using the high severity cut point improved specificity to 97% or better, but reduced sensitivity to 49 to 74%. Thus, the high cut point is more certain (i.e., not likely to produce a false positive judgment), but achieves this by erring on the side of exclusion, with under identification of real cases.

Discussion

Reprise

The results of the psychometric analysis suggest that the short comorbidity scales and subscales are reliable, show a close correspondence with their full length counterparts, and exhibit patterns of construct validity that reflect those seen in the full scales and subscales. This was accomplished in spite of reducing the number of scales from 3 (with 10 subscales and 107 items) to a shorter set of 2 scales (with 6 subscales and 33 items). Comparison of the full and short comorbidity scales with a range of substance use-related and health outcomes further demonstrates that they have similar convergent and discriminate properties in terms of construct validity. Using the moderate severity cut point (where at least 25% of the items are endorsed) seems to be the optimal cut point for distinguishing who could potentially benefit from further assessment or a brief intervention. While there is some risk of over identification at this cut point, using it is the safer alternative to missing potential cases; any remaining questions surrounding the nature and severity of difficulties should be resolved through further assessment.

Implications for Behavioral Health

These methodological findings have several important substantive implications. First, shorter and more efficient scales will make it feasible to assess individuals in more diverse settings (e.g., student or employee assistance programs, juvenile or criminal justice settings, primary or mental health care, or welfare) and with lower severity populations where detailed assessment may not be warranted. In an integrated system, the short version will also facilitate a better allocation of resources, referrals, and placement decisions.

Short versions of the GAIN-I comorbidity scales and subscales also make it feasible to measure these concepts in applied research where longer instruments are difficult to integrate into batteries of tests that are already long and/or in community or outreach situations where every minute counts. They may be particularly useful in quasi-experimental non-randomized studies as a statistical method of removing confounding effects in tests of treatment effectiveness. For example, tests of treatment effectiveness in natural sites may be confounded by a disproportionate representation of patients with comorbid conditions. Using analysis of covariance or some other partialing strategy would permit the isolation and removal of confounding effects due to the co-occurrence of other mental health problems. By removing the confounding variance in the Type III sums of squares, we can avoid the problem of overestimating the main effects of the variables in question.

For both clinical and research management, it is increasingly important to recognize the time devoted to assessment as a scarce resource. As the demands to measure increasingly more issues continue to grow, it is important to make our tools more efficient. This paper demonstrates that the assessment of co-occurring mental health issues complicating substance use problems can be made more efficient with minimal loss to utility and validity. As the number of items on the comorbidity scales dropped from 107 to 33, the time to administer dropped from 30 min to under 10 min.

The construction and validation of a short screener provides an inexpensive method of detecting co-occurring illnesses that have the capacity to drastically inflate the cost of mental and physical health treatment. For example, the health care utilization of treated alcoholics with a history of depression has been found to be much higher than that associated with treated alcoholics without a history of depression.⁴⁵ Moreover, depressed clients’ levels of health care utilization observed 3 years before entering treatment were slower to return to baseline than those of the non-depressed group. These results suggest that knowing the full mental health profile at intake could greatly improve the detection of complicating factors and, in the process, greatly improve the treatment of all health problems.

Strengths and limitations

The strengths of this study include large sample sizes, a within-subject design, and the ability to show findings that are robust across very different samples. While an important first step, it should be noted that the study was limited in the number and type of external criteria that were used to test construct validity. All of the criteria used were self-reported and, therefore, were probably affected by some common method variance. Future studies might benefit from including collateral reports and/or archival health care utilization data in a multi-method study.⁴⁶

Directions for future research

Future studies should continue to investigate the psychometric characteristics of the GAIN short comorbidity scales and subscales. There is always a need to add to the critical mass of psychometric evidence for any instrument, and linking the scales to treatment outcomes and long-term health outcomes will provide additional confidence that the scales and subscales measure what they appear to measure. To accommodate the increasing number of Spanish-speaking clients in the system, the short scales have been translated into Spanish. Investigations of the Spanish version’s psychometric properties are pending and represent an important next step. There may well be some advantage to considering translations into higher frequency languages in the USA, such as Mandarin and Vietnamese. Another important step would be to collect community-based norms on these assessments.