Test-retest reliability of self-report measures in a dually diagnosed sample

Abstract

Many individuals diagnosed with a substance use disorder are also diagnosed with another psychiatric disorder. Little is known regarding which treatments are efficacious for these dually-diagnosed individuals (DDI). Characterizing the psychometric properties of assessments used with DDI samples is essential to efficacy studies with DDI. This study examined the internal consistency and test-retest reliability of self-report instruments in DDI. Most subscales demonstrated high test-retest reliability; one subscale demonstrated poor reliability. Internal consistency was similar to that of non-DDI samples. This exploratory study suggests that, while some instruments should be interpreted cautiously, DDI samples can be accurately assessed with self-report measures.

Introduction

The prevalence of co-occurring mental health and substance use issues in clinical populations is so high that dual diagnosis is considered the norm rather than the exception. Individuals diagnosed with both a substance use disorder (SUD) and another psychiatric disorder account for 41%– 65% of substance abusers (USDHHS, 1999). While these rates may seem high, they are consistent across a range of studies (Cantor-Graae, Nordström, & McNeil, 2001; Margolese, Malchy, Negrete, Tempier, & Gill, 2004; Regier, et al., 1990; Swartz, et al., 2006). Over 40% of those seeking treatment for an alcohol use disorder also report a mood disorder, while 33% report an anxiety disorder (Grant, et al., 2004). Both epidemiological surveys and studies of clinical populations have consistently documented these high comorbidity rates.

When substance use and mental illness are comorbid, each is exacerbated (Swann, 2010). Comorbid disorders are more severe and have a greater effect on patients’ quality of life than does a single diagnosis (Burns & Teesson, 2002; Kessler, 1995). Among those with severe mental illness (SMI), substance use problems are associated with more frequent relapses and more psychosocial consequences (McLellan, Luborsky, Woody, O'Brien, & Druley, 1983). The presence of a comorbid substance use disorder may also affect symptomatology in patients with a severe mental illness (e.g., Talamo, et al., 2006). Conversely, treatment for substance use can improve the course of the comorbid mental illness (Smelson, et al., 2008).

While comorbidity is a dominant clinical reality for most treatment settings, traditional approaches have generally isolated treatment for substance abuse from treatment for other mental illnesses. Integrated treatment models that combine mental health and substance use interventions are needed to address the needs specific to those with co-occurring disorders (Drake, et al., 2001; Horsfall, Cleary, Hunt, & Walter, 2009; Weiss, et al., 2007). However, instruments examined in non-mentally ill SUD patients may not display similar psychometric characteristics in dually diagnosed individuals (DDI). Few studies assessing the reliability and validity of standard self-report instruments have been conducted in this population, and the available studies raise serious concerns. For example, the Addiction Severity Instrument ASI: (McLellan, et al., 1992) is a psychometrically-sound scale used in SUD treatment settings to assess functionality. While some evidence exists supporting the validity of ASI in DDI, most studies indicate mixed reliability and validity, impeding research using this instrument (Carey, Cocco, & Correia, 1997; Hodgins & el-Guebaly, 1992; Zanis, McLellan, & Corse, 1997). Despite this, the instrument is widely used in DDI populations (Appleby, Dyson, Altman, & Luchins, 1997; Weiss, et al., 2007).

When assessment tools designed for a specific population are adapted for another group, it is important to evaluate the relevance of their content. Scales should be reviewed to assure appropriateness and reduce ambiguity. Many instruments show inconsistent temporal stability across populations. For example, while the Beck Depression Inventory (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) has adequate test-retest reliability in mentally ill populations, it has shown poor test-retest reliability (r=.49) in a dually-diagnosed sample (Lykke, Hesse, Austin, & Oestrich, 2008). No published reliability data from dually-diagnosed populations are available for the instruments included in this paper

Current Study

Psychometric properties for many measures commonly used in DDI populations have been developed and tested in non-mentally-ill populations (e.g., Miller & Tonigan, 1996; Moyers & Miller, 1993; Tonigan, Miller, & Vick, 2000). However, to date, no psychometric data are available for many instruments with DDI populations. The present study reports the test-retest reliability and internal consistency of seven such measures. These data were collected as part of an exploratory aim in an NIH-funded clinical trial assessing the effectiveness of a twelve-step facilitation program specially adapted to meet the needs of those with co-occurring disorders. The aims of this analysis were to (1) provide the first test-retest reliability estimates for seven self-report measures in DDI, and (2) compare test-retest reliability between groups with different psychiatric diagnoses. Findings of this study will allow for further evaluation of 12-step treatment and provide data in regard to the use of traditional measurements in DDI.

Material and methods

Participants

Participants consisted of the first 39 subjects enrolled in a randomized controlled trial of a modified twelve-step facilitation program. To meet inclusion criteria, patients must have met diagnostic criteria for either a psychotic disorder or a major mood disorder in addition to alcohol abuse or dependence. The majority of participants were male (56.4%), with a mean age of 40, ranging from 24 to 54 years of age. About half (53.8%) of the participants were Caucasian, 40.5% were Hispanic. Racioethnic data were not supplied by 2 of the participants. Two-thirds of the participants were single or divorced and two-thirds were unemployed. Frequencies of psychiatric diagnoses are provided in Table 1.

Table 1.

Frequencies of Psychiatric Diagnosis.

Diagnosis	Frequency	Percent
Major Depressive Disorder	20	51.3
Depressive Disorder NOS	1	2.6
Bipolar I Disorder	4	10.3
Other Bipolar Disorder	4	10.3
Schizophrenia	6	15.4
Schizoaffective Disorder	2	5.1
Delusional Disorder	1	2.6
Psychotic Disorder NOS	1	2.6

Measures

The Alcohol Abstinence Self-efficacy Scale (AASE; DiClemente, Carbonari, Montgomery, & Hughes, 1994) is a 40-item scale asks the respondent to rate their temptation to drink and the confidence they have to avoid drinking in different situations. The Stages of Change and Treatment Eagerness Scale (SOCRATES) measures the patient’s motivation to change (Miller & Tonigan, 1996). The Understanding of Alcoholism Scale (UAS) assesses the patients’ beliefs about alcoholism (Moyers & Miller, 1993). A 3-item subscale of the 12-Step Participation questionnaire (TSPQ) is used to quantify attendance in 12-step programs (Tonigan, Miller, & Connors, unpublished instrument). The 12-step and Double Trouble in Recovery (DTR) versions of the Twelve Step Attitudes Questionnaire for Dual Diagnosis (TSAQ-DD) were developed in order to measure patients’ attitudes toward these interventions (Bogenschutz & Akin, 2000). The General Alcoholics Anonymous Tools for Recovery (GAATOR) is a 26-item scale that measures endorsement of prescribed 12-step spiritual beliefs and practices (Montgomery, Miller, & Tonigan, 1995).

Procedure

All procedures for this study were approved and overseen by the Human Research Review Committee (HRRC) of the University of New Mexico. Participants were administered a battery of seven self-report assessment instruments at baseline and at week one. The one-week interval was chosen in order to balance the concern for change in attributes measured with that of recall of previous responses. The battery included five commonly-used scales whose psychometric properties have not been characterized in this population and two new scales developed specifically to address the attitudes of DDI towards 12-step programs.

Data analysis

Two types of test-retest reliability estimates were computed: absolute agreement, as estimated by the intraclass correlation coefficient (Shrout & Fleiss, 1979), and relative agreement, as estimated by Pearson’s r. While high relative agreement would indicate that the rank orderings of scores were consistent across the one-week period, high absolute agreement would suggest that the scores themselves, not just their relative rankings, remained stable over time. The ICC is the focus of this analysis because it provides an unbiased estimate of reliability and excludes systematic variation across the time period (Berk, 1979).

Results

Descriptive Statistics and Internal Consistency

Means, standard deviations, and coefficient alphas for full scales as well as subscales are provided in Table 2. The majority of the subscale coefficients fell in the recommended range of above .70 (Nunnally & Bernstein, 1994), demonstrating sufficient intercorrelation among items.

Table 2.

Descriptive Statistics and Internal Consistencies for Baseline and One-week Assessments

Scale	Baseline		One-Week
	Mean (SD)	Cronbach’s α	Mean (SD)	Cronbach’s α
TSPQ	4.33 (9.77)	0.392	0.21 (4.05)	0.329
TSAQ-DD (Twelve Step)
Positive Attitude	0.61 (0.30)	0.834	0.61 (0.31)	0.861
Illness Related Problems	0.53 (0.32)	0.741	0.54 (0.31)	0.702
TSAQ-DD (Double Trouble)
Positive Attitude	0.75 (0.28)	0.857	0.84 (0.26)	0.922
Illness Related Problems	0.30 (0.27)	0.632	0.26 (0.26)	0.700
GAATOR Total	13.4 (7.9)	0.896	13.7 (8.8)	0.917
Higher Power	7.1 (4.7)	0.905	6.9 (4.6)	0.901
Inventory	2.8 (2.4)	0.766	2.4 (2.5)	0.777
Self-Inventory	1.2 (1.3)	0.515	1.6 (1.4)	0.455
AASE
Temptation	48.6 (13.2)	0.870	47.7 (15.6)	0.925
Confidence	41.4 (17.2)	0.929	36.7 (16.5)	0.923
SOCRATES
Recognition	29.4 (5.4)	0.868	29.2 (5.5)	0.863
Ambivalence	13.5 (3.5)	0.537	13.6 (4.4)	0.804
Taking Steps	32.3 (5.4)	0.823	33.3 (6.4)	0.921
UAS
Disease Model	3.4 (0.53)	0.778	3.4 (0.60)	0.829
Psychosocial	3.6 (0.53)	0.639	3.6 (0.61)	0.771
Heterogeneity	3.0 (0.47)	0.072	2.9 (0.51)	0.082
Moral/Spiritual	3.0 (0.73)	0.746	3.1 (0.76)	0.765

Note. AASE: Alcohol Abstinence Self-efficacy Scale ; GAATOR: General Alcoholics Anonymous Tools for Recovery; SOCRATES: Stages of Change and Treatment Eagerness Scale; TSPQ: 12-Step Participation questionnaire; TSAQ-DD: Twelve Step Attitudes Questionnaire for Dual Diagnosis; UAS: Understanding of Alcoholism Scale

Test-Retest Reliability

Intraclass correlation coefficients (ICCs) and Pearson correlation coefficients are provided in Table 3. Most ICCs were in the good (.60–.74) to excellent (.75–1.00) range (Cicchetti & Sparrow, 1981), demonstrating high reliability. One subscale, Understanding Alcoholism Scale- Moral/Spiritual, demonstrated poor (< .40) reliability. It is worth noting that the UAS was developed for use with substance abuse clinicians, not clients, and that the Moral/Spiritual subscale has not yet been validated in any population.

Table 3.

Test-Retest Estimates for Twelve-Step Instruments

Scale	ICC	95% CI		Pearson r
		Lower Bound	Upper Bound
TSPQ	0.686^^	0.406	0.850	0.729**
TSAQ-DD (Twelve Step)
Positive Attitude	0.870^	0.732	0.939	0.867**
Illness Related Problems	0.877^	0.769	0.938	0.877**
TSAQ-DD (Double Trouble)
Positive Attitude	0.533^^^	−0.047	0.841	0.519
Illness Related Problems	0.652^^	0.197	0.886	0.652*
GAATOR Total	0.766^	0.585	0.874	0.764**
Higher Power	0.740^^	0.549	0.858	0.743**
Inventory	0.551^^^	0.271	0.736	0.551**
Self-Inventory	0.744^^	0.552	0.861	0.756**
AASE
Temptation	0.744^^	0.540	0.865	0.756**
Confidence	0.436^^^	0.118	0.675	0.445*
SOCRATES
Recognition	0.839^	0.712	0.913	0.836**
Ambivalence	0.501^^^	0.217	0.707	0.507**
Taking Steps	0.482^^^	0.199	0.692	0.489**
UAS
Disease Model	0.624^^	0.385	0.785	0.625**
Psychosocial	0.744^^	0.556	0.859	0.741**
Heterogeneity	0.652^^	0.427	0.802	0.661**
Moral/Spiritual	0.370^^^^	0.062	0.615	0.369*

^*p < .05 (2-tailed)

^**p < 0.01 (2-tailed);

^{^}= Excellent reliability

^{^^}= Good reliability

^{^^^}= Fair reliability

^{^^^^}= Poor Reliability

Note. AASE: Alcohol Abstinence Self-efficacy Scale; GAATOR: General Alcoholics Anonymous Tools for Recovery; SOCRATES: Stages of Change and Treatment Eagerness Scale; TSPQ: 12-Step Participation questionnaire; TSAQ-DD: Twelve Step Attitudes Questionnaire for Dual Diagnosis; UAS: Understanding of Alcoholism Scale

Differences Associated with Diagnosis

Differences were observed between patients with depressive disorder and other disorders. ICCs and effect sizes (q; Cohen, 1988) for between-group differences can be found in Table 4. Although neither a depressive disorder nor a psychotic disorder was related to poor test-retest reliability and/or consistency in all scales, each diagnostic group showed low stability over time on at least one scale. In general, the scores in the depressive group were less reliable than were scores in the other group. No scale showed poor reliability in both diagnostic categories. It may be the case that distinct deficits observed in temporal stability are related to diagnosis. However, the effect sizes for the majority (13 of 18) of the scales and subscales were small (10–.29).

Table 4.

Test-Retest Estimates and Effect Sizes for Twelve-Step Instruments in Depressive and Other Groups

Scale	ICC		Effect Size
	Depressive	Other
TSPQ	.604*	.815**	−0.444
TSPQ-DD (Twelve Step)
Positive Attitude	.859**	.886**	−0.113
Illness Related Problems	.876**	.891**	−0.068
TSPQ-DD (Double Trouble)
Positive Attitude	.252^^	.869**	−1.071
Illness Related Problems	.359^^	.910**	−1.152
GAATOR Total	.800**	.656*	0.313
Higher Power	.761**	.706*	0.119
Inventory	.470^	.736*	−0.432
Self-Inventory	.667*	.773**	−0.222
AASE
Temptation	.680*	.809**	−0.295
Confidence	.220^^	.633*	−0.523
SOCRATES
Recognition	.784**	.909**	−0.466
Ambivalence	.294^^	.684*	−0.534
Taking Steps	.575^	.355^^	0.284
UAS
Disease Model	.770**	.540^	0.416
Psychosocial	.846**	.637*	0.489
Heterogeneity	.728*	.535^	0.327
Moral/Spiritual	.608*	.179^^	0.525

Note: “Other” disorders include both psychotic disorders and bipolar disorders. Effect size is q (Cohen, 1988).

^**= Excellent reliability

^*= Good reliability

^{^}= Fair reliability

^{^^}= Poor reliability

One scale for which diagnosis had a large (> .50) effect size was the DTR version of the TSAQ-DD. Stability for the Positive Attitude and Illness-Related Problems subscales was considerably lower in the depressed group than in the psychotic group. However, the TSAQ for Twelve Step Programs showed excellent reliability in this same group. The TSAQ for Twelve Step and for DTR are nearly identical; the only difference involves to which mutual-help program the participants refer in their responses. Both of these scales showed excellent reliability in the psychotic group. Given the high reliability of the Twelve Step version of this scale in both groups, the poor reliability of the TSAQ-DD for DTR may reflect more limited engagement in or knowledge of Double Trouble in Recovery among the depressive participants.

Discussion

The use of self-report measures has traditionally been a cause for concern in clinical and research settings, and even more so in individuals with co-occurring substance use and psychiatric disorders. The results presented here should alleviate some of this concern. Based on the findings of this study, self-report instrumentation in DDI samples can be quite reliable. Deficits related to a dual diagnosis do not appear to significantly impair the participant’s ability to fully and reliably engage in research and treatment settings. In sum, the majority of the self-report instruments evaluated in this study are stable in DDI populations. This exploratory study suggests that the participation of DDI in research studies can be accurately quantified with self-report measures.

These findings should be interpreted with caution given the heterogeneity of the sample and the small sample size, which limit power and the generalizability of these results. Definitive conclusions are not possible without a larger sample. In addition, future studies should characterize the psychometric properties of scales with specific clinical populations and conditions of interest, as specific scales may perform differently in some psychiatric populations.