Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Sep 1.
Published in final edited form as: J Subst Abuse Treat. 2012 Feb 1;43(2):137–151. doi: 10.1016/j.jsat.2011.12.004

Treating depression and substance use: A randomized controlled trial

Sarah B Hunter 1, Katherine E Watkins 1, Kimberly A Hepner 1, Susan M Paddock 1, Brett Munjas 1, Karen C Osilla 1, Suzanne Perry 1
PMCID: PMC3345298  NIHMSID: NIHMS346010  PMID: 22301087

Abstract

Few integrated substance use and depression treatments have been developed for delivery in outpatient substance abuse treatment settings. To meet the call for more ‘transportable’ interventions, we conducted a pilot study to test a group cognitive behavioral therapy (CBT) for depression and substance use that was designed for delivery by outpatient substance abuse treatment counselors. Seventy-three outpatient clients were randomized to usual care enhanced with group CBT or usual care alone, and assessed at three time points (baseline, three and six months post-baseline). Our results demonstrated that the treatment was acceptable and feasible for delivery by substance abuse treatment staff, despite challenges with recruiting clients. Both depressive symptoms and substance use were reduced by the intervention, but were not significantly different from the control group. These results suggest that further research is warranted to enhance the effectiveness of treatment for co-occurring disorders in these settings.

Keywords: addiction, co-occurring disorders, cognitive behavioral therapy, depression, group therapy

1. Introduction

Individuals with substance use disorders frequently suffer from depression, and co-occurring disorders (COD) are associated with increased morbidity and mortality along with poorer treatment outcomes (Hasin et al., 2002; Compton et al., 2007) and higher treatment costs (Dickey and Azeni, 1996; Hoff and Rosenheck, 1998; Clark, Samnaliev, & McGovern, 2009). Moreover, depressed mood is a frequently cited precipitant of relapse among individuals with substance use disorders (Marlatt & Gordon, 1980; Witkiewitz & Villaroel, 2009). While some depressive symptoms are organic brain syndromes which resolve with abstinence, epidemiologic studies examining the temporal relationship between the disorders indicate that, in most cases, the depressive disorder preceded the substance use disorder, suggesting they are independent co-occurring disorders (Gilman and Abraham 2001; Kessler et al. 1996). This independence suggests that most people may need specific treatment for both their depression and substance use problems.

Current treatment guidelines recommend that individuals with COD receive treatment for both disorders (Watkins et al., 2005). However, rates of unmet need are high (Ducharme, Knudsen, & Roman, 2006; McGovern, Xie, Segal, Siembab, & Drake, 2006). For example in 2008, among the 2.5 million adults with co-occurring serious mental illness and substance dependence or abuse, only 61% received any mental health or substance use treatment and only 11% received care for both mental health and substance use (Substance Abuse and Mental Health Services Administration, 2009a). Other studies have found that among those with a serious mental illness and substance use disorders, 46% reported not receiving any type of treatment and only 7% received treatment for both their mental health and substance use disorder (Watkins et al., 2001; Harris & Edlund, 2005). These findings indicate that the recommended treatment guidelines have not been widely implemented. In addition, research suggests that individuals treated in the substance use treatment system have particular difficulty accessing mental health care, despite having similar clinical and diagnostic characteristics to those who enter treatment through the mental health system (Havassy, Alvidrez, and Mericle, 2009).

As defined by the Substance Abuse and Mental Health Services Administration (2009b), integrated treatment occurs when individuals “receive combined treatment for mental illness and substance use disorders from the same practitioner or treatment team”. Integration can occur when multiple treatments are provided by different members of a single treatment team, or when a single treatment addresses both disorders. In both models differences in treatment philosophy, funding, regulation and oversight requirements that exist between the mental health and substance abuse treatment system should be resolved before the client begins treatment (Burnam & Watkins, 2006; Drake, Essock, Shaner, Carey, Minkoff, Kola et al., 2001). Integrated treatment should improve access, since clients only need to enter one system to receive treatment for both disorders.

Despite the need for integrated COD treatment, there is limited empirical evidence on the efficacy of such approaches. For example, a meta-review by Tiet and Mausbach (2007) identified only two tested psychosocial treatments, one for depression and alcohol (Brown, Evans, Miller, Burgess, & Mueller, 1997) and the other for depression and illicit substances (Charney, Paraherakis, Gill, 2001). Both studies included small samples (Brown et al., n = 19; Charney et al., n = 43) and the latter did not include a comparison group. A recent Cochrane review reported that many integrated treatment studies have used poor study designs with small sample sizes, high attrition rates and short follow-up periods (Cleary, Hunt, Matheson, Siegfried, & Walter, 2010; Horsfall, Cleary, Hunt & Walter, 2009). Therefore psychotherapeutic approaches for treating COD show promise but there is insufficient evidence of their effectiveness (Hesse, 2009; Flynn & Brown, 2009). Although intensive case management approaches for substance users with a severe mental illness have empirical support (e.g., Integrated Dual Disorders Treatment (IDDT) Mueser, Noordsy, Drake, & Fox, 2003), these approaches include psychiatrists and licensed mental health providers as integral members of the treatment team. Substance abuse treatment programs typically do not have licensed mental health providers on staff, and even fewer have the resources to provide care from a psychiatrist. Although there are some notable exceptions, such as Illinois and Arizona, most states do not allow substance abuse treatment providers to bill for mental health treatment services, which limit the types of services that are supported in substance abuse treatment systems (Burnam & Watkins, 2006). In summary, there is a lack of well-designed studies on effective integrated COD treatments, especially for approaches that use existing resources within the substance abuse treatment system.

We report the results of a Stage 1b study (i.e., pilot testing of a near final version of the therapy; Rounsaville, Carroll & Onken, 2001) of an integrated treatment for co-occurring depression and substance use that was designed to be delivered within the substance abuse treatment system and by trained substance abuse treatment counselors. The intervention was group cognitive behavioral therapy (CBT) for depression and substance use. Cognitive behavioral therapy is an effective treatment for both depression (e.g., American Psychiatric Association, 2010; Lambert & Davis, 2002) and substance use disorders (e.g., American Psychiatric Association, 2006; Carroll, Ball, Martino, Nich, Vabuscio, Nuro et al., 2008; Project MATCH Research Group, 1997). This study addressed whether two independently efficacious components could be combined and formatted for delivery in outpatient substance abuse treatment.

We conducted a randomized controlled trial assigning clients to the intervention (i.e., usual care enhanced with group CBT) or control (i.e., usual care) condition to address four study goals: 1) determine our ability to recruit sufficient numbers of the target population; 2) understand client acceptance of the intervention; 3) ascertain the feasibility of training outpatient substance abuse treatment counselors to deliver the intervention by examining treatment fidelity; and 4) obtain preliminary assessments of the intervention’s efficacy. We hypothesized that the treatment would be both acceptable to clients and feasible for delivery by treatment counselors as it was developed and tested with input from community substance abuse treatment providers (Osilla, Hepner, Muñoz, Woo & Watkins, 2009) and that treating both disorders concurrently would lead to greater reductions in depressive symptoms and reduced substance use as compared to usual care treatment.

2. Methods

2.1. Setting

The setting for the study was a publicly-funded outpatient substance abuse treatment program located in Los Angeles County. The outpatient program was part of a large substance abuse treatment organization that included outpatient, residential and detoxification services at different locations across the county.

2.2. Design

Clients were randomized to one of the two groups, the intervention or control condition, using a block-randomization design method (Berger & Exner, 1999) such that randomization occurred within blocks of two to four clients into intervention and control conditions, respectively. This strategy was used to balance enrollment across both study conditions during the recruitment period and to maintain at least three clients in the group CBT intervention at any one time.

2.2.1. Study Conditions

2.2.1.1. Intervention Condition

The intervention was an integrated group CBT for depression and substance use disorders; Phase Ia development work is reported elsewhere (Osilla et al., 2009). It was adapted from existing cognitive behavioral therapies with demonstrated efficacy (Cuijpers, Muñoz, Clarke, & Lewinsohn, 2009; Lewinsohn, Antonuccio, Breckenridge, & Teri, 1984; Muñoz, Ippen, Rao, Le, & Dwyer, 2000;; Marlatt & Donovan, 2005; Hepner et al 2011a). It consisted of 18 two-hour sessions, divided into three modules: Thoughts, activities, and people interactions (Hepner, Muñoz, Woo, Osilla, Wiseman, & Watkins, 2011b). The group was delivered twice a week for nine weeks and was co-led by two substance abuse treatment counselors.

2.2.1.2. Control Condition

The control condition consisted of treatment as usual. The treatment offered at the participating center was based on the Twelve-Step approach (Alcoholics Anonymous, 2001), and the Matrix Model (Rawson et al., 1995) manuals were used as guides for some of the groups offered at the site (e.g., Relapse Prevention). In usual care treatment, clients attended group sessions facilitated by one treatment counselor that lasted 90 minutes.

2.3. Participants and Recruitment

The study protocol was approved by the research organization’s Institutional Review Board, and we received a NIH Certificate of Confidentiality to protect the privacy of study participants. During the outpatient treatment intake interview, clients were informed about the opportunity to participate in a study about “Mood and Substance Use.” They were asked to provide written consent to be contacted by research staff to learn more about the study and participate in a brief screener to qualify for the study. Research staff followed up with consented clients to screen for study eligibility approximately two to four weeks after treatment entry.

Between August 2007 and January 2009, 230 clients were screened for current depressive symptoms using the Beck Depression Inventory-II (BDI-II; Beck, Steer & Brown, 1996) (see Figure 1). Study eligibility was based on BDI-II scores greater than 13, indicative of the presence of at least ‘mild’ depressive symptoms. We included clients with minor depression because both major and minor depression impair functioning and reduce life quality (Rapaport et al. 2002; Wells et al. 1996) and increase the odds of relapse (Kodl et al., 2008). We included clients regardless of whether they met criteria for an independent or substance-induced depressive disorder, given that determining such criteria would be beyond the resources available in typical settings. Therefore, we expected that in these settings, both types of clients would be offered treatment.

Figure 1.

Figure 1

Open in a new tab

Study enrollment and data collection diagram

We included clients who had one or more substance use disorders, consistent with the goal of creating an intervention that can be used in community treatment programs which typically do not provide substance-specific treatment. Clients were screened for the presence of a probable alcohol or drug disorder using the AUDIT-C to screen for an alcohol disorder using the criteria of a score > 4 for men and > 3 for women (Dawson, Grant, Stinson, & Zhou, 2005) and the DAST to screen for other substance disorder using the criteria of a score greater than 2 (Cocco & Carey, 1998; Maisto, Carey, Carey, Gordon, Gleason, 2000; Skinner, 1982). Exclusion criteria included a positive screen for a self-reported bipolar disorder (using four items from Sloan, Kivlahan & Saxon 2000), schizophrenia (using one item from the Healthcare for Communities psychoticism screener; Wells et al., 1996), or cognitive impairment (as assessed by the Short Blessed Scale Exam; Dennis et al, 2006).

Next, research staff explained to eligible clients (n = 104) that they would be randomly assigned to treatment condition. Clients were informed that depending on assignment, study participation would entail attendance at group sessions offered twice weekly for nine weeks at specific times (i.e., the intervention condition). Clients who agreed to be randomly assigned were enrolled in the study and completed a 60–90 minute baseline interview (n = 73). Following the interview, the research staff assigned clients to one of the two study conditions: the intervention (group CBT) or control (usual care).

Once clients were randomly assigned, they received instructions about the timing of the treatment groups that they were expected to attend (based on study condition). Clients assigned to the intervention condition were asked to attend groups that were held twice a week at prescribed times. Clients assigned to the control condition were asked to attend treatment groups that were offered at multiple days/times during the week.

Clients in both conditions attended groups 2–3 times a week, based on an individualized treatment plan that was determined prior to study enrollment. Clients assigned to the intervention condition were asked to attend the intervention groups in lieu of two usual care groups during the nine-week period. Clients in both conditions also attended one-on-one meetings with their primary counselor as prescribed in their individualized treatment plan. Clients in both conditions were eligible to receive usual care mental health treatment which consisted of a referral for those with severe depression to a local community mental health center.

2.3. Procedures

2.3.1. Client Data Collection

To evaluate client acceptability, we examined perceptions of the treatment, group climate, therapeutic alliance, and self-reported clinical improvement from clients assigned to the intervention condition following the intervention. We also examined client acceptability by comparing treatment satisfaction and retention rates of clients assigned to the intervention and control conditions.

To obtain preliminary assessments of the intervention’s efficacy, we collected data from participants at three time points: 1) a baseline assessment at the time of study entry (2 – 4 weeks after substance use treatment entry); 2) at three months post-baseline (corresponding to the time that clients in the intervention condition completed the intervention); and 3) at six months post-baseline (approximately three months post-intervention). We employed an intent-to-treat approach, in which all persons who completed the baseline interview were included in the analyses. Thus, the assessment of the preliminary effectiveness included any effects of attrition.

2.3.2. Counselor Training and Supervision

Three addiction counselors (2 female, 1 male) employed by the participating agency were trained to deliver the intervention. None had any specialized training in mental health or previous experience delivering CBT. Counselors received two days of didactic training including: 1) understanding depression and its relationship to substance abuse, 2) assessing depression symptoms, 3) group management skills, 4) understanding the CBT model for depression and substance use, and 5) introducing specific CBT exercises from the manuals. Counselors were judged as ready to lead the intervention when they could demonstrate minimal competence in basic CBT concepts (e.g., could accurately and clearly explain the CBT model). Throughout the study, counselors received weekly two-hour group supervision to increase and retain CBT adherence and competence over time (Sholomskas, Syracuse-Siewert, Rounsaville, Nuro, & Carroll, 2005). Supervision was led by a licensed clinical psychologist with expertise in CBT. During supervision, portions of session recordings were reviewed, challenges to group facilitation were discussed, client progress was reviewed, and role-plays were conducted.

2.3.3. Intervention Fidelity Data Collection and Coding

To address intervention feasibility, we examined whether substance abuse treatment counselors delivered the treatment with fidelity by analyzing ratings of adherence and competence using audio taped sessions from the group CBT that was delivered during the randomized controlled trial. All intervention sessions were audio taped. Three independent coders (two Masters-level and one PhD level) received 16 hours of training, which included reading the treatment manuals, coding four sessions independently of the other coders, and then meeting throughout the study to discuss their codes. More details about the training and coding can be found in Hepner et al. (2011c).

2.4. Measures

2.4.1. Client Acceptance

Treatment Perceptions

We developed a 13-item measure to assess perceptions of the intervention, which included items specific to the intervention’s content and length (e.g., “Each group session (two hours) was the right length”; “The Thoughts Module taught me about the connection between my thoughts, my alcohol/drug use, and my mood”). Clients responded on a five-point Likert scale that ranged from “completely disagree” to “completely agree. Therapeutic Alliance. The six-item Working Alliance Inventory (WAI) for group settings was used to assess each client’s perception of alliance with the group leaders and alliance with the other group members. This version of the WAI has been used successfully in the past (Johnson, Burlingame, Olsen, Davies, & Gleave, 2005). Self-Reported Clinical Improvement. To assess client’s perceptions of clinical improvement, we asked “Looking back on what your life was like just before you started the group and how it is now, how much would you say your life has improved?” Clients responded on a five-point Likert scale that ranged from “not at all improved” to “extremely improved”.

Treatment satisfaction was measured using an 11-item survey developed by Hser, Evans, Huang & Anglin (2004), (e.g., “I was asked to participate in my recovery or treatment plan”; “I would recommend this program to a friend in need of alcohol or other drug treatment/recovery services”). Treatment retention (i.e., length of stay) data were abstracted from computerized administrative records indicating the number of days between admission to the last day of service.

2.4.2. Treatment Fidelity

Treatment fidelity was assessed by examining adherence to the intervention protocol and competence in delivering the intervention using measures developed for this therapy (Hepner et al., 2011c). Adherence refers to whether the group leaders cover key aspects of the treatment, while competence refers to how well group leaders deliver the treatment (Perepletchikova and Kazdin, 2005). The adherence measure was specific to each session and required ratings on a four-point scale (ranging from 0 to 3) of how adequately group leaders covered each session element. A score of two or higher indicates adequate adherence to each session element. The number of adherence items varies based on the number of exercises covered in each session. The competence measure was adapted from the Cognitive Therapy Adherence and Competence Scale (Barber et al., 2003). Adaptations were guided by the unique characteristics of the integrated group CBT tested in this study that differed from typical individual CBT. Specifically, the tested therapy was a highly structured, group therapy in which the session agendas are largely predetermined (rather than setting the agenda with an individual at the beginning of the session) and counselors focus on the needs of many group members (rather than a single individual). Further, the tested therapy is a modular therapy in which portions of the CBT model are focused on for a series of sessions (i.e., Thoughts module focuses on cognitive restructuring, while deemphasizing behavioral interventions). We also added 4 items to assess group dynamics (Jaycox et al., 2009). The 14 competence items are rated on a 7-point scale (0–6) and were applied across all coded sessions.

2.4.3. Client Outcomes

The primary study outcome measures were changes in depressive symptoms and substance use. We assessed depressive symptoms using the Beck Depression Inventory-II (BDI-II). The BDI-II is a 21-item scale that measures level of depressive symptoms within a previous two-week reference period and is widely used to evaluate the intensity of self-reported depression. It is reliable for treatment-seeking substance users (Buckley et al. 2001), and its psychometric properties have been evaluated among substance users (Beck et al. 1988; Hepner Hunter, Edelen, Zhou & Watkins, 2009).

Alcohol use was measured using the Timeline Followback Method (Sobell & Sobell, 1992) to assess both intensity and frequency of drinking in the past 30 days (before entering treatment at baseline). We used a measure of drinking intensity by examining the average number of drinks consumed per drinking day in the past 30 days. We used items from the Addiction Severity Index (McLellan et al., 2005) to assess past 30 day use of the following substances: heroin, methadone, other opiates/analgesics, barbituates, sedatives/hypnotics/tranquilizers, cocaine, amphetamines, cannabis/marijuana, hallucinogens, and inhalants. Problem substance use was measured by asking the number of days of reported use in the past 30 days (before entering treatment at baseline) of the individual’s self-reported problem substance.

Alcohol swab and urine drug tests were conducted at the three and six-month post-baseline assessments to validate self-report. To reduce burden, specimens were only collected from consenting participants who did not admit use. We were also unable to obtain specimens from anyone who was incarcerated (n = 2 at three months post-baseline, n = 4 at six months post-baseline). The alcohol swab test identified persons with blood alcohol concentration greater than 0.02. The testing strips screened for nine substances: marijuana, cocaine, opiates, PCP, amphetamines, methamphetamines, barbiturates, methadone, and benzodiazepines and applied standard cutoff levels set by the National Institute on Drug Abuse (Hawks & Chiang, 1986). For the analyses presented herein, we aggregated the information across the different use measures to ascertain one value for use at each time point (i.e., use or no use): If a participant self-reported no use but screened positive on the biological test, then we re-coded their response to be consistent with the biological test results. Few responses were re-coded from non-users to users (n = 2 at three months post-baseline and n = 4 at six months post-baseline). The number of problem substance use days was re-coded for these clients using predictive mean matching, which involved predicting the outcome at each follow-up for all clients and for each client who screened positive imputing his/her number of problem substance use days by taking the value from the client with the closest predicted value (Little, 1998).

Secondary outcomes included changes in mental heath functioning and self-reported negative consequences from substance use. We used the mental health component summary (MCS) derived from the 12-item Short-Form General Health Survey (Ware et al., 2002) to assess mental health functioning at the three time points. Because negative consequences from substance use appear to be a separate construct from use (Blanchard, Morganstern, Morgan, Labouvie, & Bux, 2003), we assessed consequences using the Shortened Inventory of Problems modified for alcohol and drug use (SIP-AD; Tonigan & Miller 2002). The SIP-AD exhibits good internal consistency, as well as convergent and discriminant validity, and detects change over time (Blanchard et al., 2003).

Demographic Characteristics

To adjust for pre-existing participant differences, we collected at the baseline assessment comprehensive diagnostic information about mental health and substance use status using the Composite International Diagnostic Interview (CIDI 3.0; Kessler & Ustun, 2004; Haro et al., 2006). This version of the CIDI distinguishes between independent and substance-induced psychiatric disorders. We also collected information about participant age, gender, marital status, race/ethnicity, employment status, mental health medication use, and previous treatment utilization.

2.5. Analytic Strategy

2.5.1. Recruitment

We evaluated the proportion of clients entering treatment that met study criteria and were enrolled in the study. We monitored recruitment and intervention attendance throughout the study.

2.5.2. Client Acceptance

Among clients that received the intervention and completed the post-intervention interview, we calculated descriptive statistics (i.e., frequencies, means, ranges) regarding treatment perceptions, working alliance, and perceived clinical improvement. We assessed treatment satisfaction among all participants and compared responses across the intervention and control groups. We examined treatment retention in a couple of different ways. We calculated the number of individual and group sessions that clients in the intervention and control conditions received as well as the length of stay in the outpatient treatment program. Wilcoxon-Mann Whitney tests were conducted to compare the intervention and control clients with respect to these measures.

2.5.3. Treatment Fidelity

Thirty-seven percent of the CBT sessions (n = 49) were randomly selected for fidelity coding. A total of 27 tapes were coded by at least two coders. To account for the high proportion of items with high prevalence and for possible disagreement between the two coders with respect to baseline rates of each measure, we estimated interrater reliability using the prevalence-adjusted bias-adjusted kappa (PABAK; Byrt, Bishop & Carlin, 1993). PABAK estimates ranged from −0.1 to 1.0 (average 0.67), with 19 of 30 adherence and competence items with substantial inter-rater agreement (PABAK > 0.6; Landis & Kock, 1977). An adherence score was computed for each rated session by dividing the number of elements that were adequately covered (i.e., scored a “2” or “3”) by the total number of elements required for that session. Adherence was considered high if 85% of session elements were adequately covered. Competence ratings were computed for each rated session. Similar to the original Cognitive Therapy Adherence and Competence Scale (Barber et al., 2003), an average score of 4.0 indicated competent CBT delivery.

2.5.4. Treatment Efficacy

We analyzed outcomes using an intent-to-treat approach, in which data from all enrolled participants were analyzed. We compared baseline characteristics of clients assigned to the intervention and control conditions using chi-squared tests for categorical characteristics and t-tests for continuous variables. Respondents and non-respondents of both follow-up surveys were also compared. Given 6% of persons at the three months post-baseline and 13% at the six months post baseline interviews were institutionalized for the full 30 day period, we verified that results for the substance use outcome measures were not sensitive to the inclusion or exclusion of persons who were institutionalized for the full 30 days.

Each outcome was modeled using generalized linear mixed effects regression modeling using the GLIMMIX procedure in SAS Software Version 9.2 (SAS Institute Inc., 2008). Error distributions and link functions were selected so that distributional assumptions (e.g., homoskedasticity, normality) for residual errors were met. Gaussian error distributions with identity link functions were used to model BDI-II and MCS, while a square-root power link function was used to model negative consequences and average drinks consumed per drinking day, and the number of days of problem substance use was modeled assuming a Poisson error distribution with log link function. Random client effects accounted for the non-independence, or intra-cluster correlation (ICC), of outcomes repeatedly observed for clients at the three months post-baseline and the six months post-baseline interviews. A multiple membership modeling approach was used to account for the ICC of outcomes among those who attended group CBT sessions together, which involved including random CBT session effects in the model and estimating the client-specific session effect as an average of the random session effects for those sessions attended by the client (Browne, Goldstein, & Rasbash, 2001; Carey 2000). The baseline value corresponding to the outcome was entered into the model as a covariate. Additional covariates were identified by testing for significant baseline differences between intervention and control conditions. For all models, an indicator variable of intervention assignment, follow-up wave (i.e., three or six months), and an interaction of these two measures were included as predictor variables. Intervention effects at each follow-up were estimated from these coefficients and tested using two-sided t-tests, and an overall intervention effect over the full follow-up period was tested using a likelihood ratio test (Fitzmaurice, Laird, & Ware, 2004).

We also calculated the proportion of clients in each study condition whose depressive symptoms were in the minimal range (BDI-II<=13; Beck, Steer & Brown, 1996) and compared the proportion with minimal symptoms in each condition at the two follow-up points using chi-squared tests. We also calculated the proportion of clients in each study condition who were 100% abstinent at each of the follow-ups for the two study conditions. Within-condition changes between baseline and follow-up were tested using Wilcoxon signed-rank tests. Finally, the effect of incentives on outcomes within the intervention group was tested using Wilcoxon Mann-Whitney tests.

3. Results

3.1. Recruitment of Sufficient Numbers of Target Population

As can be seen in Figure 1, almost half of the participants entering this outpatient treatment program met study criteria based on the presence of current depressive symptoms. After enrollment, a comprehensive diagnostic assessment indicated that almost all clients (98.63%) met criteria for an alcohol and/or other drug disorder(s) and about a third met criteria for a depressive disorder. Two participants met criteria for a substance-induced depressive disorder.

Despite the high prevalence of co-occurring depression and substance use, we experienced initial difficulties with maintaining a sufficient number in the group CBT condition, in part because the client census at the outpatient facility had decreased since the study was proposed. Clients also reported that transportation, childcare, and work challenges prevented regular attendance at the prescribed group times. As a result, beginning in the third month of enrollment, we moved from a 50/50 to a 60/40 assignment schedule such that 60% of participants were assigned to the intervention condition and 40% of participants were assigned to the control condition in order to ensure that at least three clients were assigned to the group CBT condition at any one time. Two months later we changed the group time to a more convenient schedule for clients. We continued to have difficulty recruiting enough participants to maintain a sufficient number in the group CBT condition, so five months later we shifted to a 70/30 assignment schedule and initiated an incentive program.

The incentive program allowed for clients assigned to the intervention condition to receive $30 cash for each session attended and $50 gift card for completion of each of the three modules (i.e., thoughts, activities, and people). The $30 was to compensate for the costs of childcare, transportation or finding a replacement for job responsibilities. The $50 incentive was to encourage consistent attendance and completion of the modules. Clients in the control condition were not offered the same incentives because they had the flexibility to attend groups on multiple days/times of the week.

Sixty-four percent of the clients assigned to the intervention condition were eligible to receive an incentive (n = 30). The mean number of sessions attended for clients who received an incentive (M = 13.80 sessions; SD = 4.83) was higher than those that did not receive an incentive (M = 8.82 sessions; SD =6.58; z =−2.63, p < 0.01). As a result of this mid-course change in our study design, we conducted sensitivity analyses to determine whether receiving an incentive had an impact on client acceptance and treatment efficacy.

3.2. Client Acceptance

Treatment Perceptions

Responses to the treatment perception items are presented in Table 2. Overall, responses to these items were highly skewed to the “completely agree” option demonstrating relatively high perceived helpfulness of the group intervention across the items that asked about different aspects of the CBT group. In general, perceptions of the group did not appear to differ before versus after the incentive was introduced (average difference across items = 0.07 points). Working Alliance Inventory (WAI). The mean total score on the WAI suggested positive perceptions of therapeutic alliance with the group leaders. The mean responses did not meaningfully differ between clients that were offered the incentive (M = 4.44, SD = 0.83) from those that did not receive the incentive (M = 4.57, SD = 0.45). Self-Reported Improvement. The majority of clients reported that their life had improved ‘extremely’ or ‘quite a bit’. The group that received the incentives had lower endorsements (M = 62.96%) as compared to those that did not (M = 80.99%).

Table 2.

Mean responses to the client acceptance items

Intervention Control
Treatment Perceptions (% “completely agree”) na
      The group leaders were helpful 90.48
      I could use information from the group in my daily life 85.71
      The workbook we used was easy to use and follow 85.71
      The exercises we did in group together were helpful 80.95
      The daily practice activities I did outside of group were helpful 78.57
      I could understand the information from the group 78.57
      Going to group was helpful in improving my mood and alcohol/drug use cravings 76.19
      Each group module (6 sessions) was the right length 73.81
      Each group session (2-hours) was the right length 69.05
      Treatment for mood problems and substance use problems should be treated together in one setting 66.67
Working Alliance Inventory (Mean, SD) 4.49 (0.71) na
Self-Reported Improvement (% “extremely” or “quite a bit”) 69.05 na
Treatment Satisfaction 4.64 (0.55) 4.43 (0.92)
Treatment Retention
      Length of stay (# of days) 145.11 (60.6) 151.0 (90.04)
      Group sessions (# of sessions) 35.98 (20.04) 34.31 (19.60)
      Individual sessions (# of sessions) 17.64 (9.86) 19.85 (14.36)
Open in a new tab

Treatment Satisfaction

As shown in Table 2, we did not find statistically significant group differences in the level of substance abuse treatment satisfaction (z = −0.05, p = 0.958). The means across the study conditions suggest that clients in both treatment conditions were highly satisfied with the treatment they received. Treatment satisfaction did not appear to be influenced by the introduction of the incentives in the intervention condition (No Incentive M = 4.70, SD = 0.67; Incentive M = 4.61, SD = 0.47) and appeared to slightly lower satisfaction in the control condition (No Incentive M = 4.65, SD = 0.73; Incentive = 3.98, SD = 1.14) but the differences were not statistically significant (p = 0.17).

Treatment Retention

Also in Table 2, length of stay was similar across conditions, (z =0.52, p = 0.61) as well as the number of group (z = −0.06, p = 0.95) and individual sessions attended (z =0.28, p = 0.78). The introduction of the incentives did not appear to influence any of these variables for the intervention condition (Length of Stay: No Incentive M = 143.76, SD = 48.28; Incentive M = 145.87, SD = 80.39; Number of group sessions: No Incentive M = 38.06, SD = 14.56; Incentive M = 34.80, SD = 22.71; Number of individual sessions: No Incentive M = 18.00, SD = 7.08; Incentive M = 17.43, SD = 11.25) but the introduction of the incentive appeared to lower values in the control condition (Length of Stay: No Incentive = 170.63, SD = 103.41; Incentive M = 119.60, SD = 54.03; Group Sessions: No Incentive M = 41.38; SD = 19.49; Incentive M = 23.00, SD = 14.26; Individual Sessions: No Incentive M = 23.31, SD = 16.54; Incentive M = 14.30, SD = 7.83), although probability tests were not statistically significant (all p-values > 0.20).

3.3. Treatment Fidelity

The average adherence rate was 94.7% across all coded sessions (n = 49), suggesting that the counselors achieved high adherence to the treatment manual. Most of the sessions (87.8%) had adherence rates over 85%. The average competence score across all coded sessions was 4.05. Most of the sessions (59.2%) had competence scores of at least 4.0. Adherence and competence to the treatment did not vary by treatment module (i.e., thoughts, activities, and people).

3.4. Treatment Efficacy

3.4.1. Sample Characteristics

The characteristics of the study sample are presented in Table 1. As shown, the clients assigned to the intervention and control conditions were similar except for two characteristics: The intervention sample was older (M = 37.4 years vs. 31.1 years; p = 0.08) and less likely to meet criteria for a past 12-month drug dependence disorder (23% vs. 42%; p = 0.09). Age (indicator of being older versus younger than 30 years of age) and past 12 month drug dependence disorder were thus included as covariates in the outcome analyses.

Table 1.

Baseline characteristics of the study sample

Overall Mean (SD) or Percent
Intervention
(n = 47)		 Control
(n = 26)
Gender (% Male) 52.05 51.06 53.85
Less than 30 Years Old (%)* 43.84 36.17 57.69
Marital Status (% Married) 35.62 31.91 42.31
Ethnicity/Race
      Caucasian 26.03 23.40 30.77
      African American 31.51 38.30 17.23
      Hispanic 26.03 21.28 34.62
      Other 16.44 17.02 15.38
Employment
      Full/Part Time 24.66 21.28 30.77
      Unemployed/Laid Off 47.95 46.81 50.00
      Other Work Situation 27.40 31.91 19.23
Education
      < High School 31.51 31.91 30.77
      High School / GED 58.90 59.57 57.69
      College or More 9.59 8.51 11.54
Mental Health Measures
      Beck Depression Inventory-II Score (BDI-II) 28.1 (9.0) 27.8 (9.3) 28.7 (8.5)
      Mental Health Composite Score (MCS) 40.5 (14.2) 42.2 (14.7) 37.2 (12.9)
      Currently Taking Medication for Mental/Emotional Problem 10.96 12.77 7.69
      Currently Receiving Individual Counseling 13.70 14.89 11.54
      CIDI Depression Disorder 34.25 34.04 34.62
Substance Use Measures
      Negative Consequences from Alcohol and Drug Use 11.9 (11.8) 12.3 (12.8) 11.3 (9.9)
Lifetime
      Ever Receive AOD Treatment 64.38 59.57 40.43
      CIDI Alcohol Abuse or Dependence, Lifetime 78.08 80.85 73.08
      CIDI Drug Abuse or Dependence, Lifetime* 91.78 89.36 96.15
Past 12 Months
      AUDIT-C, % Problem Alcohol Use 53.42 53.19 53.85
Problem Substance
      Alcohol 19.18 23.40 11.54
      Heroin 1.37 0.00 3.85
      Sedatives 1.37 0.00 3.85
      Cocaine 23.29 25.53 19.23
      Amphetamines 31.51 27.66 38.46
      Marijuana 19.18 17.02 23.08
      Hallucinogens 1.37 2.13 0.0
      More than One 1.37 2.13 0.0
      Other 1.37 2.13 0.0
Open in a new tab

Note:

*

denotes p < 0.10 between the CBT and UC groups

3.4.2. Client Outcomes

Table 3 shows the treatment effect estimates, confidence intervals, and test statistics for each outcome by study condition over time. Figure 2 shows the conditional mean estimates (i.e., the response by the average client) for each study condition at the baseline, 3- and 6-month post baseline time points on the five outcome measures.

Table 3.

Treatment effect estimates, confidence intervals, and test statistics for each outcome by study condition at three and six-months post baseline

CBT Treatment Effect Estimate 95% Confidence
Interval		 t-
statistic
(60 df)		 Likelihood
ratio test
statistic
(χ2(2 df))		 p-value
BDI-II 2.24 0.33
    3 Months −4.17 −10.43 2.10 −1.33 0.19
    6 Months −1.53 −7.77 4.71 −0.49 0.63
SF-12 MCS 0.21 0.90
    3 Months −0.79 −6.30 4.71 −0.29 0.77
    6 Months 0.86 −4.45 6.18 0.33 0.75
Drinks per Days Drank (Square Root) 1.41 0.49
    3 Months −0.46 −1.29 0.38 −1.10 0.28
    6 Months −0.43 −1.23 0.38 −1.06 0.29
Negative Consequences (Square Root) 2.17 0.34
    3 Months −0.71 −1.72 0.30 −1.44 0.17
    6 Months −0.48 −1.42 0.46 −1.92 0.31
Problem Substance Use Days (Relative Risk) 1.62 0.44
    3 Months 0.40 0.09 1.70 −1.27 0.21
    6 Months 0.42 0.10 1.75 −1.21 0.23
Open in a new tab
Figure 2.

Figure 2

Open in a new tab

Baseline, three- and six-month post-baseline mean estimates for the intervention and control groups on the mental health and substance use outcomes

Mental Health Outcomes

Participants in both the intervention and control conditions demonstrated statistically significant improvements in depressive symptoms (BDI-II) and mental health functioning (MCS) at three months post-baseline (at a p < 0.05 level). The proportion of clients at 3 months post-baseline with minimal depressive symptoms (BDI-II <= 13) for the intervention versus control condition was 64 versus 45 percent, respectively (χ2= 2.099, p = 0.147).

Improvements from baseline to six months were found in both the intervention (S = −493.5, p < 0.001) and control conditions for the BDI-II (S = −139, p < 0.001). For the MCS variable, significant improvement between baseline and six months was found for the control condition (S = 73, p = 0.338) and borderline significant improvement in the intervention condition (S = 171.5, p = 0.052). The proportion of clients at 6 months post-baseline with minimal depressive symptoms (BDI-II <= 13) for the intervention versus control condition was 76 versus 54 percent (χ2= 3.297, p = 0.069).

Substance Use Outcomes

At 3 months post-baseline, clients in both the intervention and control conditions reported significantly fewer days of problem substance use (Wilcoxon signed-rank test statistic, S = −200.5 (intervention) and S= −29.0 (control), p < 0.05). Clients in the intervention condition also significantly reduced the number of drinks per drinking day and negative consequences from substance use whereas the clients in the control condition did not.

Improvements from baseline to six months in days of problem substance use were found for the intervention condition (S=−163.5, p < 0.05) and a trend for negative consequences was significant (S=−205.5, p = 0.05), but average number of drinks per drinking day was not (S=−10.5, p = 0.86). For the control condition, responses at six months were not different from baseline on problem substance use (S=−5.5, p = 0.69), drinks per drinking day (S=32.0, p = 0.24) and negative consequences from substance use (S=−43.0, p = 0.20). The proportion of clients reporting 100% days abstinent from problem substance at 3-months for the intervention versus control condition was 73 versus 62 percent (χ2= 0.75, p = 0.3866) and 63 versus 62 percent at 6-months (χ2= 0.001, p = 0.9819).

Sensitivity Analyses

We conducted sensitivity analyses to address two issues: 1) the inclusion of participants with BDI-II scores in the mild range and 2) the introduction of incentives to intervention participants part-way through the trial. Regarding the first issue, we examined the results after restricting our sample to BDI-II > 19, corresponding to at least moderate depression symptoms. The sample size was reduced to n = 58 at baseline, and we followed up with 97% of those participants at 6 months (n = 56). The pattern of results was similar and did not change significance, indicating that the findings did not appear strongly influenced by the inclusion of mildly depressed participants. We also examined self-reported receipt of individual mental health counseling (outside of outpatient substance abuse treatment). We did not find statistically significant differences between the two conditions at baseline among the BDI>19 sub-sample (p = 0.697; 16.22% in the intervention condition and 9.52% in the control condition; see Table 1 for the full sample values). Moreover the rates of self-reported receipt of individual mental health counseling did not change over the course of the study (i.e., at 3 months, p =.999, 9.38% in the intervention condition and 5.88 in the control condition; and at 6 months, p= 0.696; 17.14% in the intervention condition and 10.00 in the control condition). These findings are consistent with the literature demonstrating the lack of access to mental health treatment among co-occurring populations receiving substance use care.

Regarding the second issue, we examined change scores from baseline to the three and six month follow-ups in the intervention and control conditions pre and post- the introduction of the incentives. Albeit small sample sizes (Pre-Incentive Intervention n = 17 and Control n = 16; Post-Incentive Intervention n = 30; Control = 10), the change scores for the intervention and control conditions prior to and after the incentives did not differ on BDI-II, problem substance use, MCS, and SIP-AD scores. However the average drinks per drinking day change score from baseline to six months significantly differed in both the intervention and control conditions from pre- and post-incentive periods (intervention, z = −2.64, p = 0.01; z = 2.233 control, z = 2.33, p = 0.02). The means indicated that the days drank was reduced among the pre-incentive groups from baseline to the 6-month follow-up (Intervention M = −5.63; Control M = −.63) but increased for the post-incentive groups (Intervention M = 2.77; Control M = 7.09).

Confidence Intervals

To reconcile the concern about sample size with the Stage 1b pilot study goal of determining whether the intervention holds promise for a larger-scale Stage 2 study, we examined 95% confidence intervals for our treatment effect estimates to assess the range of effects that would be statistically compatible with our data (Rounsaville et al., 2001). At three months following baseline, reductions in BDI-II that would be compatible with our data could be as large as 10 points (as indicated by the lower end of the 95% Confidence Intervals (CI)) and increases in BDI-II are unlikely to be more than 2 points (see Table 3). The treatment effect estimate at six months is reduced, with the resulting 95% CI ranging from +4 to −7 points. A similar analysis on the substance use measures shows that a reduction in the square root of the number of drinks per drinking day for the intervention condition that would be compatible with our data could be as large as 1.29 drinks (at the 3-month follow-up), but that increases of up to 0.38 drinks (indicated by the lower and upper bounds of the 95% CI) would be compatible as well, with similar results at 6 months. A similar examination of the other 95% CIs in Table 3 suggests that the intervention could lead to greater reductions in negative consequences as compared to usual care and more ambiguous intervention effects on changing SF-12 scores and problem substance use days.

4. Discussion

4.1. Summary of Findings

In this Stage 1b treatment development study, we found that an integrated group CBT for co-occurring depression and substance use was acceptable to clients and feasible for delivery by outpatient substance abuse treatment counselors. Although clients had to manage competing demands from work and family in order to receive the intervention, pilot efficacy data suggest the treatment improves both depression and substance use outcomes. These findings are important because an integrated psychosocial treatment for depression and substance use, which is acceptable and feasible for usual care substance abuse providers to deliver, solves many of the barriers to treatment faced by clients with co-occurring disorders.

Treatments that build upon existing resources have the potential to increase access for the large group of individuals with co-morbid depression symptoms who enter the substance abuse treatment system. Given the high rates of substance disorders and the strong relationship between depressive symptoms and relapse (Curran et al., 2000; Witkiewitz & Villarroel, 2009), the need to address both disorders for clients entering the substance abuse treatment system is critical. While recent studies suggest the effectiveness of providing medications to treat depression and substance disorders (Pettinati et al., 2010), most substance abuse treatment providers do not have the resources to offer psychopharmacological treatment (McGovern et al., 2006) and previous attempts to coordinate care between the mental health and substance abuse provider system have been unsuccessful (Wenzel et al., 2005). Furthermore, effective models of integrated care for the seriously mentally ill are resource-intensive and may not be necessary for the majority of individuals with co-occurring disorders (Burnam & Watkins, 2006).

In relation to our study goals, we demonstrated high prevalence of clients with co-occurring depression and substance use being treated in an outpatient substance abuse treatment setting. However, recruiting clients to attend a two-hour group twice a week and maintaining consistent attendance proved challenging. Initially attendance was erratic, and therefore we designed an incentive system to ensure more consistent attendance in the intervention condition. We found that the incentives improved attendance. Although we were not statistically powered to detect differences, we explored whether client acceptance and outcomes appeared to be influenced by the introduction of the incentives since it could serve as a potential study confound. The pattern of findings did not suggest that the introduction of incentives increased acceptance or improved outcomes. In fact the only statistically significant effect was change in days drinking, which showed that clients in both the intervention and control conditions had greater improvements prior to the incentive being introduced as compared to afterwards. Although contingency management (CM) approaches utilize incentives to promote abstinence and other treatment goals, such as retention (Petry, 2000; Prendergast, Podus, Finney, Greenwell, Roll, 2006), the incentive approach used in this study was different in a variety of ways. First, the incentives were not introduced at the beginning of addiction treatment, but several weeks later after the participant was enrolled in the study. Second the incentives used in this study were much larger in value than typically utilized in CM. Third, the incentives were not associated with drug testing or outcome reporting. That is, different staff provided the incentives at different times than the drug testing and follow-up interviews. Moreover, we did not utilize a variable reinforcement schedule that has been shown as an effective CM strategy. Given that outpatient treatment is the most popular modality with over 80% of treatment recipients receiving this form of treatment (Substance Abuse and Mental Health Services Administration, 2006), our study findings suggest that more work is needed to address client attendance. In future work we hope to explore the use of technology to increase engagement and treatment access.

Regarding our second goal to develop a treatment that was acceptable to clients, we found that clients assigned to the intervention condition perceived the treatment positively. At the follow-up interviews, intervention clients reported high levels of therapeutic alliance and satisfaction with the intervention. The majority of clients receiving the intervention reported that the treatment had improved their life. We did not find any evidence that intervention clients stayed in treatment longer or attended more or fewer substance abuse treatment sessions than clients assigned to the usual care condition; we speculate that these behaviors were more likely influenced by the clients’ individual treatment plan that determined the number of substance abuse treatment sessions and length of stay rather than their experiences with the intervention.

The high fidelity ratings throughout the trial indicate that the treatment is feasible for delivery by outpatient counselors. While research supports the observation that substance abuse treatment counselors can deliver CBT for substance use (Morgenstern, Morgan, McCrady, Keller, & Carroll, 2001; Sholomskas et al., 2005), less evidence exists for the ability of counselors with little or no mental health training to deliver CBT for depression. In a recent systematic review of CBT for depression or anxiety, Montgomery, Kunik, Wilson, Stanley & Weiss (2010) identified only four studies with sufficient methodological rigor to include in the review and concluded that paraprofessionals (i.e., those without formal mental health training) can be effective in delivering CBT to patients with depressive symptoms. However, none of the reviewed studies examined substance abuse treatment counselors. Our results regarding treatment fidelity add to this emerging literature and suggest that increasing the capacity of the substance abuse treatment system to deliver mental health care is a viable approach to the problem of poor mental health care access. This finding is helpful in terms of reducing the gap between research and practice settings because it suggests that it is possible for substance abuse treatment counselors to deliver group CBT for depression and substance use when given adequate training and supervision.

Finally, our findings suggest that the treatment holds promise for treating both depressive symptoms and substance use. More specifically, the range in confidence intervals for depressive symptoms suggests that the intervention is likely to lead to reductions in symptoms as compared to usual care. The confidence intervals for problem substance use suggest that it would be unlikely for the intervention to be meaningfully worse than usual care. Moreover, the intervention shows potential for reducing negative consequences from use. The non-statistically significant results between the intervention and control groups are unsurprising, given the sample size, and we do not think the results should be interpreted as lack of efficacy. The large variance in treatment effect size leads to a risk that the estimated treatment effect will be understated, thus posing a danger of erroneously concluding that the intervention is not promising and prematurely aborting future study of this intervention (Kraemer, Mintz, Noda, Tinklenberg, & Yesavage, 2006). It is also possible that the treatment as usual groups (i.e., control condition), that incorporated the Matrix model may have had some conceptual overlap with the intervention, such as cognitive skills training and activity scheduling (Conrod & Stewart, 2005; Marlatt & Gordon, 1985; Rawson et al., 1995) reducing our ability to detect a difference between the two groups. Even so, the results from this study suggest that the intervention may lead to better mental health and substance use improvements over time as compared to the control condition.

4.2. Limitations and future directions

We tested the efficacy of the intervention in a relatively large, publicly-funded treatment center located in an urban setting serving a diverse population. A more homogeneous client population may have led to more convincing evidence of the treatment’s efficacy, but would have limited real-world applicability (Hughes et al. 2000). We recognize that among substance users, depression may resolve with abstinence and studies have shown that depressive symptoms decrease within the first 7 to 14 days after treatment admission (alcoholics, Brown & Schuckit, 1988; cocaine users, Husband et al. 1996; opiate users, Strain et al. 1991). Due to these findings, we screened participants two to four weeks after treatment entry, but since our study was conducted in outpatient treatment, it limits our certainty about participant abstinence and whether depressive symptoms resolved due to reduced use or abstinence. Another important limitation was the introduction of the monetary incentives in the intervention condition to encourage consistent attendance. Although the sensitivity analyses suggested that the incentives did not influence acceptance or outcomes among the intervention participants, the small study sample limits our ability to detect potential confound effects. The sample size also limited our ability to examine how the treatment may work (i.e., mechanisms). Future research may want to consider evaluating the content of the treatment as usual group sessions as well as collecting comparable data from clients in both conditions during treatment to ascertain the core therapeutic elements that lead to mental health and substance use recovery.

4.3. Summary

A key question for substance abuse treatment providers is how to effectively treat co-morbid depression and substance use disorders within the constraints of existing resources. The results from this study suggest that training substance abuse treatment counselors to deliver group CBT for depression and substance use is a promising approach to address these co-occurring disorders. Next steps for this work are studies with larger sample sizes to confirm the treatment’s efficacy and effectiveness. Additionally, improving client engagement perhaps through modifying the way the treatment is delivered in outpatient settings needs further attention. Because initial treatment gains for the substance use outcomes lessened over time, it might be useful to add an aftercare component. In summary, the study results contribute to the emerging literature about improving the capacity of the substance abuse treatment system to address co-occurring disorders.

Acknowledgments

This research was supported by the National Institute on Drug Abuse (grant no. R01DA020159) awarded to Katherine E. Watkins. We thank Stephanie Woo (Pepperdine University), Ricardo Muñoz (University of California San Francisco, San Francisco General Hospital, San Francisco State University) and Elizabeth Gilbert for their clinical expertise. We thank all of those at Behavioral Health Services, Inc., especially Jim Gilmore, Shirley Summers, and Yolanda Farley for their administrative support along with the outpatient counselors, Wanda Duffy, Laurie Snyder and Darrell Woods, without whose participation this project would not have been possible. We would also like to acknowledge the contributions of Magdalen Paskell and Jacob Solomon in formatting this document for publication.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Alcoholics Anonymous. Alcoholics Anonymous. 2001 Available at www.aa.org/bigbookonline/.
  2. American Psychiatric Association. Practice guideline for the treatment of patients with major depressive disorder. 2010 Available at www.psych.org/guidelines/mdd2010. [PubMed]
  3. American Psychiatric Association. Practice guideline for the treatment of patients with substance use disorder. 2006 Available at www.psychiatryonline.com/pracGuide/pracGuideTopic_5.aspx.
  4. Barber J, Liese B, Abrams M. Development of the cognitive therapy adherence and competence scale. Psychotherapy Research. 2003;13(2):205–221. [Google Scholar]
  5. Beck A, Steer R, Brown G. BDI-II, beck depression inventory: Manual. 2nd ed. Boston, MA: Harcourt Brace; 1996. [Google Scholar]
  6. Beck AT, Steer RA, Garbin MG. Psychometric properties of the Beck Depression Inventory: Twenty-five years of evaluation. Clinical Psychology Review. 1988;8(1):77–100. [Google Scholar]
  7. Berger VW, Exner DV. Detecting selection bias in randomized clinical trials. Clinical Trials. 1999;20:319–327. doi: 10.1016/s0197-2456(99)00014-8. [DOI] [PubMed] [Google Scholar]
  8. Blanchard KA, Morgenstern J, Morgan TJ, Lobouvie EW, Bux DA. Assessing consequences of substance use: Psychometric properties of the inventory of drug use consequences. Psychology of Addictive Behaviors. 2003;17(4):328–331. doi: 10.1037/0893-164X.17.4.328. [DOI] [PubMed] [Google Scholar]
  9. Brown RA, Evans DM, Miller IW, Burgess ES, Mueller TI. Cognitive-behavioral treatment for depression in alcoholism. Journal of Consulting and Clinical Psychology. 1997;65(5):715–726. doi: 10.1037//0022-006x.65.5.715. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Brown SA, Schuckit MA. Changes in depression among abstinent alcoholics. Journal of Studies on Alcohol. 1988;49:412–417. doi: 10.15288/jsa.1988.49.412. [DOI] [PubMed] [Google Scholar]
  11. Browne WJ, Goldstein H, Rasbash J. Multiple membership multiple classification (MMMC) models. Statistical Modeling. 2001;1(2):103–124. [Google Scholar]
  12. Buckley T, Parker J, Heggie J. A psychometric evaluation of the BDI-II in treatment-seeking substance abusers. Journal of Substance Abuse Treatment. 2001;20(3):197–204. doi: 10.1016/s0740-5472(00)00169-0. [DOI] [PubMed] [Google Scholar]
  13. Burnam MA, Watkins KE. Substance abuse with mental disorders: Specialized public systems and integrated care. Health Affairs (Millwood) 2006;25(3):648–658. doi: 10.1377/hlthaff.25.3.648. [DOI] [PubMed] [Google Scholar]
  14. Byrt T, Bishop J, Carlin J. Bias, prevalence and kappa. Journal of Clinical Epidemiology. 1993;46:423–429. doi: 10.1016/0895-4356(93)90018-v. [DOI] [PubMed] [Google Scholar]
  15. Carey K. A multilevel modeling approach to analysis of patient costs under managed care. Health Economics. 2000;9:435–446. doi: 10.1002/1099-1050(200007)9:5<435::aid-hec523>3.0.co;2-z. [DOI] [PubMed] [Google Scholar]
  16. Carroll KM, Ball SA, Martino S, Nich C, Babuscio TA, Nuro KF, et al. Computer-assisted delivery of cognitive-behavioral therapy for addiction: A randomized trial of CBT4CBT. American Journal of Psychiatry. 2008;165:881–888. doi: 10.1176/appi.ajp.2008.07111835. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Charney DA, Paraherakis AM, Gill KJ. Integrated treatment of comorbid depression and substance use disorders. Journal of Clinical Psychiatry. 2001;62(9):672–677. doi: 10.4088/jcp.v62n0902. [DOI] [PubMed] [Google Scholar]
  18. Clark RE, Samnaliev M, McGovern MP. Impact of substance disorders on medical expenditures for medicaid beneficiaries with behavioral health disorders. Psychiatric Services. 2009;60(1):35–42. doi: 10.1176/ps.2009.60.1.35. [DOI] [PubMed] [Google Scholar]
  19. Cleary M, Hunt GE, Matheson SL, Siegfried N, Walter G. Psychosocial interventions for people with both severe mental illness and substance misuse. Cochrane Database of Systematic Reviews. 2008;1:1–29. doi: 10.1002/14651858.CD001088.pub2. (CD001088) [DOI] [PubMed] [Google Scholar]
  20. Cocco K, Carey K. Psychometric properties of the Drug Abuse Screening Test in psychiatric outpatients. Psychological Assessment. 1998;10(4):408–414. [Google Scholar]
  21. Compton WM, Thomas YF, Stinson FS, Grant BF. Prevalence, correlates, disability, and comorbidity of DSM-IV drug abuse and dependence in the United States: Results from the National Epidemiologic Survey on Alcohol and Related Conditions. Archives of General Psychiatry. 2007;64(5):566–576. doi: 10.1001/archpsyc.64.5.566. [DOI] [PubMed] [Google Scholar]
  22. Conrod PJ, Stewart SH. A critical look at dual-focused cognitive-behavioral treatments for comorbid substance use and psychiatric disorders: Strengths, limitations, and future directions. Journal of Cognitive Psychotherapy: An International Quarterly. 2005;19(3):265–289. [Google Scholar]
  23. Cuijpers P, Muñoz RF, Clarke GN, Lewinsohn PM. Psychoeducational treatment and prevention of depression: The “Coping with Depression” course thirty years later. Clinical Psychology Review. 2009;29:449–458. doi: 10.1016/j.cpr.2009.04.005. [DOI] [PubMed] [Google Scholar]
  24. Curran GM, Flynn HA, Kirchner JE, Booth BM. Depression after alcohol treatment as a risk factor for relapse. Journal of Substance Abuse Treatment. 2000;19(3):259–265. doi: 10.1016/s0740-5472(00)00107-0. [DOI] [PubMed] [Google Scholar]
  25. Dawson DA, Grant BF, Stinson FS, Zhou Y. Effectiveness of the derived Alcohol Use Disorders Identification Test (AUDIT-C) in screening for alcohol use disorders and risk drinking in the US general population. Alcoholism, Clinical and Experimental Research. 2005;29:844–854. doi: 10.1097/01.alc.0000164374.32229.a2. [DOI] [PubMed] [Google Scholar]
  26. Dennis ML, White M, Titus JC, Unsicker J. Short blessed scale exam. Global appraisal of individual needs: Trainer's training manual and resources (July 2006 version) Bloomington, IL: Chestnut Health Systems; 2006. [Google Scholar]
  27. Dickey B, Azeni H. Persons with dual diagnoses of substance abuse and major mental illness: Their excess costs of psychiatric care. American Journal of Public Health. 1996;86:973–977. doi: 10.2105/ajph.86.7.973. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Drake RE, Essock SM, Shaner A, Carey KB, Minkoff K, Kola L, et al. Implementing dual diagnosis services for clients with severe mental illness. Psychiatric Services. 2001;52(4):469–476. doi: 10.1176/appi.ps.52.4.469. [DOI] [PubMed] [Google Scholar]
  29. Ducharme LJ, Knudsen HK, Roman PM. Availability of integrated care for co-occurring substance abuse and psychiatric conditions. Community Mental Health Journal. 2006;42(4):363–375. doi: 10.1007/s10597-005-9030-7. [DOI] [PubMed] [Google Scholar]
  30. Fitzmaurice G, Laird NM, Ware J. Applied longitudinal analysis. Hoboken, NJ: John Wiley & Sons; 2004. [Google Scholar]
  31. Flynn PM, Brown BS. Co-occurring disorders in substance abuse treatment: Issues and prospects. Journal of Substance Abuse Treatment. 2008;34(1):36–47. doi: 10.1016/j.jsat.2006.11.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Gilman SE, Abraham HD. A longitudinal study of the order of onset of alcohol dependence and major depression. Drug and Alcohol Dependence. 2001;63:277–286. doi: 10.1016/s0376-8716(00)00216-7. [DOI] [PubMed] [Google Scholar]
  33. Haro JM, Arbabzadeh-Bouchez S, Brugha TS, de Girolamo G, Guyer ME, Jin R, Kessler RC. Concordance of the Composite International Diagnostic Interview Version 3.0 (CIDI 3.0) with standardized clinical assessments in the WHO World Mental Health Surveys. International Journal of Methods in Psychiatric Research. 2006;15(4):167–180. doi: 10.1002/mpr.196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Harris KM, Edlund MJ. Use of mental health care and substance abuse treatment among adults with co-occurring disorders. Psychiatric Services. 2005;56(8):954–959. doi: 10.1176/appi.ps.56.8.954. [DOI] [PubMed] [Google Scholar]
  35. Hasin D, Liu X, Nunes E, McCloud S, Samet S, Endicott J. Effects of major depression on remission and relapse of substance dependence. Archives of General Psychiatry. 2002;59(4):375–380. doi: 10.1001/archpsyc.59.4.375. [DOI] [PubMed] [Google Scholar]
  36. Havassy BE, Alvidrez J, Mericle AA. Disparities in use of mental health and substance abuse services by persons with co-occurring disorders. Psychiatric Services. 2009;60(2):217–223. doi: 10.1176/appi.ps.60.2.217. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Hawks R, Chiang C. Urine testing for drugs of abuse. 1986 (National Institute for Druge Abuse (NIDA), Research Monograph, 73). Available at http://archives.drugabuse.gov/pdf/monographs/download73.html.
  38. Hepner KA, Miranda JM, Woo SM, Watkins KE, Lagomasino IT, Wiseman SH, Muñoz RF. Santa Monica, CA: RAND Corporation; 2011a. BRIGHT: A group cognitive behavioral therapy for depression in clients with co-occurring alcohol and drug use problems. [Google Scholar]
  39. Hepner KA, Muñoz RF, Woo SM, Osilla KC, Wiseman SH, Watkins KE. Santa Monica, CA: RAND Corporation; 2011b. BRIGHT-2: A group cognitive behavioral therapy for co-occurring depression and alcohol and drug use problems. [Google Scholar]
  40. Hepner KA, Stern S, Paddock SM, Hunter SB, Osilla KC, Watkins KE. Santa Monica, CA: RAND Corporation; 2011c. A fidelity coding guide for a group cognitive behavioral therapy for depression (TR-980-NIDA/NIAAA) Available at www.rand.org/pubs/technical_reports/TR980.html. [Google Scholar]
  41. Hepner KA, Hunter SB, Edelen MO, Zhou AJ, Watkins KE. A comparison of two depressive symptomology measures in residential substance abuse treatment clients. Journal of Substance Abuse Treatment. 2009;37:318–325. doi: 10.1016/j.jsat.2009.03.005. PMCID2762442. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Hesse M. Integrated psychological treatment for substance use and co-morbid anxiety or depression vs. treatment for substance use alone. A systematic review of the published literature. BMC Psychiatry. 2009;9:6. doi: 10.1186/1471-244X-9-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Hoff RA, Rosenheck RA. Long-term patterns of service use and cost among patients with both psychiatric and substance abuse disorders. Medical Care. 1998;36:835–843. doi: 10.1097/00005650-199806000-00007. [DOI] [PubMed] [Google Scholar]
  44. Horsfall J, Cleary M, Hunt GE, Walter G. Psychosocial treatments for people with co-occurring severe mental illnesses and substance use disorders (dual diagnosis): A review of empirical evidence. Harvard Review of Psychiatry. 2009;17(1):24–34. doi: 10.1080/10673220902724599. [DOI] [PubMed] [Google Scholar]
  45. Hser YI, Evans E, Huang D, Anglin DM. Relationship between drug treatment services, retention, and outcomes. Psychiatric Services. 2004;55:767–774. doi: 10.1176/appi.ps.55.7.767. [DOI] [PubMed] [Google Scholar]
  46. Husband SD, Marlowe DB, Lamb RJ, Iguchi MY, Bux DA, Kirby KC, et al. Decline in self-reported dysphoria after treatment entry in inner-city cocaine addicts. Journal of Consulting and Clinical Psychology. 1996;64(1):221–224. doi: 10.1037//0022-006x.64.1.221. [DOI] [PubMed] [Google Scholar]
  47. Hughes JR, Rose GL, Callas PW. Nicotine is more reinforcing in smokers with a past history of alcoholism than in smokers without this history. Alcoholism: Clinical and Experimental Research. 2000;24(11):1633–1638. [PubMed] [Google Scholar]
  48. Jaycox LH, Langley AK, Stein BD, Wong M, Sharma P, Schonlau M. Support for students exposed to trauma: A pilot study. School Mental Health. 2009;1(2):49–60. doi: 10.1007/s12310-009-9007-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Johnson J, Burlingame G, Olsen J, Davies D, Gleave R. Group climate, cohesion, alliance, and empathy in group psychotherapy multilevel structural equation models. Journal of Counseling Psychology. 2005;52(3):310–321. [Google Scholar]
  50. Kessler RC, Nelson CB, McGonagle KA, Edlund MJ, Frank RG, Leaf PJ. The epidemiology of co-occurring addictive and mental disorders: Implications for prevention and service utilization. American Journal of Orthopsychiatry. 1996;66(1):17–31. doi: 10.1037/h0080151. [DOI] [PubMed] [Google Scholar]
  51. Kessler RC, Ustun TB. The World Mental Health (WMH) survey initiative version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI) International Journal of Methods in Psychiatric Research. 2004;13(2):93–121. doi: 10.1002/mpr.168. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Kodl MM, Fu SS, Willenbring ML, Gravely A, Nelson DB, Joseph A, et al. The impact of depressive symptoms on alcohol and cigarette consumption following treatment for alcohol and nicotine dependence. Alcoholism: Clinical and Experimental Research. 2008;32(1) doi: 10.1111/j.1530-0277.2007.00556.x. [DOI] [PubMed] [Google Scholar]
  53. Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry. 2006;63(5):484–489. doi: 10.1001/archpsyc.63.5.484. [DOI] [PubMed] [Google Scholar]
  54. Lambert MJ, Davis MJ. Treatment for depression: What the research says. In: Reinecke MA, Davison MR, editors. Comparative treatments of depression. New York: Springer; 2002. pp. 21–46. [Google Scholar]
  55. Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. [PubMed] [Google Scholar]
  56. Lewinsohn PM, Antonuccio DO, Steinmetz-Breckenridge JL, Teri L. The Coping with Depression course: A psychoeducational intervention for unipolar depression. Eugene, OR: Castalia Publishing; 1984. [Google Scholar]
  57. Little RJA. Missing data adjustments in large surveys. Journal of Business and Economic Statistics. 1988;6:287–296. [Google Scholar]
  58. Maisto SA, Carey MP, Carey KB, Gordon CM, Gleason JR. Use of the AUDIT and the DAST-10 to identify alcohol and drug use disorders among adults with a severe and persistent mental illness. Psychological Assessment. 2000;12(2):186–192. doi: 10.1037//1040-3590.12.2.186. [DOI] [PubMed] [Google Scholar]
  59. Marlatt GA, Gordon JR. Determinants of relapse: Implications for the maintenance of behavior change. In: Davidson P, Davidson SM, editors. Behavioral medicine: Changing health lifestyles. New York: Brunner/Mazel; 1980. pp. 1410–1452. [Google Scholar]
  60. Marlatt GA, Gordon JR. Relapse prevention: Maintenance strategies in the treatment of addictive behaviors. New York: Guilford Press; 1985. [Google Scholar]
  61. Marlatt GA, Donovan DM, editors. Relapse prevention: Maintenance strategies in addictive behavior change. 2nd ed. New York: Guilford Press; 2005. [Google Scholar]
  62. McGovern MP, Xie H, Segal SR, Siembab L, Drake RE. Addiction treatment services and co-occurring disorders: Prevalence estimates, treatment practices, and barriers. Journal of Substance Abuse Treatment. 2006;31:267–275. doi: 10.1016/j.jsat.2006.05.003. [DOI] [PubMed] [Google Scholar]
  63. McLellan AT, Carise D, Coyne TH. Addiction severity index (ASI) 5th ed. Treatment Research Institute; 2005. [Google Scholar]
  64. Montgomery EC, Kunik ME, Wilson N, Stanley MA, Weiss B. Can paraprofessionals deliver cognitive-behavioral therapy to treat anxiety and depressive symptoms? Bulletin of the Menninger Clinic. 2010;74(1):45–62. doi: 10.1521/bumc.2010.74.1.45. [DOI] [PubMed] [Google Scholar]
  65. Morgenstern J, Morgan T, McCrady B, Keller D, Carroll K. Manual-guided cognitive-behavioral therapy training: a promising method for disseminating empirically supported substance abuse treatments to the practice community. Psychology of Addictive Behaviors. 2001;15(2):83–88. [PubMed] [Google Scholar]
  66. Mueser KT, Noordsy DL, Drake RE, Fox L. Integrated treatment for dual disorders: A guide to effective practice. New York: Guilford Press; 2003. [Google Scholar]
  67. Muñoz R, Ippen C, Rao S, Le H, Dwyer E. Manual for group cognitive-behavioral therapy of major depression: A reality management approach. San Francisco: Cognitive Behavioral Depression Clinic, San Francisco General Hospital, University of California, San Francisco; 2000. [Google Scholar]
  68. Osilla KC, Hepner KA, Munoz RF, Woo S, Watkins K. Developing an integrated treatment for substance use and depression using cognitive-behavioral therapy. Journal of Substance Abuse Treatment. 2009;37(4):412–420. doi: 10.1016/j.jsat.2009.04.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Perepletchikova F, Kazdin AE. Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice. 2005;12(4):365–383. [Google Scholar]
  70. Pettinati HM, Oslin DW, Kampman KM, Dundon WD, Xie H, Gallis TL, et al. A double-blind, placebo-controlled trial combining sertraline and naltrexone for treating co-occurring depression and alcohol dependence. American Journal of Psychiatry. 2010;167(6):620–622. doi: 10.1176/appi.ajp.2009.08060852. [DOI] [PMC free article] [PubMed] [Google Scholar]
  71. Petry NM. A comprehensive guide to the application of contingency management procedures in clinical settings. Drug and Alcohol Dependence. 2000;58(1–2):9–25. doi: 10.1016/s0376-8716(99)00071-x. [DOI] [PubMed] [Google Scholar]
  72. Prendergast M, Podus D, Finney J, Greenwell L, Roll J. Contingency management for treatment of substance use disorders: A meta-analysis. Addiction. 2006;101(11):1546–1560. doi: 10.1111/j.1360-0443.2006.01581.x. [DOI] [PubMed] [Google Scholar]
  73. Project MATCH Research Group. Matching alcoholism treatments to client heterogeneity: Project MATCH posttreatment drinking outcomes. Journal of Studies on Alcohol. 1997;58:7–29. [PubMed] [Google Scholar]
  74. Rapaport MH, Judd LL, Schettler PJ, Yonkers KA, Thase ME, Kupfer DJ, et al. A descriptive analysis of minor depression. American Journal of Psychiatry. 2002;159(4):637–643. doi: 10.1176/appi.ajp.159.4.637. [DOI] [PubMed] [Google Scholar]
  75. Rawson RA, Shoptaw SJ, Obert JL, McCann MJ, Hasson AL, Marinelli-Casey PJ, et al. An intensive outpatient approach for cocaine abuse treatment: The matrix model. Journal of Substance Abuse Treatment. 1995;12:117–127. doi: 10.1016/0740-5472(94)00080-b. [DOI] [PubMed] [Google Scholar]
  76. Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: Getting started and moving on from Stage 1. Clinical Psychology: Science and Practice. 2001;8(2):133–142. [Google Scholar]
  77. SAS Institute Inc. SAS/Stat 9.2 user's guide. Cary, NC: SAS Institute Inc; 2008. [Google Scholar]
  78. Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Nuro KF, Carroll KM. Train in vain: A dissemination trial of three strategies of raining clinicians in cognitive behavioral therapy. Journal of Consulting and Clinical Psychology. 2005;73(1):106–115. doi: 10.1037/0022-006X.73.1.106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  79. Skinner HA. The drug abuse screening test. Addictive Behaviors. 1982;7:363–371. doi: 10.1016/0306-4603(82)90005-3. [DOI] [PubMed] [Google Scholar]
  80. Sloan K, Kivlahan D, Saxon A. Detecting bipolar disorder among treatment-seeking substance abusers. American Journal of Drug and Alcohol Abuse. 2000;26(1):13–23. doi: 10.1081/ada-100100587. [DOI] [PubMed] [Google Scholar]
  81. Sobell LC, Sobell MB. Timeline followback: A technique for assessing self-reported ethanol consumption. In: Allen J, Litten RZ, editors. Measuring alcohol consumption: Psychosocial and biological methods. Totowa, NJ: Humana Press; 1992. pp. 41–72. [Google Scholar]
  82. Strain EC, Stitzer ML, Bigelow GE. Early treatment time course of depressive symptoms in opiate addicts. Journal of Nervous and Mental Disease. 1991;179:215–221. doi: 10.1097/00005053-199104000-00007. [DOI] [PubMed] [Google Scholar]
  83. Substance Abuse and Mental Health Services Administration. Rockville, MD: 2009a. Results from the 2008 National Survey on Drug Use and Health: National findings. (Office of Applied Studies, NSDUH Series H-36, HHS Publication No. SMA 09-4434) [Google Scholar]
  84. Substance Abuse and Mental Health Services Administration. Integrated treatment for co-occurring disorders (SMA08-4367-09) 2009b Available at http://store.samhsa.gov/shin/content/SMA08-4367/SMA08-4367-09.pdf.
  85. Substance Abuse and Mental Health Services Administration. Washington, DC: US Department of Health and Human Services; 2006. National Survey of Substance Abuse Treatment Services (N-SSATS) [Google Scholar]
  86. Tiet QQ, Mausbach B. Treatments for patients with dual diagnosis: A review. Alcoholism: Clinical and Experimental Research. 2007;31(4):513–536. doi: 10.1111/j.1530-0277.2007.00336.x. [DOI] [PubMed] [Google Scholar]
  87. Tonigan JS, Miller WR. The inventory of drug use consequences (InDUC): Test-retest stability and sensitivity to detect change. Psychology of Addictive Behaviors. 2002;16(2):165–168. [PubMed] [Google Scholar]
  88. Ware J, Jr, Kosinski M, Turner-Bowker D, Gandek B. How to score version 2 of the SF-12® Health Survey (with a supplement documenting version 1) Lincoln, RI: Quality Metric Incorporated; 2002. [Google Scholar]
  89. Watkins KE, Burnam A, Kung FY, Paddock S. A national survey of care for persons with co-occurring mental and substance use disorders. Psychiatric Services. 2001;52(8):1062–1068. doi: 10.1176/appi.ps.52.8.1062. [DOI] [PubMed] [Google Scholar]
  90. Watkins KE, Hunter SB, Burnam MA, Pincus HA, Nicholson G. Review of treatment recommendations for persons with a co-occurring affective or anxiety and substance use disorder. Psychiatric Services. 2005;56(8):913–926. doi: 10.1176/appi.ps.56.8.913. [DOI] [PubMed] [Google Scholar]
  91. Wells KB, Sturm R, Sherbourne CD, Meredith L. Caring for depression. Cambridge, MA: Harvard University Press; 1996. [Google Scholar]
  92. Wenzel SL, Ebener P, Hunter SB, Watkins KE, Gilmore JM. Research-practice partners assess their first joint project. NIDA Science and Practice Perspectives. 2005:38–45. doi: 10.1151/spp053138. [DOI] [PMC free article] [PubMed] [Google Scholar]
  93. Witkiewitz K, Villarroel NA. Dynamic association between negative affect and alcohol lapses following alcohol treatment. Journal of Consulting and Clinical Psychology. 2009;77(4):633–644. doi: 10.1037/a0015647. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES