Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Jun 1.
Published in final edited form as: J Subst Abuse Treat. 2011 Feb 23;40(4):405–413. doi: 10.1016/j.jsat.2011.01.003

Program, Counselor, and Patient Variability in the Alliance: A Multilevel Study of the Alliance in Relation to Substance Use Outcomes

Paul Crits-Christoph a,*, Jessica L Hamilton a, Sarah Ring-Kurtz a, Robert Gallop c, Bridget McClure b, Agatha Kulaga b, John Rotrosen b
PMCID: PMC3081942  NIHMSID: NIHMS265017  PMID: 21349680

Abstract

We explored patient, therapist, and program variability in the alliance in relation to drug and alcohol use during treatment, and whether alliance mediates the relation of program characteristics to drug/alcohol use. Data (N=1613 patients) were drawn from a randomized clinical trial investigating the efficacy of an intervention that provided alliance and outcome feedback to 112 counselors across 20 community-based outpatient substance abuse treatment clinics in the northeast United States. Program characteristics were measured using the Organization Readiness for Change scale. Using multilevel modeling, we found that alliance was related to both drug and alcohol use during the past week at the patient and program levels of analysis, but not the counselor level. Several program characteristics were related to average drug and alcohol use. The alliance was not a mediator of these relationships. Program variability in the alliance is important to the alliance-outcome relationship in the treatment of substance abuse. Better outcomes can be achieved by improving both organizational functioning and the patient-counselor alliance.

Keywords: Substance Abuse, Alliance, Program, Counselor

1. Introduction

Meta-analyses have shown a consistent link between a positive alliance – defined by Bordin (1979) as composed of the emotional bond between patient and therapist/counselor, agreement on tasks of treatment, and agreement on goals of treatment – and relatively more favorable treatment outcomes across a wide range of different types of psychosocial treatments and different patient populations including substance use disorders (Horvath & Symonds, 1991; Martin, Garske, & Davis, 2000). Despite the breadth of the literature on the alliance-outcome connection, little is known about what factors are responsible for the documented correlational relationship between the alliance and psychotherapy outcomes.

One basic question is whether patient or therapist factors are the primary variables responsible for the alliance-outcome correlation. A number of studies have indicated that pretreatment characteristics of patients are associated with the alliance (Gibbons et al., 2003; Muran, Segal, Samstag, & Crawford, 1994). Variability from patient to patient based on such pretreatment characteristics, or how individual patients experience a given therapist, may be driving the correlation between the alliance and outcome. In addition to, or instead of, such patient factors, therapist variables may be responsible for the alliance-outcome connection (Ackerman & Hilsenroth, 2003). Individual therapist differences in technical skills or non-specific characteristics such as empathy could affect the quality of the alliance and either directly or indirectly (through the alliance) impact treatment outcomes (Crits-Christoph, Barber, & Kurcias, 1993). To the extent that therapist variables are responsible for the alliance-outcome correlation, it may be possible to improve outcomes by training therapists to enhance their alliance with patients (Crits-Christoph et al., 2006).

To evaluate the extent to which patient or therapist variables primarily are responsible for the alliance-outcome correlation, it is useful to first determine whether between-patient variability in the alliance or between-therapist variability (in their average patient alliances within a therapist) influences the alliance-outcome link. If, for example, therapist differences in average patient alliances are not associated with treatment outcome, it would not be fruitful to explore therapist skills or other characteristics as potential variables responsible for the alliance-outcome correlation.

A statistical approach for separately estimating the relative contribution of patient variability and therapist variability to the alliance-outcome correlation has been described and implemented by Baldwin, Wampold, and Imel (2007). Using multilevel modeling, these authors conducted separate tests of the potential link between (1) therapist variability in the alliance and outcome, and (2) patient variability in the alliance and outcome. As explained by Baldwin et al. (2007), the within-therapist test examines whether patients with high alliance scores achieve relatively better outcomes than patients, treated by the same therapist, who have low alliance scores. The between-therapist test examines the relation of therapists' average alliance (averaging over all patients treated by a therapist) to the average outcome achieved by each therapist (again, averaging over patients treated by each therapist). Baldwin et al. (2007) describe how a relation at one level (e.g., therapist) could be positive and at the other level (e.g., patient) could be negative, with such effects canceling out when these sources of variability are incorporated into the statistical model and tested. In their own study, therapist variability in the alliance was found to relate to outcome for students in university counseling centers, but patient variability did not relate to outcome (Baldwin et al., 2007).

Recently, the Baldwin et al. (2007) multilevel approach has been applied to data from a study of motivational enhancement therapy (MET; Miller, Zweben, DiClemente, & Rychtarik, 1992) and counseling-as-usual in the treatment of substance use problems (Crits-Christoph et al., 2009). This study, like Baldwin et al. (2007), found that counselor variability in alliance, but not patient variability, was significantly related to treatment outcomes.

In the treatment of substance use problems, however, patient and drug counselor factors may not be the only levels that are important to consider. Independent of patient and counselor variability in alliance, program factors may also be important in regard to the alliance-outcome correlation. In the United States, treatment of substance use problems is commonly conducted within the context of specific substance abuse treatment programs (i.e., an outpatient clinic at one location) (Substance Abuse & Mental Health Services Administration, 2008). These programs can differ considerably in size, structure, services provided, theoretical orientation, and qualifications of the clinicians (McCarty et al., 2008; Courtney, Joe, Rowan-Szal, & Simpson, 2007; Broome, Flynn, Knight, & Simpson, 2007; Knight, Broome, Simpson, & Flynn, 2008). Such program factors have been found to be associated with patient substance use outcomes (Moos & Moos, 1998), as well as measures of treatment participation, treatment retention, and treatment satisfaction (Lehman, Greener, & Simpson, 2002; Greener, Joe, Simpson, Rowan-Szal, & Lehman, 2007). Moreover, programs rated by staff as having more positive organizational characteristics were found to have higher average patient-counselor rapport, as measured by a single item patient rating (Greener et al., 2007). Based on these types of findings, the alliance has been hypothesized to mediate the relationship between program factors and outcome in the treatment of substance abuse (Simpson, 2004).

The purpose of the current article is to explore patient, counselor, and program variability in patient-reported alliance in relation to drug and alcohol use during treatment for substance use problems. Based on the studies reviewed above, we hypothesized that when the alliance-outcome relation is partitioned into these three levels, the program level and counselor level would be related to drug and alcohol use. Given that certain program factors have been associated with substance use outcomes, a secondary aim of the study was to examine whether alliance might be a mediating factor between program characteristics and ongoing drug and alcohol use. These meditational analyses were conducted to help understand any multilevel effects found for the alliance-outcome relationship.

2. Materials and Methods

2.1 Study Design

For the current analyses, we used data from a randomized clinical trial investigating the efficacy of an intervention that involved providing alliance and outcomes feedback to substance abuse counselors (Crits-Christoph et al., 2010). The study was implemented in 20 community-based non-methadone maintenance outpatient substance abuse treatment programs in the metropolitan Philadelphia and New York areas. In the clinical trial, participating programs were randomly assigned to use the performance improvement system either immediately or after a 12-week delay. During a fixed 12-week time period, essentially all patients who attended a group counseling session during any of the 12 successive weeks were administered a brief alliance/outcome feedback survey, regardless of when each patient had begun their course of treatment. Thus, at the first (Week 1) assessment, patients completing the feedback survey had been in treatment for various durations. At subsequent assessments in the study, the patient sample that was assessed was changing as new patients were added and existing patients dropped out. This approach was taken because the focus was on counselors' performance (of their caseload as a whole) over time, not individual patient's improvement over time. Because of the changing patient sample over time, the primary analyses used only the Week 1 data for the immediate and delayed groups. Moreover, the feedback survey containing the alliance and outcome measures was anonymous and therefore the data for the same patients over time could not be linked. The Week 12 data (excluding any patients that reported being in treatment for 3 months or longer) were then used as a replication sample, since none of these patients would be included in the Week 1 sample. Previous studies examining program level characteristics in relation to patients outcomes (Greener et al., 2007; Lehman et al., 2002) also assessed patients at one point in time, with these patients having various durations of time in treatment. Thus, our methodology was similar, allowing us to make comparisons to these studies. However, we also examined whether results varied as a function of time in treatment.

2.2 Participants

To be eligible for participation in this study, programs had to have at least four counselors who were currently conducting group counseling sessions (at least once a week), and the counselors had to be able to attend a monthly staff team meeting. Programs also needed to have internet access for their counselors and supervisors. Group counseling was the primary clinical modality at all participating programs. All participating programs had clinical policies for the regular implementation of biological testing, and typically targeted tests for patients who were suspected of using drugs or alcohol.

Counselors provided written informed consent because they were the target of the intervention and were thus considered the human subjects in this randomized trial. All study materials, procedures, and consent forms were approved by all relevant Institutional Review Boards overseeing the research conducted at the participating programs.

Only patients receiving group counseling for substance abuse problems were eligible to participate (all programs used group counseling as the primary mode of service delivery). Because the patient assessment was anonymous and there was no perceived risk, informed consent by patients was not required by the participating Institutional Review Boards. Participation, however, was voluntary and patients were oriented to the study. The study did not recruit patients as they began a treatment episode at a participating program. Rather, all patients who were attending participating counselors' eligible group counseling sessions during a given study week were recruited to complete the assessment regardless of how long they had been in treatment.

2.3 Assessment of Patients

The patient assessment was designed to be a functional instrument that could be easily implemented as a regularly administered clinic-wide feedback measure; it was designed to be very brief. The assessment was a 1-page form that included demographic information (gender, race, month and year of birth), a brief alliance measure (4-items), three questions on treatment satisfaction, a question asking how long the patient has been in treatment (responses were 1 week, 2 weeks, 3 weeks, 4 weeks, 1 to 3 months, and greater than 3 months), and questions on recent drug and alcohol use. No patient names or identifying numbers were included on the form.

The therapeutic alliance measure (Crits-Christoph et al., 2004) was a brief 4-item scale derived from the California Psychotherapy Alliance Scale (CALPAS; Gaston et al., 1991). Each of the 4 items in the scale assesses one of the 4 main dimensions of alliance (bond, agreement on tasks, agreement on goals, therapist understanding). Respondents rate each on a 5-point scale that begins with “not at all” = 1, “a little bit” = 2, “moderately” = 3, “quite a bit” = 4 and “very much so” = 5. In a preliminary study, the total of the 4 items was found to correlate .80 with the CALPAS total score after deleting the 4 items (Crits-Christoph et al., 2004). The internal consistency reliability (Cronbach's alpha) for the total of the 4 items was found to be .87 at week 1 in the current study.

The two substance use items were adapted from the Drug and Alcohol sections of the Addiction Severity Index (McLellan, Luborsky, O'Brien, & Woody, 1980) and asked about the number of days in the past 7 days that alcohol and drugs, respectively, were used. In addition, to boost reliability, the average of these two items was calculated to measure degree of alcohol or drug use in the past week. The correlation between the alcohol item and the drug use item was .51 at Week 1 in the current study.

Assessments were conducted weekly for 12 weeks for programs that were randomly assigned to received the intervention immediately, and at week 1 (baseline) and week 12 (post-intervention) for programs that were randomly assigned to the delayed group. For each week, the patient assessment form was given to eligible patients at the end of group counseling sessions. Clinicians were not to be present when patients completed the form. Patients deposited the completed forms in a locked box to ensure anonymity and confidentiality. Patients were instructed to complete only one survey in a given week if the patient attended more than one group in a week with that clinician.

2.4 Assessment of Program Characteristics

The organizational functioning of programs was assessed by having participating clinicians and supervisors complete the Organizational Readiness to Change (ORC) scale (Lehman et al., 2002). The ORC measures organizational characteristics along 4 major categories of organizational readiness. Within these 4 major categories, 18 subscales can be computed. One category, motivation for change, includes three subscales (program need for improvement, perceived training needs, and pressure for change). A second category, institutional resources, has five subscales (adequacy of office space, staffing, training resources, computer access, and use of e-mail and Internet). A third category, staff attributes, includes subscales measuring potential for professional growth, efficacy (confidence in counseling skills), ability to influence coworkers, and adaptability. The fourth category, organizational climate, has 6 subscales: the organization's clarity of mission and goals, staff cohesiveness (trust and cooperation among staff), staff autonomy (freedom in treatment planning and clinical work), management's openness to communication from staff, perceived stress (strain and role overload), and openness to change. The reliability and validity of the ORC was established in a study involving over 500 clinicians from more than 100 substance abuse treatment programs (Lehman et al., 2002).

2.5 Statistical Analysis

A preliminary analysis examined means differences between counselors and between programs in alliance scores. The distribution of the alliance scores was non-normal, with a relative high (about 35%) of scores at the maximum (“5” on the 1 to 5 scale). A Box-Cox transformation analysis revealed that a cube transformation was most appropriate to address the non-normal distribution. Using these transformed scores, we performed a multilevel mixed effects model accounting for the three levels of nesting: patients within counselor within program, where counselor and program were treated as random effects. In these models, the test for an effect at each level controls for the effects of the other levels. The test of significance in these models follows the method described by Verbeke and Molenbergh (2000), where the test statistic is a mixture of chi-squares.

For the prediction of outcome, the measures used as primary dependent variables were the drug and alcohol use scores from the patient assessment. These drug and alcohol use measures showed a preponderance of zeros in this sample (> 80%). To address this, a mixed-effects, zero-inflated Negative Binomial (ZINB) model was used for the analysis of the relation of the alliance to drug and alcohol use (Hedeker & Gibbons, 2006; Lambert, 1992). The ZINB model, which is an extension of the better-known zero-inflated Poisson (ZIP) model, specifies the conditional distribution as being Negative Binomial and thereby is a non-linear analog to the linear mixed-effects, repeated-measures model. The ZINB model consists of two components: a logistic regression portion to examine zero vs. non-zero states (e.g., no drug use vs. any drug use) and a Negative Binomial regression part to account for differences in degrees of non-zero state (e.g., number of non-zero days of drug use). To implement the model with our data, we initially included a random intercept for program, and a random intercept and slope for counselor within program. Additional analyses also tested for a cross-level interaction (Snijders & Bosker, 1999), specifically an interaction between the counselor level and the program level. This interaction examined the possibility that counselor variability in the alliance was associated with outcome at some programs but not at other programs. Patient-reported duration of current treatment episode was included as a covariate in all models. Further models tested whether the alliance-outcome relationship (at the patient, counselor, and program levels) varied as a function of patient treatment duration. To fit these models, we used PROC MIXED and PROC NLMIXED in SAS (Littell, Milliken, Stroup, Wolfinger, & Schabenberger, 2006).

To examine whether the alliance mediated the relation between program characteristics and drug/alcohol use, the same multilevel models described above were conducted with ORC scales as covariates (predictors) at the program level. These analyses were conducted with and without the alliance in the model. If alliance mediates the relation of program characteristics to drug/alcohol use, the relationship between ORC scales and these outcome should be smaller in size (less statistically significant) when alliance is in the model compared to when alliance is not in the model. This is because the ORC scale's impact on outcome would be through its impact on the alliance, if the alliance is a mediator, so the ORC scale would no longer impact outcome, or impact it less, once the intervening mediator is statistically controlled. If the part of organization functioning that is unrelated to the alliance is predictive of outcome, then partialling out the alliance from the relationship between the ORC scale and outcome will not decrease the size of this effect.

3. Results

3.1 Sample Characteristics

Across the 20 programs, there were 112 clinicians for whom patients provided feedback surveys during Week 1 of the study. Of these 112 clinicians, 72.3% (81/112) were women, and 12.5% (14/112) were Hispanic or Latino. In terms of race, 75.0% (84/112) were Caucasian, 17.9% (20/112) were African American, 0.9% (1/112) were bi-racial, and 6.3% (7/112) reported other racial backgrounds. The average age of the clinicians was 41.8 years (range: 22 to 71). In terms of clinical experience, 2.6% (3/112) had worked as a clinician for 0 to 6 months, 11.6% (13/112) for 6 to 11 months, 25.9% (29/112/118) for 1 to 3 years, 11.6% (13/112) for 3 to 5 years, and 48.2% (54/112) for more than 5 years. Most 56.3% (63/112) of the clinicians had a master's degree; 1.8% (2/112) had a high school diploma, 8.9% (10/112) had an associate's degree, 31.3% (35/112) had a bachelor's degree, and 1.8% (2/112) had a doctoral degree.

At the Week 1 assessment (N = 1,613), the patient sample was 66% men and 34% women; 48% identified themselves as Caucasian, 39% African-American, 10% Latino, and 3% other. In terms of treatment duration, 7% percent of patients had been in treatment less than one week; 22% from one to four weeks; 26% from one to three months; 45% more than three months.

3.2 Multilevel Analysis of Counselor and Program Differences in Mean Alliance

The multilevel model showed significant differences in average alliance for counselors (χ2(1) = 8.76, p = .003) and for program (χ2(1) = 4.00, p = .046) as assessed through the multilevel mixed effects model on transformed alliance score with counselor and program treated as random effects. The average alliance scores ranged from 3.38 to 5.00 across the 112 clinicians with an overall average of 4.30 (SD = .37). The average alliance ranged from 3.76 to 4.61 across the 20 sites with an overall average of 4.28 (SD = .23).

3.3 Multilevel Prediction of Drug and Alcohol Use from Alliance

The multilevel mixed effects models revealed that alliance was significantly associated with drug and alcohol use at both the patient and program levels of analysis in the logistic part of the analysis (any use vs. no use) for all three outcomes (drug use, alcohol use, any drug or alcohol use) (Table 1). This means that variability in the alliance within a counselor's caseload was predictive of drug/alcohol use and variability between programs in the average alliance predicted average program drug/alcohol use, controlling for patient length of treatment at both the patient and program levels. Alliance variability between counselors within a program was not related to counselors' average outcomes in the logistic part of the analysis. Alliance did not significantly predict amount of usage (number of days) for those with any (non-zero) use of drug or alcohol, at the patient or program levels of analysis. There was a significant relation of the alliance to (non-zero) days of alcohol use at the counselor level. However, no significant relationship of alliance with days of drug use, or days of any alcohol or drug use, was evident at the counselor level in this non-zero part of the alliance. There were no significant cross-level interactions (counselor by program) effects for either the logistic part of the analyses or the amount of non-zero use portion of the analysis for alcohol use, drug use, or any alcohol/drug use.

Table 1. Results of Multilevel Analysis Predicting Drug and Alcohol Outcomes From Alliance.

Any Use
(zero vs. non-zero)		 Count of
Non-Zero Use
Regression
Coefficient		 p value Regression
Coefficient		 p value
Any Drug or Alcohol Use
 Patient Level 0.367 0.001 0.018 0.83
 Counselor Level 0.396 0.38 -0.349 0.14
 Program Level 2.136 0.0008 0.220 0.71
Alcohol Use
 Patient Level 0.518 0.001 0.053 0.61
 Counselor Level 0.670 0.20 -0.212 0.47
 Program Level 2.122 0.006 -0.097 0.82
Drug Use
 Patient Level 0.261 0.039 0.037 0.70
 Counselor Level 0.687 0.200 -0.130 0.67
 Program Level 2.060 0.005 0.632 0.15
Open in a new tab

Note. Analysis is based on 1613 patients, 112 counselors, and 20 programs. Length of treatment is a covariate in all analyses.

To more clearly understand the strength of these findings and to compare the findings with the literature, we calculated Cohen's d effect sizes at the patient, counselor, and program levels between the alliance and outcome based on the test statistics from the logistic part of the ZINB model. At the patient level (controlling for counselor and program differences and controlling for treatment duration), effect sizes for predicting any alcohol use, any drug use, and any drug or alcohol uses were .62, .40, and .63, respectively. At the program level, these effect sizes were .52, .53, and .66. At the counselor level, the effect sizes were .24 .24, and .17.

3.4 Impact of Treatment Length on Alliance-Outcome Relationship

Although there was a main effect for patient treatment length in relationship to outcome (reported above), there was a high rate of reported non-use of drugs and alcohol at each category of treatment length. For the Week 1 data, the percent of non-use of drugs at each duration of treatment was: 1 week: 78.9%; 2 weeks: 85.5%; 3 weeks: 81.9%; 4 weeks: 87.5%; 1-3 months: 90.0%; > 3 months: 87.7%. The percent of non-use of alcohol for patients at each duration of treatment was: 1 week: 76.9%; 2 weeks: 78.3%; 3 weeks: 80.0%; 4 weeks: 83.7%; 1-3 months: 88.9%; > 3 months: 88.0%. Despite these relatively small differences in rates of non-use for patients at different treatment durations, we examined whether the alliance-outcome relationship varied depending on length of treatment. Inclusion of a treatment duration by alliance interaction term in the statistical models revealed no significant interaction effects for alcohol use, drug use, or any alcohol or drug use for the logistic (any use vs. no use) at either the patient, counselor, or program levels. However, inclusion of the interaction term in the prediction of days used (non-zero part), revealed several significant treatment duration by alliance interactions. Significant interaction effects were found at the program level in predicting days of alcohol use (t [111]=2.33, p = .022) and days of any drug or alcohol use (t [111] = 2.47 p = .015), with larger effects (r's =-.29 and -.24, for alcohol and any drug/alcohol, respectively) for the alliance-outcome relationship among programs that were below the average patient length of treatment at the program level. In addition, a significant (t [111]= 2.59, p = .011) treatment duration by alliance interaction was evident at the counselor level for days of drug use. Effects for those counselors with above average patient lengths of treatment (r = .07) and those with below average patient lengths of treatment (r = -.07) were both relatively small.

3.5 Stability of Effects

The stability of these effects was examined by conducting the same analyses on the sample of patients who completed the alliance and outcome measures at Week 12 of the randomized trial. For this sample, we excluded all patients who reported they had been in treatment at the program for 3 months or longer (so that a different set of patients was selected at Week 12 than those that provided data at Week 1). This resulted in a sample of 812 patients who were treated by 96 counselors. Using this Week 12 data, the pattern of results from the multilevel model was very similar to the Week 1 results. In the logistic part of the ZINB model, there were significant effects at the program level for alcohol use (coefficient = 2.982, p = .0008, d = .71), drug use (coefficient = 2.807, p = 0.0001, d = .62) and any alcohol or drug use (coefficient = 3.404, p < .0001, d = .93). There were also significant effects at the patient level in the logistic part of the ZINB model (drug use: coefficient = .440, p = .003, d = .62; any alcohol or drug use: coefficient = .335, p = .021, d = .48). No significant effects were apparent at the counselor level for any of the outcome measures and no significant effects were evident for the analysis of non-zero amount of drug/alcohol use. No significant interactions between the alliance and length of treatment were evident at the patient, counselor, or program levels in the Week 12 data.

3.6 Relation of Program Characteristics to Outcome

Table 2 provides results of the logistic part of multilevel ZINB models in which ORC scales are used to predict drug and alcohol use outcomes at the program level. Effect sizes are expressed as correlation coefficients calculated from the test statistics; only significant results are shown. With alcohol use as the outcome, Cohesion, Program Needs, and Influence were statistically significant predictors (Table 2). With drug use as the outcome, Cohesion, Communication, and Influence were statistically significant predictors. With any drug or alcohol use as the outcome, Cohesion, Staffing, and Influence were statistically significant predictors. Better average program outcomes were associated with programs that were perceived to have better organizational cohesion, better communication among staff, more program needs, greater motivation to change, adequate staffing, and more positive staff attributes (influence).

Table 2. Organization Characteristics as Predictors of Drug and Alcohol Use Outcomes.

Outcome Measure
Any Alcohol or Drug Use Any Alcohol Use Any Drug Use
ORC scales (predictors) Without Alliance in Model With Alliance in Model Without Alliance in Model With Alliance in Model Without Alliance in Model With Alliance in Model
Cohesion .25 (.007) .25 (.008) .21, (.024) .20 (.034) .24 (.012) .24 (.011)
Communication - - - - -.21 (.037) -.25 (.007)
Mission - - - - - -
Program Needs - - -.19, (.048) -.18 (.053) - -
Staffing -.20, (.036) -.29 (.002) - - - -
Influence -.29 (.002) -.34 (.0002) -.26 (.006) -.30 (.002) -.21 (.026) -.27 (.004)
Open in a new tab

Note. Effect sizes (p-value) expressed as correlation coefficients are shown. Correlation coefficients are calculated from test statistics from the logistic part of the ZINB multilevel model that included 1613 patients, 112 counselors, and 20 programs. Only significant results of ORC scales in relation to outcome at the program level, without alliance in the model, are displayed (along with the corresponding coefficient when alliance is added to the model). Bolded values indicate cases where the relation of the ORC scale to outcome decreased when alliance was in the model (i.e., potential evidence for mediation); non-bolded values indicate cases where the relation of the ORC scales to outcome increased when alliance was in the model.

Other ORC scales, including the Autonomy, Stress, Openness to Change, Training Needs, Pressure to Change, Office Facilities, Training, Equipment, Internet, Growth, Efficacy, and Adaptability, showed no statistically significant relations to average program drug or alcohol use outcomes. Thus, out of a total of 54 analyses (18 scales; 3 outcomes), there were 9 statistically significant results.

3.7 Does Alliance Mediate the Relation between Program Characteristics and Outcome?

Based on the finding that alliance at the program level predicted drug and alcohol use outcomes, and that certain program characteristics were also significantly related to outcome at the program level, we assessed whether or not the alliance at the program level mediated the relation between program characteristics and outcome at the program level. When alliance was included in multilevel ZINB models along with ORC scales as predictors of outcome, there was little evidence that the relation between program characteristics and outcome was mediated by alliance. For six of the nine significant relationships between ORC scales and outcome, the magnitude of the relationship actually increased when alliance was in the model compared to when alliance was not in the model (Table 2). For three of the significant relationships between ORC scales and outcome, the inclusion of alliance in the model only slightly reduced the significance of the ORC-outcome relationship.

4. Discussion

The current study is unique in conducting a tri-level (patient, counselor, and program) analysis of the alliance-outcome relationship. The primary finding of the study was that alliance variability at the patient and program level, but not the counselor level, was significantly related to presence/absence of any drug and alcohol use during the past week. These findings were strongest for patients who had been in treatment for a relatively shorter period of time (less than 3 months). This general pattern of results was replicated using a second sample of patients that participated at a later point in time in the randomized trial from which these data were drawn. However, in the replication sample there was no evidence for differential alliance-outcome relationship depending on length of treatment. An additional finding was that certain program characteristics predicted treatment outcome, but these relationships were not mediated by the alliance.

The lack of finding at the counselor level is discrepant from recent studies (Baldwin et al., 2007; Crits-Christoph et al., 2009) that found that clinician variability in alliance was related to outcome. In the context of the current study, it might be hypothesized that counselors experiencing burn-out at poorly functioning programs that have difficult-to-treat patient populations would have poorer alliance and outcomes compared to counselors at relatively better functioning programs, thus producing a confound between the program level and the counselor level. In fact, we did find significant overall mean differences between counselors in their average alliance, indicating that some counselors typically form better alliances than do other counselors. This counselor variability in alliance was not related to outcome, with the exception of a finding of the alliance in relation to alcohol use. This effect, however, was small and was not found in the replication sample at Week 12. Moreover, the lack of counselor effect in alliance-outcome relationship was consistent across programs (i.e., no significant counselor by program cross-level interaction), suggesting that even at the better functioning programs there was no evidence for a counselor level effect in the alliance-outcome relationship. One of the previous studies (Crits-Christoph et al., 2009) that did find a counselor-level effect in the alliance-outcome relationship also used a sample from community-based substance abuse treatment clinics, so the differences between that study and the current one are not likely due to patient population or clinician differences. It may be that the methodology used here, including the use of a single assessment of alliance and drug/alcohol use, was not sensitive enough to pick up on potential counselor effects. Alternatively, the lack of modeling program effects in the Crits-Christoph et al. (2009) study may have generated apparent counselor effects if the counselor effect was confounded with program in that study.

Another difference between the current study and previous multilevel studies of the alliance-outcome relationship was our finding that patient variability (within counselor) in alliance was related to outcome. The large sample size used here (N=1613) and associated high statistical power compared to other studies may be in part responsible for the current study detecting the patient level effects. However, the effect sizes for the relationship between alliance and any alcohol use, any drug use, and any drug or alcohol use at the patient level (converting Cohen's d to r, these were r's of .30, .20, and .30, respectively) were similar in magnitude, or higher, than that reported for the alliance-outcome relationship in meta-analyses (i.e., r = .22 from Martin et al., 2000). Thus, it may be more perplexing why the Baldwin et al. (2007) and Crits-Christoph et al. (2009) studies failed to find that patient variability in alliance predicted outcome than it is perplexing why the current study did find such a relationship. From a clinical point of view, differences between patients in their experience of the alliance, within a clinicians' caseload, has long been thought to be a determinant of outcome. Thus, the current results are more consistent with clinical expectations.

The findings in the current study, namely that certain aspects of organizational functioning of programs are associated with ongoing patient-reported drug/alcohol use, replicates previous research in substance abuse treatment programs (Moos & Moos, 1998; Lehman et al., 2002; Greener et al., 2007). In particular, Lehman, Greener, and Simpson (2002) reported that ORC scales of Staffing, Influence, Mission, Cohesion, Autonomy, Communication, and Openness to Change were related to treatment satisfaction and/or counselor rapport at the program level. The current study extends these results by documenting that Staffing, Influence, Cohesion, and Communication also predict drug/alcohol use outcomes at the program level. Assuming a causal connection between these program characteristics and drug/alcohol use outcomes, clinical treatment programs could make use of such research findings by assessing and intervening at the organizational level to improve treatment outcomes. Furthermore, accrediting agencies, policy makers, and third-party payers should also reflect on such results when considering how best to accredit, structure, and finance the treatment of substance use problems in order to develop a service delivery system that achieves better treatment outcomes. From a research point of view, intervention studies should incorporate assessment of the treatment context and existing program infrastructure to fully understand treatment outcomes. This is especially relevant for multi-center trials in the treatment of substance use problems.

Of special interest in the current study are the findings that (1) programs differ in average alliances, and (2) variability in the average alliance for programs is related to the average program outcomes. The mean differences in alliance between programs suggests that patients are able to relate better to their counselors in some programs compared to other programs (independent of any impact of the particular counselors working in these program). It may be that patients are picking up expectations about treatment and about counselors from the program environment and these expectations impact alliance. Expectations about treatment have been found to be associated with the alliance in previous investigations (Gibbons et al., 2003; Joyce & Piper, 1998). Another possibility is that there is some patient variable that is highly confounded with program and is also correlated with alliance, such as severity of drug/alcohol dependence.

Although variability in programs in their average alliances was found to be related to variability in average program drug and alcohol use outcomes, alliance was not a mediator of the relationship between program functioning variables and drug and alcohol use outcomes. Thus, the data do not support the hypothesis offered by Simpson (2004) that alliance mediates the relationship between program factors and outcome in the treatment of substance abuse. If poor organizational functioning has a causal impact on outcome, the current data suggest that it does so through a different mechanism than does the alliance. One implication of these findings is that, assuming causality based on these correlations, better outcomes could be achieved by both improving the organizational functioning of substance abuse treatment programs and improving the alliance of counselors with their patients.

Several limitations of the current study are important to keep in mind. One issue that limits the potential generalizability of our results is that many patients (both with drug/alcohol problems and other disorders) are not treated within a program setting. In the context of office practice (private practice) treatment of substance use problems, clinician differences may have a relatively greater influence on the alliance-outcome relationship. A second limitation was that the alliance measure was very brief (4 items). Stronger relationships may be apparent with an alliance measure with more items, although internal consistency reliability of the brief alliance measure was good (.87) in the current study. Similarly, the outcome measures were single items of recent drug and alcohol use. Again, multi-item outcome measures would more likely show stronger relationships with alliance.

A further limitation of the current study was that assessments were conducted as a “snapshot” of all patients within a program in a given week. Patients therefore varied on the duration of their treatment, with almost half of the patient sample having been in treatment for 3 months or longer. Because many patients report no current alcohol or drug use when they begin a treatment episode for substance use problems (they have already stopping using drugs and/or alcohol prior to their intake visit), the level of ongoing use throughout the course of treatment reflects the extent to which they have “slipped” or “relapsed” during their recovery. From this perspective, the level of reported use is an appropriate short-term outcome measure. However, the amount of use over a longer period of time than one week would be a more reliable index of treatment outcome.

Another limitation was that alliance and drug/alcohol use were assessed at the same point in time. Thus, the current study does not document that alliance predicts further change beyond the time point at which alliance was assessed. Additional studies will be needed to examine whether the program level, and patient level, effects in the alliance-outcome relationship found here are also evident when outcome is measured longitudinally for each patient beginning with their intake assessment. Such longitudinal assessments would be especially useful in understanding the causal direction of influence among these variables.

In summary, the current study found that variability in alliance at the program level and variability in alliance at the patient level were both related to drug and alcohol use outcomes. Alliance, however, was not found to be a mediator of the relationship between program characteristics and drug and alcohol use during treatment.

Acknowledgments

The preparation of this manuscript was funded in part by NIDA grants R01- DA020799 and R01- DA020809. We wish to thank the participating program administrators, clinicians, and clinic staff from the following programs: In the Philadelphia region: Northeast Treatment Center; Rehab After Work – Northeast; Rehab After Work - Center City; Rehab After Work – Lansdale; Rehab After Work – Paoli; CHANCES; Brandywine Counseling (in Delaware); Princeton House Behavioral Health (in New Jersey); In New York: Outreach Development – Greenpoint; Turning Point; Odyssey House; Lexington Center – Poughkeepsie; Lexington Center - Mt. Kisco; Lexington Center - New Rochelle; Addiction Research and Treatment Corporation; Horizon - Main Amherst; Horizon - Hertel Elmwood; Horizon - Bailey LaSalle; Horizon - Niagara Falls; and Horizon - Boulevard.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Ackerman SJ, Hilsenroth MJ. A review of therapist characteristics and techniques positively impacting the therapeutic alliance. Clinical Psychology Review. 2003;23:1–33. doi: 10.1016/S0272-7358(02)00146-0. [DOI] [PubMed] [Google Scholar]
  2. Baldwin SA, Wampold BE, Imel ZE. Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology. 2007;75:842–852. doi: 10.1037/0022-006X.75.6.842. [DOI] [PubMed] [Google Scholar]
  3. Bordin ES. The generalizability of the psycho-analytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice. 1979;16:252–260. doi: 10.1037/h0085885. [DOI] [Google Scholar]
  4. Broome KM, Flynn PM, Knight DK, Simpson DD. Program structure, staff perceptions, and client engagement in treatment. Journal of Substance Abuse Treatment. 2007;33:149–158. doi: 10.1016/j.jsat.2006.12.030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Courtney KO, Joe GW, Rowan-Szal GA, Simpson DD. Using organizational assessment as a tool for program change. Journal of Substance Abuse Treatment. 2007;33:131–137. doi: 10.1016/j.jsat.2006.12.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Crits-Christoph P, Barber JP, Kurcias JS. The accuracy of therapists' interpretations and the development of the therapeutic alliance. Psychotherapy Research. 1993;3:25–35. [Google Scholar]
  7. Crits-Christoph P, Connolly Gibbons MB, Crits-Christoph K, Narducci J, Schamberger M, Gallop R. Can therapists be trained to improve their alliances? A pilot study of Alliance-Fostering Therapy. Psychotherapy Research. 2006;13:268–281. doi: 10.1080/10503300500268557. [DOI] [Google Scholar]
  8. Crits-Christoph P, Gallop R, Temes CM, Woody G, Ball SA, Martino S, Carroll KM. The alliance in motivational enhancement therapy and counseling as usual for substance use problems. Journal of Consulting and Clinical Psychology. 2009;77:1125–1135. doi: 10.1037/a0017045. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Crits-Christoph P, Gibbons MB, Forman R, Hu B, Hearon B, Worley M. A brief alliance scale for community-based research. Unpublished manuscript, University of Pennsylvania; 2004. [Google Scholar]
  10. Crits-Christoph P, Ring-Kurtz S, McClure B, Temes C, Kulaga A, Gallop R, Forman R, Rotrosen J. A randomized controlled study of a web based performance improvement system for substance abuse treatment providers. Journal of Substance Abuse Treatment. 2010;38:251–262. doi: 10.1016/j.jsat.2010.01.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Gaston L. The reliability and criterion-related validity of the California Psychotherapy Alliance Scale. Psychological Assessment. 1991;3:68–74. doi: 10.1037/1040-3590.3.1.68. [DOI] [Google Scholar]
  12. Gibbons MB, Crits-Christoph P, de la Cruz C, Barber JP, Siqueland L, Gladis L. Pretreatment expectations, interpersonal functioning, and symptoms in the prediction of the therapeutic alliance across supportive-expressive psychotherapy and cognitive therapy. Psychotherapy Research. 2003;13:59–76. doi: 10.1093/ptr/kpg007. [DOI] [PubMed] [Google Scholar]
  13. Greener JM, Joe GW, Simpson DD, Rowan-Szal GA, Lehman WEK. Influence of organizational functioning on client engagement in treatment. Journal of Substance Abuse Treatment. 2007;33:139–147. doi: 10.1016/j.jsat.2006.12.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Harmon SC, Lambert MJ, Smart DM, Hawkins E, Nielsen SL, Slade K, et al. Enhancing outcome for potential treatment failures: Therapist-client feedback and clinical support tools. Psychotherapy Research. 2007;17:379–392. doi: 10.1080/10503300600702331. [DOI] [Google Scholar]
  15. Hedeker D, Gibbons RD. Longitudinal Data Analysis. John Wiley & Sons; New Jersey: 2006. [Google Scholar]
  16. Horvath AO, Symonds BD. Relation between working alliance and outcome in psychotherapy: A meta-analysis. Journal of Counseling Psychology. 1991;38:139–149. doi: 10.1037/0022-0167.38.2.139. [DOI] [Google Scholar]
  17. Joyce AS, Piper WE. Expectancy, the therapeutic alliance, and treatment outcome in short-term individual psychotherapy. Journal of Psychotherapy Practice and Research. 1998;7:236–248. [PMC free article] [PubMed] [Google Scholar]
  18. Knight DK, Broome KM, Simpson DD, Flynn PM. Program structure and counselor-client contact in outpatient substance abuse treatment. Health Services Research. 2008;43(2):616–634. doi: 10.1111/j.1475-6773.2007.00778.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Lambert D. Zero-inflated Poisson regression, with an application to defects in manufacturing. Technometrics. 1992;34:1–14. Retrieved from http://www.jstor.org/stable/1269547.
  20. Lehman WEK, Greener JM, Simpson DD. Assessing organizational readiness for change. Journal of Substance Abuse Treatment. 2002;22:197–209. doi: 10.1016/S0740-5472(02)00233-7. [DOI] [PubMed] [Google Scholar]
  21. Littell RC, Milliken GA, Stroup WW, Wolfinger RD, Schabenberger O. SAS System for Mixed Models. 2nd. Cary NC: SAS Institute Inc.; 2006. [Google Scholar]
  22. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical Psychology. 2000;68:438–450. doi: 10.1037/0022-006X.68.3.438. [DOI] [PubMed] [Google Scholar]
  23. McCarty D, Fuller BE, Kaskutas LA, Wendt B, Nunes EV, Miller M. Treatment programs in the National Drug Abuse Treatment Clinical Trials Network. Drug and Alcohol Dependence. 2008;92:200–207. doi: 10.1016/j.drugalcdep.2007.08.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. McLellan AT, Luborsky L, O'Brien CP, Woody GE. An improved diagnostic instrument for substance abuse patients, The Addiction Severity Index. Journal of Nervous and Mental Diseases. 1980;168:26–33. doi: 10.1097/00005053-198001000-00006. [DOI] [PubMed] [Google Scholar]
  25. Miller WR, Zweben A, DiClemente CC, Rychtarik RG. Motivational enhancement therapy manual: A clinical research guide for therapists treating individuals with alcohol abuse and dependence. Vol. 2. Rockville, MD: National Institute on Alcohol Abuse and Alcoholism; 1992. (NIAAA Project MATCH Monograph Series). DHHS Publication No. (ADM) 92-1894. [Google Scholar]
  26. Moos RH, Moos BS. The staff workplace and the quality and outcome of substance abuse treatment. Journal of Studies on Alcohol. 1998;59:43–51. doi: 10.15288/jsa.1998.59.43. [DOI] [PubMed] [Google Scholar]
  27. Muran JC, Segal ZV, Samstag LW, Crawford CE. Patient pretreatment interpersonal problems and therapeutic alliance in short-term cognitive therapy. Journal of Consulting and Clinical Psychology. 1994;62:185–190. doi: 10.1037/0022-006X.62.1.185. [DOI] [PubMed] [Google Scholar]
  28. Simpson DD. A conceptual framework for drug treatment process and outcomes. Journal of Substance Abuse Treatment. 2004;27:99–121. doi: 10.1016/j.jsat.2004.06.001. [DOI] [PubMed] [Google Scholar]
  29. Snijders T, Bosker R. Multilevel analysis: An introduction to basic and multilevel analysis. London: Sage; 1999. [DOI] [Google Scholar]
  30. Substance Abuse & Mental Health Services Administration National Survey of Substance Abuse Treatment Services (N-SSATS) 2008 Retrieved from: http://wwwdasis.samhsa.gov/webt/state_data/US08.pdf.
  31. Verbeke G, Molenberghs G. Linear Mixed models for Longitudinal Data. Springer-Verlag; New York: 2000. pp. 64–74. [Google Scholar]
  32. Whipple JL, Lambert MJ, Vermeersch DA, Smart DA, Nielsen S, Hawkins EJ. Improving the effects of psychotherapy: The use of early identification of treatment and problem-solving strategies in routine practice. Journal of Counseling Psychology. 2003;50:59–68. doi: 10.1037/0022-0167.50.1.59. [DOI] [Google Scholar]

RESOURCES