Organizational Readiness for Change and Opinions toward Treatment Innovations

Abstract

Program administrators and staff in treatment programs participating in the National Drug Abuse Treatment Clinical Trials Network (CTN) completed surveys to characterize participating programs and practitioners. A two-level random effects regression model assessed the influence of Organizational Readiness for Change (ORC) and organizational attributes on opinions toward the use of four evidence-based practices (manualized treatments, medication, integrated mental health services, and motivational incentives) and practices with less empirical support (confrontation and noncompliance discharge). The ORC Scales suggested greater support for evidence-based practices in programs where staff perceived more program need for improvement, better Internet access, higher levels of peer influence, more opportunities for professional growth, a stronger sense of organizational mission and more organizational stress. Support for confrontation and noncompliance discharge, in contrast, was strong when staff saw less opportunity for professional growth, weaker peer influence, less Internet access, and perceived less organizational stress. The analysis provides evidence of the ORC’s utility in assessing agency strengths and needs during the implementation of evidence-based practices.

1.0 Introduction

The Institute of Medicine’s Crossing the Quality Chasm series recommended increased use of treatments with empirical evidence of efficacy and effectiveness for health care (Institute of Medicine, 2000; Institute of Medicine, 2001) and for the treatment of alcohol, drug, and mental health disorders (Institute of Medicine, 2006). There is, however, a significant lag between science-based treatment innovations and the widespread adoption of those strategies in health care (Balas & Boren, 2000; Institute of Medicine, 2001). A comprehensive review of implementation research identified six core components that must be present to initiate and sustain the use of proven programs in new locations and environments: 1) select staff who can implement the program, 2) train staff, 3) require ongoing coaching, 4) use supervision and fidelity assessments to provide performance feedback to staff, 5) evaluate overall program functioning, and 6) facilitate implementation and sustainability with administrative supports (Fixsen et al., 2005). Many programs neglect one or more core implementation components and struggle to develop the skills required to effectively use evidence-based practices. Thus, it is critical to assess organizational strengths and weaknesses prior to the initiation of large scale implementation.

1.1 Organizational Readiness for Change

Simpson offers a process model of program change to promote the adoption of evidence-based practices (Simpson, 2002). The conceptual model includes four stages. Training exposes program staff to the new concepts or techniques. Next, the organizational leadership with input from counselors and other stakeholders makes a decision to adopt the technology and try the technique (Stage 2). Stage 3 is an implementation phase where the feasibility and practicality of the intervention is tested. If this stage is successful, the innovation is institutionalized (Stage 4) and becomes a facet of routine clinical practice. Variables that can facilitate or inhibit progress through each aspect of the program change are included in the model (Simpson, 2002).

Dimensions that influence organizational change are assessed using the Organizational Readiness for Change (ORC) Scales (Lehman et al., 2002). Eighteen domains are grouped into four categories of organizational readiness. Motivation for change is assessed on three dimensions (program need for improvement, perceived training needs, and pressure for change). Institutional resources are measured on five dimensions and reflect the adequacy of resources needed for daily activities and for supporting change: adequacy of office space, staffing, training resources, computer access, and use of email and internet. Four dimensions examine staff attributes that influence organizational change: potential for professional growth, efficacy (confidence in counseling skills), ability to influence co-workers, and adaptability. Finally, organizational climate is assessed on six facets: the organization’s clarity of mission and goals, staff cohesiveness (trust and cooperation among staff), staff autonomy (freedom in treatment planning and clinical work), management’s openness to communication from staff, perceived stress (strain and role overload) and openness to change.

The initial validation study suggested that the psychometric properties were strong and that the 18 dimensions were generally single factors (Lehman et al., 2002). Differences in responses to the instrument between directors and counselors were consistent with roles and responsibilities. Directors perceived greater immediate needs for training and had more access to computers and e-mail. Directors also reported more opportunity for personal growth, and greater perceived efficacy and influence. Directors rated management as more open to communication and perceived greater organizational willingness to change. Residential programs perceived higher training needs, had less adequate office space, and experienced greater stress when compared to outpatient services. Further, treatment units with low stability reported that resources (offices, staffing, training, and computer access) were less adequate. Management in less stable programs were perceived as less open to communication, stress was elevated, and staff were less open to change. Overall, the observed differences were consistent with expectations and supported the validity of the ORC.

1.2 National Drug Abuse Treatment Clinical Trials Network (CTN)

The National Drug Abuse Treatment Clinical Trials Network (CTN) is an alliance of 17 regional research centers and about 150 community-based organizations providing alcohol and drug treatment services. (The number of participating programs has increased over-time and expands as protocols require.) Investigators and practitioners collaborate to design and implement multi-site clinical trials and assess the effectiveness of behavioral, pharmacological and integrated therapies in real clinical settings with a full range of patient complexity. The Network was constructed to generate evidence that research-based innovations are effective and to promote adoption of science-based practices.

1.3 Evidence-Based Treatment

Treatment manuals facilitate the introduction of evidence-based practices. Manualized treatments standardize content and promote consistent implementation of proven practices. The manuals summarize intervention strategies, offer guidance on clinical techniques, and provide worksheets and structured exercises for clients. Opinions on the use of treatment manuals are often polarized. Opponents perceive manuals as inhibiting individualized treatment and restricting professional judgment and autonomy. Proponents, conversely, frame the manuals as treatment suggestions that facilitate efficacious treatment rather than prescriptions that must be followed without alteration (Institute of Medicine, 1992; Nathan, 1998).

CTN protocols, moreover, documented the value of treatments that incorporate medications (Amass et al., 2004; Ling et al., 2005) and use motivational incentives (contingency management) to reduce use of stimulants among methadone patients (Peirce et al., 2006) and to increase retention in care among stimulant users in outpatient services (Petry et al., 2005). The CTN Workforce survey, therefore, included items assessing opinions toward the use of evidence-based treatments including manualized treatment, medications, mental health services, and motivational incentives as well as support for the use of confrontation and discharge for non-compliance.

Early in implementation, the Clinical Trials Network recognized the importance of characterizing participating members. Chief executive officers, program directors, medical staff, support staff, supervisors and counselors working within the CTN completed organizational, treatment unit, and workforce surveys in order to describe the treatment programs and practitioners participating in the CTN. Analysis of the CTN data may add support to the validity of the ORC, assess its influence on workforce openness to empirical interventions, and further elucidate its value in organizational assessments. Staff opinions toward the use of evidence-based practices and support for the use of confrontation and discharge for noncompliance are examined in this analysis of organizational readiness for change.

2.0 Methods

2.1 CTN Surveys

Three inter-related surveys (Organizational, Treatment Unit, and Workforce) were completed sequentially within the treatment programs participating in the CTN. Complete details on the data collection methods for the Organizational, Treatment Unit and Workforce Surveys are presented elsewhere (McCarty et al., 2006a). A protocol coordinator helped each of the regional networks to distribute materials to the treatment programs, facilitate participation, and monitor response rates.

Surveys were either completed on-line or via paper surveys. The Organizational Survey included information on ownership status (e.g., not-for-profit, for-profit, government), primary service setting (e.g., health care, mental health, free-standing addictions treatment program) and corporate size (e.g., annual revenues, employees, facilities). Respondents listed distinct treatment units and provided contact information. The response rate was 95%. Treatment Unit Surveys provided information on the levels of care and services, facility accreditation and licensure, patient characteristics, sources of revenue, staffing, staff retention, and scales to assess program environment and philosophy. The response rate was 91%. Employees with direct care responsibilities were listed and categorized as “counseling staff,” “medical staff,” “other staff with patient contact (including reception and aides)” and “management staff and clinical supervisors.”

The Workforce Survey was distributed in sealed envelopes addressed to the identified employees in each treatment unit. Distribution methods varied. Most often surveys were distributed during staff meetings and completed at that time. Completed surveys were either mailed directly to the study’s data management center in business reply envelopes or sealed envelopes were handed to the protocol coordinator and returned in batch using overnight delivery services. Survey items assessed years of experience, education, training, credentials, and opinions toward the use of specific alcohol and drug treatments. The last section of the survey was the 115 item ORC Scale (Lehman et al., 2002). The workforce survey had a 71% response rate.

2.2 Human Subject Protections

The Oregon Health & Science University Institutional Review Board (IRB) reviewed and approved study procedures. Forty local IRBs also reviewed and approved the protocol. The first two pages of the surveys were an information sheet that outlined the study’s purpose and noted that respondents could decline to participate.

2.3 Data analysis

Opinion items were factor analyzed to form factor scores and hierarchical linear modeling assessed the influence of organizational characteristics and ORC dimensions on the factor scores.

2.3.1 Factor Scores

Opinion items were measured on a 5-point Likert scale (strongly disagree to strongly agree). A maximum likelihood principal components factor analysis rotated with an orthogonal (Varimax) solution generated five factors. Six items that failed to load on a specific factor and one item that loaded onto two factors were eliminated. Items and factor loadings are reported in Table 1. Five items formed Factor 1: "Evidence-Based Practices and Manualized Treatments" and accounted for 21% of the variance with an internal consistency estimate of 0.62. Factor 2: "Confrontation and Noncompliance Discharge" contained three items accounting for 14% of the variance with an internal consistency estimate of 0.52. Three items formed Factor 3: "Medications" accounted for 20% of the variance with a coefficient alpha of 0.61. Factor 4: "Mental Health Services" consisted of three items, explained 24% of the variance with an alpha of 0.75. Factor 5: "Motivational Incentives” also had three items, accounted for 21% of the variance with an alpha of 0.72. It is not uncommon for three item scales to have coefficients alpha lower than 0.80. When the alpha is below 0.80, it is best to use factor scores rather than unit scaling to generate a summated score (Nunnally & Bernstein, 1994). Factor scores were generated by weighting items by their factor loadings and calculating their sum.

Table 1.

Factor loadings and items for the five factor solution.

Item	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5
	Treatment manuals and evidence-based treatment	Confrontation and noncompliance discharges	Medications	Integrated mental health services	Motivational incentives
Evidence-based practices are useful to improve quality of care	.54	−.11	.11	.16	.08
Treatment manuals are useful tools for learning new interventions	.52	0	2	.09	0
Scientifically supported treatments can be useful	.49	−.11	.21	.19	.21
Evidence-based practices guidelines promote over-simplified "cookbook" care	−.35	.15	.01	−.01	−.07
Treatment manuals Interfere with treatment	−.55	.11	.01	−.06	0
Clients who continue to abuse substances are not committed to treatment	−.08	.53	−.01	−.05	0
Non-compliant patients should be discharged.	−.11	.51	−.06	−.08	−.10
Confrontational approaches should be used more in addiction treatment.	−.13	.48	−.07	−.06	−.04
Naltrexone should be used more in the treatment of alcohol dependence	.08	−.05	.64	.06	.01
Buprenorphine is an effective treatment for opiate dependence	.18	−.18	.56	.06	.03
Methadone maintenance should be used more to treat heroin dependence	.05	−.11	51	.10	.15
Addiction treatment services should routinely provide mental health services	.13	−.06	.03	.73	.05
Addiction treatment programs should provide pharmacotherapy for psychiatric disorders	.14	−.13	.13	.70	.06
Substance abuse patients should be assessed for psychiatric disorders routinely	.21	−.09	.13	.63	.06
It’s okay for patients to have the opportunity to earn prizes worth as much as $100 for abstinence	.08	−.06	.17	.07	.88
It is okay to pay patients for attending treatment	.05	−.17	.18	.07	.63
Incentives can have a positive effect on the patient/counselor relationship	.18	.02	.13	.08	.49

2.3.2 Multilevel modeling

Each factor score was modeled as the outcome variable in a 2-level random-effects regression model (Hierarchical Linear Modeling) (Fitzmaurice et al., 2004). The level 1 predictors were the ORC scales centered on the group mean for each treatment unit in order to model within group variation. This centering decision has no impact on the significance of the results but only the interpretation of the coefficients. In this solution, each individual is measured against the mean of that score for their treatment unit. Eight organizational variables were specified as level 2 predictors: 1) free standing treatment centers, 2) for profit status, 3) annual revenue (categorized as small, medium and large), 4) national accreditation, 5) offering residential treatment services, 6) offering outpatient/intensive outpatient treatment, 7) offering methadone and 8) offering detoxification services. The mixed model controlled variance within treatment units while calculating appropriate standard errors for each statistical test. A restricted maximum likelihood estimation technique estimated parameters. Robust standard errors were used because a large number treatment unit was available for analysis. The general form of the random effects mixed model is:

$${\mathrm{Y}}_{\mathrm{i}\mathrm{j}}={\mathrm{\beta }}_{0\mathrm{j}}+{\mathrm{\beta }}_{1\mathrm{j}}({\mathrm{X}}_{\mathrm{i}\mathrm{j}}-{\mathrm{X}}_{\mathrm{j}})+{\mathrm{r}}_{\mathrm{i}\mathrm{j}}$$ Level 1

$$\begin{array}{c}{\mathrm{\beta }}_{0\mathrm{j}}={\mathrm{\gamma }}_{00}+{\mathrm{\gamma }}_{01}\left(\mathrm{Z}\mathrm{j}\right)+{\mathrm{u}}_{0\mathrm{j}}\\ {\mathrm{\beta }}_{1\mathrm{j}}={\mathrm{\gamma }}_{10}+{\mathrm{\gamma }}_{11}\left(\mathrm{Z}\mathrm{j}\right)+{\mathrm{u}}_{\mathrm{i}\mathrm{j}}\end{array}$$ Level 2

where Y_ij was the dependent variable for subject i in group j, β_0j was the intercept (in these models the intercept is usually group centered so the intercept is the value of Y when X is at its mean value), (X_ij − X_j) was the group mean-centered predictor, β_1j was the slope of the relationship between Y and (X_ij − X_j), Z_j was the level 2 predictor for each group, and r_ij was the residual variance at level 1 whereas u_0j and u_1j were the residuals among the level 2 effects. In the level 2 model the intercept from level 1 β_0j was predicted by γ₀₀ (the grand mean of the predictor across all groups), γ₀₁ (the slope of the relationship between the mean predictor and the intercept) and u_0j (the random unexplained variance). The parameter γ₀₁ was the intercept of the slopes for the level 2 variables and Z_J was a predictor of between unit variance.For this paper, the models run were of the following form:

$${\text{Attitude}}_{\mathrm{i}\mathrm{j}}={\mathrm{\beta }}_{0\mathrm{j}}+{\mathrm{\beta }}_{1\mathrm{j}}({\text{ORC}1}_{\mathrm{i}\mathrm{j}}-{\text{ORC}1}_{\mathrm{j}})+{\mathrm{\beta }}_{2\mathrm{j}}({\text{ORC}2}_{\mathrm{i}\mathrm{j}}-{\text{ORC}2}_{\mathrm{j}})+\dots .+{\mathrm{r}}_{\mathrm{i}\mathrm{j}}$$ Level 1

$$\begin{array}{c}{\mathrm{\beta }}_{0\mathrm{j}}={\mathrm{\gamma }}_{00}+{\mathrm{\gamma }}_{01}\left(\mathrm{Org}1\right)+{\mathrm{\gamma }}_{02}\left(\mathrm{Org}2\right)+\dots +{\mathrm{u}}_{0\mathrm{j}}\\ {\mathrm{\beta }}_{1\mathrm{j}}={\mathrm{\gamma }}_{10}+{\mathrm{\gamma }}_{01}\left(\mathrm{Org}1\right)+{\mathrm{\gamma }}_{02}\left(\mathrm{Org}2\right)+\dots +{\mathrm{u}}_{\mathrm{i}\mathrm{j}}\end{array}$$ Level 2

where attitude toward manualized treatment and attitude toward traditional counseling were predicted by an intercept β_0j plus the ORC scale (ORC1) of employee i in treatment unit j minus the mean of that ORC scale for treatment unit j (ORC1_ij − ORC1_j) plus an error term (r_ij) that modeled the variance not predicted in scores on the dependent variables. This model identifies relationships between the outcome of interest and the set of centered predictors in a regression model controlling for staff clustering within treatment units.

The level 2 model predicted the variances of the means of factor scores between treatment units. The predictors were agency-level variables measured in the organizational and treatment unit surveys. Therefore each mixed model had 18 group-centered ORC scales (within variance) and eight organizational predictors of the differences between treatment units (between variance). Effects were evaluated at alpha equal to or less than .05.

Intercept-only models were used to calculate intraclass correlations (ICC) for each of the five factors. Although the first ICC for manualized treatment is lower than the commonly accepted level at which data is aggregated to the unit level (ρ = .066), a multilevel approach was used because of the clear distinction between employee-level and agency-level predictors. All of the other ICC values were above .10. Compound symmetry was used to model the covariance matrix for the predictors as this structure provided the best model fit. The degrees of freedom were calculated with the between-within specification. All models were run in SAS using the PROC MIXED module (SAS Institute, 1999).

3.0 Results

3.1 Participants

The analysis was restricted to treatment units with at least five staff members responding to the Workforce Survey because the models used group mean centering. Use of small numbers of observations per group makes mean centering either impossible (in the case of n = 1) or highly erratic and inhibits solution convergence. The 249 participating treatment units included outpatient treatment programs (n = 79), residential and inpatient services (n = 95), detoxification programs (n = 34) and) methadone programs (n = 41). These agencies reported a mean of 977 (± 5462) admissions per year. Two-thirds (63.6%) of the treatment units claimed national accreditation. The typical counselor caseload was 28.5 (± 53.4) patients. Two-thirds of the respondents to the workforce survey were women (65%). One-third (35%) of the workforce reported a masters or doctoral degree and 67% reported certification (36%) or licensure (45%). The number of cases with complete data on the level 1 ORC scales and the outcome variables of attitude factor scores varied slightly with each regression model: use of treatment manuals (n = 2,142), medications (n = 2,136), integrated mental health services (n = 2,234), motivational incentives (n = 2,233), and use of confrontation and noncompliance discharge (n = 2,207). The five models contained 205 treatment units with complete data for the level 2 models.

3.2 Level 1 Influences

Staff responses to the ORC scale were used as level 1 variables in the analysis. Mean scale scores map the 50^th percentile of scale scores as presented in the introductory paper in this issue (Simpson & Flynn, this issue): program needs (mean = 31.5, ±8.2), training needs (mean = 30.3, ±8.2), pressure for change (mean = 32.3, ±6.1), offices (mean = 33.6, ±8.1), staffing (mean = 30.6, ±7.,0), training resources (mean = 32.9, ±8.0), computer equipment (mean = 28.6, ±6.9), internet access (mean = 29.6, ±10.0), growth (mean = 34.8, ±6.3), efficacy (mean = 38.7, ±5.1), influence (mean = 35.2, ±6.3), adaptability (mean = 38.4, ±5.0), mission (mean = 34.9, ±6.6), cohesion (mean = 33.8, ±8.4), autonomy (mean = 33.8, ±5.8), communication (mean = 32.1, ±7.5), stress (mean = 32.5, ±8.4), and openness to change (mean = 33.1, ±6.3). The similarity with the national norms suggests that the CTN as a group is similar to other programs that have completed the ORC.

Parameter estimates and standard errors are tabled for each factor score: manualized treatments (Table 2), medications (Table 3), mental health services (Table 4), motivational incentives (Table 5), confrontation and non-compliance discharge (Table 6). Significant parameters are set in bold typeface. The four primary dimensions of the ORC were examined for each factor score and were used to structure the presentation of results.

Table 2.

Level 1 and Level 2 influences on opinions toward treatment manuals

Effect ICC = .066	Parameter	Standard Error	t value
Level 1 (df = 1919)
ORC Motivation/Needs
Program needs	.1031	.0315	3.27 ***
Training needs	.0184	.0307	0.45
Pressure for change	−.0034	.0267	−0.13
ORC Institutional Resources
Offices	−.0140	.0304	−0.46
Staffing	.0240	.0335	0.72
Training resources	.0460	.0329	1.40
Computer equipment	−.0752	.0307	−2.45 *
Internet	.0834	.0290	2.87 **
ORC Staff Attributes
Growth	.0500	.0343	1.46
Efficacy	.0285	.0311	0.92
Influence	.0981	.0317	3.09 ***
Adaptability	.0551	.0279	1.97 *
ORC Organizational Climate
Mission	.1365	.0354	3.86 ***
Cohesion	.0572	.0344	1.66
Autonomy	.0309	.0317	0.97
Communication	−.0477	.0388	−1.23
Stress	.0441	.0334	1.32
Openness to change	.0025	.0361	0.07
Level 2 (df = 196)
Residential services	.0525	.0536	0.98
Outpatient services	.2061	.0547	3.77 ***
Methadone services	.1139	.0597	1.91
Detox services	−.1273	.0525	−2.43 *
Free-standing program	.0177	.0492	0.36
National accreditation	.0338	.0502	0.67
For-profit	.0029	.1016	0.03
Revenue	.0525	.0536	0.98

^*p < .05

^**p < .01

^***p < .005

ICC = Interclass Correlation

Table 3.

Level 1 and Level 2 influences on opinions toward medications

Effect ICC = .228	Parameter	Standard Error	t value
Level 1 (df = 1913)
ORC Motivation/Needs
Program needs	.0720	.0307	2.35 **
Training needs	.0512	.0298	1.71
Pressure for change	.0167	.0258	0.06
ORC Institutional Resources
Offices	.0016	.0294	0.06
Staffing	−.0127	.0325	−0.39
Training resources	.0135	.0321	0.42
Computer equipment	−.0035	.0297	−0.12
Internet	.0865	.0283	3.05 ***
ORC Staff Attributes
Growth	.0890	.0335	2.66 **
Efficacy	.0114	.0304	0.37
Influence	.0372	.0309	1.21
Adaptability	−.0179	.0270	−0.66
ORC Organizational Climate
Mission	−.0116	.0345	−0.34
Cohesion	.0436	.0334	1.30
Autonomy	.0030	.0309	0.10
Communication	.0336	.0381	0.88
Stress	.0026	.0328	0.08
Openness to change	.0033	.0352	0.09
Level 2 (df = 196)
Residential services	−.2474	.0552	−4.48 ***
Outpatient services	.1084	.0564	1.92
Methadone services	.8394	.0619	13.55 ** *
Detox services	.0105	.0540	0.19
Free-standing program	−.0747	.0506	−1.47
National accreditation	.0957	.0517	1.85
For-profit	−.0159	.1016	−1.57
Revenue	−.0130	.0350	−0.37

^*p < .05

^**p < .01

^***p < .005

ICC = Interclass Correlation

Table 4.

Level One and Level Two Influences on Opinions toward Integrated Mental Health Services

Effect ICC = .188	Parameter	Standard Error	t value
Level 1 (df = 2011)
ORC Motivation/Needs
Program needs	.0281	.0373	0.75
Training needs	.0219	.0364	0.60
Pressure for change	.0199	.0315	0.63
ORC Institutional Resources
Offices	−.0075	.0363	−0.21
Staffing	−.0154	.0398	−0.39
Training resources	−.0347	.0392	−0.89
Computer equipment	−.0532	.0365	−1.46
Internet	.0100	.0347	0.29
ORC Staff Attributes
Growth	.0728	.0408	1.78
Efficacy	.0390	.0370	1.05
Influence	.1503	.0378	3.98 ***
Adaptability	.0734	.0331	2.22 *
ORC Organizational Climate
Mission	.0837	.0425	1.97 *
Cohesion	.1174	.0410	2.86 **
Autonomy	.0435	.0380	1.15
Communication	−.1298	.0464	−2.80 **
Stress	.1753	.0398	4.40 ***
Openness to change	.0165	.0430	0.38
Level 2 (df = 196)
Residential services	.0056	.0662	0.08
Outpatient services	−.0115	.0675	−0.17
Methadone services	.3324	.0736	4.52 ***
Detox services	−.3044	.0648	−4.70 ***
Free-standing program	−.1610	.0608	−2.65 **
National accreditation	.1311	.0628	2.11 *
For-profit	−1764	.1245	−1.42
Revenue	.0600	.0420	1.43

^*p < .05

^**p < .01

^***p < .005

ICC = Interclass Correlation

Table 5.

Level 1 and Level 2 Influences on Opinions toward Motivational Incentives

Effect ICC = .149	Parameter	Standard Error	t value
Level 1 (df = 2010)
ORC Motivation/Needs
Program needs	.1185	.0482	2.46 *
Training needs	.0324	.0469	0.69
Pressure for change	.0651	.0406	1.60
ORC Institutional Resources
Offices	−.0608	.0466	−1.30
Staffing	.0199	.0512	0.39
Training resources	−.0201	.0505	−0.04
Computer equipment	−.0387	.0470	−0.82
Internet	.2241	.0446	5.03 ***
ORC Staff Attributes
Growth	.0604	.0526	1.15
Efficacy	−.0770	.0475	−1.62
Influence	.1367	.0486	2.81 **
Adaptability	−.0526	.0427	−1.23
ORC Organizational Climate
Mission	.0295	.0545	0.05
Cohesion	.0631	.0529	1.19
Autonomy	−.1051	.0488	−2.15 *
Communication	−.0061	.0600	−0.10
Stress	−.0281	.0512	−0.55
Openness to change	.1072	.0555	1.93
Level 2 (df = 196)
Residential services	.0131	.0852	0.15
Outpatient services	.2610	.0868	3.01 ***
Methadone services	.5338	.0947	5.64 ***
Detox services	−.3405	.0833	−4.09***
Free-standing Program	−.0528	.0781	−0.68
National accreditation	−.2409	.0799	−3.02 **
For-profit	−.5211	.1604	−3.25 ***
Revenue	−.2730	.0539	−5.07 ***

^*p < .05

^**p < .01

^***p < .005

ICC = Interclass Correlation

Table 6.

Level 1 and Level 2 Influences on Use of Confrontation and Noncompliance Discharge

Effect ICC = .115	Parameter	Standard Error	t value
Level 1 (df = 1984)
ORC Motivation/Needs
Program needs	−.0061	.0325	−0.19
Training needs	.0091	.0316	0.29
Pressure for change	.0408	.0272	1.50
ORC Institutional Resources
Offices	.0912	.0314	2.91 ***
Staffing	−.0428	.0344	−1.24
Training resources	.0086	.0341	−0.25
Computer equipment	.0625	.0314	1.99 *
Internet	−.1051	.0299	−3.51 ***
ORC Staff Attributes
Growth	−.1285	.0355	−3.62 ***
Efficacy	−.0610	.0320	−1.91
Influence	−.0833	.0325	−2.56 *
Adaptability	.0095	.0286	0.33
ORC Organizational Climate
Mission	−.0073	.0370	−0.20
Cohesion	−.0528	.0356	−1.48
Autonomy	−.0231	.0328	−0.70
Communication	.0042	.0401	0.11
Stress	−.0783	.0347	−2.26 *
Openness to change	.0450	.0374	1.20
Level 2 (df = 196)
Residential services	.1781	.0584	3.05 **
Outpatient services	−.1191	.0595	−2.00 *
Methadone services	.0882	.0651	−0.35
Detox services	.0882	.0570	1.55
Free-standing program	−.0115	.0535	−0.22
National Accreditation	−.0407	.0549	−0.74
For-profit	−.0677	.1094	−0.62
Revenue	.0070	.0370	0.19

^*p < .05

^**p < .01

^***p < .005

ICC = Interclass Correlation

ORC Motivation for Change

Perceived need for program improvement is a key facet of motivation for change. Programs with staff that saw more opportunities for program improvement tended to have significantly greater support for three of the four factors reflecting evidence-based treatments: manualized treatments, medications, and motivational incentives. The remaining motivation for change dimensions did not have a significant influence on the factor scores.

ORC Institutional Resources

Staff perceptions of more adequate Internet access (e.g., e-mail at work, seeking information, participation in list-serves) significantly influenced greater support toward the use of manualized treatments, medications, and motivational incentives. Conversely, support for using noncompliance discharge and confrontation was strongest in agencies where staff rated Internet access as less adequate and office space was rated as more adequate. Computer access, however, had a negative relationship with manualized treatments and a positive relationship with use of confrontation and noncompliance discharge.

ORC Staff Attributes

Opinions toward treatment manuals, mental health services, and motivation incentives were more positive in treatment units where respondents perceived more willingness and ability to influence peers. Programs where staff rated themselves as more adaptable were more positive toward the use of treatment manuals and the integration of mental health services. Programs where staff saw more opportunity for professional growth, moreover, were more open to the use of medications. Conversely, support for noncompliance discharges and the use of confrontation was stronger when respondents perceived less opportunity for professional growth and less influence with their peers.

ORC Organizational Climate

A stronger sense of agency mission was associated with more support for the use of treatment manuals and mental health services. Support for integrated mental health services was more apparent in organizations with higher ratings on measures of organizational stress, and organizational cohesiveness, and lower ratings on openness of communication. Ratings of more staff autonomy were associated with less support for the use of motivational incentives. Finally, programs where staff perceived less organizational stress expressed more support for the use of confrontation and discharge for noncompliance.

3.3 Level 2 Influences

The analysis included eight organizational variables as level 2 predictors and controlled for intraclass correlations among counselors within each treatment unit. The level 2 variables characterized the programs on dimensions typically used to classify addiction treatment agencies: type of services (detoxification, residential, outpatient, and methadone), corporate size (operationalized as annual revenues and coded small, medium, large), national accreditation, free-standing addiction treatment services, and for-profit status.

Residential programs were more supportive of the use of confrontation and noncompliance discharge and less supportive of medications. Outpatient programs, on the other hand, were less supportive of confrontation and noncompliance discharges and more supportive of using treatment manuals and motivational incentives. Detoxification programs had less support for treatment manuals and integrated mental health services. Methadone services were more likely to support medications, integrated mental health services, and motivational incentives. Programs with national accreditation (e.g., JCAHO and CARF) were more likely to support integrated mental health services but less likely to support the use of motivational incentives. Free-standing alcohol and drug treatment centers were less supportive of integrated mental health services. Finally, for-profit corporations and programs that reported more revenues were less likely to support the use of motivational incentives.

4.0 Discussion

The CTN provided a large and heterogeneous collection of programs and staff for assessing the influence of ORC scales and organizational attributes on opinions that may either support or inhibit adoption of evidence-based practices. Within the four ORC domains one or two dimensions seemed to have more influence on the support for evidence-based practices. Perceived need for improvement was a key facet of motivation for change. Programs with higher staff scores on program need for improvement were more supportive of treatment manuals, medications, and motivational incentives. Internet access appears to be a critical institutional resource. Agencies where staff rated Internet access as more adequate were more supportive of treatment manuals, medications, and motivational incentives and less supportive of using confrontation and noncompliance discharge. ORC dimensions assessing staff attributes found that staff who rate themselves as more influential with peers had stronger support for treatment manuals, integrated mental health services, and motivational incentives. Conversely, lower ratings on peer influence were found in programs where there was more support for confrontation and noncompliance discharge. Perceived professional growth opportunities had a positive effect on willingness to use medications and a negative influence on the use of confrontation and noncompliance discharge. Finally, measures of organizational climate were also influential. Ratings of perceived stress were lower when there was support for using confrontation and noncompliance discharges. It appears that staff were comfortable with the status quo. When perceived stress was higher, there was more support for integrated mental health services (presumably reflecting pressures for change and the stress of integrating services). A strong organizational sense of mission also was associated with more support for the use of treatment manuals and integrated mental health services.

These differences are intriguing and suggest that the likelihood of organizational change and the adoption of new treatment technology may be greater when needs for improvement are apparent and the treatment organizations promote professional growth and have a clear sense of mission. The ORC organizational climate dimensions appear to be especially influential on the integration of mental health services with addiction treatment. Staff attributes seem to have direct positive and negative effects on support for evidence-based practices. Counselors who were more supportive of confrontation and noncompliance discharge may accurately recognize they lack skills, are less likely to influence peers and have less autonomy. They seem to realize that in a quickly changing environment they are ‘old school’.

Better internet access had a strong influence on support for the use of treatment manuals, medications, and motivational incentives and a negative relationship with the use of confrontation and noncompliance discharges. The Internet is a technology innovation that offers immediate access to training and implementation resources from federal agencies, research centers, and entrepreneurial enterprises. Internet users may be more willing to innovate.

Curiously, the analysis suggested that increased computer access had a weak but statistically significant negative influence on support for treatment manuals. Overall, the zero-order correlation between Internet Access and computer access was relatively strong (r = 0.46, p <.0001). The correlation between manualized treatment scores and computer access was not significant (r = .01) while the correlation with Internet access was significant (r = .12, p < .0001). The strong zero-order correlation between Internet access and computer access, and the strong influence of Internet access on support for treatment manuals seems to create a statistical suppression effect - a negative effect for computer access in the regression models when the univariate effect is not negative. The negative weight for the computer access toward treatment manuals is probably best ignored rather than interpreted.

Computer access also had a positive influence on support for confrontation and noncompliance discharge and Internet access had a negative influence. In this case, the univariate correlations (Internet: r = −.07; p < .0001; Computers: r = .04, p < .05) are consistent with the model results. One explanation is that some treatment facilities have computers for record keeping and word processing but may not provide staff with Internet access. Counselors in these treatment environments may have less access to emerging evidence-based practices.

Differences between residential and outpatient services were apparent and reflect the differences reported in the ORC validation study (Lehman et al., 2002). Outpatient services were more supportive of the use of treatment manuals and motivational incentives and less supportive of the use of confrontation and noncompliance discharge. Residential services were more supportive of using confrontation and noncompliance discharge and less responsive to medications. As might be expected, methadone services were more supportive of medications and integrated mental health services. National accreditation also seemed to be influential. Accredited programs were more supportive of integrated mental health services and less supportive of motivational incentives. For profit corporations and corporations with more revenues were also less supportive of motivational incentives.

4.1 Limitations

The workforce survey relies on self-report data. Self-report measures can be unreliable or subject to response and recall bias. Additionally, this analysis includes only those with complete data; participants with missing variables were eliminated creating a possible bias toward those who completed the survey. Individuals who declined to participate are also not included. The CTN is also limited to larger programs and findings may not generalize to smaller free-standing programs.

The three item factors reflect a need to minimize the number of items included in the Workforce Survey and, unfortunately, result in scales with relatively low coefficient alphas. The three item factors, therefore, are likely to have weak reliability and should be used carefully. The survey items, however, were derived from and are consistent with prior work (Forman et al., 2001). Overall impressions, moreover, suggest consistent differences in the influence of ORC dimensions on evidence-based practices when compared to use of confrontation and noncompliance discharge.

4.2 Conclusions and Observations

Many of the nation’s more than 13,000 specialty clinics for the treatment of alcohol and drug disorders are struggling to respond to increased demands for economic efficiency and evidence of treatment effectiveness (McLellan et al., 2003). Long-term survival requires unleashing organizational capacities to respond to external and internal pressures for change. The analysis of ORC dimensions within the CTN suggests that agency leaders should systematically involve their staff in the change process. There was more support for the use of treatment innovations when staff perceived that they influenced peers and conversely, more support for use of confrontation and noncompliance discharges when staff perceptions of professional growth opportunity and peer influence were low.

A few resources are available to guide organizational change efforts and develop capacities for change. The Addiction Technology Transfer Centers (ATTC) produced The Change Book to help treatment centers facilitate the adoption and use of new treatment technologies (Addiction Technology Transfer Centers, 2000;Addiction Technology Transfer Centers, 2004). A ten-step process structures planning and promotes change: 1) identify the problem, 2) organize a team to address the problem, 3) identify desired outcomes, 4) assess the organization, 5) assess the target audience, 6) identify the approach most likely to work, 7) design action and maintenance plans, 8) implement action and maintenance plans, 9) evaluate the change initiative, 10) revise action and maintenance plans. Change agents lead a group through structured exercises that address each facet and, in theory, emerge with changes that are institutionalized. This approach was used in rural communities to promote the adoption of buprenorphine for the treatment of opioid dependence (McCarty et al., 2004).

The Network for the Improvement of Addiction Treatment (NIATx) is another resource; NIATx teaches treatment programs to foster change through the use of process improvement strategies (www.niatx.net). The Chief Executive Officer empowers a change leader and a change team to address specific improvement needs. The team identifies a potential improvement strategy and uses rapid cycle testing to try out changes. The first test is with one patient. If the change works, the test is spread to more individuals. Modifications are made as problems are identified or as improvements plateau. Changes are scaled up to full implementation if test results continue to document improved organizational functioning. The process is most effective when the change addresses key problems, the change leader is strong, customers are involved, the team looks outside the corporation for improvement ideas, and the team uses rapid cycles. This approach draws heavily on strategies used to reduce errors in manufacturing (Imai, 1986; Juran, 1988); these techniques have also been applied to health care (Institute of Medicine, 2000; Institute of Medicine, 2001) and have been extended to behavioral health services (Institute of Medicine, 2006) and to a host of other service environments (Barney & McCarty, 2003). NIATx members made dramatic reductions in days to treatment entry; increased admissions, improved retention in care, and reductions in no-show rates were also targeted for improvements (McCarty et al., 2006b). Participating agencies learn that action is important and that small changes can build to larger changes.

Our analysis of organizational and workforce variables adds to the validation of the ORC. Differences support the notion that the ORC measures important dimensions of organizational readiness for change. Subsequent analyses should examine the influence of ORC dimensions on patient outcomes in clinical trials and program performance in clinical trials. For example, programs with weaker communication ratings and poor sense of mission may have more difficulty recruiting study participants and implementing clinical trial protocols. Motivation for change may also influence clinic performance. Programs with strong leadership and more willingness to change may be more successful implementing trials, recruiting patients, and conducting follow-up interviews.