Program Structure, Staff Perceptions, and Client Engagement in Treatment

Abstract

A key goal for drug abuse treatment providers is to get their clients engaged and participating in therapeutic activities, as a first step toward deriving longer-term benefits. Much research has focused on personal characteristics that relate to client engagement, but characteristics of the program have received less attention. This study explores client and program differences in engagement ratings, using data from a nationwide set of 94 outpatient drug free (ODF) treatment programs in a hierarchical linear model (HLM) analysis. Results show that elements of program context, including both structural features (e.g., smaller size and JCAHO/CARF accreditation) and staff perceptions of personal efficacy, organizational climate, and communal workplace practices, relate to better overall client engagement. These findings add further evidence that treatment providers should also address the workplace environment for staff as part of quality-improvement efforts.

1. Introduction

As consensus grows that drug abuse treatment is effective (Hubbard et al., 1989; Hubbard, Craddock, Flynn Anderson, & Etheridge, 1997; Prendergast, Podus, Chang, & Urada, 2002; Simpson, Joe, Fletcher, Hubbard, & Anglin, 1999), researchers, practitioners, and policymakers alike focus increasingly on the interim goals and components of treatment (Prendergast et al., 2002; Simpson 2004). Specifically, clients “engaging,” or actively participating in treatment is a critical step leading to better outcomes (Barber et al., 2001; Simpson, 2004). In recognition of the clinical importance of early engagement, it has been recommended as a program- and system-level performance measure (Garnick et al., 2002). Although the importance of engagement is generally well understood, the factors that contribute to it have been less consistently studied.

A number of engagement studies have focused on clients and characterizing their treatment experience. Client engagement has been operationalized as session attendance (Garnick et al., 2002; Simpson, Joe, Rowan-Szal, & Greener, 1997), or through a broader set of personal reactions, including building rapport or “therapeutic alliance” with a counselor (Barber et al., 2001), openness and participation within sessions (Simpson, 2004), and general helpfulness or satisfaction with treatment (Mavis & Stöffelmayr, 1994; Siqueland et al., 2004). These studies have explored a variety of pretreatment characteristics (e.g., Joe, Simpson, & Broome, 1999; Meier, Donmall, Barrowclough, McElduff, & Heller, 2005; Simpson, Joe, Rowan-Szal, et al., 1997) or treatment activities and services (e.g., Joe et al., 1999; Moos & King, 1997; Siqueland et al., 2004) that relate to engagement. The hope underlying such an approach is that understanding the personal characteristics of clients who do or do not engage will offer a way to identify at-risk individuals, and help to target resources in ways that will improve engagement in the future.

Until recently, however, little attention has been given to differences in engagement levels among programs and how program organization might contribute to the client’s subjective experience. In part, research on program factors has lagged because of the challenges inherent in obtaining reasonably large samples of treatment programs. Nevertheless, several multisite studies have examined variations in aspects of client engagement in Veterans Administration substance abuse programs (Moos & Moos, 1998), alcohol treatment programs (Mavis & Stöffelmayr, 1994), and psychiatric facilities (e.g., Holland, Konick, Buffum, Smith, & Petchers, 1981), which were related to dimensions of the staff working environment. For example, drug abuse clients were most satisfied in programs where staff reported having a supportive and task-focused work environment. Building upon these findings, two recent reports (Broome, Simpson, & Joe, 1999; Joe, Broome, Rowan-Szal, & Simpson, 2002) considered larger samples from a range of treatment settings and showed modest but important variations in several client-rated engagement measures. Using data from the national Drug Abuse Treatment Outcome Studies (DATOS), Broome et al. (1999) found that programs sharing the same basic treatment modality (Long-Term Residential, Outpatient Drug-Free, or Outpatient Methadone Treatment) differed substantially in their clients’ ratings of the helpfulness of treatment. These differences were associated with the specific clientele served, but also with the availability of supplemental services. Joe et al. (2002) found similar variations across programs in clients’ rapport with their counselor, and these were also related to clientele served. However, in a companion study, based on a closely-related sample, Lehman, Greener, and Simpson (2002) identified significant relationships between average rapport (and other engagement measures) and aggregate program staff ratings of the working environment.

Taken together, this research suggests consistent program differences in client engagement that reflect differences in clientele, therapeutic approach, and staff workplace perceptions. Integrating these streams of research, and identifying structural and normative features of treatment programs that relate to client engagement would contribute to efforts for improving treatment effectiveness.

1.1. Conceptual background

A central theme of the present research is that a treatment program is an organization, and the way treatment is structured and managed, as well as the social norms that develop, impact the client and staff participants. Counselors are key actors in the process (especially in drug-free treatments) because, on the one hand, they are the ones delivering services and interacting with clients, and, on the other hand, they function as part of a formal unit. Consequently, relevant organizational elements can extend well beyond clinical practices to include description of the program as a workplace (Milne, Blum, & Roman, 2000; Moos & Moos, 1998). Of particular interest are program features that may be indicative of greater internal differentiation and separation among participants, versus a more integrated and communal approach.

Many efforts at assessing the impact of program structure have involved classifying programs according to services offered and clinical policies (e.g., Ball & Ross, 1991; James, Hammond, Hartman, & Sells, 1976; Timko, 1995) or theoretical orientation (Cole & Watterson, 1976; Swindle, Peterson, Paradise, & Moos, 1995). While these general features do relate to both client retention and participation (Joe, Simpson, & Hubbard, 1991; Moos, King, Burnett, & Andrassy, 1997), a trend among outpatient treatment providers has been to provide varying intensities of services (Gerstein & Harwood, 1990; Timko, Sempel, & Moos, 2003), even within the same program. For example, 46% of outpatient drug-free (ODF) programs in the DATOS project indicated they offered both “regular” (less than 6 hours of contact per week) and “intensive” (6 or more hours of weekly contact) levels of care (Etheridge, Hubbard, Anderson, Craddock, & Flynn, 1997). Such variations can also extend to length of stay and supplemental services. Although there are good clinical reasons to match a client with appropriate care, the impact on programs of trying to offer “something for everyone” has not been carefully evaluated. One concern is that diverse client tracks could lead to a disjointed and less engaging environment.

Similar problems could arise in larger programs. Programs with more clients or with larger counselor caseloads can have poorer outcomes (McCaughrin & Price, 1992; but see also Moos et al., 1997). Recognizing that engagement is a precursor to outcomes (as described above), it is implicated as a possible link between larger size and poorer outcomes. As organizations grow in size, they tend to cope with the greater volume of work by relying on more bureaucratic procedures and increased internal specialization (Weber, 1947). The result may be a more efficient use of labor, but reduced social interaction. For clients, the chances of getting “lost in the shuffle” increase, and they may fail to receive appropriate attention.

Conversely, accreditation procedures exist, in part, as a quality assurance mechanism, including a routine assessment of whether adequate standards of care are met. Accreditation by national bodies, such as the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO) or the Commission on Accreditation of Rehabilitation Facilities (CARF), is a rigorous process that suggests a strong commitment to quality by programs willing to expend the resources and effort (D’Aunno, 1997). Programs meeting these standards tend to provide better access to health services (Friedmann, Alexander, & D’Aunno, 1999) and have better client outcomes (e.g., McCaughrin & Price, 1992). Based on the association between engagement and outcomes, it is reasonable to expect that JCAHO or CARF accredited programs have better client engagement as well.

In addition to these measures of “global” structure, many relevant features can be defined through aggregation of individual staff perceptions (c.f., Lazarsfeld & Menzel, 1961). First, and perhaps most obviously, counselors must have the basic skills for the job. A sense of personal efficacy and perceived opportunities for professional growth can help counselors cope with the varied demands of drug counseling.

Second, broader aspects of the work environment can impinge upon performance. One set of influences are subjective elements of the job itself, or “psychological climate” (James & James, 1989; James & McIntyre, 1996). Various elements can be identified, but these concern primarily issues of stress and job roles, job design (e.g., autonomy), work-group cohesion and cooperation, and management (e.g., clarity of mission, responsive communication, openness to change). Lehman et al. (2002) found these climate features to be key correlates of average client engagement, and Moos and Moos (1998) reported similar dimensions were related to client satisfaction. They deserve further attention within a wider context of program influences.

Finally, the nature of staff interactions and workplace practices can be delineated. A “professional community” (Louis, Kruse, & Bryk, 1995) or “occupational community” (Van Maanen & Barley, 1984) arising within a workplace is argued to be a positive force. A communal orientation emphasizes social interaction and personal connection that minimizes isolation within the organization (Louis et al., 1995). It emphasizes social control of behavior, rooted in effective professional standards, over external control structures (Kruse, Louis, & Bryk, 1995). These conditions foster work commitment and satisfaction (Van Maanen & Barley, 1984). Although the construct originates outside the drug abuse treatment arena, the key issues of “professionalism” and “community” have bearing for drug counselors’ work (see below). A professional community orientation among staff should contribute to a more engaging environment, buffering against the pressures of internal differentiation described above.

Broadly, “professional community” describes an environment where staff interaction is frequent and in-depth, and staff share values and goals. More specifically, it has several key behavioral (i.e., social interaction practices) and normative (i.e., shared values and goals) features (Bryk, Camburn, & Louis, 1999). In a drug abuse treatment context, three core practices would characterize counselor behavior in a professional community: (a) peer collaboration in which counselors engage in actual shared work, (b) deprivatized practice where counselors observe and learn from each other’s approach, and (c) a reflective dialogue among counselors about therapeutic techniques and client change. These practices are supported by a shared focus on client outcomes, and a sense of collective responsibility for program operations and improvement. Finally, core practices and social norms are maintained through deliberate efforts at socializing and integrating new members of the community.

The goal of the present study is to examine these features of organizational context as they relate to client engagement, while also considering characteristics of the clientele. The intent is to gain some understanding of how these features integrate to define a successful program (i.e., having higher levels of engagement). Because the processes under investigation are intrinsically multilevel – involving individual clients nested within a treatment program context – a hierarchical linear model (HLM; Raudenbush & Bryk, 2002) framework is used to assess the interrelated influences.

2. Method

2.1. Sample

As part of the Treatment Costs and Organizational Monitoring (TCOM) project, data were collected in 2004 and 2005 from 115 treatment programs in 9 states: Florida, Idaho, Illinois, Louisiana, Ohio, Oregon, Texas, Washington, and Wisconsin. These data represent an initial assessment of organizational structure and the first of three annual surveys of clinical staff, clients, and costs. Programs were selected to reflect major types of Outpatient Drug-Free (ODF) treatment for adults in several diverse geographic areas of the United States. The Addiction Technology Transfer Centers (ATTCs; including the Southern Coast ATTC, Great Lakes ATTC, Gulf Coast ATTC, and Northwest Frontier ATTC) assisted with recruitment.

This study includes 94 programs with both staff and client data. The study sample consists of 5,013 clients from 94 programs; data from 546 counseling staff are used for describing the program environment. The remaining 21 of the 115 treatment programs did not provide this data. Six of these 21 programs closed between the time of the initial assessment and the first annual survey administration, three others were undergoing significant reorganization, and two were rebuilding following Hurricane Katrina; these 11 were therefore ineligible for the first annual data collection. Ten other programs withdrew from the study. Thus, the 94 programs included here represent 90% of the eligible programs. Comparisons between the 94 programs with staff and client surveys and the 21 without that information showed no significant differences on the structural characteristics used in the study (described below). Within these 94 programs, survey return rates were 77% for staff and 55% for clients.

Descriptive information for the sample appears in Table 1. Clients were predominantly male, white, and in their mid-thirties. Treatment tenure at the time of data collection was diverse. Their engagement ratings were generally high. The programs generally offered a mixture of regular and intensive outpatient services, but 30% offered only regular. Thirty nine percent were nationally accredited.

Table 1.

Sample description

Characteristic	Percent
Clients (n = 5,013)
Female	33
Ethnic minority	41
Age, median (min-max)	33 (18–75)
Treatment tenure
30 days or less	27
31–90 days	39
91 days to 1 year	28
More than 1 year	6
Programs (n = 94)
Level of care
Regular	30
Intensive	11
Mixed regular & intensive	59
JCAHO/CARF accredited	39
Has planned length of stay	31
Supplemental service areas, median (min-max)	2 (0–4)
Capacity, median (min-max)	100 (12–1,200)
Average caseload, median (min-max)	23 (3–80)

2.2. Procedure

Data collection procedures focused on obtaining a cross-sectional view of treatment program functioning. Upon enrollment in the project, a program director or clinical manager completed a Survey of Structure and Operations (SSO), which gathered information about general program characteristics, organizational relationships, clinical assessment and practices, services provided, staff and client characteristics, and recent changes. Later, during a period of approximately 1 month, clients and staff completed surveys regarding their attitudes. Clients completed the Client Evaluation of Self and Treatment (CEST; Joe et al., 2002), assessing their motivation for treatment, psychological and social functioning, and the treatment experience.

This evaluation was the source of the client engagement measures used in this study. Clinical staff completed a Survey of Organizational Functioning (SOF), which included the Organizational Readiness for Change (ORC; Lehman et al., 2002) instrument. This survey measured needs and pressures for change, general resources, staff attributes, organizational climate, job attitudes, and several specific workplace practices. A program cost assessment was also completed, but these data were not included in the present study.

2.3. Measures

Each of the composite measures described below was evaluated by applying a Rasch rating scale model (Wright & Masters, 1982) to sets of survey items. Rasch analysis assesses the relative difficulty of items in the set (i.e., the likelihood that respondents agree with the item), the fit of the item within the set (i.e., the consistency of item responses with responses to the other items), and the person separation reliability (analogous to Cronbach’s alpha). Taken together, this information supported the use of the scales. Mean square fit statistics were generally near 1.0, indicating most respondents’ answers were consistent with item placement in the scales. All reliability statistics were above .60, and most were above .70 (see below). These analyses were conducted with Winsteps 3.47 computer software (Linacre, 2003).

2.3.1. Treatment engagement

Clients’ engagement in treatment was measured using three composite rating scales (satisfaction, rapport, and participation), which were then averaged to create an overall measure. Treatment satisfaction was based on seven items, such as “this program is organized and run well” and “you get plenty of personal counseling at this program” (reliability = .80). Rapport was a 13 item scale including “you trust your counselor” and “your counselor respects you and your opinions” (reliability = .88). Finally, 12 items measured treatment participation, such as “you are willing to talk about your feelings during counseling” and “you always attend the counseling sessions scheduled for you” (reliability = .85). Reliability of the overall engagement measure was .88. All ratings were made using a 1 to 5 response scale, where 1 indicated “strongly disagree” and 5 indicated “strongly agree;” scale scores were rescaled to range from 10 to 50.

2.3.2. Client background

Client gender, ethnic minority status, and current age were used to represent pretreatment background. Time spent in treatment was considered an important client-level covariate, because of the cross-sectional nature of the survey design. It also serves as a proxy for other, unmeasured treatment-related variables. Clients indicated whether, at the time of the survey, they had been in treatment less than 1 month, 1 to 3 months, more than 3 months but less than 1 year, or more than 1 year. Not all respondents answered all of these background questions, but it was preferable to retain them for analysis, to avoid possible bias and loss of statistical power. For these individuals, missing values were recoded to zero and a separate “missing” indicator was created. This approach corresponds, in essence, to adding a “don’t know” category to each measure.

2.3.3. Organizational structure

Several elements of program structure, based on reports from directors, were considered as possible influences on client engagement. The service model was represented in three ways. Programs were classified according to their level of care: regular (less than 6 hours of programming per week), intensive (at least 2 hours of programming on 3 days per week), or a mixture of the two. Having a planned length of stay for all clients was distinguished from having no planned length, or having one that varied by client type (coded “1” for set planned length, and “0” otherwise). Availability of on-site supplemental services was measured with a count across seven areas: medical, psychological, family, legal, vocational, educational, and financial (for a possible range of 0 to 7 on-site service areas). Program size was characterized both by total client capacity, and by the typical counselor caseload. Accreditation by a national body – either the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) or the Commission on Accreditation of Rehabilitation Facilities (CARF) – was used to indicate ongoing quality assurance efforts. The same procedures described above were used for handling missing data on the structural measures.

2.3.4. Organizational functioning

Selected composite rating scales represented clinical staff views on organizational functioning. These all used the same 5-point rating scale described above for treatment engagement. Two composites addressed staff skills (growth and efficacy). Growth was a five-item scale including “this program encourages and supports professional growth” and “you regularly read professional journal articles or books on drug abuse treatment” (reliability = .78). Efficacy was measured with five items, such as “you are effective and confident in doing your job” and “you have the skills needed to conduct effective group counseling” (reliability = .70).

Organizational climate was represented with six scales (mission, cohesion, autonomy, communication, stress, and openness to change). Mission was based on five items including “this program operates with clear goals and objectives” and “management has a clear plan for this program” (reliability = .76). Six items represented cohesion among staff members (reliability = .82), and included “staff here get along very well” and “the staff here always work together as a team.” Autonomy in decision-making was measured by five items, such as “management here fully trusts your professional judgment” and “counselors here are given broad authority in treating their own clients” (reliability = .61). Communication between staff and management was represented with a five-item composite, and included “ideas and suggestions from staff get fair consideration by program management” and “program staff are always kept well informed” (reliability = .79). Four items measured perceptions of stress, for example “you are under too many pressures to do your job effectively” and “staff members often show signs of stress and strain” (reliability = .84). Finally, openness to change included five items, such as “it is easy to change procedures here to meet new conditions” and “you are encouraged here to try new and different techniques” (reliability = .70). These six dimensions were highly intercorrelated (see Greener, Joe, Simpson, Rowan-Szal, & Lehman, this issue; James & McIntyre, 1996), and were combined into a single, general climate measure (reliability = .83).

Six specific “professional community” workplace practices were included (peer collaboration, deprivatized practice, reflective dialogue, focus on outcomes, collective responsibility, and counselor socialization). These were derived from measures used elsewhere (Bryk et al., 1999). Each practice was measured with its own scale, which were then combined to create an overall professional community composite. Four items indicated peer collaboration, or shared work, including “counselors at this program make a conscious effort to coordinate with other service professionals” and “counselors here design therapeutic interventions together” (reliability = .72). Deprivatized practice reflected the tendency to observe one another’s methods and philosophies, and was measured with five items regarding activities in the past year, such as “invited someone in to help facilitate your sessions” or “received meaningful feedback on your performance from colleagues” (reliability = .79). Reflective dialogue about counseling was represented with five items, including “in the past year, you have had regular conversations with colleagues about what helps clients improve” and “counselors talk about counseling in staff meetings, in the break room, etc.” (reliability = .76). A shared focus on outcomes was measured with five items, including “when making important decisions, this program always focuses on what’s best for client improvement” and “this program has well-defined expectations for all clients” (reliability = .79). A collective responsibility for operations and improvement was defined by six items, such as “many counselors in this program feel responsible to help each other do their best” and “counselors support the director in enforcing program policies and rules” (reliability = .84). Finally, two items measured the approach to counselor socialization, and these were “a conscious effort is made by staff to make new counselors feel welcome here” and “experienced counselors invite new counselors into their sessions to observe, give feedback, etc.” (reliability = .65). Reliability for the overall professional community composite was .85.

2.4. Statistical analysis

The central focus of the study was on the relationship between treatment program context and client engagement. A key feature of the data is measurement at two levels, clients and programs. Such data call for an analytic model that explicitly incorporates program membership (and the similarity among clients that it introduces), without losing valuable information about individual client variability (Raudenbush & Bryk, 2002). Accordingly, a two-level hierarchical linear model (HLM) was used to test hypotheses. At level 1, treatment engagement is related to the set of client-level controls. At level 2, the adjusted means for treatment engagement become program-level outcomes to be predicted by organizational factors. Analyses were conducted with HLM 6 computer software (Raudenbush, Bryk, & Congdon, 2005).

3. Results

Program-level correlations of organizational features with average client engagement appear in Table 2. These reflect small to medium effect sizes (Cohen, 1988), and follow the hypothesized pattern. Engagement tends to be higher in programs with a less differentiated service approach (e.g., offering one level of care, or having a set planned length of stay), that are JCAHO or CARF accredited, where staff skills are greater (more professional growth and efficacy), or where staff report a stronger climate and sense of professional community. In contrast, engagement ratings tended to be lower in larger programs (higher capacity or caseload).

Table 2.

Correlations of program-level covariates with engagement ratings (94 programs)

Variable	Correlation
Service approach
Intensive outpatient	.12
Mixed intensive & regular	−.27
Has planned length of stay	.14
Supplemental service areas	.17
Size
Capacity	−.34
Caseload	−.37
Quality assurance
JCAHO/CARF Accreditation	.18
Staff skills
Growth	.13
Efficacy	.24
Climate
Organizational Climate Index	.17
Work practices
Professional community	.18

Correlations greater than ±.20 are significant, p < .05.

3.1. Prediction analyses

The HLM analyses begin with an assessment of the overall variation in the engagement outcome, to provide a baseline against which other models can be compared. This is accomplished by fitting a model with no predictor variables. Results from this model showed the grand mean of engagement ratings (across all programs) to be 40.61, with a program-level standard deviation of 1.56. About 8% of the total variation in engagement reflected program differences. These differences can also be understood by defining a “high engagement program” as one being 2 standard deviations above the mean (average engagement rating = 43.73), and a “low engagement program” as being 2 standard deviations below the mean (average rating = 37.49); ratings in the “high” program are about 17% above the “low” program. Thus, although program-level differences are modest, particularly in comparison to differences among individual clients, there are noteworthy variations in engagement levels between programs. The purpose of the following analyses was to understand those variations better.

3.2. Client-level prediction model

Client gender, minority status, age, and treatment tenure were considered as predictors of engagement. A key feature of the HLM framework is the ability to assess whether the strength of individual-level relationships vary from one program to another, or whether these effects are largely consistent. For both gender and minority status the homogeneity hypothesis was supported (p’s > .10). For age, the homogeneity test was significant (p < .01), indicating treatment programs differed in the impact of age on engagement ratings. The homogeneity hypothesis was also rejected for the time in treatment indicators, but further examination revealed that three programs had much steeper slopes than the rest of the sample; when these three programs were removed from the sample, the test was no longer significant (p’s > .15). Thus, it was sensible to estimate a common time in treatment relationship for all 94 programs.

The outcome of these tests has bearing for the way client factors are represented in the analysis. In HLM terminology, gender, minority ethnicity, and treatment tenure are all “fixed” effects, or common across programs. These variables are also “centered” around their respective grand means (like covariates in a classical Analysis of Covariance model), to control statistically for differences across programs in these clientele characteristics. Age, however, is considered a “random” effect that differs from one program to another. Consequently, client ages are centered around the mean for their respective program, so that the regression coefficient represents a program-specific effect. To control fully for age differences among programs, mean client age is included in the program-level portion of the model.

Table 3 shows the client-level model. Females and older clients tended to give higher engagement ratings. Clients who had spent a longer time in treatment also gave higher ratings. Because this comparison is a cross-sectional one, it probably reflects both a tendency for clients to become more engaged over time, and a tendency for less engaged clients to drop out. Together, these clientele characteristics account for 5% of the variation in client-level ratings, and 21% of the variation in mean ratings across programs. Thus, a portion of the observed difference among programs has to do with the clients they serve, and when these clients are assessed.

Table 3.

Client-level prediction of engagement ratings

Predictor	Coefficient
Base engagement	40.546**
Female	1.792**
Minority	0.322
Age (years)	0.031**
Tenure (30 days or less)a	−1.188**
Tenure (91 days to 1 year)a	0.680**
Tenure (More than 1 year)a	0.686*
Missing on gender	0.034
Missing on minority status	−0.912
Missing on age	0.861
Missing on tenure	−0.427

^aReference group for tenure comparisons has between 30 and 90 days in treatment.

^*p < .05;

^**p < .01

3.3. Program-level model comparisons

Each of the potential program domains was first considered separately. For each model, the client-level covariates remained as described above, so the effects of program features control for these clientele differences. The models are summarized in Table 4. With the exception of the service approach area, each explains additional variation in adjusted mean engagement ratings. The strongest effects are observed for the areas of staff skills and workplace practices. Clients reported better engagement in programs where they interact with staff who have a greater sense of efficacy and where staff follow a more communal approach.

Table 4.

Comparison of alternative models for program engagement ratings

Model	Coefficient	Variation Explained
Service approach		21%
Base	40.646**
Intensive outpatient	0.368
Mixed intensive & regular	−0.833
Has planned length of stay	0.479
Supplemental service areas	0.220
Missing on length	−0.164
Missing on services	0.511
Size		31%
Base	40.678**
Capacity	−0.022*
Caseload	−0.019
Missing on capacity	−0.166
Missing on caseload	−0.443
Quality assurance		25%
Base	40.564**
JCAHO/CARF accreditation	0.748*
Staff skills		38%
Base	40.592**
Growth	0.013
Efficacy	0.181**
Climate		30%
Base	40.600**
Climate index	0.134*
Work practices		36%
Base	40.590**
Professional community	0.164**

Models also include client-level covariates, as described in Table 2.

^*p < .05;

^**p < .01

Program size, quality assurance mechanisms, and staff views of the working environment are also associated with engagement. Clients report less engagement in larger programs. Ratings are higher in programs accredited by JCAHO or CARF. Clients are also better engaged in programs where staff report a more positive organizational climate.

3.4. Composite model for engagement

As a final step in the analysis, the significant program-level predictors from the separate models were combined in a single composite model. When combined, counselor efficacy and the climate index were no longer significant predictors (p > .40), and were deleted from the model. The final composite model appears in Table 5. It shows client engagement to be higher in programs where staff feel a stronger sense of community. In contrast, engagement is lower in programs with a larger capacity. Accreditation was also retained as a predictor because, even though it was not significant by a 2-tailed test (p = .054), its effect was in the expected direction and would be significant by a 1-tailed test; it therefore merits further consideration in the future. Taken together, these measures explain 45% of the variation in programs’ average engagement levels.

Table 5.

Composite model for engagement ratings

Predictor	Coefficient
Base engagement	40.648**
Mean client age	0.098
Capacity	−0.021**
JCAHO/CARF accredited	0.649
Professional community	0.157*
Missing on capacity	−0.395
Client-level controls
Female	1.785**
Minority	0.343*
Age (years)	0.032**
Tenure (30 days or less)a	−1.205**
Tenure (91 days to 1 year)a	0.675**
Tenure (More than 1 year)a	0.673*
Missing on gender	0.033
Missing on minority status	−0.892
Missing on age	0.852
Missing on tenure	−0.412
Program-level variation explained	45%
Client-level variation explained	5%

^aReference group for tenure comparisons has between 30 and 90 days in treatment.

^*p < .05;

^**p < .01

The composite model suggests that no single program dimension can account for differences in client engagement. Rather, more successful programs are characterized by an array of features that interrelate in complex ways. In particular, larger capacity programs appear to be less productive environments for both clients and staff, as underscored by the lower sense of efficacy (r = −.26), professional community (r = −.14), and poorer climate (r = −.08) that prevails there. More detailed study of these issues clearly is needed.

4. Discussion

The study began with expectations that certain organizational features of programs – including the service delivery approach, size, and accreditation, along with staff skills, view of climate, and workplace practices – would influence client engagement in treatment. These expectations were generally supported. Engagement was higher in smaller programs, those with JCAHO or CARF accreditation, and where staff felt more confidence in their skills, reported a more positive work climate, and engaged more in professional community practices. The combined analysis of these factors showed that no single organizational dimension accounts for program differences; rather, it is a constellation of interrelated structural features and social norms that link to engagement.

The findings with regard to size deserve comment, as they tap a fundamental tension in program design, between an efficient use of resources and maintaining productive social interaction and manageable staff loads. Indeed, while some studies indicate larger programs to be problematic (e.g., McCaughrin & Price, 1992), others argue that larger size suggests more resources and more benefits for clients (Moos et al., 1997). In practice, however, even though cost efficiencies can come with increased size (Harwood, Kallinis, & Liu, 2001), it is not clear that larger programs actually offer more or higher quality services (Delany, Broome, Flynn, & Fletcher, 2001; Harwood et al., 2001; Friedmann, D’Aunno, Jin, & Alexander, 2000). Results of the present study show larger capacity programs to be less engaging environments for clients and perhaps for staff as well. This suggests that the barriers to interaction and greater workload may outweigh any potential resource advantage associated with increased size. The challenge that faces programs is to work toward an optimal size, neither too small nor too large, to balance the benefits of efficiency and social interaction.

With the various staff measures, the present study extends previous findings that staff workplace perceptions have implications for clients (Greener at al., this issue; Lehman et al., 2002; Moos & Moos, 1998) and illustrates the key role that counselors play. The emergence of “professional community” practices – based on sustained and in-depth social interaction and shared goals and values – appeared to be a particularly salient feature, and deserves further attention. In part, many counseling staff factors may have an indirect impact on clients by facilitating or inhibiting the acquisition of new therapeutic skills and techniques (Joe, Broome, Simpson, & Rowan-Szal, this issue; see Bryk et al., 1999 for an illustration from an educational context). That is, the organization of counselors’ work life could improve the quality of the counseling they give by supporting their learning and sharing of “best practices,” which in turn benefits clients. Another equally important possibility, however, is that the social system described by these counselor factors has a direct impact on clients’ experience. This is the view articulated by Moos and Moos (1998), who suggested that staff with a positive work environment tend to develop a comparable environment for clients. Thus, the engagement and commitment shown by staff would be reflected by clients who share the interactive social system within the program. Selecting between these two possible explanations is beyond the scope of the present study, but in either case the staff workplace would be a promising avenue for program improvements, and future research should examine these issues more closely.

It is important to reiterate, of course, that clients and staff constitute two interrelated but distinct stakeholder groups for treatment programs. Each has its own perspective on program functioning, and the organizational features that matter most to clients may not be the same ones that matter most to counseling staff. This view is outlined by Schneider and colleagues (1975; Schneider & Bowen, 1985; see also Burke, Borucki, & Hurley, 1992), who describe the importance of climate “for something,” such as customer service. That is, the relevant organizational features will vary by the specific task and stakeholder. Research with clients and staff represent two interrelated lines of inquiry for the TCOM project. For instance, when combined with information about accounting and economic costs of treatment, a program’s well being can be described on a variety of dimensions.

Support was mixed for the hypothesis that a differentiated service approach would be associated with less engagement. Offering a single level of care and having a set planned length of stay did correlate with better average engagement ratings, but these were not significant predictors in the HLM analysis. Thus, it appears difficult to separate the potential impact of these structural features from the impact of the clientele that tend to appear with them. More attention to this issue is needed.

The findings presented here supplement the traditional client-level focus on treatment process because they help to clarify the organizational foundation upon which therapeutic activities take place. The TCU Treatment Process Model (Simpson, 2004), for example, extensively describes the sequence of events and activities (including early engagement) leading to positive outcomes for clients, but until recently research has had less to say about the impact of the organizational context in which these activities take place. The present study supported earlier findings that clientele factors are important, even in understanding program differences (Broome et al., 1999; Joe et al., 2002), and that the staff working environment has implications for client engagement (Moos & Moos, 1998); in addition, the study highlighted the role of structural design features, such as size.

More broadly, this work is part of a larger concern with why different programs offering the same basic model of care can have widely differing client outcomes. The same pattern has been reliably identified not only in ODF programs (McCaughrin & Price, 1992; McLellan et al., 1993), but also among methadone maintenance (Ball & Ross, 1991; Magura, Nwakeze, Kang, & Demsky, 1999) and inpatient programs (McLellan et al., 1993). In recognition of the connection between engagement and eventual outcomes (Simpson, 2004), studies such as this one that identify program features associated with engagement are a valuable corollary to outcome-focused research.

Of course, present results must be interpreted within the context of certain limitations. The sample is not intended to be nationally representative, although it is large and geographically diverse. Some risk remains that the importance of some organizational characteristics might be obscured through nonrandom selection. Within the sample, however, reasonable coverage of survey respondents was obtained, particularly among staff (77% return rate). Respondents’ portrayal of their programs therefore seems trustworthy. Furthermore, comparisons between the 94 included programs and the 21 without client and staff survey data showed no significant differences in the structural features used in the study; these results argue against bias from attrition. Finally, the list of potential predictors was clearly not exhaustive, particularly at the client level. It is possible that some relevant features have been omitted. However, in order for an omitted client variable to bias the program-level results, the average of the omitted variable must have an independent relationship to organizational features in the model (and to engagement), above and beyond the effect of client-level measures (including treatment tenure) already included. Given the wide variety of client measures related to early dropout from treatment (Simpson, Joe, Broome, et al., 1997), the likelihood of an unmeasured client variable having substantial independent covariation seems small.

In summary, this study adds to a growing mosaic of research showing that drug abuse treatment program organization matters. Emerging from these analyses is a rough picture of the distinctive organizational environment in an ODF program that is successful at engaging clients (see Greener et al., this issue, for a multi-modality sample). Such a program is smaller, focused on maintaining quality, has more confident counselors, and promotes opportunities to interact and collaborate with colleagues. In part, then, programs that are better able to engage clients are also better able to engage their counseling staff. Focusing on staff and addressing their workplace concerns, through specialized training for example, could ultimately pay dividends in terms of client performance, which underscores the need for ongoing monitoring of the program as a whole. This would include key components of the foundation upon which the treatment process occurs: structure, staff and climate, and capital resources.