Training and retaining staff to competently deliver an evidence-based practice: The role of staff attributes and perceptions of organizational functioning

Abstract

Within the context of an initiative to implement evidence-based practices (EBPs) for adolescents with substance use disorders, this study examined the extent to which staff factors measured at an initial EBP training workshop were predictive of EBP-competence and turnover status of staff (N=121) measured 6, 9, and 12 months post-training. By the final assessment point, 52.3% of staff transitioned to the Employed/EBP-Competent category, 26.6% transitioned to the Not Employed/Not EBP-Competent category, 4.6% transitioned to the Not Employed/EBP-Competent category, and 16.5% had not transitioned out of the initial category. Multilevel multinomial regression analysis identified several measures that were significant predictors of staff transitions to the Not Employed/Not EBP-Competent category (e.g., program needs, job satisfaction, burnout) and transitions to the Employed/EBP-Competent category (e.g., months in position, pressures for change, influence). Findings have implications for the development and testing of strategies to train and retain staff to deliver EBPs in practice settings.

In 2008, an estimated 34 million Americans received services for problems related to mental health, alcohol use, or illicit drug use (Substance Abuse and Mental Health Services Administration [SAMHSA], 2009). Unfortunately, given the widely acknowledged “research-to-practice gap” (e.g., Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Garner, 2009; Hogan, 2003; Institute of Medicine, 1998, 2001, 2006), it is likely that only a small percentage of these individuals received services that have been empirically supported (i.e., evidence-based practices [EBPs]). Given the critical importance of providing effective treatment, an emphasis has been put on facilitating the implementation of EBPs. Consistent with Brown and Flynn’s (2002) recommendation for the federal government to play a larger role in transferring EBPs to practice settings, one strategy currently being used in the United States to help promote greater use of EBPs in community-based practice settings is providing funding for EBP implementation via time-limited grants. As one example, since 2003, SAMHSA’s Center for Substance Abuse Treatment (SAMHSA/CSAT) has funded a total of 118 grants (each approximately $300,000 per year for three years) to improve implementation of EBPs for adolescent substance use treatment, which represents a substantial federal investment in improving the dissemination and implementation of EBPs.

In addition to funding, researchers have identified other factors important to EBP implementation. For example, Fixsen and colleagues (2005) have proposed seven “core implementation components” necessary for successful implementation, including: a) systems interventions, b) staff selection, c) staff training, d) staff consultation and coaching, e) staff evaluation, f) program evaluation, and g) facilitative administrative supports. Literature reviews have noted, however, that research informing the selection and training of staff in EBPs is in the very early stages (Fixsen et al., 2005; Garner, 2009; Miller, Sorensen, Selzer, & Brigham, 2006; Walters, Matson, Baer, & Ziedonis, 2005) and that these important areas of implementation need greater research attention.

As with research on staff selection and training, research on staff turnover (i.e., when a staff person voluntarily or involuntarily separates from an organization) is an important topic that has received only limited attention within EBP implementation research. While considerable turnover-related research has been conducted within the broader fields of substance use and mental health treatment (e.g., Aarons & Sawitzky, 2006; Eby, Burk, & Maher, 2010; Glisson et al., 2008; Knight, Broome, Edwards, & Flynn, 2011; Knudsen, Ducharme, & Roman, 2006, 2007, 2008; McNulty, Oser, Johnson, Knudsen, & Roman, 2007; Rothrauff, Abraham, Bride, & Roman, 2011; White & Garner, 2011), few studies have examined staff turnover within the context of EBP implementation (for exceptions, see Aarons, Fettes, Flores, & Sommerfeld, 2009; Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009; Sheidow, Schoenwald, Wagner, Allred, & Burns, 2007; Woltmann et al., 2008). Based on a sample of staff that participated in a Multisystemic Therapy (MST) implementation study, Sheidow et al. (2007) found that staff who reported greater perceptions of an emotionally demanding organizational climate were significantly more likely to leave their employment. Aarons and colleagues, who examined data from staff participating in a study related to implementation of an EBP called SafeCare, found EBP implementation and ongoing fidelity monitoring may have some protective effect against staff turnover (Aarons, Sommerfeld, et al., 2009) and that reports of emotional exhaustion were lowest among staff in the EBP implementation condition (Aarons, Fettes, et al., 2009). Thus, given that the extant literature is relatively sparse in regard to turnover in the context of EBP implementation, there is a need for further investigation of the extent to which staff turnover impacts EBP implementation.

Because successful implementation of EBPs in practice settings requires both the ability to train staff to competently deliver an EBP (i.e., EBP-competence) and retain (i.e., prevent turnover) staff for some period of implementation after the staff has demonstrated EBP-competence, we argue it is useful to consider these two important outcomes simultaneously. Intuitively, this makes sense given the turnover of staff who have demonstrated EBP-competence would be less desirable than the turnover of staff who have not yet demonstrated EBP-competency. Likewise, retaining staff who have demonstrated EBP-competence would be more desirable than continuing to employ staff who are unable to demonstrate EBP-competence. Thus, in an effort to advance existing knowledge regarding how best to train and retain staff to competently deliver EBPs in practice settings, the current study’s objective was to examine the extent to which staff factors measured at an initial EBP training predicted staff’s subsequent transitions from the starting status of Employed/Not EBP-Competent to the other three possible categories: Employed/EBP-Competent, Not Employed/Not EBP-Competent, and Not Employed/EBP-Competent.

Staff factors examined in this study included basic staff background characteristics (e.g., age, race, gender, education, experience) and staff factors measured with an expanded version of the Texas Christian University’s Organizational Readiness for Change instrument (Lehman, Greener, & Simpson, 2002). These TCU factor measures were developed specifically for assessing readiness to implement EBPs in practice and have demonstrated good psychometric properties (Lehman et al., 2002). Additionally, prior research has shown these measures to have good convergent and concurrent validity (Saldana, Chapman, Henggeler, & Rowland, 2007), be correlated with attitudes toward treatment innovations (Fuller et al., 2007) and client engagement in treatment (Greener, Joe, Simpson, Rowan-Szal, & Lehman, 2007), and be predictive of greater self-reported use of treatment innovations following a training workshop (Simpson, Joe, & Rowan-Szal, 2007). To the best of our knowledge, however, research to date has not examined the predictive validity of these scales with regard to training and retaining staff to competently deliver EBPs in practice settings. Such research is important given Saldana et al. (2007) noted that “the ultimate value of the highly promising ORC scale domains will be determined by the prospective associations with the adoption, implementation, and sustainability of EBPs” (p. 168).

Because this is the first known study to prospectively examine the extent to which staff factors predict staff transitions through the EBP implementation process, we consider this study and its analyses to be exploratory. In general, however, we expected that more positive perceptions of organizational functioning (e.g., greater satisfaction, lower burnout) would significantly increase the likelihood of staff transitioning from the initial status of Employed/Not EBP-Competent to the preferred status of Employed/EBP-Competent and decrease the likelihood of staff transitioning to the less desirable status of Not Employed/EBP-Competent.

Method

Study Context

Data for this study were collected within the context of a large SAMHSA/CSAT-funded EBP implementation initiative. As has been described elsewhere (see Godley, Garner, Smith, Meyers, & Godley, 2011), the general goal of this initiative was to improve adolescent substance use treatment by providing several community-based treatment organizations with funding so that their clinical staff could learn and implement treatments that have been shown to be effective in reducing adolescent substance use and substance-related problems. Each organization received approximately $300,000 per year (for up to three years) and was able to have up to five staff attend training in the Adolescent Community Reinforcement Approach and Assertive Continuing Care (A-CRA/ACC) and receive ongoing technical assistance at no additional cost. The training in A-CRA/ACC was funded by SAMHSA/CSAT and provided by model developers (Godley et al., 2001; Meyers & Smith, 1995) who have published several articles on the effectiveness and cost-effectiveness of these approaches (e.g., Dennis et al., 2004; Garner, Godley, Funk, Dennis, & Godley, 2007; Garner, Godley, et al., 2009; Godley, Godley, Dennis, Funk, & Passetti, 2002, 2007; Godley et al., 2010). The training process included components identified by Carroll, Kadden, Donovan, Zweben, and Rounsaville (1994) and supported in clinical training research (Miller, Yahne, Moyers, Martinez, & Pirritano, 2004; Sholomskas et al., 2005). Specifically, this process included a treatment manual, 3.5-day initial workshop, multiple coaching/supervision sessions, and ongoing individual performance feedback following reviews of recorded therapy sessions by experts trained in the use of a rating manual. A more detailed description of the training process has been provided by Godley et al. (2011).

Procedures

Between December 2006 and November 2009, all staff who attended one of the 12 three and a half day initial A-CRA/ACC training workshops were invited to participate in this study. In accordance with Chestnut Health Systems’ Institutional Review Board, staff were informed that no explicit incentives were being offered for study participation and that participation was both voluntary and confidential. On the last day of the training, staff were provided study packets that contained the study’s informed consent, the study survey, and a postage-paid return envelope to return the signed informed consent and completed survey.

Sample

Of the 154 staff who attended one of the A-CRA/ACC training workshops, 121 (79%) agreed to participate in the current study. Participants ranged in age from 22 to 64 years (M = 35.2, SD = 10.7), and the majority were female (73.6%). Almost half reported their race as Caucasian (54.5%), 18.2% were African American, 2.5% were Asian, 2.5% were American Indian/Alaska Native, and the remaining 22.3% reported being either “more than one race” or “other.” Most staff had a Master’s Degree (57.9%) or Bachelor’s Degree (34.7%), while a smaller number reported either having something less than a Bachelor’s degree (6.6%) or a Doctoral Degree (less than 1%). Prior experience in drug abuse counseling ranged from less than a month to 35 years (Median = 1.5 years, SD = 4.9 years), and 16.5% reported currently having an addictions treatment license or certification.

Measures

Dependent Variable

Staff status regarding employment (Employed vs. Not Employed) and achievement of A-CRA/ACC competence (EBP-Competent/Not EBP-Competent) were abstracted from administrative records. Each of the SAMHSA/CSAT-funded organizations were required to report staff turnover for those who had attended the initial training, and there were several cross checks on each staff member’s status through regular contact with the organizations. The determination of EBP-competency was based on the measurement of in-session clinical behaviors used to determine basic A-CRA competence. The EBP-competence process was developed because the funder’s goal for this initiative was for each organization to have staff reliably delivering the EBP with fidelity. Thus, it was important to have a means of regularly assessing staff progress towards and attainment of competent delivery of the intervention. A-CRA/ACC is based on a philosophical approach that emphasizes using individualized reinforcers to motivate change and a number of procedures that treatment staff can call upon based on adolescent needs (see Godley et al., 2001 for intervention details). Each clinician was required by SAMHSA/CSAT to upload digital recordings of actual treatment sessions to a web-based application and could request specific sessions be reviewed by raters during the certification process. A number of trained individuals were taught to evaluate these sessions using a very detailed rating manual (see Garner, Barnes, & Godley, 2009 for details on rater training). Raters also provided specific narrative feedback regarding what staff did well and what they could do to improve procedure delivery. The over 150-page rating manual contains operational definitions of each rating possibility (1–5) for each component of each treatment procedure (Smith, Lundy, & Gianini, 2007), and general clinical skills (i.e., warm/understanding, non-judgmental, maintaining session focus, and appropriately active). Each session also was rated on an overall dimension that included two components: a) stayed within treatment protocol (as far as philosophy and objectives) and b) introduced treatment procedures at appropriate times. Achievement of EBP-competence was operationally defined as demonstrating on the digital recordings: a) a positive, supportive, relationship-enhancing tone to the sessions (i.e., receiving rating of 3 or greater in each of the general clinical skills areas, shown with not less than six recordings); and b) competency in the following treatment procedures (i.e., receiving rating of 3 or greater in each of the following areas, across six recordings): Functional Analysis of Use, Functional Analysis of Pro-social Behavior, use of the Happiness Scale, Treatment Plan/Goals of Counseling, Communication Skills, Problem Solving Skills, Adolescent-Caregiver Relationship Skills, and Homework. Thus, achievement of EBP-competence was a discrete measurable event based on this process. Once clinicians achieved EBP-competence, it was not possible to transition back to the Not EBP-Competent status. However, to monitor continued faithfulness to the model, fidelity checks were conducted bimonthly by randomly selecting and rating a recorded session from each clinician. If needed, a clinician received supplemental supervision sessions. As reported by Godley et al. (2010), achievement of EBP-competence took a median of 19 (range of 8 – 49) recorded sessions and 9 (range of 3–24) months.

Predictor measures

Staff completed a one-page form about their background characteristics (e.g., age, race, gender, education, years of drug abuse counseling experience) and an expanded version of the Organizational Readiness for Change (ORC; Lehman et al., 2002) instrument called the Survey of Organizational Functioning (SOF; Broome, Knight, Edwards, & Flynn, 2009). In total, the SOF has 27 scales, which include the 18 original ORC scales and nine additional scales related to Job Attitudes and Workplace Practices. Each of the items making up each scale were rated using a five point Likert-type scale (1 = Strongly Disagree, 2 = Disagree, 3 = Uncertain, 4 = Agree, 5 = Strongly Agree). Each scale score was then multiplied by 10, resulting in a possible scale range of 10 – 50 (Lehman et al., 2002). Brief descriptions of each of these measures and descriptive statistics are provided in Table 1.

Table 1.

Descriptive statistics for Survey of Organizational Functioning (SOF) scales

Measures (# items) – Sample item.	Alpha	Mean (SD)
Motivation for Change
Program Needs (8 items) - Your program needs additional guidance in assessing client needs.	.86	28.11 (7.33)
Training Needs (8 items) - You need more training for monitoring client progress.	.79	28.26 (6.47)
Pressures for Change (7 items) - Current pressures to make program changes come from accreditation or licensing authorities.	.65	30.74 (5.73)
Resources
Offices (4 items) - Your offices and equipment are adequate.	.68	33.26 (8.54)
Staffing (6 items) - There are enough counselors here to meet client needs.	.70	32.57 (6.16)
Training (4 items) - This program holds regular in-service training.	.48	36.37 (6.39)
Computer Access (7 items) - You have a computer to use in your personal office space at work.	.50	34.28 (5.31)
e-Communications (4 items) - You have convenient access to e-mail at work.	.36	37.35 (6.21)
Staff Attributes
Growth (5 items) - You have enough opportunities to keep your counseling skills up-to-date.	.59	35.89 (5.87)
Efficacy (5 items) - You are effective and confident in doing your job.	.67	40.38 (4.61)
Influence (6 items) - You often influence the decisions of other staff here.	.77	34.88 (5.76)
Adaptability (4 items) - You are willing to try new ideas even if some staff members are reluctant.	.66	39.33 (4.96)
Organizational Climate
Mission (5 items) - This program operates with clear goals and objectives.	.71	35.57 (6.25)
Cohesion (6 items) - The staff here always work together as a team.	.84	36.74 (7.15)
Autonomy (5 items) - Counselors here are given broad authority in treating their own clients.	.62	34.67 (5.72)
Communication (5 items) - The formal and informal communication channels here work very well.	.81	32.85 (7.45)
Stress (4 items) - Staff members often show signs of stress and strain.	.77	30.40 (8.15)
Openness to Change (5 items) - The general attitude here is to use new and changing technology.	.74	34.10 (6.38)
Job Attitudes
Burnout (7 items) - You feel like you aren’t making a difference.	.71	22.99 (5.66)
Satisfaction (6 items) - You are satisfied with your present job.	.77	39.57 (6.03)
Director Leadership (10 items) - My program director gets people to work together for the same goal.	.93	37.21 (7.91)
Workplace Practices
Peer Collaboration (4 items) - The director, counselors, and staff collaborate to make this program run effectively.	.68	36.63 (6.24)
Deprivatized Practice (5 items) - You have received meaningful feedback on your performance from colleagues.	.73	32.00 (7.48)
Collective Responsibility (6 items) - Many counselors in this program feel responsible to help each other do their best.	.90	36.41 (7.09)
Focus on Outcomes (5 items) - When making important decisions, the program always focuses on what’s best for client improvement.	.77	35.55 (6.59)
Reflective Dialogue (5 items) - Counselors in this program regularly discuss assumptions about counseling and behavior change.	.72	35.19 (6.34)
Counselor Socialization (2 items) - Experienced counselors invite new counselors into their sessions to observe, give feedback, etc.	.42	35.93 (7.98)

Analytic plan

Participants in the study represented 34 different treatment organizations, and it is likely that staff from the same organization shared more similarities with one another than they did with staff from other organizations. Since intercorrelations among participants may violate the ordinary least squares regression assumption of independent error terms and can lead to misestimated standard errors, HLM version 6.08 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2004) software was used because it allows the relationship between the variables of interest to vary by higher-level groupings. The Employed/Not EBP-Competent status served as the referent category because it represented the initial status of all staff, and two-level (therapists [level-1] nested within organization [level-2]) multinomial regression analyses were conducted for all staff background factors and those scales with acceptable internal consistency (i.e., alpha greater than or equal to .60; see Murphy & Davidshofer, 1988) at each of the three assessed time points. The conventional p < .05 was used to define statistical significance.

Results

Staff classifications

By default, all 121 staff began in the Employed/Not EBP-Competent category. At 6 months post-training, 80 (66.1%) staff remained in the Employed/Not EBP-Competent category, 20 (16.5%) had transitioned to the Employed/EBP-Competent category, 20 (16.5%) had transitioned to the Not Employed/Not EBP-Competent category, and 1 (< 1%) had transitioned to the Not Employed/EBP-Competent category. Six staff (5.0%) were not able to be categorized 9 months after the training workshop because it had not been nine months since their training attendance or because their organizations’ three-year project ended before this time point. Of the 115 staff who were eligible to be classified after nine months, 42 (36.5%) remained in the Employed/Not EBP-Competent category, 44 (38.3%) transitioned to the Employed/EBP-Competent category, 27 (23.5%) transitioned to the Not Employed/Not EBP-Competent category, and 2 (1.7%) transitioned to the Not Employed/EBP-Competent category. At the 12-month time point, there were 12 staff (9.9%) not able to be categorized because it had not been twelve months since their training attendance or because their organizations’ three-year project ended before this time point. Of the remaining 109 eligible staff, 18 (16.5%) had not transitioned out of the Employed/Not EBP-Competent category, 57 (52.3%) transitioned to the Employed/EBP-Competent category, 29 (26.6%) transitioned to the Not Employed/Not EBP-Competent category, and 5 (4.6%) transitioned to the Not Employed/EBP-Competent category.

Predictors of staff transitions

Few staff could be classified into the Not Employed/EBP-Competent category at each of the three respective time points (n = 1 at 6-month, n = 2 at 9-month, and n = 5 at 12-month). Thus, this group was excluded from subsequent analyses (Raundenbush & Liu, 2000). Results reported below focus on staff transitioning from the initial status of Employed/Not EBP-Competent (referent group) to either Employed/EBP-Competent or Not Employed/Not EBP-Competent. As shown in Table 2, 13 measures (4 staff background and 9 organizational functioning) were found to be statistically significant predictors of staff transition status during the 12 months following initial training.

Table 2.

Predictors of transitioning to the Employed/EBP-Competent or Not Employed/Not EBP-Competent status.

	6 months			9 months			12 months
Measure	OR	(95%	p	OR	(95%	p	OR	(95%	p
Background
Male
E/EBP-C	1.32	(0.39,4.	.650	1.64	(0.50,5.	.413	1.12	(0.22,5.	.894
Not E/Not	2.94	(1.02,8.	.047	3.87	(1.22,12	.022	2.93	(0.59,14	.186
Master’s Degree or
E/EBP-C	0.34	(0.10,1.	.092	1.16	(0.43,3.	.761	2.40	(0.61,9.	.210
Not E/Not	0.56	(0.17,1.	.348	0.35	(0.12,0.	.047	0.90	(0.22,3.	.887
Licensed or
E/EBP-C	0.57	(0.08,3.	.552	0.19	(0.04,0.	.035	2.80	(0.24,32	.408
Not E/Not	0.98	(0.18,5.	.986	0.83	(0.24,2.	.761	5.36	(0.46,61	.177
Months in position
E/EBP-C	0.95	(0.91,1.	.060	0.96	(0.92,1.	.026	0.96	(0.92,1.	.095
Not E/Not	0.98	(0.94,1.	.352	0.97	(0.94,1.	.166	0.96	(0.92,1.	.118
Program Needs
E/EBP-C	1.04	(0.94,1.	.420	1.03	(0.95,1.	.481	1.05	(0.94,1.	.390
Not E/Not	1.19	(1.08,1.	.001	1.18	(1.07,1.	.002	1.16	(1.03,1.	.018
Pressures for
E/EBP-C	1.03	(0.93,1.	.577	1.14	(1.03,1.	.016	1.02	(0.90,	.802
Not E/Not	0.94	(0.85,1.	.299	1.05	(0.94,1.	.409	1.00	(0.87,1.	.970
Staff Attributes
Influence
E/EBP-C	0.85	(0.76,0.	.007	0.91	(0.83,1.	.049	0.96	(0.85,1.	.442
Not E/Not	0.97	(0.87,1.	.501	0.92	(0.83,1.	.110	0.93	(0.82,1.	.264
Organizational
Mission
E/EBP-C	0.96	(0.87,1.	.481	1.01	(0.92,1.	.803	0.96	(0.85,1.	.442
Not E/Not	0.86	(0.78,0.	.004	0.88	(0.79,0.	.013	0.88	(0.77,1.	.051
Autonomy
E/EBP-C	0.91	(0.81,1.	.073	0.93	(0.84,1.	.140	0.99	(0.87,1.	.824
Not E/Not	0.92	(0.83,1.	.126	0.86	(0.77,0.	.013	0.87	(0.75,1.	.053
Job Attitudes
Burnout
E/EBP-C	1.01	(0.90,1.	.873	1.00	(0.89,1.	.930	0.97	(0.84,1.	.713
Not E/Not	1.13	(1.01,1.	.042	1.10	(0.97,1.	.143	1.06	(0.91,1.	.479
Satisfaction
E/EBP-C	1.01	(0.91,1.	.895	1.00	(0.92,1.	.927	1.01	(0.89,1.	.854
Not E/Not	0.88	(0.80,0.	.014	0.89	(0.81,0.	.028	0.88	(0.77,1.	.075
Director
E/EBP-C	0.95	(0.88,	.182	0.98	(0.91,1.	.513	1.03	(0.94,1.	.571
Not E/Not	0.90	(0.84,0.	.011	0.91	(0.84,0.	.025	0.95	(0.86,1.	.258
Workplace
Collective
E/EBP-C	0.99	(0.90,1.	.764	0.98	(0.90,1.	.650	1.02	(0.92,1.	.678
Not E/Not	0.90	(0.82,0.	.027	0.91	(0.83,1.	.053	0.93	(0.93,1.	.178

Note: E/EBP-C = Employed/EBP-Competent; Not E/Not EBP-C = Not Employed/Not EBP-Competent. Variables tested but not statistically significant at any of the three time points were: age, white, months experience, training needs, offices, staffing, efficacy, adaptability, cohesion, communication, stress, openness to change, peer collaboration, deprivatized practice, focus on outcomes, and reflective dialogue.

Predictors of 6-month transitions

At the first assessment point (i.e., 6 months post-training), the odds of transitioning to the Employed/EBP-Competent status were significantly lower for staff reporting greater influence (OR = 0.85). The odds of transitioning to the Not Employed/Not EBP-Competent status, however, were significantly higher for males (OR = 2.94) and staff reporting greater levels of program needs (OR = 1.19) or burnout (OR = 1.13), and significantly lower for staff reporting higher levels of mission (OR = 0.86), satisfaction (OR = 0.88), director leadership (OR = 0.90), and collective responsibility (OR = 0.90).

Predictors of 9-month transitions

At the second assessment point (i.e., 9 months post-training), the odds of transitioning to the Employed/EBP-Competent status were significantly lower for staff who reported having other licenses/certifications (OR = 0.19), more months in their current position (OR = 0.96), and greater influence (OR = 0.91), but were significantly higher for staff who reported greater pressures for change (OR = 1.14). The odds of transitioning to the Not Employed/Not EBP-Competent status by the 9-month assessment point were significantly lower for staff with a Master’s degree or higher (OR = 0.35) and staff reporting greater levels of mission (OR = 0.88), autonomy (OR = 0.86), satisfaction (OR = 0.89), and director leadership (OR = 0.91). However, the odds of transitioning to the Not Employed/Not EBP-Competent status by the 9-month assessment point were significantly higher for males (OR = 3.87) and staff reporting greater program needs (OR = 1.18).

Predictors of 12-month transitions

By the third assessment point (i.e., 12 months post-training), none of the measures were found to be significant predictors of staff transitioning to the Employed/EBP-Competent status. However, the odds of transitioning to the Not Employed/Not EBP-Competent status by the 12-month assessment point were found to be significantly higher for staff reporting greater program needs (OR = 1.16).

Discussion

Staff training and retention are important issues for all organizations, yet they have been the topics of relatively limited attention in substance use and mental health treatment research. This study contributes to this important literature and is the first known study to examine predictors of staff transitions over time in terms of both EBP-competence and employment status. As noted previously, this distinction has important clinical and program management implications, given turnover of staff who have achieved competence in an EBP is more concerning than turnover of staff who have not demonstrated this competence, and retaining competent staff is more desirable than continuing to employ staff unable to demonstrate competence.

Overall rates of staff turnover and EBP-Competence

We examined staff status with regard to both EBP-competence and employment at three time points (6, 9, and 12 months post the initial training workshop) for 121 staff participating in a large EBP dissemination and implementation initiative (Godley et al., 2011). Examining each of these outcomes separately, which is useful for making comparisons with the existing literature, we observed staff turnover rates of 17% at 6 months, 25% at 9 months, and 31% at 12 months. Relatively speaking, the 31% 12-month turnover rate observed in this study is similar to the 33% 12-month turnover rate reported by Eby et al. (2010) and 48% higher than the 21% annualized turnover rate reported by Sheidow et al. (2007) for their sample of mental health treatment staff. Additionally, we observed staff EBP-competence rates of 17% at 6 months, 40% at 9 months, and 57% at 12 months. The 12-month EBP-competence rate (57%) was similar to the approximately 61% of staff who met or exceeded the proficiency criterion at 12 months in Miller and colleagues’ (2004) workshop plus feedback and coaching condition.

Four-level classification of staff transitions

When examining the categorization of staff into one of the four mutually exclusive combinations of EBP-competence and employment, we found that at the 12-month time point, the majority of staff (52.3%) could be classified as Employed/EBP-Competent, which is the most desirable of the four possible outcomes. Additionally, it seems to be a positive outcome that the smallest percentage (4.6%) of staff was classified in the Not Employed/EBP-Competent category. Although achievement of EBP-competence would possibly make staff more marketable and thus potentially increase the likelihood of their turnover, it appears staff achieving EBP-competence generally remained committed to their organization. This finding is both encouraging and similar to a finding by Aarons, Sommerfeld, et al. (2009), who found that EBP fidelity monitoring had a protective effect on staff turnover. Approximately one in six (16.5%) staff were unable to achieve EBP-competence within a year after the initial training workshop, yet remained employed at their original organizations. Although there are potential unmeasured reasons that could be responsible for staff not achieving EBP-competence within 12 months (e.g., personal health issues, competing work assignments), it is encouraging to note that the majority of these staff (13 of 18; 72%) did eventually achieve EBP-competence. Finally, more than a quarter (26.6%) of staff left (either voluntarily or involuntarily) their employment after participating in the 3.5-day initial EBP training workshop, but before obtaining EBP-competence. On average, staff in this category remained on the project for 5.3 months (SD = 3.2 months; range 4 to 355 days). Although individual staff member’s therapeutic skills might have improved and he/she might have used new skills within the substance use treatment field at other organizations (both of which are not known), it is a concern that the extensive training-related resources provided to each individual were not realized for more than a quarter of the staff. Because information about staff was collected at the time of the initial training workshop, we had the ability to examine the extent to which these staff factors were predictive of staff transitioning to the other possible statuses at different points in time after training.

Predictors of staff turnover prior to EBP-competence

Across all three time points, nine staff factors (two background and seven organizational functioning) were found to be significant predictors of turnover prior to achieving EBP-competence. Of these nine measures, the most consistent predictor was program needs. Specifically, staff who reported greater “program needs” (e.g., program needs help increasing participation by clients, program needs help raising the overall quality of counseling) were significantly more likely to turnover prior to achieving EBP-competence at each of the three assessment points. Similarly, Fuller et al. (2007) found the TCU program needs scale to be a consistent positive correlate of staff attitudes toward treatment innovations (e.g., treatment manuals, medications, motivational incentives). Thus, it appears that at least based on the findings of Fuller et al. (2007) and this study, that although staff who perceive greater program needs also report greater attitudes toward EBPs, greater perception of program needs is predictive of greater likelihood of staff turnover prior to demonstration of EBP-competence.

Consistent with several meta-analyses findings based on turnover research (e.g., Griffeth, Hom, & Gaertner, 2000; Hom & Griffeth, 1995), we found the likelihood of turnover prior to EBP-competence was significantly lower for staff with higher reported levels of job satisfaction and higher perceptions of director leadership. Although consistent with the broader turnover literature, these findings are important, given that job satisfaction has not consistently been found to be a significant predictor of staff turnover within the context of implementing EBPs. For instance, although Sheidow et al. (2007) found lower job satisfaction to be predictive of turnover, a significant relationship between job satisfaction and staff turnover was not observed in the Aarons, Sommerfeld, et al. (2009) study. As noted by Aarons, Sommerfeld, et al., (2009), however, the lack of statistical significance between satisfaction and turnover in their study may have been due to either the covariance among predictors attenuating the relationship or this relationship being less salient compared to other predictors included in the model (e.g., turnover intentions). Thus, it is possible that job satisfaction may have had a significant direct relationship with staff turnover that was simply not detected.

Although only statistically significant at one of the three assessment points, it is notable that six months after attending the EBP training, staff reporting greater levels of burnout had significantly greater odds of turning over prior to achieving EBP-competence. This finding is important, given considerable turnover-related research in the substance use and mental health fields (e.g., Aarons, Fettes, et al., 2009; Broome et al., 2009; Ducharme, Knudsen, & Roman, 2008; Garner, Knight, & Simpson, 2007; Knudsen et al., 2006, 2008) has focused on examining correlates of emotional exhaustion (i.e., a dimension of the burnout concept), despite little empirical evidence that emotional exhaustion or burnout is a significant predictor of actual staff turnover among treatment staff.

Finally, relative to results from the two other known studies that have examined staff background factors as predictors of staff turnover within the context of implementing EBPs (Aarons, Somerfeld, et al., 2009; Sheidow et al., 2007), this study is the first to have found a significant gender difference related to turnover, with male staff being significantly more likely to turnover prior to achieving EBP-competence at both the six- and nine-month post-training assessment point. More research is needed to explore the relationship between gender and turnover. However, given many treatment organizations strive to have both male and female staff for staff balance and for matching with client needs, it is important to understand if there are aspects of learning or implementing an EBP that might be perceived differently by males and females and contribute to greater turnover by males.

Predictors of achieving EBP-competence and remaining employed

Across all three time points, only four staff factors (2 background and 2 organizational functioning) were found to be significant predictors of staff being able to achieve competence in the EBP and remain employed. Consistent with Schoenwald, Letourneau, & Halliday-Boykins’ (2005) findings regarding MST implementation, neither staff age, race, gender, nor education were found to be significant predictors of EBP implementation (i.e., model adherence). We found, however, that staff who had the most longevity at their organizations or held other treatment licenses or certifications were significantly less likely to have obtained EBP-competence at the nine-month post-training assessment point. These findings are consistent with the view that it may be easier to learn new practices when one does not have established procedures that must be changed to adopt an EBP (Liddle et al., 2002). As might be expected, we also found that staff who had greater perceptions of pressures for change had significantly higher odds of achieving EBP-competence at the nine-month assessment point. A somewhat unexpected finding, however, was that staff who reported having greater influence within their organization (e.g., viewed as a leader by other staff, influenced the decisions of other staff) had significantly lower odds of achieving EBP-competence at both six and nine months after the initial training. It is possible that these staff may have had more established counseling styles that were in conflict with the EBP. It also is possible that staff with greater influence had been in their position for a longer period of time and thus felt less organizational pressure for change. Indeed, we found a significant positive correlation between influence and months in position, but found no significant relationship between influence and pressures for change. Thus, additional research is needed to better understand the relationship between influence and achievement of EBP-competence.

Limitations

This study examined staff turnover and EBP-competence within the context of a federally funded national EBP implementation project. Thus, one potential limitation is that it is not known to what extent these findings would generalize to other settings or other EBPs. It is important to note, however, how similar the rates of turnover and EBP-competence found in this study were to rates in prior studies (Eby et al., 2010; Miller et al., 2004). Moreover, given the increasing efforts to implement EBPs in practice settings and that all of the staff in this study were employed by community-based organizations, we believe the study context has the strength of reflecting staff in usual practice settings. As noted previously, another limitation of the current study is that we do not know to what extent staff who left their organization did so involuntarily (as opposed to voluntarily). When possible, distinguishing between voluntary and involuntary turnover is useful, given there are likely very different reasons for and organizational impacts of these two types of turnover. A final study limitation, is that the number of exploratory tests increased the likelihood that some finding may be spurious. Thus, additional research is needed to replicate these findings.

Conclusions and future research

There are two conclusions of primary importance based on this study. First, although the majority of staff were able to demonstrate EBP-competence within 12 months of their initial training, nearly one in four staff left their organization prior to demonstrating EBP-competence. Thus, it appears that even within the context of a well-resourced implementation initiative that staff turnover can occur and adversely impact an organization’s return-on-investment regarding the resources (both financial and time) spent on training individuals. Second, there appear to be several staff factors that can be used to predict an individuals’ likelihood of being successfully trained and retained to deliver an EBP in practice settings. Thus, if selecting staff for an EBP implementation effort from within existing staff, it may be useful to assess staff perceptions of organizational functioning (e.g., program needs) and work attitudes (e.g., job satisfaction) to inform the staff selection process. In addition to examining the extent to which such staff factors are predictive of training and retaining staff to deliver other EBPs, future research is needed to develop practical criteria/guidelines for how best to use these staff factors to inform the staff selection process.

Figure 1.

Four-level classification of staff status at 6, 9, and 12 months post initial training workshop. Sample size decreases over time due to the fact that some staff were not able to be categorized at the 9- and 12-month time points because they either had not had enough time elapse post-training workshop or because their three-year grant ended before the time point.