The Impact of Evidence-Based Practice Implementation and Fidelity Monitoring on Staff Turnover: Evidence for a Protective Effect

Abstract

Staff retention is an ongoing challenge in mental health and community-based service organizations. Little is known about the impact of evidence-based practice implementation on the mental health and social service workforce. The present study examined the effect of evidence-based practice implementation and ongoing fidelity monitoring on staff retention in a children’s services system. The study took place in the context of a statewide regionally randomized effectiveness trial of an evidence-based intervention designed to reduce child neglect. Twenty-one teams consisting of 153 home-based service providers were followed over a 29 month period. Survival analyses revealed greater staff retention in the condition where the evidence-based practice was implemented along with ongoing fidelity monitoring presented to staff as supportive consultation. These results should help to allay concerns about staff retention when implementing evidence-based practices where there is good values-innovation fit and when fidelity monitoring is designed as an aid and support to service providers in providing a high standard of care for children and families.

Across organizational types, staff turnover increases costs and limits optimal organizational functioning. For example, turnover negatively impacts staff morale, short and long-term productivity, and organizational effectiveness (Gray, Phillips, & Normand, 1996) (Jayaratne & Chess, 1984; Mowday, Porter, & Steers, 1982). Turnover leads to poor work-team performance and production (Argote, Insko, Yovetich, & Romero, 1995). Public sector service organizations that provide mental health and social services have ongoing concerns about service provider retention with ramifications not only for the costs of recruitment and training, but also for the quality of products and services (Glisson, 2002; Knudsen, Johnson, & Roman, 2003). Turnover can be a particularly serious problem in human service agencies (Howard & Gould, 2000) where annual rates often exceed 25% (Gallon, Gabriel, & Knudsen, 2003) and in child and adolescent services where annual turnover rates can exceed 50% (Aarons & Sawitzky, 2006a; Glisson, Dukes, & Green, 2006; Glisson & James, 2002). Such turnover can impact the quality and outcomes of services provided by mental health and social service staff working with clients (Glisson et al., 2006).

In recent years child and adolescent mental health and social service agencies and programs have become increasingly interested in implementing evidence-based practices (EBPs) (Aarons, 2005). For the purposes of this study EBPs are defined as those behavioral and social service interventions that have sufficient support from well-conducted, rigorous research studies and allow for both clinical judgment and consumer choice, preference, and culture (APA Presidential Task Force on Evidence-Based Practice, 2006; Institute of Medicine, 2001).

Two common features of EBPs are a high degree of structure or manualization and the use of some form of monitoring to insure that the intervention is delivered with fidelity. Staff retention is a critical concern when considering EBP implementation because of the increased resources required for additional training and clinician support needed to promote adherence to particular intervention protocols. Although there have been studies of turnover in the context of EBP delivery (Sheidow, Schoenwald, Wagner, Allred, & Burns, 2007), a critical next step is examining the impact of EBP implementation and fidelity monitoring on staff retention in public sector service organizations, the subject of the current study. In the following sections we briefly review literature regarding factors pertinent to staff turnover, with particular attention to those factors most closely related to EBP implementation.

Staff Turnover and Organizational Change

Turnover

Turnover occurs when an employee actually leaves an organization (Knudsen et al., 2003). Turnover in response to organizational change frequently results in decrements in organizational performance (Baron, Hannan, & Burton, 2001). However, reductions in turnover can be accomplished when management decisions are sensitive to staff needs (Knowles, 1965). Service provider turnover rates in mental health and human service organizations are often high and are an ongoing concern in managing mental health case-management, clinical, and therapeutic services programs (Aarons & Sawitzky, 2006a).

The effects of turnover on clinical organizations can include poor staff morale, ineffectiveness of staff, reduced productivity, weaker work teams, and increased costs of training new employees (Bingley & Westergaard-Nielsen, 2004) as well as inconsistent services and weaker staff-consumer relationships (Albizu-Garcã­a, Rã­Os, Juarbe, & Alegrã­a, 2004). In public sector service organizations, turnover has been attributed to factors such as the high stress environment, lack of support, and low pay. A recent study in child and adolescent mental health agencies found that poor organizational culture and climate both contribute to lower job satisfaction and organizational commitment which, in turn, lead to higher staff turnover (Aarons & Sawitzky, 2006b). Another study reported similar results in youth social service settings (Glisson & James, 2002). Two related factors implicated in turnover are particularly relevant to EBP implementation and are reviewed below: organizational change and job autonomy.

Organizational Change

The implementation of an EBP and fidelity monitoring represent significant changes to organizational structure and process. Service providers who experience organizational change report that role changes may be inconsistent with their professional identity leading to negative effects on job satisfaction (Neuman, 2003). In another study, turnover was highest (39%) during the organizational change process, but turnover rates moderated during program stabilization (Gill, Greenberg, & Vazquez, 2002). Implementation of EBP generally requires organizational changes that impact how work is carried out as well as subsequent staff attitudes and behaviors (Aarons & Sawtizky, 2006b; Harris & Mossholder, 1996; Howard & Frink, 1996). Although models have been proposed to explain turnover (Chan, 1996; Lee & Ashforth, 1993), to our knowledge the effects of organizational change represented by EBP implementation and fidelity monitoring on staff retention have never been explicitly examined and the mechanisms by which staff retention can be maximized is an area in need of investigation (Aarons & Sawitzky, 2006b; Hoge, et al., 2005).

Job Autonomy

For the purposes of this study, job autonomy is defined as the degree of perceived control that an employee has over how they perform tasks and the degree to which they operate independently. Job autonomy varies across work contexts (Dobbin & Boychuk, 1999), has a direct effect on turnover intentions (Knudsen et al., 2003), and mediates the relationship between employment status, work attitudes, and performance (Marchese & Ryan, 2001). Common elements in EBP (conforming to rigorous standards, performance monitoring) appear likely to reduce job autonomy. Still, we do not know if the introduction of EBP and fidelity monitoring might alter staff perceptions of control over their work (Kennedy & Griffiths, 2003). However, a first step is to determine whether or not perceived job autonomy has an impact on staff turnover in the context of EBP implementation.

Turnover Intentions

Turnover intention is commonly defined as the degree to which employees are considering leaving their current job and/or are actively seeking another job. Turnover intention is related to organizational characteristics and to actual decisions to terminate employment. In addition, it is related to other withdrawal behaviors such as tardiness and absenteeism (Halfhill, Huff, Johnson, Ballentine, & Beyerlein, 2002). The relationships among these variables can be complex. For example, the effect of job satisfaction and organizational commitment on turnover intention may also vary by personal characteristics (Farkas & Tetrick, 1989). Turnover intention is a reliable predictor of actual turnover behavior (Knudsen, Ducharme, & Roman, 2007).

Fidelity Monitoring

Transitioning mental health and social service agencies into the implementation of EBP typically involves changes in organizational processes due, in part, to the need for adherence to a more structured service model. For example, successful translation of laboratory models of child therapy and social services into the field depends upon maintaining protocol adherence (Elliott & Mihalic, 2004). One example of this comes from ongoing efforts to disseminate the Multi-Systemic Therapy (MST) model for delinquency. Henggeler and colleagues (1997) have noted significant decrements in effectiveness, which were found to be associated with drift from the MST protocol, prompting dissemination efforts that emphasize fidelity monitoring. Fidelity monitoring and feedback facilitates the assessment of and compliance with adherence to an EBP. The current study investigated the effects of implementation and fidelity monitoring on job retention by systematically disentangling model (EBP or services as usual [SAU]) and monitoring (fidelity monitoring or no monitoring) effects in a child social service system implementing an EBP to reduce child neglect.

The Evidence-Based Practice: SafeCare® Intervention for Child Neglect

Most EBPs require a higher degree of structure relative to usual care. The EBP that is the focus of the present study is SafeCare®, a home-based intervention designed to reduce child-neglect and improve parent-child interactions (Gershater-Molko, Lutzker, & Wesch, 2003; Lutzker & Bigelow, 2002). SafeCare targets proximal parental maltreatment behaviors and was developed for and evaluated with multi-problem families with young children involved in the child welfare system. SafeCare (with origins in “Project 12-Ways”) grew out of the behavior analysis field, and is manualized, structured, and uses classic behavioral intervention techniques (e.g., ongoing measurement of observable behaviors, skill modeling, direct skill practice with feedback, training skills to criterion). SafeCare is comprised of three components derived from the original 12 units of Project 12-Ways: (a) infant and child health, (b) home safety and cleanliness, and (c) parent-child bonding.

Infant and child health

The goals of the infant and child health care component are to train parents to use health reference materials, prevent illness, identify symptoms of childhood illnesses or injuries, and provide or seek appropriate treatment by following steps of a task analysis (Lutzker & Bigelow, 2002). Parents are provided with a validated health manual that includes a symptom guide, information about planning and prevention, caring for a child at home, calling a physician, and emergency care. Health recording charts and basic health supplies (e.g., thermometer) are also provided.

Home safety and cleanliness

The home safety and cleanliness component teaches skills and uses structured checklists for identifying, correcting, and preventing environmental hazards, filth, and clutter.

Parent-Child Bonding and Parent Activity Training

The parent-child bonding and attachment component consists of parent-infant interaction training (birth to 8–10 months) and parent-child interaction training (8–10 months to 5 years). This teaches parents child engagement and stimulation activities, increases positive interactions between parents and their children, and increases time spent verbalizing with very young children (Lutzker, Bigelow, Doctor, & Kessler, 1998). Positive behaviors are reinforced and problematic behaviors are addressed and modified. Providers teach parents to use checklists to help structure their activities.

SafeCare has demonstrated support for parent behavior change and reduced recidivism across a series of studies including case studies, multiple baseline studies of behavior change, quasi-experimental recidivism studies, and randomized trials of behavior change (Gershater-Molko et al., 2003; Lutzker, 1984; Lutzker & Bigelow, 2002; Lutzker, Bigelow, Doctor, Gershater, & Greene, 1998; Lutzker & Rice, 1984; Lutzker & Rice, 1987). There is robust evidence to support the conclusion that SafeCare produces in situ behavior changes, which are in behavioral domains directly proximal to child maltreatment. Furthermore, these changes appear to be related to the activities of the home-based service providers, not merely the product of the materials alone. Together this pattern of findings suggests that the model, when delivered with fidelity, is responsible for the observed behavior changes.

The Present Study

The present implementation study (PI: Aarons) examined the impact of EBP implementation and fidelity monitoring on staff retention in the context of an effectiveness trial of a statewide system change and EBP implementation. The state authorities, in collaboration with academic researchers, selected SafeCare as an EBP to implement in the state child welfare family preservation/family reunification service system. The effectiveness trial (PI: Chaffin) used a 2 (EBP vs. SAU) by 2 (fidelity monitoring vs. no fidelity monitoring) design to examine intervention effectiveness and the impact of fidelity monitoring. The State of Oklahoma Child Welfare System is divided into six areas or regions for administrative purposes. Because of variation in area population characteristics, areas and sites were assigned to intervention conditions in a deliberate rather than a random manner. Such a logical approach allowed the investigators to come closest to obtaining comparable pre-intervention cell characteristics at the client level. Although this moves away from a classic randomized design, this approach solves some of the problems that randomization of sites would entail while preserving key aspects of a true experimental design. In particular, intervention conditions were assigned, not self-selected, and assignment was made independent of factors related to intervention condition. Teams within each condition (EBP or SAU) were then assigned to fidelity monitoring or no fidelity monitoring.

The SAU approach, while not manualized, included intensive case-management, identifying risk factors, facilitating access to services to address identified risks, family and social support, empowerment strategies, and didactic parenting education. Motivational Interviewing (Miller, 1996) and safety planning for domestic violence was integrated into the services statewide, but were the same across all conditions and not experimentally manipulated in the present study. The agencies in the regions selected to implement the EBP had previously been providing SAU, thus the implementation of EBP involved adding SafeCare to their model of care. All regions in the EBP condition received a week-long didactic and interactive training in SafeCare.

Fidelity monitoring and feedback was provided by professionals trained in the service model (EBP or SAU) who attended home-based services, observed sessions, and after the session, provided feedback and additional training as needed to providers in the fidelity monitoring conditions. The program referred to the fidelity monitors as “ongoing consultants” to enhance receptivity by home-based providers. Ongoing consultants were selected by the agencies and often were providers who were viewed as highly skilled by their colleagues and supervisors. Conscious efforts were made to address service providers’ receptivity and acceptability of the ongoing consultants. For example, to facilitate the providers’ perception of the consultant as a support rather than as someone to “monitor” them, consultants were presented as someone to help the providers learn how to deliver the SafeCare or SAU model with their complex families. The ongoing consultants were distinct from the organizational supervisor. Consultants were not to provide feedback about providers to agency supervisors, except in rare cases of ethical concerns. However, the consultants were indeed fidelity monitors in addition to supportive coaches and were trained in evaluating providers and turned in fidelity data for each session observed. Each provider was observed by their consultant for two sessions each month.

All ongoing consultants were trained in the provision of consultation utilizing Stoltenberg’s Integrated Developmental Model of supervision (Stoltenberg, 2005). Consultants were trained to adjust their provision of feedback to the professional developmental stage of the provider. Ongoing consultants were provided instruction and practice in rating providers, counseling, rapport building, and cultural sensitivity skills. The SafeCare ongoing consultants were additionally trained (practice by coding videotapes and feedback) in rating providers in provision of the three SafeCare modules using fidelity measures. Specific forms were developed (collaboratively with SafeCare intervention developers) for each SafeCare module focusing on each key component in the module, and were completed for every session observed.

The ongoing consultants in the two treatment conditions had slightly different roles. In the EBP group, the focus of the consultation was assisting the service providers in applying the SafeCare protocol in the home with their clients. The consultant also was available to model and assist with motivational interviewing and other issues related to the case. In the SAU condition, the role of the consultant was to aid in providing social support, case management resources, and didactic parenting, as well as to assist with motivational interviewing and general supportive counseling skills.

Given previous research results supporting the negative impact of organizational change and reduced job autonomy on staff retention, we proposed the following hypothesizes: 1) being in the EBP implementation condition would increase the risk of staff turnover, 2) fidelity monitoring would increase the risk of staff turnover, 3) there would be an additive effect such that employees in the EBP implementation + fidelity monitoring condition would have the highest risk of turnover, 4) lower perceived job autonomy would predict greater staff turnover, and 5) higher turnover intentions would predict greater staff turnover.

Methods

Study Context

Data used in the present study were collected over four waves of data collection across a 29 month period as part of a larger longitudinal study examining organizational factors likely to impact the statewide implementation of an EBP, SafeCare, throughout a statewide network of nonprofit organizations contracted with the Oklahoma State Children Services System. As part of this larger study, home-based service providers and supervisors employed by the contracted agencies were asked to complete periodic web-based surveys. The surveys took approximately 45–90 minutes to complete and response rates ranged from 90.2% to 96.8% (average 94.5%) over the four waves. The organizational participation rate was 100%. Participants received a written description of the study and informed consent was obtained prior to the survey. This study was approved by the appropriate institutional review boards.

Participants

A total of 153 home-based service providers were included in the analyses. Of these, 85.6% were female, 63.4% were Caucasian, 19.6% African American, 12.4% American Indian, and 4.6% Hispanic. At the time of their first survey, the mean age of the home-based providers was 36.8 years (SD=10.2). The highest educational attainment for the home-based providers consisted of high school graduate (0.7%), college graduate (41.8%), some graduate level education (25.5%), and master’s degree (32.0%). Their educational backgrounds included social work (39.9%), psychology (25.5%), human relations (13.1%), child development (7.2%), marriage and family therapy (5.2%) and “other” (8.2%). Mean job tenure at the time of the first survey was 31.1 (SD=36.7) months.

Measures

Provider Demographics

A provider survey incorporated questions regarding home-based provider demographics including age, sex, race, education level, and job tenure (Aarons, 2004). Sex was binary coded to indicate whether the home-based provider was female. For this study, race was treated dichotomously indicating if the home-based provider was Caucasian or non-Caucasian. Provider education was also collapsed into a dichotomous measure indicating whether the participant had received at least some graduate level education.

Job tenure represents length of employment (measured in months) and operates as the indicator of time in the analyses. Job tenure was determined by calculating the employment start date from the participant’s self reported duration of employment on their initial completed survey. Rather than a single continuous measure of job tenure, multiple job tenure intervals are utilized, with the first interval consisting of participant observations with job tenure ranging from 0 to 11 months, functioning as the reference category. This interval approach for handling analysis time allows for a flexible parameterization of the relationship between time (job tenure) and the hazard or risk of turnover since each interval can reflect a specific, unique hazard rate. This approach also minimizes the influence of outliers by collapsing the observations on the right tail of the job tenure distribution into one interval. A more fine-grained analysis utilizing 10 different time intervals did not substantively alter the model findings.

Job Autonomy

Job autonomy was assessed with items drawn from three sources, all with good psychometric and measurement characteristics: 1) three items from Knudsen, Johnson, and Roman (2003) adapted from previous work by Pritchard and Karasick (1973) with factor loadings ranging from .86 to .89; 2) four items from Wang and Netemeyer (2002) derived from the Job Diagnostic Survey (1976) and Spreitzer’s (1995) self-determination subscale (alpha=.75); and 3) four items from Marchese and Ryan (2001) adapted from Hackman and Oldham’s (1975) definition of autonomy with factor loadings ranging from .73 to .87 (alpha=.81). Example items include: “I can decide on my own how to go about doing my work” and “This job allows me to use personal initiative or judgment in carrying out the work.”

Work Attitudes

Work attitudes were assessed with two scales from the Children’s Services Survey (Glisson & James, 2002) that assess job satisfaction and organizational commitment. Job satisfaction is the degree to which respondents are satisfied with various aspects of their job (e.g., “How satisfied are you with your working conditions?”). Organizational commitment is the extent to which the respondent is committed to the agency (e.g., “I am proud to tell others that I am a part of this organization”). These scales have excellent psychometric properties and have been used in numerous studies in children’s mental health and social services (Aarons & Sawitzky, 2006a; Glisson & James, 2002; Schoenwald, Sheidow, Letourneau, & Liao, 2003). The scales have been shown to be related to organizational climate and culture and staff turnover (Aarons & Sawitzky, 2006a). In a study of cross-level effects in child and adolescent service organizations, coefficient alphas for job satisfaction and organizational commitment scales were good (.84 and .89 respectively) (Glisson & James, 2002). Consistent with prior research, these measures are included in the analyses as the average of the mean scores for each of the two subscales.

Turnover Intentions

Turnover intentions are defined as the degree to which the respondent intends to leave or stay at their organization (e.g., “I am actively looking for a job at another agency”) and were assessed with five items derived from organizational studies and adapted for use in human service agencies (Knudsen et al., 2003; Walsh, Ashford, & Hill, 1985). Respondents were asked questions about their intentions to leave or stay at their present job, measured on a five-point scale Likert scale. The scale has good factor structure and validity with factor loadings ranging from .82 to .88, and excellent reliability (5-items; alpha=.91).

Staff Turnover

For all home-based providers employed during at least one data collection wave from wave one (May, 2004) through wave three (February, 2006), we determined the presence and timing of participants’ turnover events (if any) by the start of the fourth survey wave (October, 2006). For employees not eligible for participation in subsequent survey waves due to leaving their agency, employment separation dates and departure reason (involuntary versus voluntary) were collected from the agencies and/or the employee.

With a primary study goal of examining employee volitional behavior during EBP implementation, voluntary departures represent the turnover event of interest. Our specific focus on voluntary turnover is consistent with a large body of prior research regarding the determinants and correlates of employee turnover (Griffeth, Hom, & Gaertner, 2000; Tett & Meyer, 1993).

Employment status changes involving promotions to supervisory positions or home-based provider transitions to other positions within the same agency not involving the direct provision of SafeCare removed the participant from continued observation as a SafeCare home-based provider, but were not considered to represent turnover events. Based on these definitions, 57 of the 153 home-based providers recorded a turnover event during the study period. Exploratory analyses using different turnover classification approaches (e.g. including involuntary terminations and internal transitions) are discussed in the Analyses section below. Turnover was indicated dichotomously with 1 representing participant turnover and 0 representing no turnover. The timing of turnover was measured in months.

Experimental Condition

A unique aspect of this research project is the 2 × 2 experimental design, in which EBP vs. SAU is crossed with the level of fidelity monitoring (monitored vs. non-monitored). In this study, there were 21 teams of home-based service providers operating in six regions covering the entire state, with approximately one quarter of the teams operating in each study condition. Efforts to promote stratified randomization of the study sites were pursued by purposefully distributing rural and urban regions within each study condition. For the analyses, models were developed to examine both the independent effects of EBP and fidelity monitoring on employee turnover as well as the interaction of EBP and monitoring conditions on employee turnover. The four experimental groups are defined as follows:

• SC/M - participating in SafeCare and receiving fidelity monitoring;

• SC/Non - participating in SafeCare but not receiving fidelity monitoring;

• SAU/M - services as usual and receiving fidelity monitoring;

• SAU/Non - services as usual/and not receiving fidelity monitoring.

Analyses

Survival analysis techniques are particularly suited to the study of employee turnover by allowing the timing of the event (or lack thereof) as well as any covariate changes over time to contribute information to the analyses. We utilized two different survival analysis approaches. First, we explored the bivariate relationship between the 2 × 2 experimental conditions and the timing of employee turnover as measured by job tenure (i.e., number of months working for the agency). This was accomplished by developing an estimate of the survivor function, S(t), which states the probability of a participant “surviving” with an agency past time t. More specifically, we use the Kaplan-Meier (KM) computation of the survivor function, which gives the product limit estimate of S(t) at any time t, as specified by:

$$\widehat{S}(t)={\displaystyle \prod _{j|t_j\le t}\left(\frac{n_j-d_j}{n_j}\right)}$$

where n_j represents the number of individuals at risk during time t_j and d_j represents the total number of failures at t_j (Cleves, Gould, & Gutierrez, 2004). The probability of surviving past a given time interval is determined by the probability of surviving that specific period multiplied by the cumulative probability of surviving up to the start of that specific time period. Therefore, the KM technique provides the running product of the probability of surviving for each successive time period. The resulting KM chart illustrates, at a bivariate level, whether employees in the four different experimental conditions exhibit different patterns of survival (or the inverse, turnover) in relation to job tenure. The KM function only utilizes information from those at-risk of turnover within a given time period (e.g. job tenure equal to “x” number of months) in the calculation of the survival probability for that specific time interval.

Next, we developed multivariate survival analysis models for hypotheses testing of the factors associated with the “hazard” or risk of employee turnover by utilizing a discrete-time, exponential proportional hazards modeling approach. Two important advantages of this parametric modeling strategy relative to semi-parametric approaches such as the Cox model, are that covariate information is utilized more efficiently resulting in improved estimates of covariate coefficients (Cleves et al., 2004) and that direct evaluation of the effects of analysis time on turnover risk are possible (Box-Steffensmeier & Jones, 2004). For this study, analysis time is represented by employee job tenure as indicated by time intervals reflecting the number of months since employment start date.

Another benefit of discrete-time parametric modeling is the ability to readily incorporate time varying covariates into the models. This aspect is particularly relevant as this study is relatively unique in capturing multiple assessments across time from home-based providers who persist from wave to wave. An important advantage of this approach is that the most temporally proximate known values of covariates (e.g. turnover intentions) are associated with each turnover event. More dynamic modeling of covariates using multiple observation points across time substantially increases the explanatory capacity of voluntary turnover models over models that only capture one initial baseline measurement from employees (Kammeyer-Mueller, Wanberg, Glomb, & Ahlburg, 2005). Following standard procedures, updated values are incorporated into the analyses by transforming the data from the level of individual participants into person-time records (Allison, 1984; Cleves et al., 2004; Willett & Singer, 1993). Since analysis time is measured in months, individual participants will have as many person-month records as the number of months under observation. The observation period for this study runs from May 2004 to October 2006, resulting in a maximum of 29 person-month records per participant. Updated values are recorded at the start of each wave and those values stay constant within a wave.

Consistent with survival analysis protocols for addressing subject “late entry” (i.e. participants who become at risk for an event prior to study observation) (Cleves et al., 2004) and prior employee turnover research (Hom & Kinicki, 2001), participants were only considered to contribute information regarding the risk of employee turnover following their initial observation. Therefore, the period of time that an employee worked for an agency prior to their first observation does not provide any information about the risk of turnover associated with that period of their job tenure. It is only after they have been observed that their job tenure informs the analysis of turnover and its relationship to job tenure more generally.

In order to explore the issue of classifying involuntary and internal transitions as turnover events, we conducted additional analyses in which we incorporated both involuntary turnover and internal transitions into our turnover event models, which increased the total number of turnover events to 77. Although these turnover events reflect three different types of turnover activity, with presumably overlapping, but different sets of relevant predictors, similar relationships to turnover were still evident between the four experimental conditions in this analysis as were found using models removing involuntary termination and internal transitions from the definition of turnover. However, job autonomy, while having the same direction of effects, was not statistically significant in the more inclusive turnover model. Therefore, because the primary outcome of interest (i.e., the relation of turnover to EBP and monitoring conditions) was similar we elected to consider involuntary agency exits and internal transitions as not representing turnover for the purposes of this study on employee response to EBP implementation and report results below for the originally proposed survival model.

Because home-based providers were nested in 21 different work teams there was a concern of potential within-group dependency. This potential heterogeneity based upon team membership is addressed in the survival analyses by incorporating a “shared frailty” approach that includes an extra term in the modeling equation to account for systematic variation in turnover propensity at the team level. This technique is analogous to random effect models that incorporate nested data structures into modeling strategies (Cleves et al., 2004). Given the similarities in the discrete-time exponential model approach to a logistic model, we compared our survival analysis results to a multi-level random intercept model that accounted for both multiple observations of the same individual (i.e. the number of person-months under observation) as well as the nesting of home-based providers within teams using Stata’s Generalized Linear Latent and Mixed Models procedures (StataCorp LP., 2005). The results from both models were essentially identical.

To utilize shared frailty techniques clustered at the team level, individual respondents must be associated with only one team for their entire time under observation. A small number of participants (n=10) changed from one team to another during the course of the study. For clustering purposes, these individuals were assigned to their final team so that turnover events would be associated with their team at the time of turnover. In analyses where we removed the ten home-based providers that changed teams, results were essentially identical. We conducted all analyses using the STATA 9 statistical package (StataCorp LP., 2005).

Results

Table 1 presents the proportions, means, and standard deviations for the study variables as measured at initial participation for the 153 home-based providers. The home-based providers included in this study contributed a total of 2293 observation-months and 57 instances of employee turnover. This equates to an overall turnover rate of .298 per observation-year ((57 events / 2293 observation months) × 12 months) or the equivalent of 29.8 turnover events for every 100 person-years observed. Turnover rates (i.e., rate per 100 person-years) are more easily interpreted as percentages and for each of the four experimental conditions were: SC/M = 14.9%, SC/Non = 33.4%, SAU/Non = 37.6% and SAU/M = 41.5%.

Table 1.

Sample Characteristics

	Nominal Variables		Continuous Variables
Characteristic	Value	%	Mean	S.D.
Gender
Male	0	14.4
Female	1	85.6
Race
Non-Caucasian	0	36.6
Caucasian	1	63.4
Education
Less than graduate	0	42.5
At least some graduate	1	57.5
EBP condition
Services as usual (SAU)	0	41.8
In SafeCare (SC)	1	58.2
Monitoring condition
No monitoring (Non)	0	50.3
Monitoring (M)	1	49.7
Indicators for 2×2 design
SC/M	1	28.1
SC/Non	1	30.1
SAU/M	1	21.6
SAU/Non	1	20.2
Age (years)			36.84	10.23
Job tenure (months)			31.05	36.67
Job Autonomy			5.50	.97
Turnover intentions			1.04	.95
Work attitudes			2.85	.67

Notes: N = 153. Data are from baseline survey. EBP = evidence-based practice; SC/M = participating in SafeCare and fidelity monitoring; SC/Non = participating in SafeCare, but not fidelity monitoring; SAU/M = services as usual and receiving fidelity monitoring; and SAU/Non = services as usual and not receiving fidelity monitoring. SC/M is the reference category in the analytic models.

Bivariate Survival Analyses

Figure 1 graphically illustrates the four separate Kaplan-Meier (KM) survival functions evident during the first five years of job tenure (60 months) for employees in each experimental condition. Differences in the survival patterns across the four experimental groups were assessed using multiple statistical tests (log-rank, Peto-Peto-Prentice, and Wilcoxon) that vary in their relative emphasis on early versus later failure times (Allison & SAS Institute, 2001; Kleinbaum, 1996). Since all three tests yielded similar results only the log-rank results are reported.

Figure 1.

Kaplan-Meier Survival Function Estimates (Retention Probability) by Study Condition. Note: SC/M = participating in SafeCare and fidelity monitoring; SC/Non = participating in SafeCare, but not fidelity monitoring; SAU/M = services as usual and receiving fidelity monitoring; and SAU/Non = services as usual and not receiving fidelity monitoring.

As shown in Figure 1, the results of the KM survivor functions suggest that home-based providers in the SC/M study condition had a greater likelihood of staying with their agencies for a longer period of time. For example, the probability of persisting with an agency for more than a year (12 months) is 86.2% for home-based providers in the SC/M group, whereas the probability of surviving past one year for the SC/Non, SAU/M, and SAU/Non groups were 61.4%, 76.2%, and 75.7%, respectively. This survival gap continues to increase such that 57.6% of those participating in the SC/M group are expected to persist past three years (36 months), but the SC/Non, SAU/M, and SAU/Non groups were again substantially lower at 21.9%, 27.3%, and 16.7%, respectively.

The log-rank test of survival function equality across the four different groups approached traditional significance levels (X² = 6.67, df = 3, p = .08). Based on the visually apparent differences between the SC/M group survival pattern and the other three groups, we conducted a second log-rank test that compared SC/M to the combined survival pattern of all other groups of home-based providers. The results of this statistical test confirmed that the survival pattern of the SC/M condition was significantly different from the combined survival curve for the other home-based providers (X² = 6.63, df = 1, p = .01).

Multivariate Survival Analyses

Table 2 presents the results of the multivariate exponential proportional hazards survival analyses for the three models that were tested. Model 1 represents a baseline model that only contains provider demographic variables and the time interval indicators controlling for differing durations of employment (job tenure). Model 2 adds multiple work-related provider attitude measures as well as individual indicators of the direct effects for the EBP and monitoring conditions. In contrast, Model 3 includes indicators representing the interaction of the two experimental conditions to create the four different study groups discussed above. The SC/M group operates as the reference category in Model 3.

Table 2.

Results from the Multivariate Exponential Proportional Hazards Survival Analyses

	Model 1				Model 2				Model 3
Variable	b	Std. Err.	HR	CI	b	Std. Err.	HR	CI	b	Std. Err.	HR	CI
Constant	−2.024	.742 **			−1.124	1.268			−2.047	1.253
Female	.042	.448	1.043	.434–2.510	.238	.449	1.269	.526–3.060	.257	.445	1.293	.540–3.096
Caucasian	−.136	.294	.873	.490–1.553	-.094	.295	.911	.511–1.622	−.101	.293	.904	.510–1.605
Graduate school	.476	.230	1.610	.895–2.897	.270	.320	1.311	.699–2.456	.343	.313	1.409	.763–2.603
Age	−.054	.018 **	.947	.915–0.980	−.055	.018 **	.946	.914–0.980	−.058	.018 **	.944	.911–0.978
Job tenure (12–23 mos.)	.505	.362	1.656	.815–3.366	.427	.360	1.533	.757–3.105	.474	.357	1.607	.798–3.237
Job tenure (24–35 mos.)	.326	.404	1.385	.628–3.057	.276	.399	1.318	.603–2.882	.331	.397	1.393	.640–3.032
Job tenure (36–54 mos.)	−.589	.543	.555	.192–1.608	−.720	.539	.487	.169–1.340	−.714	.536	.490	.171–1.399
Job tenure (55+ mos.)	−.096	.538	.909	.316–2.611	−.266	.536	.767	.268–2.191	−.214	.537	.807	.282–2.311
Job Autonomy					−.336	.168 *	.715	.514–0.993	−.353	.162 *	.702	.511–0.965
Turnover intentions					.361	.175 *	1.434	1.017–2.023	.358	.169 *	1.430	1.027–1.992
Work attitudes					.296	.313	1.344	.727–2.485	.278	.292	1.321	.746–2.339
SafeCare EBP					−.214	.293	.807	.455–1.433
Consulting					−.486	.304	.615	.339–1.117
SC/Non									1.087	.417 **	2.966	1.311–6.710
SAU/M									.918	.438 *	2.504	1.062–5.902
SAU/Non									.809	.451	2.246	.929–5.433

Notes: EBP = Evidence-based practice; SC/M = participating in SafeCare and monitoring (reference category); SC/Non = participating in SafeCare, but not monitoring; SAU/M = services as usual, but receiving monitoring; SAU/Non = services as usual and not receiving monitoring. Job tenure is included in the model via intervals of months, with the job tenure interval of 0–11 months operating as the reference category.

HR = relative hazard ratio; CI = 95% confidence interval for the hazard ratio;

^*p < .05

^**p<.01

As shown in Table 2, in all three models no significant or consistent patterns relating gender, race, educational attainment, or job tenure interval to employee turnover were observed. Model 1, however, shows that older age was associated with a reduced risk of employee turnover. As shown in Model 2, controlling for analysis time (job tenure), and a range of home-based provider characteristics and employment attitudes, neither the indicator of participation on a SafeCare team nor participation on a team receiving fidelity monitoring had a significant influence on employee turnover. However, significant differences between the four experimental conditions were evident in Model 3. Specifically, in comparison to the SC/M condition, the SC/Non condition demonstrated a higher risk of turnover (HR = 2.966; p < .01). The hazard ratio is interpreted similarly to an odds ratio and is an indicator of effect size. For example, the hazard ratio of 2.966 indicates an almost threefold greater risk of staff turnover for the SAU/M group relative to the reference group (i.e., SC/M). Similarly, home-based providers in the SAU/M condition demonstrated a higher risk of turnover relative to those who were in the SC/M group (HR = 2.504; p < .05). The hazard ratio for the final group, SAU/Non, was also elevated in comparison to the SC/M reference group (HR = 2.246), and was marginally significant (p = .07). In a separate analysis there was a 2.6 times greater likelihood of turnover for a combined SAU/M, SAU/M, SC/Non group relative to the SC/M group (p < .05; full model results available from the authors).

In both Models 2 and 3, greater perceived job autonomy predicted a reduced risk of employee turnover (p < .05). Additionally, higher turnover intentions were associated with a greater risk of turnover (p < .05) in both models. Employee age had a significant negative relationship with employee turnover in all models (p’s = .001) such that older employees had a reduced likelihood of leaving an agency. In analyses not presented here, a quadratic term for employee age was included to check for a potential curvilinear relationship between age and turnover risk, however no such relationship was identified. In all of the models, the variance of the shared frailty was non-significant indicating that there was no evidence of a significant clustering effect due to the nesting of providers within teams.

Discussion

The present study examined the separate and interactive impact of implementation of EBP and fidelity monitoring on staff retention. The study took place in the context of a statewide implementation of an EBP designed to reduce child neglect, SafeCare®. Contrary to predictions, EBP implementation along with ongoing fidelity monitoring (SC/M) predicted greater staff retention relative to all other experimental conditions except the SAU/Non condition (which approached statistical significance). These effects were significant even when controlling for effects of perceived job autonomy, turnover intentions, and work attitudes. Job tenure and work attitudes did not significantly predict turnover and collinearity may have attenuated relationships of some independent variables with the dependent variable. However, consistent with previous research, lower perceived job autonomy and higher turnover intentions predicted greater turnover.

A review of organization theory led to our hypotheses that decreasing job autonomy through the implementation of a more rigid work structure and fidelity monitoring, common characteristics of EBP, would lead to higher turnover intentions and poor staff retention. However, this was not the case in the present study. Indeed, EBP implementation and fidelity monitoring contributed to staff retention. It is likely that SafeCare is highly congruent with the philosophy and approach of the home-based service providers in this study, representing a good values-innovation fit (Klein & Sorra, 1996) and mitigating concerns regarding job-redesign. In addition, the fidelity monitoring in this project was a supportive/coaching model and this approach may have limited potential negative impacts of increased oversight. Further research is needed to examine the ways in which EBP and fidelity monitoring are implemented, delivered, and perceived, and how we can improve on intervention design, fidelity monitoring approach, and overall implementation approach in order to capitalize on these apparent protective effects.

The highest turnover rate was found in the EBP/non-monitored group. While further study is needed to examine this issue, it is possible that implementation of the EBP without ongoing consultation and support leads to a greater sense that the new service model is just another change or demand with attendant paperwork and administrative demands. In addition, there was no opportunity for providers to have help in gaining expertise in the model to support an increasing sense of mastery, competency, and self-efficacy. Indeed, training without support could lead to reductions in perceived mastery and self-efficacy. In addition, much of the Oklahoma service area is extremely rural and some providers in the more rural areas are quite isolated with little peer or supervisor interaction. The ongoing consultants play a key supportive role and this may be even more important in rural areas.

The critical factors that differentiate providers in the SafeCare condition who received consultation from those who did not are a deeper understanding of the EBP and how to apply it flexibly to a variety of families, having someone with whom to discuss emergent issues related to the case, and a general sense of support. When providers are given initial training, but not ongoing consultation, the model may be more likely viewed as a rigid mandate rather than a flexible tool to help families (i.e., less autonomy with little perceived benefit), and helping families is a primary motivation that brings service providers to this type of work. In contrast, when ongoing consultation is present the provider is better able to understand how to flexibly apply the model with a variety of families while remaining adherent to the model. While the provider may have somewhat less job autonomy, this is likely mitigated because the support of the consultant and flexibility of the model are perceived as additional factors that increase their ability to attain core job goals (i.e., to help families).

The momentum for transition into EBP implementation in mental health and social services agencies in the United States and abroad has been wrought with challenges. Agency and provider buy-in to EBP models of services is a repeated barrier to implementation, often related to concerns about training demands, impact on work load, and perceived reduced flexibility in services, which in turn could increase job turnover. In some cases such concerns can lead organizations to “de-adopt” innovative service models (Massatti, Sweeney, Panzano, & Roth, 2008). Recent studies have found serious workforce problems in mental health and social service fields (Hoge et al., 2005). Indeed, leaders in the agencies involved in this study initially expressed worry about staff turnover due to many areas having a limited qualified employee pool. However, agency directors and managers recognize that for current staff, implementation of an EBP should provide tangible benefits (Palinkas & Aarons, In press). The results of the present study are promising in that over time, when an efficacious EBP provides a good fit with needs of agencies, service providers, and clients, and when fidelity monitoring is delivered as ongoing support and consultation, the climate for implementation (Klein & Sorra, 1996) is improved along with job retention. For the organization, future cost effectiveness research should examine the relative cost savings of job retention to balance out the additional initial costs required for training and fidelity monitoring. In the present study, the fidelity monitor was separate personnel than the agency supervisor. Future research should examine the potential integration of fidelity monitoring and feedback activities in the role of the supervisor, providing additional cost savings.

Limitations

One potential limitation of the parametric modeling strategies is the possibility of incorrectly specifying the relationship of analysis time to the baseline hazard. To mitigate this concern, an exponential modeling approach was adopted that includes separate indicator variables for different time intervals in the model. This technique requires minimal a priori assumptions regarding the shape of the baseline hazard by allowing the hazard to fluctuate freely within each time category (Box-Steffensmeier & Jones, 2004). Additionally, comparable multivariate Cox models that require no distributional assumptions about the relationship between analysis time and failure risks were also assessed. The results from the Cox model were essentially identical to those of the discrete-time exponential model.

The initial training and implementation of SafeCare and monitoring began prior to the first wave of data collection for this implementation study. In this regard, the current study may have greater relevance to our understanding of ongoing implementation strategies and institutionalization of EBP within social service and mental health systems and agencies rather than the initial effects of EBP adoption and implementation on staff retention.

This study examined the impact of EBP and fidelity monitoring with home-based service providers in child social services agencies. The pattern of results may differ in agencies with higher educated and licensed service providers. However, a recent study suggests that there may be even greater openness to EBPs among mental health service providers in other settings (Stahmer & Aarons, In press).

Future Studies and Implications

While not directly assessed in the present study, it is likely that potential negative effects of the increased structure of the EBP were mitigated because SafeCare adds structure that directly supports the mission of home-based service providers. The construct of values-innovation fit posits that implementation of innovation will be successful to the degree that the innovation matches the values (e.g., theoretical orientation, organizational mission) of the organization and individuals within that organization (Klein & Sorra, 1996). Clinical training is highly variable and when service providers are practicing in the field they may carry perspectives and theoretical orientations that may or may not be congruent with EBP models. Research has found that providers may be more accepting of an EBP when they are earlier in their career (Aarons, 2004) and when the EBP fits with client needs (Aarons & Palinkas, 2007). Further study is needed to investigate whether or not similar effects can be found with other types of EBPs in other service settings.

Another critical issue in EBP implementation is how best to conduct fidelity monitoring in real world service settings and in a way that is seen as supportive rather than punitive. In the present study, fidelity monitoring was framed and delivered as a coaching model of ongoing consultation. In this approach consultation was provided in a supportive way with a learning orientation to promote skill building and excellence. In addition, general support (e.g., brainstorming, troubleshooting, social support, camaraderie) was provided for home-based workers (who often work in isolation) in addition to coaching on specific elements of SafeCare. This is akin to a more global quality improvement approach that also has specific behavioral and performance targets.

The impact of EBP implementation on staff retention has been understudied and while some studies have examined methods of fidelity monitoring for clinical purposes (Baer et al., 2007; Essock, Covell, Shear, Donahue, & Felton, 2006), the consultation/coaching model appears to hold promise for positive organizational outcomes. The findings presented here suggest that in some settings supportive coaching models may be preferable to checklists or less engaging methods of fidelity monitoring.

There is evidence that organizational factors are associated with outcomes of services (Glisson & Hemmelgarn, 1998). We followed this line of research in further examining the association of organizational change on staff retention in a statewide system change and EBP implementation. The results of this study address the real-world implications of EBP implementation and are relevant for national, state, and local service systems and organizations and also for clinical practice. Retention of a well-well trained workforce provides benefit to both organizations and the clients that they serve. The present study supports new optimism that implementation done well, can lead to both positive organizational outcomes and ultimately to better client outcomes.