Evaluation of an Implementation Initiative for Embedding Dialectical Behavior Therapy in Community Settings

Abstract

We examined the effectiveness of Dialectical Behavior Therapy (DBT) training in community-based agencies. Data were gathered at four time points over a two-year period from front-line mental health therapists (N = 64) from 10 community-based agencies that participated in a DBT implementation initiative. We examined change on therapist attitudes towards consumers with Borderline Personality Disorder (BPD), confidence in the effectiveness of DBT, and use of DBT model components. All measures were self-report. Participating in DBT training resulted in positive changes over time, including improved therapist attitudes toward consumers with BPD, improved confidence in the effectiveness of DBT, and increased use of DBT components. Therapists who had the lowest baseline scores on the study outcomes had the greatest self-reported positive change in outcomes over time. Moreover, there were notable positive correlations in therapist characteristics; therapists who had the lowest baseline attitudes towards individuals with BPD, confidence in the effectiveness of DBT, or who were least likely to use DBT modes and components were the therapists who had the greatest reported increase over time in each respective area. DBT training with ongoing support resulted in changes not commonly observed in standard training approaches typically used in community settings. It is encouraging to observe positive outcomes in therapist self-reported skill, perceived self-efficacy and DBT component use, all of which are important to evidence-based treatment (EBT) implementation. Our results underscore the importance to recognize and target therapist diversity of learning levels, experience, and expertise in EBT implementation.

Expert panels have recommended incorporating evidence-based treatments (EBTs) into standard clinical practice, calling it a priority for improving the quality of mental health services (President’s New Freedom Commission on Mental Health, 2003). These efforts are particularly important for the public mental health sector (Adelmann, 2003; Mental Health: A Report of the Surgeon General, 1999), which serves individuals with severe and chronic mental health disorders (Adelmann, 2003), yet only 10% of public health systems deliver EBTs (Rones & Hoagwood, 2000). A number of factors have been identified that contribute to the success or failure of implementation efforts (Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010) including positive stakeholder attitudes towards EBTs, therapist professional background (degree type), organizational culture and climate, agency resources (financial, including post-training funding, leadership), and training strategies.

The field has highlighted the need for effective training strategies, but there is a lack of both comprehensive guidelines to support the transfer of EBTs to community therapists (McHugh & Barlow, 2010) and empirical information regarding effective knowledge and skill transfer (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Gotham, 2004). There is a particular paucity of data about how to most effectively train those who provide care in community settings (Herschell et al., 2010). To date, the most common way to train community therapists in EBTs has been to ask them to read written materials (e.g., treatment manuals) or attend workshops, but there is little to no evidence that this ‘train and hope’ approach (Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002), similar to continuing education formats, will result in positive, sustained increases in skill and competence (Beidas & Kendall, 2010; Herschell et al., 2010). More extensive training models that include multiple training days with time in between for therapists to practice skills with consumers and receive feedback from experts through coaching or consultation seem to be necessary (Beidas & Kendall, 2010; Herschell et al., 2010; Sholomskas et al., 2005). There also is often a need for considering the organizational context (e.g., culture, climate, resources, leadership engagement) in which the intervention will be implemented (Damschroder et al., 2009).

One example of a therapy for which a comprehensive training approach has been developed is Dialectical Behavior Therapy (DBT; Linehan, 1993a, 1993b). DBT is a Cognitive Behavioral Therapy that has been identified as an EBT for individuals diagnosed with Borderline Personality Disorder (BPD; Kliem, Kröger, & Kosfelder, 2010; Lynch, Trost, Salsman, & Linehan, 2007). DBT has been shown to improve outcomes for individuals with emotion regulation difficulties in adolescence through adulthood (e.g., Groves, Backer, van Der Bosch, & Miller, 2011) across disorders (e.g., Bipolar Disorder; Eating Disorders; Bankoff, Karpel, Forbes, & Pantalone, 2002) and settings (e.g., Dimeff & Koerner, 2007; Ritschel, Cheavens, & Nelson, 2012). Effectiveness trials (e.g., Pasieczny & Connor, 2011) and multiple efficacy trials support the effectiveness of DBT in diverse settings.

DBT is principle-based and includes specific modes and components (Linehan, 1993a). Specific DBT treatment modes include individual outpatient psychotherapy, group-based skills training (e.g., mindfulness, distress tolerance, emotion regulation, interpersonal effectiveness), telephone consultation, and case consultation meetings for therapists. Specific treatment components include core strategies (validation, problem solving, behavior therapy, dialectics), consumer-oriented therapy agreements (e.g., attendance, suicidal behavior, therapy-interfering behavior, skills training), therapist-oriented agreements (e.g., “every reasonable effort,” ethics, personal contact, respect-for-consumer), treatment targets (e.g., decreasing life-threatening behavior such as suicide behavior and self-harm behavior; decreasing therapy-interfering behavior such as non-attendance; decreasing behaviors that interfere with quality of life such as heavy alcohol use; and increasing skills), and monitoring of treatment targets (daily diary card). The DBT protocol also acknowledges the occasional need for ancillary treatment (e.g., medication management, vocational rehabilitation). While some studies have found benefits for using selected modes (individual only rather than combined individual and group: Andion et al., 2012; group only rather than group in addition to other modes: Blackford and Love, 2011) or components of DBT (e.g., Salamin, Guenot, Bénon, Walther, & Surchat, 2011), the largest empirical base and assumed optimal outcomes are found for DBT when it is implemented in its entirety (rather than only implementing selected modes or components).

The primary population for whom DBT was developed and has accumulated an evidence base (consumers with BPD) is one that has been described as difficult to treat and has experienced antagonistic judgments from professionals (e.g., Bodner, Cohen-Fridel, & Iancu, 2011; Bourke & Grenyer, 2010). Surveys have demonstrated professionals’ negative feelings toward (Westwood & Baker, 2010) and reluctance to treat consumers with BPD (e.g., Jonst, Horz, Birkhofer, Martius, & Rentrop, 2010). In fact, targeted trainings have been developed to improve professionals’ attitudes toward and confidence in treating consumers with BPD (e.g., Krawitz, 2004; Shanks, Pfohl, Blum, & Black, 2001). There often is a need to change therapists’ attitudes about consumers with BPD before an EBT for consumers with BPD can be implemented.

While attitudes toward and confidence in treating consumers with BPD cannot predict professionals’ behavior, positive attitudes have been described as fundamental to high quality treatment of consumers with BPD (e.g., Ma, Shih, Hsiao, Shih, & Hayter, 2009; Woollaston & Hixenbaugh, 2008). The DBT model recognizes the importance of this through highlighting the need for a strong therapeutic relationship between the therapist and consumer; the impact of the therapist on the consumer (e.g., therapist-interfering behavior) and the necessity of support for therapists working with consumers with BPD (Linehan, 1993b). DBT also recognizes that a therapist cannot have a negative opinion of a consumer and be helpful (i.e., genuine, validating) at the same time.

DBT has been widely disseminated, with qualitative (e.g., Herschell, Kogan, Celedonia, Gavin, & Stein, 2009) and quantitative studies (e.g., Dimeff et al., 2009; Dimeff, Woodcock, Harned, & Beadnell, 2011) examining issues related to implementation (Ben-Porath, Peterson, & Smee, 2004; Dimeff et al., 2009; Dimeff et al., 2011; Frederick & Comtois, 2006; Hawkins & Sinha, 1998; Herschell et al., 2009). Topics have included training methods for mental health professionals before (Frederick & Comtois, 2006) or after (Dimeff et al., 2011; Hawkins & Sinha, 1998) completion of their terminal degree as well as factors that facilitate or impede implementation (Herschell et al., 2009; Van den Bosch, Verheul, Schippers, & van den Brink, 2002). Specific DBT training methods that have been studied include: self-study (Dimeff et al., 2011), two-day workshops (Dimeff et al., 2009), electronic-learning (Dimeff et al., 2009; Dimeff et al., 2011), a residency program (Frederick & Comtois, 2006), and multi-component implementation processes (Hawkins & Sinha, 1998).

The training approach recommended for DBT is the DBT Intensive Training Model (ITM; Landes & Linehan, 2012), developed iteratively from 1991 (the publication of the first Randomized Controlled Trial demonstrating DBT’s efficacy) until now. This extensively-used model includes two 5-day trainings separated by a 6 month self-study and trial implementation, team building, contingency management procedures, and targeted coaching on specific strategies to reduce barriers to full DBT implementation.

Since 1993, ITM has been used to train 600 teams in 19 countries (Landes & Linehan, 2012). Annually, 5 to 8 ITM courses are offered in the United States and 9 to 10 are conducted internationally. The model has been included in efficacy and effectiveness trials to train study therapists (Koons et al., 2001; Trupin, Stewart, Beach, & Boesky, 2002; Verheul et al., 2003) where positive client outcomes have been obtained.

However, there has been only one empirical examination of the ITM itself. This study, similar to the current study, examined implementation of the full DBT model with community-based mental health professionals participating in a state initiative. Hawkins and Sinha (1998) evaluated the conceptual mastery of therapists trained using the recommended multi-component training protocol (ITM; Landes & Linehan, 2012) within a State Department of Mental Health Initiative. Using a knowledge questionnaire administered to participants at varying points within the training, results indicated that 1) therapists with diverse training backgrounds and disciplines were able to acquire a sophisticated understanding of DBT; 2) the sophistication of knowledge acquired correlated strongly with the amount of training received; 3) reading, peer support, consultation, study group attendance, and time spent applying treatment were all important components of training; and 4) learners benefited most from expert consultation after acquiring a substantial DBT knowledge base.

The present study is an empirical examination of a real-world DBT implementation initiative launched by a partnership among a large managed-care behavioral health organization, four Eastern Pennsylvania counties, and 10 community mental health centers. This study is only the second of its kind for DBT. It adds to and extends the growing field of research on factors influencing EBT implementation in real-world settings. More specifically, this study offers several unique contributions and improvements. First, the implementation method studied was not influenced by the research team. This was a real-world examination of the type of training that is recommended by the DBT developer (Landes & Linehan, 2012) rather than an experimental manipulation. Second, the research team was independent of the trainers and training group. Third, implementation included the full EBT model, rather than components. Fourth, data were collected over multiple time points (4) and across an extended period of time (2 years). Fifth, data were gathered on multiple constructs (therapist attitudes towards consumers with BPD, confidence in the effectiveness of the DBT model, and use of DBT model components) rather than only one construct (e.g., therapist knowledge; Hawkins & Sinha, 1998). Specific attention was devoted to examining the reported change in these variables over the course of the multi-component implementation initiative.

Two aims guided the research: 1) to evaluate the effectiveness of the implementation process on key outcome variables including therapist attitudes towards consumers with BPD, confidence in the effectiveness of the DBT model, and use of DBT model components, and 2) to understand practitioner background characteristics that might impact key outcome variables. It was anticipated that key outcome variables would show improvement and that therapist background characteristics (e.g., years of experience, degree type) would impact key outcome variables. Given that previous literature has produced mixed findings, directionality was not predicted for associations between therapist background characteristics and outcome variables.

Method

Participants

Sixty-four therapists from 10 previously mentioned community-based agencies were invited to participate in the study. After obtaining University of Pittsburgh Institutional Review Board approval, agencies provided names and contact information of expected trainees, who were then invited to participate.

Procedure

DBT implementation process

The implementation process was facilitated by two professional trainers employed by a well-established and respected DBT training company, Behavioral Tech, LLC (www.behavioraltech.org). The implementation process took place over 18 months from October 2006 to April 2008 and utilized the DBT recommended training method (ITM; Landes & Linehan, 2012). Implementation began with a two-day clinical and a one-day administrative and treatment overview (October 2006), which were attended by therapists, administrators, and county leaders to help participants better understand the DBT model and the implementation process (e.g., training schedule and content, potential changes to service delivery). This first session (October 2006) was meant to be a precursor, rather than a part of the formal clinical training. Essentially, this session was meant to introduce DBT to the agencies and counties so that they could be more prepared for the actual trainings. Afterward, agencies then selected therapists to attend the full clinical training, which included two five-day clinical training sessions (February 2007, August 2007), one two-day clinical training session (April 2008), and extensive phone consultation. Trainees participated in group and individual activities; lecture style presentations were mixed with behavioral rehearsal opportunities; and skills were explained, modeled, and role-played by participants within training sessions. Additionally, participants received feedback on skill implementation with clients through phone consultation that occurred approximately weekly from February 2007 (after the first five-day training) through April 2008 (the final two-day training). Each participating agency was assigned one trainer who provided group phone consultation sessions.

Therapist selection for training

After obtaining in depth information on DBT and its implementation during the October 2006 meeting, agency administrators were responsible for selecting which therapists from their organization they would send to training. Administrators commonly described carefully selecting therapists for their DBT teams, weighing factors such as volunteers for the program, therapist seniority, credentials, familiarity with DBT, the agency’s staffing needs, diversity, and those more likely to remain with the agency (trying to guard against staff turnover). A separate paper (Herschell et al., 2009) provides a more detailed description for how administrators selected therapists for training.

Data collection

Surveys were administered to participating therapists at the in-person training sessions (February 2007, August 2007, April 2008) and then in December 2008/January 2009. Time 1 was administered February 2007 before the initial 5-day training. This served as a baseline, providing information before therapists received training in DBT. Time 2 was administered August 2007, six months after the initial training and immediately before the second 5-day training. At that point, therapists had received one 5-day training and six months of phone consultation. Time 3 was administered April 2008, fourteen months after the initial training and immediately before the 2-day training. Training concluded April 2008 so Time 3 was meant to provide post-training data. Time 4 was administered December 2008/January 2009, approximately 22 months after the initial training and eight months after completing training. Intervals between assessments from baseline were 0, 6-, 8-, and 8-months, which coincided with the training schedule.

For the first three data collection points (Time 1, 2 & 3) surveys and an accompanying cover letter explaining the study were mailed or e-mailed to the therapists two weeks before each training session, asking them to mail completed surveys directly to the first author (ADH) or hand in a completed survey to a research team member (KLC) at the sign-in table on the first day of each session. The research team member was on site at each training to provide information about the study, answer questions, encourage participation, and provide copies of the survey to therapists who forgot their surveys or had not received one prior to the training. For the final data collection point (Time 4, a follow-up time point) surveys and an accompanying cover letter explaining the study were mailed or e-mailed to the therapists, and therapists were asked to return the survey via mail (self-addressed, stamped envelope was provided), email, or fax to the first author (ADH). The procedure varied for Time point 4 because training concluded with time point 3; at time 4 there was not a training session available to collect data. Therapists returning surveys were entered into a drawing for a prize (e.g., gift cards and baskets), which varied for each of the four data collection points.

Therapists could participate in the training without participating in the study. Therapist study response rates for the four training sessions were 92%, 91%, 90%, and 76%, respectively. Of these four data collection points, only one person directly declined participation. Others were considered to have declined by failing to respond to multiple reminders.

Measures

Training dose

Detailed records were kept on participant attendance at training sessions. Therapists were asked to report on their phone consultation attendance.

Therapist survey

The therapist survey was developed by the research team for this study because a previously developed, reliable and valid measure could not be found. The developed therapist survey tapped three primary areas: 1) Confidence in the effectiveness of DBT, 2) Use of DBT Components, and 3) Attitude Towards Consumers with BPD. Survey questions were developed from a systematic review of the literature, and were revised based on feedback from DBT experts as well as managed behavioral health organization and county mental health administrators.

The Confidence in the effectiveness of DBT scale was comprised of 15 items measuring therapists’ confidence that DBT would be effective in helping their clients with BPD. Therapists were asked to respond to the following question “What is the likelihood that appropriate use of DBT for treating borderline personality disorder will be effective in achieving each of the following goals for your clients?” for each of 15 outcomes (e.g., reducing suicide attempts, holding a part-time job, having close and supportive friends, improving emotion regulation, reducing crisis treatment utilization, improving family relationships). Participants rated the likelihood from 1 (very unlikely) to 5 (very likely). The scale had excellent reliability (Cronbach’s alpha = .91).

The Use of DBT Components scale was comprised of 9 items measuring therapists’ self-reported use of DBT model-specific components in their clinical work with consumers with BPD. For each of 9 components (e.g., behavior therapy; consumer agreements; daily diary cards; dialectic strategies; problem solving; protocol around suicidal behavior; therapist agreements; treatment targets; validation) participants were asked to rate “to what extent do you use the following types of services and treatment components for consumers with borderline personality disorder” on a five point scale from 1 (never) to 5 (most of the time). Additional items were included in this list that would not necessarily be considered DBT specific (e.g., case management, peer counseling) to guard against response bias. The scale had acceptable reliability (Cronbach’s alpha = .77).

The Attitude Towards Consumers with BPD scale was comprised of 6 items measuring both positive and negative attitudes towards consumers with BPD. Participants were asked to rate their agreement with specific statements on a five point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Two items included positive attitudes (I think consumers with BPD are doing the best they can; Treating consumers with BPD can be rewarding). Four items were reverse scored and tapped negative attitudes (mental health treatment rarely benefits consumers with BPD, Consumers with BPD have a poor prognosis, The way that consumers with BPD relate to others represent fixed character traits, if I had a choice, I would refuse to work with consumers with BPD). The scale had borderline reliability (Cronbach’s alpha = .68).

Additional questions also assessed therapist background, training experiences, and demographics.

Data Analyses

For primary analyses, we used hierarchical linear modeling (HLM; Raudenbush & Bryk, 2002) with time (assessment) nested within participants. We used full maximum likelihood estimation for all models. The level 1 equation for all models was Y_ti= π_0i+ π_1i(assessment) + e_ti, where Y_ti is the observed outcome score at session t for participant i. Each of the four assessments was coded 0, 1, 2, and 3. This allowed for a straightforward interpretation of the level-1 intercepts as the average score at the first assessment. First, we examined change in our primary outcome variables by running linear growth models that were unrestricted at level-2. This allowed us to test whether there was significant change over the course of the training.

Next, we examined individual differences in change over the course of the training for each of our outcome variables. The professional background variables were entered into the models as level-2 predictors. These variables included education, years of experience, degree area, and professional licensure. Our education variable was coded as “Master’s Degree or Higher” = 1 vs. Bachelor’s Degree or Lower” = 0. Professional licensure was coded “Yes” = 1 vs. “No” = 0. A set of dichotomous contrasts were created to examine degree area. These were “Psychology Degree” = 1 vs. “Non Psychology Degree” = 0, “Social Work Degree” =1 vs. “Non Social Work Degree” = 0, and “Psychology or Social Work Degree” = 1 vs. “Other Degree” = 0. We tested each of these variables as level-2 predictors in separate models. We also examined whether change in our outcome variables was associated with the degree to which participants reported changes in their job descriptions. To test this, our Job Change variable was entered as a level-2 predictor variable.

We also examined if baseline (first assessment) levels of outcome variables predicted change over the course of the training. To test this, we examined the bivariate correlations between the intercepts and slopes of our level-1 models. A positive correlation indicated that participants with higher baseline scores improved more over the course of the training than participants with lower baseline scores. Conversely, a negative correlation indicated that participants with lower baseline scores improved more over the course of the training than participants with higher baseline scores.

Finally, we assessed associations among attitudes towards consumers with BPD, confidence in the effectiveness of DBT, and use of DBT components, focusing on whether improvements in attitudes towards consumers with BPD and increased confidence in the effectiveness of DBT were associated with greater use of DBT components. To test this, we examined the bivariate correlations between the slopes of these variables from the level-l models. Each slope represents change over time (the course of training). Positive correlations indicate that an increase in one variable was associated with an increase in the other variable.

Follow-up repeated measures ANOVAs were conducted to determine differences in DBT-Specified Modes, DBT-Specified Components, and Ancillary Treatment use pre and post training. These analyses were only conducted for training completers. DBT-Specified Modes were defined as types of treatment that are part of the DBT model as articulated by the developer in the original DBT text (Linehan, 1993a) individual therapy, skills training, groups, and 24-hour phone consultation). DBT-Specified Components were defined as important therapeutic elements of the DBT model as specified by the developer (Linehan, 1993a; e.g., validation, problem solving, daily diary cards). Ancillary treatments are not part of DBT model, but it is recognized that in treating such a diverse and complex population, that additional supports like these ancillary treatments may be needed.

Results

Participant Characteristics

78 percent (n=50) of the therapists were female (Table 1). The average age was 44 years (SD=13.5 years) and 95% were Caucasian. The majority of the therapists were Master’s level (70%); 14% had a Bachelor’s degree. Fifty-two percent were licensed, with licensed professional counselor (LPC; 39%), social work (LSW or LCSW; 36%), psychologist (13%), and registered nurses (RN; 13%) being the most common licenses. Therapists averaged approximately 14 years of experience in human services work (M=13.81, SD=9.4) and six years at their present agencies (M=6.2, SD=6.1).

Table 1.

Demographic Characteristics of Participants

	Total Sample (n = 64) n (%) or M (SD)
Gender (n=64)
Male	14 (21.9%)
Female	50 (78.1%)
Age (n=59)	44.1 (13.5)
Racea(n=64)
American Indian/Alaska Native	0.0
African American	0.0
Asian	2 (3.1%)
Caucasian	61 (95.0%)
Hispanic	1 (1.6%)
Native Hawaiian or Pacific Islander	0.0
Highest Degree (n=64)
Some college	1 (1.6%)
Bachelor’s degree	9 (14.0%)
Some graduate school	6 (9.4%)
Master’s degree	45 (70.3%)
Doctoral degree	3 (4.7%)
Years experience
In full-time human services work (n=54)	13.81 (9.4)
At present agency (n=58)	6.2 (6.1)
Highest level degree is in: (n=64)
Education	2 (3.2%)
Nursing	4 (6.3%)
Psychology	22 (34.9%)
Social Work	16 (25.4%)
Other	19 (30.2%)
Professionally licensed (n=63)	33 (52.4%)
License type (n=31)
Licensed Professional Counselor	12 (38.7%)
Social Work	11 (35.5%)
Psychologist	4 (12.9%)
Registered Nurse	4 (12.9%)

Training dose

The amount of time spent in training sessions ranged from 32 to 96 hours (total possible number of hours=96) with a Mean of 74.0 hours (SD=23.45); 70% of participants who had the opportunity to complete all trainings completed at least 75% of the training. Phone consultation time ranged from 6 to 110 hours with a Mean of 25.67 hours (SD=30.67); 13% of participants who had the opportunity to complete all consultation completed at least 75% of the consultation.

Therapist turnover and study retention

Nine of the 10 agencies that began the study remained involved through the conclusion of the four assessments. One agency dropped out because of substantial changes in management and organizational structure. The number of therapists steadily declined through the course of the study due to therapists leaving their agencies (time point 1: n=64; time point 2: n=50; time point 3: n=41; time point 4: n=35). By the end of the training initiative, 35 of the original 64 therapists (55%) remained at their agencies. There was some effort to replace therapists who left their agencies within the DBT training initiative so that at the last time point 38 therapists were involved in DBT training. Study completers (those who were present at the final training session; n=29) did not differ significantly from the non-completers (n=35) on any of the background variables presented in Table 1 except for “Years experience in full-time human services work.” Completers were more experienced (M=18.5 years) than non-completers (M = 10.4 years), F(1,52) = 11.80, p = .001.

Attitudes Towards Consumers with Borderline Personality Disorder

We found that therapists’ attitude toward individuals with BPD improved over the course of the training, as indicated by the positive slope (β =0.21, p < .01) in the unrestricted linear growth model presented in Table 2. Individuals with lower attitudes towards consumers with BPD at baseline had significantly greater improvements in attitudes over the course of the training(r = −.69, p < .01). However, there were no significant differences in changes in attitudes toward individuals with BPD associated with therapist level of education, years of experience, degree area, licensure status, or having a change in job description. Figure 1 depicts the mean scores for the Attitude Towards Consumers with BPD scale over the course of the training.

Table 2.

Linear Model of Attitude Towards Consumers with BPD --Unconditional at Level-2

Fixed Effect	Coefficient	se	t Ratio	p Value
Intercept, β00	3.59	0.07	48.18	0.00
Slope, β10	0.21	0.03	6.02	0.00

Random Effect	Variance Component	χ2	p Value
Level 1
Temporal variation in scores, eti	0.10
Level 2 (Between Subjects)
Intercept, r0i	0.28	213.37	0.00
Slope, r1i	0.03	84.74	0.00

Figure 1.

Mean scores for the three primary outcome variables over the course of the training. Error bars represent standard errors (SEs).

Confidence in the Effectiveness of DBT

Confidence in the effectiveness of DBT also increased over the course of the training as indicated by the positive slope (β = 0.16, p < .01) presented in Table 3. Level of education predicted change in confidence in the effectiveness of DBT over the course of the training. Specifically, participants with a bachelor’s degree or lower gained more confidence in the effectiveness of DBT over the course of the training relative to participants with a master’s degree or higher (β = −0.15, p < .05). None of the other professional background variables predicted change in confidence in the effectiveness of DBT over the course of the training. Change in job description also did not predict the amount of change over the course of the training. However, it should be noted that more experienced participants had greater baseline confidence in the effectiveness of DBT compared to less experienced participants (β = 0.02, p < .01). There was also a strong negative correlation (r= −.46, p < .01) between the intercept and slope of the level-1 model, indicating that the greatest improvement in confidence in the effectiveness of DBT was seen for participants who reported relatively low confidence in the effectiveness of DBT at baseline. Figure 1 depicts the mean scores for the Confidence in the effectiveness of DBT scale over the course of the training.

Table 3.

Linear Model of Confidence in the Effectiveness of DBT--Unconditional at Level-2

Fixed Effect	Coefficient	se	t Ratio	p Value
Intercept, β00	3.77	0.06	59.41	0.00
Slope, β10	0.16	0.04	4.11	0.00

Random Effect	Variance Component	χ2	p Value
Level 1
Temporal variation in scores, eti	0.09
Level 2 (Between Subjects)
Intercept, r0i	0.16	110.97	0.00
Slope, r1i	0.03	96.18	0.00

Use of DBT Components

We found that self-reported use of DBT components increased over the course of the training (Table 4; β = 0.33, p < .01). More experienced participants reported more use of DBT components at baseline compared to less experienced participants (β=0.02, p < .05). None of the other professional background variables, nor change in job expectations predicted baseline use of DBT components. There was a moderate negative correlation (r= −.29, p < .05) between the intercept and slope of the level-1 model, indicating that the greatest improvement in the use of DBT components was seen for participants who reported relatively low use of DBT components at the first assessment. Figure 1 depicts the mean scores for the Use of DBT Components scale over the course of the training.

Table 4.

Linear Model of Use of DBT Components--Unconditional at Level-2

Fixed Effect	Coefficient	se	t Ratio	p Value
Intercept, β00	3.49	0.08	42.58	0.00
Slope, β10	0.33	0.04	8.46	0.00

Random Effect	Variance Component	χ2	p Value
Level 1
Temporal variation in scores, eti	0.26
Level 2 (Between Subjects)
Intercept, r0i	0.25	80.46	0.00
Slope, r1i	0.00	58.34	0.20

Using follow-up ANOVAs, there were significant increases in components that were consistent with the DBT model (Table 5). Missing data analyses showed that completers and non-completers did not significantly differ on baseline DBT component use, suggesting that any missing data were missing at random (MAR). As a result, we determined that it was reasonable to use completer analyses for our ANOVAs. When a Bonferroni correction was applied to correct for multiple comparisons (p=.0017), the four components that increased significantly were: Skills Training (F=19.07, p < .001), Treatment Targets (F=25.88, p < .001), Daily Diary Cards (F=23.75, p < .001), and Dialectical Strategies (F=25.26, p < .001). Little change was observed in Ancillary Treatments. Although not significant using the conservative Bonferroni correction, there was a trend (p < .01) for intensive, high cost, ancillary treatments (e.g., hospitalizations and emergency room use) to decrease from time 1 to time 4 - as reported by therapists.

Table 5.

Summary of Pre – Post DBT-Specified Mode, Component, and Ancillary Treatment Use

Item	PRE (n=61)		POST (n=29)		Difference
“To what extent do you use _______ for consumers with BPD”	Mean	SD	Mean	SD	F
DBT-Specified Modes
24-hour Phone Consultation	2.90	1.83	3.57	1.73	.84
Individual Outpatient Therapy	4.34	1.03	4.45	1.27	.02
Process Groups	3.02	1.34	2.69	1.17	4.34*
Skills Training	3.15	1.39	4.48	.69	19.07***
DBT-Specified Components
Behavior therapy	3.77	.94	4.21	1.08	2.51
Consumer agreements	3.66	.79	4.31	.93	10.19**
Daily diary cards	2.26	1.37	4.20	1.05	23.75***
Dialectic strategies	3.25	1.01	4.38	.90	25.26***
Problem solving	4.23	.69	4.41	.73	.24
Protocol around suicidal behavior	3.70	1.16	4.24	1.02	1.35
Therapist agreements	2.97	1.28	4.10	1.19	7.42*
Treatment targets	2.65	1.25	4.28	.96	25.88***
Validation	4.08	1.16	4.72	.53	7.12*
Ancillary Treatments
After hour crisis line	3.54	1.62	3.76	1.50	.04
Case management	3.67	1.29	3.46	1.35	.02
Crisis services	3.10	1.62	2.72	1.39	4.09
Day treatment or clubhouse	3.03	1.55	2.96	1.34	1.26
Emergency Room Visits	2.93	1.21	2.55	.91	9.85**
Hospitalization	2.97	1.09	2.48	.87	10.21**
Medication management	4.15	1.22	4.20	1.20	.16
Occupational/recreational therapy	2.48	1.35	2.21	1.18	.11
Peer counseling	2.28	1.34	2.48	1.33	.74
Psychoeducation groups	3.23	1.23	3.31	1.11	.00
Respite services	2.05	1.27	1.71	1.05	.66
Substance abuse/dual diagnosis	3.69	1.12	3.34	1.17	.72
Support Groups	3.00	1.18	3.25	.93	.71
Vocational rehab	3.02	1.16	2.72	1.16	6.29*

Note. p<.05; p<.01; p<.001; all scores have a possible range of 1 to 5.

Associations between Attitudes, Confidence, and Use

Finally, we examined the associations between attitudes towards consumers with BPD, confidence in the effectiveness of DBT, and use of DBT components. Improvement in attitudes towards consumers with BPD was positively correlated with increased use of DBT components, r= .30, p < .05. Increased confidence in the effectiveness of DBT was also positively correlated with increased use of DBT components, r =.45, p < .01. In other words, both greater improvement in attitudes towards consumers with BPD and greater confidence in the effectiveness of DBT were associated with greater increase in the use of DBT components.

Discussion

Participating in DBT training was associated with positive changes over time in the study outcomes including improved therapist attitudes toward consumers with BPD, improved confidence in the effectiveness of DBT, and increased use of DBT-specified treatment modes and components, providing support for the recommended training approach for DBT – the Intensive Training Model (Landes & Linehan, 2012). In fact, substantial improvements in the use of DBT-specified treatment modes and components were found over time with training and implementation support. The use of ancillary treatments remained unchanged; however, there was a trend for therapists to report less use of emergency room visits and hospitalization for consumers with whom they were using DBT, which could have financial implications for consumers, organizations, and health care systems. This observed pattern demonstrates the value of providing ongoing support to therapists after initial training to ensure that the use of the EBT skills continues to grow and strengthen over time. This level of support is quite different than current training models in community mental health where there might be a 1 or 2-day training with little to no personalized follow-up. In examination of intensive ongoing training and support, we found increased reported use of DBT-specified treatment modes and components. This finding is consistent with the other study of the DBT Intensive Training Method that included extended consultation and supervision, and found it to be important for maximizing the uptake and use of DBT (Hawkins & Sinha, 1998).

There was a differential level of change over time across study outcomes based on therapist baseline attitudes and characteristics. For example, therapists who had the lowest baseline attitudes towards individuals with BPD, confidence in the effectiveness of DBT, or who were least likely to use DBT modes and components were the therapists who had the greatest reported increase over time in each respective area. Bachelor’s level therapists gained more confidence in the effectiveness DBT than their Master’s level colleagues. These finding might have been related to: 1) the therapists with lower baseline confidence and attitudes had more room for growth and improvement, and 2) the therapists who already had high baseline confidence and attitudes didn’t have as much room to grow (ceiling effects) and/or 3) the training process having a greater impact on less experienced and confident trainees. Regardless, this highlights the importance of understanding how therapist characteristics impact implementation. After all, the characteristics of those who receive the training and provide the treatment could affect implementation on multiple levels such as treatment competence (Siqueland et al., 2000) and client outcomes (Vocisano et al., 2004).

Despite high attendance and interest in the training, only 55% of the original group completed training. On average, participants completed over 100 hours of training and consultation time and reported high satisfaction with the training experience. In fact, drop-out was not reportedly related to the training itself. Instead, it was due to therapists leaving their agencies (i.e., staff turnover). This seemingly high level of staff turnover should have been expected. Turnover rates within the mental health and substance abuse workforce are estimated to be as high as 50% to 60% each year (Ben-Dror, 1994; McLellan, Carise, & Kleber, 2003). Some have suggested that the entire workforce turns over every 5–7 years (Kamis-Gould & Staines, 1986). Additional research is needed to understand why therapists leave their agencies and what strategies are needed to prevent it. Not only is this an important issue for the field, generally, it also is specifically important to EBT implementation. If agencies continue to lose their trained therapists after investing substantial time, resources, and energy into their EBT training, it is not likely that fiscally-conscious administrators will continue to invest resources in EBT implementation.

Even though more experienced therapists were more likely to complete training, less experienced therapists (and those lower on variables of interest) were likely to report more improvement in attitudes, confidence, and DBT use. This highlights the importance of including different experience levels within training and the potential impact of differing level of experiences on a training initiative. A noteworthy caveat is the possible relationships among therapist experience, workforce stability, and completion of training. It may be that as administrators have suggested (Herschell et al., 2009) more experienced therapists may stay longer at their agency, and in turn, be able to complete lengthier, agency-sponsored trainings.

Increases in DBT component use were associated with changes in favorable attitudes toward consumers with BPD and confidence in the effectiveness of DBT. While causality cannot be determined, the association may be important. It is possible that the negative attitudes therapists have of consumers with BPD are due to a perceived inability to effectively treat this population. Learning and routinely using an effective treatment like DBT may bolster therapists’ hope in achieving clinically-meaningful progress with consumers, thereby changing their attitudes toward consumers with BPD. Perhaps, with additional research, measures of attitudes or confidence could be used as a more feasible proxy to measures of component use. Or perhaps, behaviorally-oriented training strategies in which participants are required to practice frequently are particularly important (Herschell et al., 2010) and effective in changing attitudes and perceived competence.

It also should be noted that the items tapping negative attitudes produced a lower alpha than for other scales; these items may have been confusing to interpret, especially if participants considered the topic more broadly. For example, “mental health treatment rarely benefits consumers with BPD” may be true for other treatments. Similarly “the way that consumers with BPD relate to others represents fixed character traits” could be conceived as true if therapists were thinking about personality traits rather than state characteristics.

Limitations

The findings of the study should be considered within the context of the limitations. The study used a naturalistic design with only one study group, and we do not know to what extent our findings would have been different with a different training model. Additionally, the data collection used only one self-report measure that was specifically created for the study and has not been tested for reliability and validity. Similarly, important measurement limitations are the lack of behavior observation data (e.g., adherence coding to support therapist self report of use of DBT strategies) as well as consumer-level outcome data to demonstrate that DBT was effective in these settings. In order to have high response rates it seemed that sending surveys in advance and following up in person at the trainings would be important, which was true. In person data collection response rates were 90–92% whereas requiring participants to mail back a survey resulted in a response rate of 76%. Substantial staff turnover during the course of the training limited the number of individuals for whom we had data at all time points and contributed to a wide range of training and consultation participation. The majority of the participants were Caucasian females, which reflects the demographic characteristics of the clinics involved in the project, and is consistent with participants included in other implementation trials (Dimeff et al., 2009; Dimeff et al., 2011). However, we do not know to what extent our findings would be generalizable to other demographic groups. We also examined the implementation of DBT, and do not know if our findings would generalize to efforts to implement other EBTs.

Lessons Learned

Limitations aside, this study makes an important contribution to our efforts to understand the course of implementation of EBTs over time. Our finding that training with ongoing implementation support was associated with therapist reported gains in attitudes, confidence, and DBT use is encouraging, and suggests that implementation models like this one are a promising approach toward supporting the mental health system in successfully implementing new treatments. Yet our findings also highlight the heterogeneity among behavioral health professionals, and suggest that, future efforts to improve therapist skills will benefit from recognizing and targeting this diversity of learning levels, experience, and expertise as well as appreciating the impact of staff turnover on EBT implementation.

Highlights.

• We conducted an implementation study of Dialectical Behavior Therapy (DBT).

• We examined the effectiveness DBT training in 10 community-based agencies.

• DBT training improved therapist attitudes, confidence, and use of DBT components.

Biographies

Oliver J. Lindhiem, Ph.D. is an assistant professor of Psychiatry at the University of Pittsburgh School of Medicine. His research focuses on developing innovative ways to optimize treatment delivery for childhood disruptive behavior disorders (DBDs). His research has an emphasis on methodology and developing assessment tools for measuring and promoting the utilization of cognitive behavioral therapy skills. Dr. Lindhiem was recently awarded an Early Career Research Scientist Development Award (K01) from the National Institute of Mental Health (NIMH) on assessing and enhancing skill-use in evidence-based treatments for childhood DBDs.

Karen L. Celedonia, MPH is a freelance public health researcher and holds an adjunct research staff position at the RAND Corporation. Ms. Celedonia has worked on multiple projects studying the implementation of evidence-based treatments in community mental health settings. Currently, she is involved in projects using technology to facilitate the implementation of evidence-based treatments in the community and to develop novel, community-based health interventions. Additional research interests include the etiology and trajectory of serious mental illness; the subjective experience of contending with mental illness; and the intersection of art and public health.

Bradley D. Stein, M.D., Ph.D. is associate director for mental and behavioral health in the RAND Center for Domestic and International Health Security. His research includes a study examining the implementation of the LAUSD Youth Suicide Prevention Program, a study of the social and functional impact of depression in teens identified in school-based health clinics and other primary care settings, an examination of treatment patterns for youth diagnosed with PTSD, and an ongoing analyses of national survey data regarding American’s emotional and behavioral response to terrorism.

Amy D. Herschell, Ph.D. is an assistant professor of Psychiatry and Psychology at the University of Pittsburgh School of Medicine. Her core research interests lie in studying the implementation of evidence-based treatment (EBT) for children in community settings. She is particularly interested in understanding the variables that support or inhibit practitioner implementation of EBTs in communities. She has conducted multiple studies on this topic, completed a five-year, National Institute of Mental Health (NIMH)-funded career development award on implementation, and is now studying the implementation of Parent-Child Interaction Therapy across Pennsylvania.

Jane N. Kogan, Ph.D. is an assistant professor of Psychiatry at the University of Pittsburgh School of Medicine and the previous Director of Outcomes and Research Administration at Community Care Behavioral Health Organization. Her research focuses on health care delivery innovations such as use of technology to support access and delivery of mental health services and consumer-driven care; payment reform; and implementation of evidence-based practices.