Qualitative and quantitative feedback following workshop training in evidence-based practices: A dissemination study

Introduction

To date more than a thousand clinical trials of addiction treatments have been published. As a result of the progress in the development and validation of treatments for addiction, we have more science based answers than ever to the question of “what works” for those struggling with an addiction. However, there is no single best treatment for substance use disorders; instead, there is a range of treatments that have evidence of efficacy. Behavioral therapies including contingency management (Petry, Martin et al., 2000), behavioral marital counseling (Powers, Vedel et al. 2008), motivational interviewing and modifications of motivational interviewing, (Hettema, Steele, & Miller, 2005) cognitive behavioral therapy (Carroll, Galanter et al. 2008) and the community reinforcement approach (Miller, Meyers et al., 1999) are among these evidence-based practices (EBP) that have demonstrated efficacy across diverse populations and substances.

Generally brief behavioral interventions should be considered for use as the initial treatment in a stepped-care program within a specialist agency (Project MATCH Research Group 1997; Dunn, Deroo et al. 2001; Moyer, Finney et al. 2002; UKATT Research Team 2005), unless there are specific reasons, such as a medical crisis, psychiatric comorbidity, homicidality, suicidality, or a need for inpatient detoxification, to offer patients more intensive treatments as a first resort. Equally promising are findings suggesting that increased treatment effects can be obtained by combining pharmacotherapies with behavioral therapies (Anton et al., 2006). On the whole, effective behavioral treatments for substance use disorders appear to be empathic (Crits-Christoph, Baranackie et al. 1991), focusing on the substance behavior (Carroll, Galanter et al. 2008), reinforcing an alcohol and drug-free life (Meyers, Miller et al. 2002), enhancing motivation for change (Miller and Rollnick 2002), attending to the social support system (McCrady 2006), and promoting the use of adjunctive medication (Garbutt 2009; Ross and Peselow 2009).

The Gap between Research and Clinical Practice

It is difficult for new techniques, including many of the treatments described above, to find their way into an established system of treatment. The Institute of Medicine reported that a 17-year gap exists between the publication of research results and its impact on treatment delivery (Institute of Medicine 1998). There remains a significant gap between the knowledge of what works and the treatments that are currently in use in addiction treatment programs. Unfortunately, the most commonly used U.S. treatment methods, such as group therapy, mandated 12-step, confrontational counseling, and educational lectures and films are some of the treatments with evidence of little or negative efficacy (Miller and Wilbourne 2002). Weakest evidence for efficacy appears to be found in methods designed to educate, confront, or shock regarding the nature and causes of a substance use disorder. The approaches most commonly used, therefore, may in fact be increasing the likelihood of a return to problematic use (Kraus, Castonguay & Boswell, 2011).

Barriers to Implementing EBPs in Clinical Programs

Although there is a substantial base of scientific evidence available to inform and improve clinical practice and more than half of the states in the U.S. already require the use of evidence-based practices in order to receive reimbursement for addiction treatment (Miller, Zweben et al. 2005), there are many barriers preventing the successful implementation of EBPs in community programs (Glasner-Edwards and Rawson 2010). Continuing education is intended as a means to bridge this gap, yet neither continuing education or license maintenance require any education in best practices.

There are personal and systemic barriers in the adoption of EBPs. Clinicians may have negative attitudes toward changing the status quo. Specifically, clinicians previously in recovery and those who endorse a 12-step self-help model seem to have less interest in the use of evidence-based practices (McGovern, Fox et al. 2004) Another major barrier to disseminating research into practice is that frontline providers are often too overburdened with clinical duties to stay current with research findings. Busy providers and program managers cannot be expected to digest the entire treatment outcome literature and come to their own conclusions about EBPs (Miller, Sorensen et al. 2006). Furthermore, providers are understandably wary of research findings since few clinical trials on EBPs are delivered under normal practice conditions in community settings. Even if a clinician has a desire to learn how to implement new EBPs, it is difficult to implement new treatment modalities without intensive training and supervision.

What do we know about the Effectiveness of the Dissemination Process?

While 2-day trainings are the most common approach to training clinicians in new treatments, one-shot methods tend to be ineffective in changing practice behavior and increasing clinical skillfulness. Miller and Mount (2001) examined skill levels in motivational interviewing after trainees attended a two day workshop, viewed videotapes, and completed readings on motivational interviewing. They found a minimal increase in skill in motivational interviewing, though trainees believed they had developed competence. In a subsequent study, Miller, Yahne, Moyers et al. (2004) found that providing ongoing feedback and coaching improved proficiency in motivational interviewing over workshop alone, suggesting that skill maintenance requires these ongoing training enrichments. Subsequent work in frontline providers supported this conclusion (Moyers et al., 2008).

Apart from the Miller and Mount (2001) study, there is little research on the perceived effectiveness of trainings in evidence-based approaches for addiction. Most training evaluations are conducted immediately after the completion of training, with limited information available to understand provider perceptions of the long-term usefulness of the trainings or how easily specific techniques are subsequently used in an attendees’ clinical setting. In studies of workshop training, information about provider satisfaction in particular may be an important predictor of provider retention and participation in behavioral follow-ups. The goal of the present study, then, was to evaluate participant’s long-term impressions following workshop training.

Methods

Over a 13-month period, The University New Mexico hosted a set of thirteen local workshops in evidence-based practices for addiction. Each 1–2 day workshop was led by leading experts in the topic areas. Across all trainings, attendees were encouraged to form practice groups to maintain their skill level and provide ongoing feedback and coaching based on observed practice.

Workshops included training in thirteen EBPs, including 1) Motivational Interviewing (Miller and Rollnick 2002); 2) Buprenorphine Awareness—(Addiction Technology Transfer Centers); 3) Motivational Incentives (Addiction Technology Transfer Centers); Seeking Safety (Najavits 2007); 5) Motivational Interviewing Supervisor Training; 6) Community Reinforcement Approach and Family Training (Meyers, Miller et al. 1999); 7) Smoking Cessation ; 8) SMART Case Management with the Addiction Severity Index; 9) Treatment Planning M.A.T.R.S.: Utilizing the Addiction Severity Index (ASI) to Make Required Data Collection Useful; (Addiction Technology Transfer Centers); 10) Dialectical Behavior Therapy (Linehan, Schmidt et al. 1999); 11) Behavioral Couples Therapy (Powers, Vedel, & Emmelkamp, 2008); 12) Twelve Step Facilitation (Nowinski, Baker et al. 1992); 13) Do-It-Yourself Program Evaluation.

Participants

Prior to each scheduled workshop, announcements were e-mailed to Southwest Node affiliated community treatment programs (CTPs) in New Mexico, as well as other treatment programs in and around New Mexico. A total of 327 individuals attended one or more of the workshops. Each active CTP was able to have two staff members attend without cost. Other attendees were required to pay a nominal fee to attend. Of the workshop participants, 249 (76%) of attendees were from within New Mexico and approximately 69% of workshop attendees were female. All workshop attendees were eligible to receive continuing education credits.

Procedure

Immediately following completion of each workshop, the participants were asked to complete an online evaluation of the training content, the facilitator and the materials. In addition to this evaluation, participants were asked to provide telephone, address and email contact information for future follow up. This second follow up was designed to assess the long term success of the material having been taught, and the effectiveness of its implementation in a treatment setting.

Email requests for the completion of anonymous web-based surveys were sent fourteen months after each workshop. The survey took approximately 15 minutes to complete and queried demographic information, clinical relevance, skills gained, and knowledge application. This evaluation also assessed the usefulness of materials and ability to implement specific techniques in an attendee’s workplace setting. Specific questions inquired about relevancy of materials, amount learned about the topic of the workshop, and ease with which the clinician could incorporate the method into their normal clinical duties. Attendees were also asked to indicate the ways in which their approach to treatment may have changed since the workshop. The questionnaire ended with open-ended questions designed to help understand how trainings could be more useful in the future. Weekly reminders were sent via email for twelve months following the initial survey request.

Qualitative data coding and analyses

Participants were asked to respond to four open questions to give additional detail about their perceptions of the trainings. These questions were: “How will it apply to your clinical work?”; “What did you like most about the trainings?”; “What did you like least about the trainings?”; and “What suggestion do you have for improving future trainings?”

The first step in the coding process involved open coding by two readers who reviewed the responses and discussed the emerging themes. Using a content analysis approach (Hsieh and Shannon 2005), readers refined their notions about the themes and potential ways of coding the responses. We developed themes inductively, allowing the data to dictate the analytic categories. To increase methodological rigor, we had two researchers participating in data coding and analysis to offer alternative viewpoints and perceptions of responses.

Results

Online surveys were completed by 248 participants (75.8% of total workshop attendees). Most attendees were female (67.3%), had a Master’s degree or higher (88.7%), and worked in a clinical setting (58.9%). Across the 13 workshops conducted, 69.1% of respondents reported that the trainings were relevant to the clinical services they provide, and 65.2% reported that they used what they had learned from the workshops in their work. A majority of respondents indicated that the workshop content was applicable to their jobs: 73.2 % found that more than half of the content was applicable, and 52.4% reported that 75% or more of the content was applicable. The occupations reported by workshop participants included nurse, psychiatrist, psychologist, social worker, substance abuse counselor, and student. There was little variability between participants attending the 13 workshops.

Quantitative analysis

Some workshops had a significant number of a single occupation type compared to others. For example, substance abuse counselors were the most represented group in the Buprenorphine workshop. The Smoking Cessation workshop was attended by a significant number of nurses, and psychologists were the majority in attendance for the BCT workshop.

There was substantial interest in the training outside of Clinical Trials Network (CTN) CTPs. Across all workshops approximately half (49.04%) of participants traveled from outside of the New Mexico metro area to attend a workshop. On average attendees traveled 357.1 miles. Attendees who were unaffiliated with a CTN CTP were required to pay a fee to attend. These non-CTP attendees represented 55.13% of all attendees and paid an average of $182.20 to attend a workshop. On the whole, attendees were willing to spend expend significant time and resources to attend these training workshops.

There was significant variability in participants’ perceptions of the applicability of the information learned and the relevance of the trainings to their work. Perceptions of the ability to use the skills learned in the trainings varied at the trend level of significance. Follow-up tests using logistic regression indicated significant associations between relevance ratings and MIN (OR=0.29) and MIST workshop attendance (OR=2.50); as well as applicability ratings and MI (OR=2.21), SS (OR=3.73), and MIST attendance (OR=2.62).

Qualitative Analysis Themes

How did it apply?

Four themes emerged from the analysis examining how the training applied to the attendees’ work. Overall, participant’s responses were very positive and reflected creative and broad applications of the training. Themes included Teaching others/further dissemination, Increased programming/options available to patients, Increased awareness of EBPs, and General application.

What I Liked Most about the Training

Four themes emerged from our coding of responses to this question. Participants seemed enthusiastic about the quality of the trainer and training, the ability to apply this to clinical practice, and learning about the research evidence supporting specific approaches. Themes addressed Instructor’s expertise, Ability to practice skills during the training, Applicability, and Rationale/Research evidence.

What did you Like Least about the Training

When analyzing responses to this question, we were surprised that we did not see any responses reflecting attendees finding the training not applicable to their setting. It appeared universal that attendees all seemed to get something out of the trainings attended and were extremely appreciative of having this opportunity. Themes included Nothing, Amount of info, Length, and Logistical/setting criticisms.

Suggestions for Improving Future Trainings

Our intent in asking this question was to find out what suggestions attendees might have as a way to offer feedback for trainers and training content. Attendees instead seemed to use this open question to express their gratitude and request additional training opportunities. Themes here were Need for more trainings, Gratitude, and Need for ongoing coaching.

Discussion

Our results indicated that in the long term, providers had positive perceptions of the workshops; that is, providers generally thought them helpful and believed that their client care improved as a result of attendance. Many of the participants felt that the content was strongly applicable to their jobs or clinical practices. Participants indicated an interest in attending future workshops, especially those reflecting new evidence-based practices. In addition, the willingness of many participants to travel and to pay fees to attend suggests that treatment providers are hungry for further training in evidence-based practices. It appears that providers generally did not have access to training in EBPs in their local setting; our data suggest that when such training is made available, provider attitudes will generally be favorable and long-term satisfaction will be high. These provider attitudes increase the likelihood not only of attendance but also of engagement, an important consideration in studies incorporating the long-term behavioral follow-ups required to assess skill acquisition in EBPs.

Clearly, provider demand for training is high while funding for training in these settings is low. The one-shot training workshop continues to be common, despite limited evidence of efficacy. In addition, there is limited research examining the evaluation of programs to transfer research-based addiction treatments into community treatment programs. Assessing attendees’ perceptions provides a starting point for understanding the appeal and potential pitfalls of such efforts. Although our data does not provide evidence of skill acquisition, it does provide information to guide future dissemination and implementation efforts.

In workshops such as these, long-term success is contingent on the motivation of the individual attendee to maintain their new skills as well as the ability of treatment programs in providing ongoing coaching, supervision, and training opportunities to their clinicians. The dissemination efforts of the Southwest Node of NIDA’s Clinical Trials Network have included repeat workshops as well as workshops that build upon previously taught skills (e.g., advanced trainings in motivational Interviewing, contingency management, buprenorphine awareness, etc.) to assist trainees and treatment programs. Additional dissemination efforts are needed, but for effective dissemination, it is important that the trainers are capable of training attendees in the clinical method and that the attendees achieve proficiency after the training and maintain that learning through the use of learning communities. For our program, there was control in ensuring that the trainers were highly proficient in training using the workshop method. Watered-down trainings, such as sessions with trainers who are not appropriately trained in a specific technique before training others in the method and/or trainings that are too short in length for adequate coverage for the material, are a significant problem. There is less control over the assessment of fidelity to the method and less access to continued feedback and coaching in the method.

Limitations

Despite its strengths, the present study also included several limitations. First, this initial dissemination effort included no objective assessments of skill acquisition. Research has consistently demonstrated a tendency for trainees to overestimate their skills (Kruger & Dunning, 1999), an effect that has been repeatedly demonstrated in both medicine (Davis et al., 2006; Violato & Lockyer, 2006) and psychotherapy (Mathieson, Barnfield, & Beaumont, 2009; Wain et al., 2015). The relationship between the perceived and actual benefits of training is unknown in the present sample of workshop attendees.

Another potential confound is the effect of any ongoing coaching or local learning communities. Some trainees may have obtained training enrichments in their job setting, such as ongoing supervision or coaching, while others may not have had access to such resources. Such enrichments are a well-established means of preventing skill decline post-training (Miller, Yahne, Moyers, Martinez, & Pirritano, 2004; Moyers et al., 2008), but were not evaluated here.

Conclusions

Our results suggests that following a workshop training in EPBs, participants reported long-term (14 months) high satisfaction with the training and believed that the trainings had been incorporated into their daily practice. Participant enthusiasm for training was indicated by the willingness to travel up to a thousand miles and independently pay for travel expenses. These results indicate that frontline providers are aware of EBPs and eager for training in these methods. However, future dissemination studies in front-line settings should involve formal assessments of training needs, objective measures of skill acquisition, and inclusion of training enrichments to supplement the workshop format.

Biographies

JON M. HOUCK received his M.S. and Ph.D. in psychology from the University of New Mexico. He is currently a Research Assistant Professor in the Center on Alcoholism, Substance Abuse, and Addictions and the Department of Psychology at the University of New Mexico in Albuquerque, New Mexico. His areas of professional interest include translational neuroscience, neurobiological mechanisms of behavior change, psychotherapy process research, and health disparities research.

ALYSSA A. FORCEHIMES received her M.S. and Ph.D. in clinical psychology from the University of New Mexico. She is currently the Executive Director of Train for Change, a national behavior change training and consulting company. Her areas of professional interest include examining the effectiveness of behavioral treatments for addiction and developing effective methods for disseminating evidence-based behavioral treatments in real-world settings

MEREDITH M. A. DAVIS received her M.S. in Criminal Justice, with a concentration in forensic psychology, from the University of New Haven. She is currently a Counselor at Acadia Healthcare Charleston Comprehensive Treatment Center in Charleston, West Virginia. Her areas of professional interest include brief interventions, motivational interviewing, medication-assisted treatment for substance use disorders, and the relationship between addictions, mental health, and the criminal justice system.

MICHAEL P. BOGENSCHUTZ received his M.D. from Harvard Medical School. He is currently a Professor (Research) in the Department of Psychiatry at New York University School of Medicine in New York, New York. His areas of professional interest include and the application of novel medications and psychosocial interventions to treatment of addictions and co-occurring disorders.