Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Aug 22.
Published in final edited form as: Eval Program Plann. 2011 Mar 2;34(4):390–398. doi: 10.1016/j.evalprogplan.2011.02.006

Using a NIATx based local learning collaborative for performance improvement

Mathew Roosa a, Joseph S Scripa a, Thomas R Zastowny b, James H Ford II c,*
PMCID: PMC3424268  NIHMSID: NIHMS392460  PMID: 21371751

Abstract

Local governments play an important role in improving substance abuse and mental health services. The structure of the local learning collaborative requires careful attention to old relationships and challenges local governmental leaders to help move participants from a competitive to collaborative environment. This study describes one county’s experience applying the NIATx process improvement model via a local learning collaborative. Local substance abuse and mental health agencies participated in two local learning collaboratives designed to improve client retention in substance abuse treatment and client access to mental health services. Results of changes implemented at the provider level on access and retention are outlined. The process of implementing evidence-based practices by using the Plan-Do-Study-Act rapid-cycle change is a powerful combination for change at the local level. Key lessons include: creating a clear plan and shared vision, recognizing that one size does not fit all, using data can help fuel participant engagement, a long collaborative may benefit from breaking it into smaller segments, and paying providers to offset costs of participation enhances their engagement. The experience gained in Onondaga County, New York, offers insights that serve as a foundation for using the local learning collaborative in other community-based organizations.

Keywords: Local learning collaborative, Evidence based practice, Process improvement, Vision, Use of data, Pay for participation, Relationships

1. Introduction

A shift from traditional approaches in chemical dependency treatment to the use of evidence-based practices (EBPs) presents opportunity and challenge for treatment providers. Efforts to change how institutions and individuals deliver care are often hampered by the day-to-day responsibilities of delivering treatment. Access to trainings is a barrier to EBP implementation and such meetings may take away from direct-service time with clients (Ruzek & Rosen, 2009). While training workshops can improve skills (Miller, Yahne, Moyers, Martinez, & Pirritano, 2004) and confidence (Bennett-Levy & Beedie, 2007), the process of incorporating new methodologies and ways of approaching treatment can require considerable investment on the part of practitioners. Additionally, fewer client hours during transition and planning times can have financial repercussions for individuals and treatment centers. Concern about administrative and executive support can also influence willingness to engage in process improvement. Researchers have suggested different potential frameworks for advancing implementation science (Damschroder et al., 2009; Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005; Wandersman et al., 2008). These frameworks highlight the importance of leadership engagement, resource support, internal opinion leaders and champions, and training opportunities to build knowledge and skills, which are supported by consultation, and coaching to support innovation implementation.

A multi-organizational learning collaborative focused on improvement is a mechanism that helps accelerate the diffusion of innovations (Wilson, Berwick, & Cleary, 2003). The use of a learning collaborative to improve the quality of care can exist at the local, regional, or national level. States have used a regional learning collaborative approach to improve services in mental health (Cohen, Adams, Dougherty, Clark, & Taylor, 2007); the public health department (Kushion, Tews, & Parker, 2007; Riley et al., 2009) and substance abuse treatment agencies (Rutkowski et al., 2010). However, fewer examples illustrate how county-based or local learning collaboratives improve services. One such collaborative sought to improve maternal and child health services. Their work highlighted the importance of partnerships with local funders, stakeholder involvement, barrier identification by local providers, and learning from other promising national and local initiatives as key steps in redesigning local systems of care (Keyser et al., 2010).

In 2006, leaders in the Onondaga County Department of Mental Health (OCDMH) created a local learning collaborative to support the implementation of evidence-based clinical practices in their chemical dependency programs and evidence-based business practices to improve client access to mental health services. A local learning collaborative is a data-driven network of change teams from organizations within a single system of care that work collectively to enhance performance through process improvement. OCDMH’s prior experience with the NIATx (Formerly the Network for the Improvement of Addiction Treatment, now simply NIATx) process improvement model prompted their decision to use a local learning collaborative. This model also supports the ongoing development of a person-centered service delivery system in Onondaga County. The local learning collaborative brought together key stakeholders to support providers interested in making content and process changes to their programming that would improve the quality of care in Onondaga County. The stakeholder groups included licensed chemical dependency and mental health treatment providers in Onondaga County, the Office of Alcoholism and Substance Abuse Services (OASAS, the single state agency), and Onondaga County as the local funder of services. Since 2006, these organizations have used the local learning collaborative model to support EBP implementation and improve access to mental health services. The purpose of this paper is to discuss the use of a local learning collaborative approach to support the implementation of evidence-based clinical and business practices to improve the quality of care. The lessons learned may be particularly relevant to states and counties interested in facilitating a collaborative approach among community-based treatment providers. It may also offer guidance on how to implement a local learning collaborative.

1.1. Local learning collaborative in chemical dependency

When the pilot began, Onondaga County, as a member of the Western New York Care Coordination Program (www.carecoordination.net), was already developing a person-centered, service-delivery system with emphasis on natural supports, individualized recovery approaches, and client choice (Raskin & Rogers, 1995; Schwartz, Jacobson, & Holburn 2000). Primarily applied within the mental health service system, this person-centered approach fosters communication between chemical dependency and mental health treatment providers while offering comprehensive support to clients seeking services for co-occurring chemical dependency and mental illness. Person-centered planning is a comprehensive, life-planning approach originally developed to meet the needs of developmentally disabled adults. At its core, person-centered planning is about developing services and supports based upon the needs of the individual. This is in contrast to a traditional-service approach that enrolls individuals into pre-existing service elements that often provide a poor fit and yield modest progress toward recovery. Recovery management emphasizes partnerships between clients and providers, while focusing on long-term-care management (Scott, Dennis, & Foss, 2005; White, Boyle, & Loveland, 2003). The recovery-management approach is familiar to the chemical-dependency treatment community, and its values and practices align with the person-centered planning model. More recently, recovery management (RM) has been promoted as a parallel approach for the chemical dependency service system, with comparable values regarding individualized recovery-oriented approaches (Adams & Grieder, 2005). Therefore, recovery management became a stepping-stone to the person-centered approach for participating chemical-dependency treatment providers. To encourage collaboration between providers while offering a common understanding of person-centered treatment principles, participating providers received training in person-centered planning and recovery management. These trainings provided a basis for implementing EBPs within a person-centered treatment system.

The “Evidence-Based Practice in Chemical Dependency” local learning collaborative was a service enhancement project of the Onondaga County Department of Mental Health. It was designed to improve chemical dependency services through the following four goals: (a) encourage person-centered/recovery-management approaches and values; (b) increase the utilization of evidence-based practices, (c) foster collaboration among providers and (d) encourage a process improvement approach using rapid cycle change. Providers in this local learning collaborative focused on the implementation of three evidence based practices: Contingency Management, Seeking Safety and Motivational Interviewing Assessment: Supervisory Tools for Enhancing Proficiency.

1.2. Contingency Management

OCDMH’s first EBP priority was testing a method to decrease no-show rates and increase persistence in treatment. Based upon a review of EBPs in this area and a dialogue to build consensus, the collaborative selected Contingency Management (CM) as the first EBP to implement (Petry, 2000). CM is a behavioral therapy that supports treatment goals with incentives and consequences (Prendergast, Podus, Finney, Greenwell, & Roll, 2006). Previous studies reveal significant increases in attendance, abstinence, and treatment compliance using CM (Carroll & Rounsaville, 2007; Lott & Jencius, 2009; Petry, Martin, Cooney, & Kranzler, 2000). The use of CM has successfully increased abstinence among such diverse populations as cocaine abusers, drug dependent pregnant and postpartum woman, and in a community-based marijuana treatment program (Budney, Higgins, Radonovich, & Novy, 2000; Petry, Alessi, & Hanson, 2007; Silverman, Svikis, Robles, Stitzer, & Bigelow, 2001).

1.3. Seeking Safety

Seeking Safety is a manual-based treatment for clients with a dual diagnosis of Post-Traumatic Stress Disorder (PTSD) and Substance Use Disorder (SUD) (Najavits, 2002). The treatment methodology is group-based and uses principles of Cognitive Behavioral Therapy to address both PTSD and substance abuse simultaneously. PTSD is significantly more common in SUD populations and is linked to higher rates of treatment noncompliance and relapse (Jacobsen, Southwick, & Kosten, 2001). Several studies show integrative treatment of PTSD and SUD, in general, and with Seeking Safety in particular, to have positive outcomes in areas of client satisfaction, treatment compliance, increase in abstinent behaviors, and reduction in PTSD symptoms (Cohen & Hien, 2006; Desai, Harpaz-Rotem, Najavits, & Rosenheck, 2008; Hien, Cohen, Miele, Litt, & Capstick, 2004; Morrissey et al., 2005; Najavits, Gallop, & Weiss, 2006).

The selection of Seeking Safety is due, in part, to a perceived positive outcome from an existing Seeking Safety group already active in one of the partner organizations. Analysis of the current group demonstrated that patients completed treatment at higher rates than the general treatment population. Attendance and patient-reported satisfaction within the Seeking Safety group were consistently positive. This result led to the selection of Seeking Safety as a means of targeting rates of treatment completion among clients with co-current experiences of trauma and substance abuse. The collaborative contracted with the author of Seeking Safety, to conduct a two-day workshop on the model. Subsequently, all four provider–partners began a Seeking Safety group (Najavits, 2004).

1.4. Motivational Interviewing Assessment: STEP

Based on the work of the SAMHSA/NIDA Blending Team Initiative, Motivational Interviewing Assessment: Supervisory Tools for Enhancing Proficiency (MIA: STEP) is designed to assist counselors in the use of motivational interviewing skills during assessments. It includes tools to aid in clinical supervision and is for direct use by practitioners (Martino et al., 2006; Martino, Gallon, Ball, & Carroll, 2008; Martino et al., 2010). Initially, providers participated in two MIA: STEP training workshops. A wide variability among practitioners’ knowledge of motivational interviewing techniques and therefore their ability to apply those techniques in clinical practice was apparent, and two levels of training were required. Level one addressed basic MI concepts and skills, while the second level focused on the supervisory and advanced skills needed to implement MIA: STEP. Currently the EBPs used in chemical-dependency treatment provider organizations are in varying stages of MIA: STEP implementation. Some have implemented MIA: STEP, while others have engaged in MIA: STEP-informed practice.

1.5. Local learning collaborative for clinic access

Buoyed by their success, Onondaga County Mental Health Department decided to implement a second local learning collaborative with their mental-health providers. The Clinic Access Project’s (CAP) focus was to use the NIATx Plan-Do-Study-Actmodel to implement person-centered, family-driven changes that increase clinic access. This focus area was selected to help minimize sizeable waiting lists and related access challenges for mental health therapy and outpatient psychiatric services throughout the community.

1.6. Role of process improvement

In a recent article, McCarty, Gustafson, Capoccia, and Cotter (2009) state how the NIATx process improvement model provides a framework for meeting the Institute of Medicine (IOM) goals to improve the quality of behavioral health care. However, organizational effort to begin a change process can be daunting. It is often difficult to identify and prioritize an agency’s treatment needs and subsequently connect those needs with the implementation of specific evidence-based practices. Using a learning collaborative approach, the NIATx process improvement model integrates five key principles of organizational change with the use of PDSA change cycles (Capoccia et al., 2007). The NIATx learning collaborative approach (including the key values of transparent dialogue and collective learning) allows providers to come together for collective benefit, despite a history of competition for funding and program support (Evans, Rieckmann, Fitzgerald, & Gustafson, 2008). Treatment agencies have used the NIATx model to focus their efforts on implementation of business practices to reduce waiting times and increase continuation in treatment (Hoffman, Ford, Choi, Gustafson, & McCarty, 2008; McCarty et al., 2007; Wisdom, Hoffman, Rechberger, Seim, & Owens, 2009). The NIATx model has also helped improve patient care in behavioral health systems (Prescott, Madden, Dennis, Tisher, & Wingate, 2007). A recent NIATx initiative, Advancing Recovery, expanded the NIATx model to test its usefulness in the implementation of evidence-based practices (EBPs) related to pharmacological and behavioral therapies among chemical dependency providers and to create changes at state and payer levels that would encourage EBP adoption (Bornemeier, 2010).

The local learning collaborative in Onondaga County used the NIATx (www.niatx.net) rapid cycle change model as the vehicle to assist providers in rapidly implementing various targeted EBPs. It was offered to providers as the best way to insure their investment because the model uses clear data to fuel decision making (Wisdom et al., 2006). County leaders pointedly conveyed the message that if properly implemented EPBs did not yield results, providers would not have to continue them.

2. Methods

2.1. Participants

Onondaga County, with a population of over 450,000 residents, is located at the center of New York State, and includes the City of Syracuse (population of approximately 139,000) and surrounding suburban and rural communities. OCDMH’s Director of Planning and Quality Improvement directed the project, assisted by staff from Onondaga Case Management Services, Inc. (OCMS). The collaborative hired OCMS as a trainer for person-centered planning and as an ongoing consultant to the project. Through the work of the Dual Recovery Coordinator, OCMS also provided guidance in choosing EBPs for clients with co-occurring chemical dependency and mental illness. The state participated in the initiative through the OASAS regional field office. To offset any lost revenue and remove barriers to participation, OCDMH provided funding to replace any loss of direct-service revenue and cover the cost of training and outside expert consultation.

Four chemical-dependency treatment provider organizations participated in the Evidence-Based Practice in Chemical Dependency Initiative (EBP in CD). Agency A is exclusively devoted to serving people with co-occurring chemical dependency and mental illness. Agency B is the region’s only methadone clinic, and Agency D is a federally funded comprehensive health service clinic. The two larger providers, Agencies B and C, represent more than 80% of the chemical dependency services provided in Onondaga County. Six mental-health providers participated in the Clinic Access Project (CAP) local learning collaborative: four community-based mental health providers (Agencies E–H), the local county department of mental health (Agency I), and a local psychiatric hospital.

Table 1 provides summary information about the participants in the two local learning collaborative projects. It includes focus, number of participants, annual admissions, total full-time equivalents, collaborative budget, and other key stakeholders.

Table 1.

Information about the local learning collaboratives.

Local learning collaborative 1 Local learning collaborative 2
Collaborative focus Implementing EBP in CD initiative with a set of common aims Improving Clinic Access Project with focus on client needs
Participating agencies 4 participating CD providers 6 mental health clinics
Annual admissions 5000 annual admissions 2262 annual admissions
Number of FTEs 300 staff 164 staff
Collaborative budget $225,000 over 27 months including $15,000 in annual mini-grants, training costs and reimbursement for lost billings $125,000 over 2 years
Key stakeholders Field Office of the Single State Agency (OASAS)
Trainers and consultants (local and national experts)
Onondaga County Department of Mental Health		 Field Office of the Single State Agency (OMH)
Trainers and consultants (local and national experts)
Onondaga County Department of Mental Health
Training received Recovery management
Person-centered planning
NIATx process improvement model
Contingency Management
Seeking Safety
MIA-STEP		 NIATx process improvement model only
No person centered planning or specific EBP training
Coaching provided Yes Yes
Open in a new tab

2.2. Measures and analysis

Key-outcome measures differed for each local learning collaborative. The EBP in CD initiative examined measures relevant to Contingency Management, Seeking Safety, and Motivational Interviewing. The measures included appointment no-show rates, treatment continuation, and treatment completion. The initiative also measured participant satisfaction. The absence of a clear focus within the CAP local learning collaborative resulted in each participating agency tracking individual outcomes specific to their change project. These included the number of scheduled appointments, percent of kept appointments, and prescription refills.

In keeping with the NIATx process improvement model, providers used spreadsheets and presentations to summarize the impact of their change projects. This information was used to summarize the impact on outcomes within the local learning collaborative.

At the end of the first local learning collaborative, participating providers evaluated the process using an online survey sent to all providers who remained in the project. The survey used a four-point scale (Very Valuable (4) to No Value (1)) to measure the value of each evidence-based practice to clients and staff as well as the value of sustaining the practice. The second part of the survey used a similar four-point scale (4 = high motivation, 1 = low motivation) to rate the level of motivation to participate in the EBP in CD initiative. The motivation level of three variables was assessed (funding provided, opportunity to receive EBP training, and opportunity to work in a learning collaborative with peers)

3. Results

This section of the paper outlines the results from change efforts for each local learning collaborative. In the case of the first local learning collaborative, results are shown by evidence-based practice implemented and are followed by an overall evaluation of the local learning collaborative. Results from the second local learning collaborative are summarized at the participating provider level.

3.1. Contingency Management

Implementation of contingency-management strategies began in late 2006 and early 2007 with an emphasis on reducing client no shows and increasing retention rates for outpatient services. Four agencies (Agencies A, B, C, and the local psychiatric hospital) implemented change projects related to Contingency Management. These changes included providing day planners, refreshments, and light meals to recipients of group therapy, as an incentive to increase attendance. Data was available from all agencies except the psychiatric hospital. Changes were implemented for a single group with an average attendance of eight to twelve people. No-show rates over a six-month period improved for: (a) an evening out-patient group from 61% to 33.3% by serving a light meal (Soup for Group); (b) second appointments for outpatients and intensive outpatients from 49% to 13% by offering day planners to recipients; and (c) an outpatient treatment group from 30% to 9% by offering refreshments (see Fig. 1). Agency B provided food vouchers to improve perfect weekly attendance for its clients. Over a four-month period, the percent perfect weekly attendance improved from 82.5 to 90.5%. Sustainability was mixed. Staff turnover and lack of funding were the reasons most cited for failure to sustain the change.

Fig. 1.

Fig. 1

Open in a new tab

Improvements in no-show rates using Contingency Management.

Aside from tangible improvements in no-show rates, the implementation of Contingency Management (CM) helped cultivate an entrepreneurial spirit among participating organizations. Nonetheless, it is important to anticipate and address issues associated with CM implementation. Providers accustomed to tight budgets were concerned about paying for incentives such as food. CM helped reveal the need to spend money to make money, and the implemented CM projects produced a significant return on investment through increased billings. However, providers encountered organizational challenges related to getting checks cut to buy gift cards. It is also important to consider the cultural shift of “paying” people to come to treatment.

3.2. Seeking Safety

Three providers, Agencies A, B and C, implemented PDSA change cycles related to Seeking Safety. While all four providers reported positive results from the implementation of Seeking Safety, only two were able to provide data to support their change efforts. Agency B reported that the organization started a Seeking Safety group for women, which averages 14 clients per session. Over a six-month period, Agency C reported strong improvements in retention in outpatient, women services, and completion while maintaining a high degree of client satisfaction (see Fig. 2). Given that individuals with a history of trauma tend to relapse (Desai et al., 2008) or complete treatment at a lower rate than other recipients (Gatz et al., 2007), these results are a strong indication of Seeking Safety’s value. Clients involved with Seeking Safety groups reported that the program helped them “discover ways of preventing unsafe behaviors,” to “be more honest about … feelings/emotions,” and to “move on.”

Fig. 2.

Fig. 2

Open in a new tab

Results from Seeking Safety implementation.

3.3. Motivational Interviewing: Supervisory Tools for Enhancing Proficiency

Despite efforts by three agencies (Agencies A, B and C), the MIA: STEP EBP was not widely implemented in this small local learning collaborative. The rigorous fidelity requirements and the time consuming supervisory review of audiotapes made MIA-STEP a tough EBP to implement in treatment providers with no fiscal margin.

3.4. Provider evaluation of the chemical dependency collaborative

Twelve individuals from three organizations completed the survey. Half of these respondents were administrators and half were direct-care staff. The summary results table reflects very high value ratings for all of the practices implemented and for the initiative overall (Table 2).

Table 2.

Providers’ rating of the EBP in CD initiative.

Evidence-based practice Average response for question related to
Value to
clients		 Value to
staff		 Sustainability
Contingency Management 3.42 3.08 3.00
Seeking Safety 3.50 3.50 3.50
MIA-STEP 3.92 3.83 3.83
Overall rating 3.92 3.83 3.83
Open in a new tab

Note: Number of responses was 12. Survey used a four point scale where 4 = Very Valuable to 1 = No Value.

Table 3 indicates that while all three variables were highly motivating, the opportunity to participate in learning collaborative with peers was the strongest motivational factor. This data supports responses to an open question regarding the ‘best and worst parts’ of this initiative with comments focusing on training/staff development and collaboration. The results reflected the challenges of engaging staff and implementing the project in the midst of staff turnovers. Respondents asked for more training, and more opportunities for “peer exchange.”

Table 3.

Level of motivation to participate in the EBP in CD initiative.

Level of motivation variable (n = 12) Avg. response
The funding provided 3.40
The opportunity to receive training in these EBPs 3.75
The opportunity to work in a learning collaborative with peer organizations 3.82
Open in a new tab

Note: Used a four point scale where 4 = Very Valuable to 1 = No Value.

3.5. Client Access Learning Collaborative

The Clinic Access Project (CAP) was built on a modest foundation of periodic clinic directors meetings that predated the collaborative. Further developing these relationships became a critical and unforeseen asset as clinics were confronted with a state-wide Clinic Restructuring initiative in 2010. The collaborative approach to problem solving generated by CAP served the clinics well, as they implemented new regulations and reimbursement structures to improve client access to treatment. The local psychiatric hospital sought to reduce the unplanned discharge rate by 5% from a baseline of 53%. The hospital staff identified areas for improvement – intake treatment planning and revising the discharge planning process through client inclusion. However, the hospital was unable to report data on these outcomes which county leadership attributed to a lack of resources to adequately support data collection. Three of the remaining five agencies participating in the Client Access Local Learning Collaborative provided data to support their change efforts. The results of their PDSA change cycles are presented below.

At Agency E, reminder phone calls helped improve no-show rates. The calls were limited to adult patients at risk of a no-show for a treatment appointment, and were placed to 72 clients over three months. During that time, the percent of appointments attended increased from 58.5% to 64.7%.

Agency F implemented a centralized intake system that improved telephone triage, allowed for a multi-disciplinary triage team to be established, and assigned clients based on counselor skills and interest. Specific outcome measures included no-show rates, number of clients accessing services, staff productivity and client satisfaction. In six-months, the no-show rate was reduced from 26 to 18%. From September 2009 through February 2010, clinic appointments increased from an average of 540 to 700 per week. Center productivity increased and ranged from 93 to 105%. The agency reported improvements in client satisfaction and a $156,000 increase in client revenues over six months.

Agency G implemented changes in medication procedures to help improve the quality of care and in staffing patterns to improve staff productivity. The first change used client education to influence prescription refill requests. This change reduced the number of calls to the medication room for unplanned prescription refill requests from ten per day in March’09 to two per day in June’09. The agency has continued to sustain this rate. To improve staff productivity, the agency sent the team a daily no-show list, which allowed therapist to reschedule appointments. Over a four-month period, one therapist improved productivity by 5% and increased available service hours from 34 in March 09 to 46 h in June 09. Agency G also implemented a change project in their adolescent outpatient clinic to streamline the number of refill requests between medical appointments. Changes focused on the re-education of families and only writing bridge prescriptions between appointments. As a result, eight new families receiving prescription services and the number of refill requests between appointments were reduced by 54%.

4. Discussion

Each of the local learning collaborative initiatives achieved positive results. However, differences in focus, success, and implementation existed. A major reason for success in the EBP in CD initiative was the agreement that participating agencies would focus only on a core set of EBPs. The significant upfront planning of the EBP in CD initiative resulted in more substantive gains. Conversely, the diversity of provider interest in the CAP initiative required Onondaga County Mental Health Department to stop “insisting” on using the EBP plan. As a result, providers were less committed and united in action, and the initiative had a smaller impact on change-project aims. However, it is important to note that the CAP initiative took place during a much tougher fiscal time in Onondaga County.

The two local learning collaborative initiatives used the Plan-Do-Study-Act rapid-cycle change in combination with EBP implementation and found that it is a powerful tool for change at the local level. Health care organizations have used Plan-Do-Study-Act (PDSA) cycles to improve organizational processes (Clemmer, Spuhler, Oniki, & Horn, 1999; Pearson et al., 2005; Ragsdale & Mueller, 2005). The use of PDSA cycles has also improved patient care in behavioral health systems (Prescott et al., 2007) and operational efficiency in drug treatment agencies (Hoffman et al., 2008; McCarty et al., 2007). The combination provides rapid feedback to providers about the impact of their change efforts to improve client services. This process engages staff in provider agencies, which helps speed the implementation and dissemination of an evidence-based practice in the organization. As a result, the EBP is more likely to be sustained. The approach also highlights the business case for change, which allows providers to see that the evidence-based practice not only improves quality of care but also offers a return on their investment. Otherwise, providers may implement a particular evidence based practice because it is the right thing to do without any resources to support implementation.

Local government leadership also learned that EBP implementation requires a cultural change to achieve success. Culture, leadership engagement and external collaboration are examples of attributes necessary to support implementation research (Damschroder et al., 2009; Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). The EBP in CD initiative’s original plan had called for the implementation of five EBPs over the course of 27 months. It became apparent, over time, that this was an overly ambitious agenda. The initiative fully implemented two EBPs (Contingency Management and Seeking Safety) and continued to work to implement MIA: STEP. For other local governments considering the use of a local learning collaborative, an initial focus on the successful implementation of less rigorous evidence-based practices may dramatically enhance efficiency and might provide the capacity to properly implement an EBP such as MIA: STEP. Future EBP implementation efforts should roll out at a rate of approximately one new practice per year.

Participants found the initiative too long. They might have been more enthusiastic if the project had been defined as three separate initiatives, rather than one 27-month-long project. Multiple celebrations to recognize successes during the course of the project might also help enhance motivation and collaboration. Some provider participants dropped out of the local learning collaborative due to time commitments or other competing quality initiatives. For example, the Syracuse Community Health Center dropped out of the EBP in CD local learning collaborative because they were not invested in the project, and a small service provider had difficulty dedicating representatives and getting administrative support to do any projects after several months. In the CAP local learning collaborative, Saint Joseph’s, a hospital-based program, stopped participating because of competing quality obligations within the hospital.

In a local learning collaborative, training plays an important role in contributing to overall success. At the beginning of the EBP initiative, it was unclear if planned supports (monthly meetings and as needed assistance) would be adequate to aid providers in effective EBP implementation. The EBP in CD initiative experimented with a range of training approaches that had high participant ratings. The collaborative hired national experts to facilitate large trainings that were open to the entire community. Small sessions conducted by local trainers were open only to the EBP in CD participants. It was clear that not all trainings required national experts, and that local training resources were of high quality. However, providers appreciate the opportunity to gather for larger training sessions with national experts. The resulting “buzz” generated energy for the project and connected the EBP in CD initiative to the community at large. Future initiatives should strategically use these high-profile training events for the energy they bring to the project. The design of smaller local training efforts should harness this energy to move forward with implementation efforts.

While monthly meetings appeared to keep providers connected in a collaborative effort and motivated to continue to work on the changes, individualized coaching is a critical supplement to community-wide collaborative meetings. External coaching helps in a number of ways: (1) It allows members to speak freely without the ties and anchors of local commitment; (2) provides a another set of expertise and experience to bear on the collaborative development and performance improvement work; (3) adds to “external” validity of the process; and (4) provides a new resource to help guide, focus and support during the demanding and hectic work week. For this project, regular coaching might have resulted in stronger outcomes and better sustainability. Coaching may also have prevented the loss of one of the providers, by helping them over their unique hurdles. It also may have enabled a sharper focus on process improvement techniques and the NIATx Plan-Do-Study-Act rapid-cycle change model. The limited budget only allowed for training of process improvement techniques during a brief orientation at the beginning of the project. When selecting a coach, local leadership must carefully consider the individual’s aptitudes as it relates to their project’s needs. Local leadership should also ensure that the coach brings a new voice and vision to the local learning collaborative.

5. Application principles and lessons learned

The local learning collaborative offers an opportunity for payers to work together to improve services for individuals within a set geographic catchment area. In contrast, a national or regional collaborative may require significant monetary support to enable such in-person gatherings. In the absence of such support, these collaborative approaches often rely on technology to offer low-key training, which may compromise connections associated with a local learning collaborative. For example, the Onondaga County EBP in CD initiative had a substantial effect on chemical dependency service delivery in Onondaga County. In the clear language of one participant, EBP in CD changed “how care is delivered to patients.” The two local learning collaboratives in Onondaga County resulted in care improvements. They also offered insight into key lessons that serve as a foundation to promote local learning collaborative success in other community-based organizations. These include:

  1. Establish a clear plan and shared vision: A shared vision enhances engagement, support and creates the basis of the collaborative.

  2. One size does not fit all: Diversity of implementation among collaborative members ensures ongoing relevance for all participants.

  3. Data fuels engagement: Using the NIATx model and a data-driven approach to discuss how to adjust changes based on results led to strong learning and community wide teamwork.

  4. Quality over quantity-“Smaller Chunks” strategy: Implementing a local learning collaborative is like driving on ice: traction first, then carefully accelerate.

  5. Pay for participation: Providers are less likely to volunteer to participate if they feel they will have to lose money (lost billings, etc.).

These lessons learned, discussed in more detail below, will assist program planners, and improve future research implementing EBPs into field and clinical settings. They also provide insight to help organizations and change teams make improvements in learning collaborative environments, thus setting the stage for practical and specific applications.

5.1. Lesson 1: Establish a clear plan and shared vision

Establishing a clear plan and shared vision for a local learning collaborative allows for rapid engagement by participating providers. It enhances the likelihood that adequate progress toward collaborative goals will be achieved within a reasonable time frame. Project complexity may lead some participants to express uncertainty about the broader vision and structure of the collaborative. While efforts were made to orient participants, the first collaborative evolved as it was implemented, and in its early stages lacked clarity of purpose. In this collaborative, a key lesson was the importance of dedicating more resources to the planning phase of the project, which in turn would have enabled a more focused vision, and more rapid engagement of stakeholders. The development of future local learning collaborative should establish a shared vision during the planning process, which enhances engagement and creates the basis of the collaborative.

5.2. Lesson 2: One size does not fit all

When implementing a local learning collaborative model, local payer leadership must recognize that one size does not fit all. This recognition should guide local government leaders to find common interests that bind participants together and allow for variation within the local learning collaborative. This approach is important to exploring the tension for change within each participating organization and using that information to find an appropriate balance with the collaborative. For example, the Syracuse Community Health Center (SCHC) stopped participating halfway through the 27-month EBP in CD initiative. While SCHC is a large health care provider, they were the smallest CD service provider engaged in the initiative. Even though accommodations were made regarding the number of staff expected to attend training and meetings, the investment of time and resources proved to be unsustainable for them. Such challenges may be amplified by the degree of diversity among participants. The establishment of a local learning collaborative should focus on engaging the entire collaborative in developing a clear strategy for addressing individual needs of each participating organization prior to commencing the initiative.

5.3. Lesson 3: Data fuels engagement

A data-driven approach to the local learning collaborative helps fuel provider engagement. Participants reported an enhanced appreciation for the importance of using data to make decisions and to drive the implementation of new practices. The use of objective outcomes data and recipient feedback data fueled the changes necessary to move EBP implementation forward and improve clinic access. While some EBP practices were harder to measure than others, the participants generally appreciated the focus on measurement. By considering the relative measurement challenges related to the EBPs during the planning process, county leadership could have helped provider organizations avoid some of the data challenges that they encountered. Conversely, the clinic access collaborative measured operational outcomes such as prescriptions filled, or number of appointments. Collection of these measures can often occur more rapidly than a measure tied to a long-term outcome (e.g. treatment completion). Thus, it is important to identify and select the right outcome measures for the collaborative in partnership with participating treatment providers.

5.4. Lesson 4: Quality over quantity-“Smaller Chunks” strategy

The size and scope of the local learning collaborative may affect long-term provider motivation. Within the substance abuse disorder local learning collaborative, county leadership tried to motivate providers to engage in the heavier lift of MIA: STEP implementation by urging them to remember that the earlier EBPs (Contingency Management and Seeking Safety) had required modest labor when compared to the fiscal supports that were provided. With regard to time invested and fiscal compensation, the essential message to providers was that they were “ahead of the game” and “would now need to spend some extra resources” in order to complete this last practice. However, the project length (27 months) did not help motivate participating treatment agencies. Providers appeared to approach each EBP in isolation, instead of considering the whole project from a global budget perspective. In order not to over burden providers, future projects in a local learning collaborative should focus on quality versus quantity and insure that each portion of a project has a well-defined fiscal and time budget.

5.5. Lesson 5: Pay for participation

Paying for participation helps offset lost revenue and removes some common obstacles that prevent provider participation in change projects. When providers realized that their involvement in the local learning collaborative would not result in their organization making any short-term financial sacrifices, they became enthusiastic participants. While the funding did not insure success, it built good will by acknowledging the value of staff time. Even a modest amount of funding enabled providers to participate without perceiving the project as a short-term loss. In the words of one of the participating organization’s administrators, “Without the money we may have come to the table, but we would not have stayed as long.” Conveners of a local learning collaborative will need to consider paying providers for participation but the compensation does not have to be significant.

6. Conclusions

The unique structure of the local learning collaborative requires attention to old relationships. The challenge for leaders is moving participants from a competitive to collaborative environment through transparency. In developing a culture of collaboration, providers have to move toward an altruistic viewpoint and understand that this collaboration is a means to improve behavioral health services in their locality. Achieving this level of transparency represents a unique challenge. It is individual and organizational shame that drives us to hide flaws rather than openly share them. This resulted in missed opportunities for change, as organizations, especially with local peers, are fearful of asking for support and acknowledging errors or imperfections. When an organization blames individuals for problems, it discourages transparency. When an organization encourages open sharing, team members can come together without shame to explore potential improvements. As a result, team members begin to notice flaws in the process and are encouraged to call attention to these valuable improvement opportunities.

In Onondaga County, the collaborative approach fostered transparent collaboration, resulting in a significant cultural shift that enhanced lines of communication between the participating organizations. The flexibility of a local learning collaborative offers opportunities for small, local, in-person gatherings that foster cooperation across treatment providers and with their peers, leverage their actions to improve service delivery. Onondaga County leadership also had an important role in modeling and encouraging the value of collaboration and transparency; in this case, leading by example is crucial. One noted example of role modeling was OCDMH’s rapid acknowledgement and repair efforts related to an unsuccessful training session. Participants appreciated this, and OCDMH’s example helped cultivate the transparent collaboration necessary for the project to be successful. Without transparency, the potential for improvements may be non-existent.

Now more than ever, fiscally tough times require efficient and effective tools for navigation and bringing change to the treatment system. In Onondaga County, providers needed to develop effective and efficient practices and work to abandon traditional practices that do not achieve measurable recovery goals. It is important to use the local government unit as a mechanism to support change through rapid cycle leadership and the local learning collaborative appears to be an effective system to bring about change. The local government unit represents a change infrastructure that is flexible, cost efficient, self-sustaining and ripe with local knowledge; consider it as a powerful platform of innovation and springboard to EBP implementation. Using the NIATx approach to support the local learning collaborative not only will help bring about effective change, but will establish an infrastructure to improve the quality of the local delivery system.

Acknowledgements

This study was funded by the Onondaga County Mental Health Department. Article preparation was supported by the National Institute of Drug Abuse (R01 DA020832). The authors would like to thank all of the staff from Central New York Services, Crouse Chemical Dependency, Syracuse Brick House and the Syracuse Community Health Center who participated in the EBP-CD Local Learning Collaborative and from Arise, Brownell Center, Hutchings Psychiatric Center (children and adult clinics), Onondaga County Department of Mental Health Children and Youth Clinic, and Saint Joseph’s Hospital who were a part of the Clinic Access Local Learning Collaborative. Special thanks are also addressed to Andrea Gianopoulos, for her assistance in the revision of the manuscript.

Biographies

Mathew Roosa, ACSW, LCSW-R, is the Director of Planning and Quality Improvement for the Onondaga County Department of Mental Health. In that capacity he works with providers of mental health, chemical dependency, and developmental disabilities services on a range of County and state wide projects and initiatives. Mr. Roosa also works as a NIATx coach, provides process improvement consultation and training, and is an adjunct faculty member in the Syracuse University Social Work Department.

Joseph S. Scripa, LMSW, CASAC, as a Dual Recovery Coordinator for Onondaga County, is responsible for facilitating the development of integrative services and a recovery oriented system of care for those impacted by co-occurring mental health and substance use issues. He has worked in behavioral healthcare providing services for inpatient and outpatient treatment agencies including treatment and education programs in the Corrections field. He has served in roles as a clinician, supervisor, trainer, educator, consultant, and program developer.

Thomas R. Zastowny, Ph.D., is a Clinical Psychologist, NIATx Coach and Healthcare Consultant in Upstate NY. His work has focused on health care delivery, quality and performance improvement, health care accreditation, and outcomes.

James H. Ford II, Ph.D., is an Assistant Scientist and NIATx Research Director at the University of Wisconsin-Madison. His health services research in substance abuse treatment and mental health focuses on organizational change, employee commitment to change, information technology and the role of performance management in improving how organizations use data to manage change.

Contributor Information

Mathew Roosa, Email: MathewRoosa@ongov.net.

Joseph S. Scripa, Email: JosephScripa@ongov.net.

Thomas R. Zastowny, Email: Drzwy@aol.com.

James H. Ford, II, Email: Jay.ford@chess.wisc.edu.

References

  1. Adams N, Grieder D. Treatment planning for person-centered care: The road to mental health and addiction recovery: Mapping the journey for individuals, families and providers. Burlington: Elsevier Academic Press; 2005. [Google Scholar]
  2. Bennett-Levy J, Beedie A. The ups and downs of cognitive therapy training: What happens to trainees’ perception of their competence during a cognitive therapy training course? Behavioural and Cognitive Psychotherapy. 2007;35(01):61–75. [Google Scholar]
  3. Bornemeier J. The Robert Wood Johnson Foundation’s Efforts to Combat Drug Addiction. In: Isaacs SL, Colby DC, editors. The Robert Wood Johnson Foundation Anthology to improve health and health care. Vol. XIII. 2010. pp. 146–173. [Google Scholar]
  4. Budney AJ, Higgins ST, Radonovich KJ, Novy PL. Adding voucher-based incentives to coping skills and motivational enhancement improves outcomes during treatment for marijuana dependence. Journal of Consulting & Clinical Psychology. 2000;68(6):1051–1061. doi: 10.1037//0022-006x.68.6.1051. [DOI] [PubMed] [Google Scholar]
  5. Capoccia VA, Cotter F, Gustafson DH, Cassidy EF, Ford JH, Madden L, et al. Making stone soup: Improvements in clinic access and retention in addiction treatment. Joint Commission Journal on Quality and Patient Safety. 2007;33(2):95–103. doi: 10.1016/s1553-7250(07)33011-0. [DOI] [PubMed] [Google Scholar]
  6. Carroll KM, Rounsaville BJ. A perfect platform: Combining contingency management with medications for drug abuse. The American Journal of Drug and Alcohol Abuse. 2007;33(3):343–365. doi: 10.1080/00952990701301319. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Clemmer TP, Spuhler VJ, Oniki TA, Horn SD. Results of a collaborative quality improvement program on outcomes and costs in a tertiary critical care unit. Critical Care Medicine. 1999;27(9):1768–1774. doi: 10.1097/00003246-199909000-00011. [DOI] [PubMed] [Google Scholar]
  8. Cohen E, Adams N, Dougherty R, Clark JD, Taylor SA. The intersection of transformation and quality in mental health treatment: Implementing the California learning collaborative. International Journal of Mental Health. 2007;36(2):21–35. [Google Scholar]
  9. Cohen LR, Hien DA. Treatment outcomes for women with substance abuse and PTSD who have experienced complex trauma. Psychiatric Services. 2006;57(1):100. doi: 10.1176/appi.ps.57.1.100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. 2009;4(5) doi: 10.1186/1748-5908-4-50. Available at http://www.implementationscience.com/content/4/1/50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Desai R, Harpaz-Rotem I, Najavits L, Rosenheck R. Impact of the seeking safety program on clinical outcomes among homeless female veterans with psychiatric disorders. Psychiatric Services. 2008;59(9):996. doi: 10.1176/ps.2008.59.9.996. [DOI] [PubMed] [Google Scholar]
  12. Evans AC, Rieckmann T, Fitzgerald MM, Gustafson DH. Teaching the NIATx model of process improvement as an evidence-based process. Journal of Teaching in the Addictions. 2008;6(2):21–37. [Google Scholar]
  13. Fixsen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F. Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005. (FMHI Publication #231). [Google Scholar]
  14. Gatz M, Brown V, Hennigan K, Rechberger E, O’Keefe M, Rose T, et al. Effectiveness of an integrated, trauma-informed approach to treating women with co-occurring disorders and histories of trauma: The Los Angeles site experience. Journal of Community Psychology. 2007;35(7):863–878. [Google Scholar]
  15. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly. 2004;82(4):581–629. doi: 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Hien DA, Cohen LR, Miele GM, Litt LC, Capstick C. Promising treatments for women with comorbid PTSD and substance use disorders. The American Journal of Psychiatry. 2004;161(8):1426. doi: 10.1176/appi.ajp.161.8.1426. [DOI] [PubMed] [Google Scholar]
  17. Hoffman KA, Ford JH, II, Choi D, Gustafson DH, McCarty D. Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment. Drug and Alcohol Dependence. 2008;98(1–2):63–69. doi: 10.1016/j.drugalcdep.2008.04.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Jacobsen LK, Southwick SM, Kosten TR. Substance use disorders in patients with posttraumatic stress disorder: A review of the literature. American Journal of Psychiatry. 2001;158(8):1184–1190. doi: 10.1176/appi.ajp.158.8.1184. [DOI] [PubMed] [Google Scholar]
  19. Keyser DJ, Pincus HA, Thomas SB, Castle N, Dembosky J, Firth R, et al. Mobilizing a region to redesign a local system of care: Lessons from a community-based learning collaborative. Family Community Health. 2010;33(3):216–227. doi: 10.1097/FCH.0b013e3181e4bc59. [DOI] [PubMed] [Google Scholar]
  20. Kushion ML, Tews DS, Parker MD. Enhancing Michigan’s local public health accreditation program through participation in the multistate learning collaborative. Journal of Public Health Management and Practice. 2007;13(4):410–414. doi: 10.1097/01.PHH.0000278036.54141.c3. [DOI] [PubMed] [Google Scholar]
  21. Lott DC, Jencius S. Effectiveness of very low-cost contingency management in a community adolescent treatment program. Drug and Alcohol Dependence. 2009;102(1–3):162–165. doi: 10.1016/j.drugalcdep.2009.01.010. [DOI] [PubMed] [Google Scholar]
  22. Martino S, Ball SA, Gallon SL, Hall D, Garcia M, Ceperich S. Motivational interviewing assessment: Supervisory tools for enhancing proficiency. Salem, OR: Northwest Frontier Addiction Technology Transfer Center, Oregon Health and Science University; 2006. from http://www.attcnetwork.org/explore/priorityareas/science/blendingintiative/index.asp. [Google Scholar]
  23. Martino S, Brigham GS, Higgins C, Gallon S, Freese TE, Albright LM, et al. Partnerships and pathways of dissemination: The national institute on drug abuse – substance abuse and mental health services administration blending initiative in the clinical trials network. Journal of Substance Abuse Treatment. 2010;38:S31–S43. doi: 10.1016/j.jsat.2009.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Martino S, Gallon S, Ball SA, Carroll KM. A step forward in teaching addiction counselors how to supervise motivational interviewing using a clinical trials training approach. Journal of Teaching in the Addictions. 2008;6(2):39–67. [Google Scholar]
  25. McCarty D, Gustafson D, Capoccia V, Cotter F. Improving care for the treatment of alcohol and drug disorders. The Journal of Behavioral Health Services and Research. 2009;36(1):52–60. doi: 10.1007/s11414-008-9108-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. McCarty D, Gustafson D, Wisdom J, Ford J, Choi D, Molfenter T, et al. The Network for the Improvement of Addiction Treatment (NIATx): Strategies to enhance access and retention. Drug and Alcohol Dependence. 2007;88(2–3):138–145. doi: 10.1016/j.drugalcdep.2006.10.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano MA. A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting Clinical Psychology. 2004;72(6):1050–1062. doi: 10.1037/0022-006X.72.6.1050. [DOI] [PubMed] [Google Scholar]
  28. Morrissey JP, Jackson EW, Ellis AR, Amaro H, Brown VB, Najavits LM. Twelve-month outcomes of trauma-informed interventions for women with co-occurring disorders. Psychiatric Services. 2005;56(10):1213. doi: 10.1176/appi.ps.56.10.1213. [DOI] [PubMed] [Google Scholar]
  29. Najavits LM. ‘Seeking safety’ therapy for trauma and substance abuse. Corrections Today. 2002;64(6):136. [Google Scholar]
  30. Najavits LM. Treatment of posttraumatic stress disorder and substance abuse: Clinical guidelines for implementing seeking safety therapy. Alcoholism Treatment Quarterly. 2004;22(1):43–62. [Google Scholar]
  31. Najavits LM, Gallop RJ, Weiss RD. Seeking safety therapy for adolescent girls with PTSD and substance use disorder: A randomized controlled trial. Journal of Behavioral Health Services & Research. 2006;33(4):453–463. doi: 10.1007/s11414-006-9034-2. [DOI] [PubMed] [Google Scholar]
  32. Pearson ML, Wu S, Schaefer J, Bonomi AE, Shortell SM, Mendel PJ, et al. Assessing the implementation of the chronic care model in quality improvement collaboratives. Health Services Research. 2005;40(4):987–996. doi: 10.1111/j.1475-6773.2005.00397.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Petry NM. A comprehensive guide to the application of contingency management procedures in clinical settings. Drug and Alcohol Dependence. 2000;58(1–2):9–25. doi: 10.1016/s0376-8716(99)00071-x. [DOI] [PubMed] [Google Scholar]
  34. Petry NM, Alessi SM, Hanson T. Contingency management improves abstinence and quality of life in cocaine abusers. Journal of Consulting and Clinical Psychology. 2007;75(2):307–315. doi: 10.1037/0022-006X.75.2.307. [DOI] [PubMed] [Google Scholar]
  35. Petry NM, Martin B, Cooney JL, Kranzler HR. Give them prizes and they will come: Contingency management for treatment of alcohol dependence. Journal of Consulting and Clinical Psychology. 2000;68(2):250–257. doi: 10.1037//0022-006x.68.2.250. [DOI] [PubMed] [Google Scholar]
  36. Prendergast M, Podus D, Finney J, Greenwell L, Roll J. Contingency management for treatment of substance use disorders: A meta-analysis. Addiction. 2006;101(11):1546–1560. doi: 10.1111/j.1360-0443.2006.01581.x. [DOI] [PubMed] [Google Scholar]
  37. Prescott DL, Madden LM, Dennis M, Tisher P, Wingate C. Reducing mechanical restraints in acute psychiatric care settings using rapid response teams. Journal of Behavioral Health Services Research. 2007;34(1):96–105. doi: 10.1007/s11414-006-9036-0. [DOI] [PubMed] [Google Scholar]
  38. Ragsdale MA, Mueller J. Plan, Do, Study, Act model to improve an orientation program. Journal of Nursing Care Quality. 2005;20:268–272. doi: 10.1097/00001786-200507000-00013. [DOI] [PubMed] [Google Scholar]
  39. Raskin N, Rogers C. Person-centered therapy. In: Corsini RJ, Wedding D, editors. Current psychotherapies. Itasca, Ill: F.E. Peacock Publisher; 1995. [Google Scholar]
  40. Riley W, Parsons H, McCoy K, Burns D, Anderson D, Lee S, et al. Introducing quality improvement methods into local public health departments: Structured evaluation of a statewide pilot project. Health Service Research. 2009;44(5p2):1863–1879. doi: 10.1111/j.1475-6773.2009.01012.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Rutkowski BA, Gallon S, Rawson RA, Freese TE, Bruehl A, Crèvecoeur-MacPhail D, et al. Improving client engagement and retention in treatment: The Los Angeles County experience. Journal of Substance Abuse Treatment. 2010;39(1):78–86. doi: 10.1016/j.jsat.2010.03.015. [DOI] [PubMed] [Google Scholar]
  42. Ruzek JI, Rosen RC. Disseminating evidence-based treatments for PTSD in organizational settings: A high priority focus area. Behaviour Research & Therapy. 2009;47(11):980–989. doi: 10.1016/j.brat.2009.07.008. [DOI] [PubMed] [Google Scholar]
  43. Schwartz AA, Jacobson JW, Holburn SC. Defining person centeredness: Results of two consensus methods. Education and Training in Mental Retardation and Developmental Disabilities. 2000;35(3):235–249. [Google Scholar]
  44. Scott CK, Dennis ML, Foss MA. Utilizing recovery management checkups to shorten the cycle of relapse, treatment reentry, and recovery. Drug and Alcohol Dependence. 2005;78(3):325–338. doi: 10.1016/j.drugalcdep.2004.12.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Silverman K, Svikis D, Robles E, Stitzer ML, Bigelow GE. A reinforcement-based therapeutic workplace for the treatment of drug abuse: Six-month abstinence outcomes. Experimental & Clinical Psychopharmacology. 2001;9(1):14–23. doi: 10.1037/1064-1297.9.1.14. [DOI] [PubMed] [Google Scholar]
  46. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology. 2008;41(3):171–181. doi: 10.1007/s10464-008-9174-z. [DOI] [PubMed] [Google Scholar]
  47. White W, Boyle M, Loveland D. A model to transcend the limitations of addiction treatment. Behavioral Health Management. 2003;23(3):38. [Google Scholar]
  48. Wilson T, Berwick DM, Cleary PD. What do collaborative improvement projects do? Experience from seven countries. Joint Commission Journal on Quality and Safety. 2003;29(2):85–93. doi: 10.1016/s1549-3741(03)29011-0. [DOI] [PubMed] [Google Scholar]
  49. Wisdom JP, Ford JH, Hayes RA, Hoffman K, Edmundson E, McCarty D. Addiction treatment agencies’ use of data: A qualitative assessment. Journal of Behavioral Health Services and Research. 2006;33(4):394–407. doi: 10.1007/s11414-006-9039-x. [DOI] [PubMed] [Google Scholar]
  50. Wisdom JP, Hoffman K, Rechberger E, Seim K, Owens B. Women-focused treatment agencies and process improvement: Strategies to increase client engagement. Women & Therapy. 2009;32:69–87. doi: 10.1080/02703140802384693. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES