Improving quality of care in substance abuse treatment using five key process improvement principles

Abstract

Process and quality improvement techniques have been successfully applied in health care arenas, but efforts to institute these strategies in alcohol and drug treatment are underdeveloped. The Network for the Improvement of Addiction Treatment (NIATx) teaches participating substance abuse treatment agencies to use process improvement strategies to increase client access to, and retention in, treatment. NIATx recommends five principles to promote organizational change: 1) Understand and involve the customer; 2) Fix key problems; 3) Pick a powerful change leader; 4) Get ideas from outside the organization; and 5) Use rapid-cycle testing. Using case studies, supplemented with cross-agency analyses of interview data, this paper profiles participating NIATx treatment agencies that illustrate application of each principle. Results suggest that the most successful organizations integrate and apply most, if not all, of the five principles as they develop and test change strategies.

Introduction

Health care providers are increasingly under pressure to implement evidence-based decision making and reduce inefficiency and errors in general medical care.^{1, 2} The Institute of Medicine’s Crossing the Quality Chasm called for health care organizations to improve their quality of care through redesigning care processes and systems.¹ More recently, these recommendations were extended to services for alcohol, substance, and mental health problems.³ Yet, the organizational structures of most substance abuse treatment agencies are weak and demand is great, leaving the system unable to meet additional demands for access or for better quality care.⁴

Incorporating process improvement approaches into the management of addiction treatment services helps address this problem. Treatment centers modify service delivery to enhance access and retention with only marginal expansions of agency infrastructure and resources.^5–7 The rationale for process improvement recognizes that demand for treatment services has grown beyond current service capacity and that dramatic increases in resources for addiction treatment are unlikely. Thus, the efficiency and effectiveness of existing service systems must improve if treatment goals are to be met, particularly in the areas of admission processes and early engagement efforts.⁶

Network for the Improvement of Addiction Treatment

The Robert Wood Johnson Foundation (RWJF) and the Substance Abuse and Mental Health Services Administration’s Center for Substance Abuse Treatment (CSAT) combined resources to support a demonstration project that would use process improvement techniques within participating drug and alcohol treatment organizations. The project, entitled The Network for the Improvement of Addiction Treatment, is a learning community of addiction treatment centers seeking to improve the delivery of treatment. Participating agencies strive to remove inefficiencies that delay admissions and contribute to missed appointments and early drop out by testing organizational changes.

This paper provides in-depth case study examples of the five NIATx (Network for the Improvement of Addiction Treatment) process improvement principles, applied in real-world settings. The five principles, designed to foster improvements in addiction treatment processes, evolved through a review of product innovation and process improvement research⁸: 1) Understand and involve the customer; 2) Fix key problems; 3) Pick a powerful change leader; 4) Get ideas from the outside; and 5) Use rapid-cycle testing. NIATx provides collaborative learning opportunities and technical assistance to treatment agencies so that they can improve organizational processes and increase capacity. Member agencies attend learning conferences, receive coaching from process improvement experts, participate in topic-specific conference calls, and have access to tools from the NIATx website (www.niatx.net) and its process improvement examples, learning kits, and suggestions. All are implemented using rapid-cycle change processes (Plan, Do, Study, Act or PDSA cycles) with the goal of to reducing days to admission and enhancing retention in care. For example, a powerful tool for promoting client-oriented changes is the “walk-through”.⁹ In this practice, a staff person takes on the role of a customer in order to experience first-hand the organization’s processes. A typical walk-through includes the first contact, intake and assessment process, and the first treatment visit.⁹ The staff person records his/her impressions of the services received and notes needed improvements. These experiences are shared with the change team which then engages in Plan Do Study Act change efforts to improve their system.

A cross-site evaluation of NIATx agencies found significant improvements in delivery of care, including a 37% decline in days to admission, an 18% gain in retention in care between the first and second treatment session, and a 17% gain in retention between the first and third treatment session.⁶ A second NIATx cohort replicated the improvements.¹⁰

This paper elucidates the story behind these statistics, providing qualitative analyses and examples of processes adopted by agencies that successfully implemented the NIATx principles. The following section reviews the theoretical underpinnings behind the five principles.

The NIATx Five Principles of Change

Principle 1: Understand and Involve the Customer

Successful organizational innovations are customer-oriented.^11–14 The first NIATx principle calls for a “continued commitment to understand the needs and expectations of people who can benefit from the products and/or services produced by the organization”. ⁸ Customer-oriented organizations often involve those they serve in developing and assessing improvements, confirming that improvements meet expectations.¹⁵ This principle can pose significant challenges in alcohol and drug addiction treatment resulting from clients’ disparate but related needs, such as co-occurring mental health disorders, criminal justice involvement, homelessness, domestic abuse and child-care. These concerns require comprehensive, client-focused approaches to adequately meet treatment needs.^16–20 Demand for treatment often exceeds capacity leading some programs to use standard protocols rather than individualizing each treatment plan. NIATx promotes alternative approaches for promoting organizational efficiency while maintaining sensitivity to clients’ needs.

Principle 2: Fix Key Problems (and Help the Chief Executive Officer Sleep at Night)

Principle 2 links organizational change to agency goals by directing change agents to focus on processes that are critical to the organization. Although improvement projects can either focus on incremental or global changes, participating NIATx agencies are encouraged to focus on the most pressing of problems, those “that keep the chief executive officer up at night.” The underlying rationale for this approach is that problems plaguing agency leadership tend to gain support and resources for process improvements. Specific targets can vary, but agencies are counseled to link changes to salient organizational objectives such as reducing staff turnover or improving client retention. Agencies are urged to use process improvements and organizational changes to improve business financial status; for example, by reducing no-show rates and time to admission, reduce inefficiencies and budget shortfalls.¹⁵

Principal 3: Pick a Powerful Change Leader

Principle 3 addresses how to select change leaders (individuals directing PDSA cycles and coordinating organizational change teams) to successfully lead process improvements. Powerful change champions need to 1) have adequate authority and respect to assemble and motivate groups,⁸ 2) have enough power and social capital to lead change efforts,²¹ 3) be able to encourage participation and buy-in by winning the “hearts and minds” of staff,²² and 4) provide teams with a clear vision that is attractive to them.²³ To be successful, change leaders must have access to the CEO, be able to allocate resources to the change effort, and be committed to change goals, processes, and results.⁸ In addition, making change a part organizational culture requires top leadership to actively articulate and demonstrate commitment to changes and the change process.⁸ The probability of measurable success is likely to improve when a CEO transparently authorizes and supports the change leader. Staff resistance to change, a common barrier to process improvements, can be mitigated by strong leaders who bridge “organizational segmentalism” and department barriers that prevent or slow the process of change.²⁴ Leadership strategies that use consistent communications create a vision for change, promote staff engagement in the change process, and can help overcome staff resistance.^{21, 25}

Principle 4: Get Ideas from Outside the Organization

The fourth principle encourages organizations to seek input from outside their field. In fact, process improvement’s emphasis on effectiveness and efficiency draws heavily on continuous improvement models initially developed to improve manufacturing processes.^{26, 27} Applications of process improvement to business management and health care processes have demonstrated the capacity of organizational change to enhance operational efficiency ²⁸ and strengthen patient outcomes.^{29, 30} Change teams are encouraged to seek out these ideas and capitalize on other organizations’ successful improvements, including production and business practices from other domains (e.g., the fast food, transportation, hospitality industries).¹⁵

Principle 5: Use Rapid Cycle Testing

The fifth principle entails the planning and testing of process improvements. NIATx suggests using rapid-cycle PDSA methods to identify problems and generate solutions (Plan), implement new processes (Do), measure and assess outcomes (Study), and institutionalize the change or make additional changes (Act).^{31, 32} Agencies participating in NIATx receive training in conducting PDSA cycles, then are coached as they use a series of linked PDSA cycles designed to improve processes. Changes are piloted with a small sample of staff and patients to assess feasibility and initial effectiveness. If effective, the intervention is institutionalized. If results are ineffective, or only partially effective, the change initiative is modified and tried again until successful, or until a decision is made to abandon the approach. A crucial component of the PDSA method involves “statistical mindedness,” meaning that agencies need to “confirm the existence of problems, identify opportunities for improvement, and evaluate the effectiveness of process improvements” based on data from PDSA cycles or customer reports.⁸

Although a substantial body of theoretical knowledge about the application of the aforementioned five principles can be found in other settings (e.g., business and management literature,³³ few clear examples exist in behavioral health. Consistent with Fernandopulle and colleagues’ ³⁴ suggestion that “case studies provide essential qualitative insight into attributes associated with quality performance” (p. 182), this paper uses case studies from participating NIATx agencies, combined with cross-agency analyses, to illustrate successful applications of the principles in addiction treatment agencies.

Methods

Participant selection and characteristics

The sample consists of NIATx members chosen to participate in the Paths to Recovery and Strengthening Treatment Access and Retention programs. Substance abuse treatment agencies seeking participation in NIATx submitted six-page letters of intent that highlighted their agency. Eligibility was limited to nonprofit agencies with at least a 50% publicly funded client base. Agencies were also required to have at least 100 admissions per year and the ability to control a continuum of care either within the organization or through arrangements with other organizations. As part of the application process and as an introduction to process improvement, applicants received instructions for completing an admissions walk-through exercise and described their agency’s strengths and weaknesses in their letter of intent, basing their answers on the walk-through findings. Applicant organizations submitted 327 six-page letters of intent which were then adjudicated by NIATX program staff based on the following criteria: 1) potential community impact, 2) evidence of a leadership commitment, 3) ability to move patients between levels of care, 4) systems to collect and report data, 5) suitable financial health, 6) perceived strength of the project team, and 7) a willingness to participate in a learning community. RWFJ made 10 awards to community treatment programs in August 2003 and 15 more awards in January 2005 (awards were for 18 months). One additional program participated without RWJF resources. CSAT awarded 13 grantees in September 2003, providing them with 36 months of support. Further details about the initial round of applications and awards can be found in Ford and colleagues.⁹ Treatment agencies were located in rural and urban settings in 17 states, varied by size and services provided, and represented agencies providing multiple levels of care, including detoxification, inpatient treatment, programs for pregnant and parenting women, intensive outpatient treatment, outpatient treatment, and adolescent care. Annual revenues ranged from less than $300,000 to over $11 million. (For additional information on applicant characteristics, see Ford and colleagues;⁹ for additional sample characteristics, see McCarty and colleagues⁶).

Procedures

Annual face-to-face site visits and quarterly telephone interviews were completed between October 2003 and May 2006. The qualitative interview guide was used semi-structured and included both individual and focus group interviews. All interviewees were informed about confidentiality, freedom to participate and the right to withdraw from the study at any point. The Oregon Health & Science University Institutional Review Board for the Protection of Human Subjects approved the study.

Interviews were conducted by the OHSU evaluation team and all interviewers held PhD or masters level degrees. Interviews generally lasted about one hour and covered the following dimensions: new change initiatives and impacts, how staff were sustaining previous change initiatives, how staff were using data to make data-driven decisions, change leader effectiveness, staff buy-in, staff empowerment, agency stability, communication about changes, adaptive culture, customer focus, and learning from others. Interviewers audio-recorded interviews, summarized findings, and documented reflections about the interviews and site visits. After completion of the interviews, the researcher would write a summary that was then shared with the change leader to discuss and clarify the interpretation, and contribute new or additional perspectives. An iterative approach to data collection and analysis was taken to ensure saturation was reached.

Data analysis took a deliberate and systematic approach. The research team developed a Manual of Procedures for investigators to follow during coding and analysis. The team began with initial read-throughs of a number of interviews, making marginal notes regarding possible themes and ideas which were then compared by the group. Thematic codes were developed inductively as the transcripts were reviewed, allowing the data to dictate the analytic categories (Glaser & Strauss, 1967). After coding each transcript using coding categories that were mutually agreed upon by the first two authors, a series of automated text searches were subsequently run to cross-check several of the categories. The contents of each coding category and query were reviewed in detail by the evaluation team to ensure agreement on the nature of participants’ responses to the interview questions. Common responses were highlighted and grouped, as were quotations that best illustrated the most common themes. Evaluation staff took several steps to increase methodological rigor: (a) multiple evaluators participated in data collection and analysis to ensure multiple viewpoints and discussion of perceptions of data, (b) evaluators sought consensus on coder agreement to ensure more accurate coding, (c) evaluators considered rival explanations while analyzing data to facilitate trimming and validating the theoretical scheme, and (d) agency representatives reviewed text to ensure clarity and provide detail when needed.

To ensure consistency across coders, we check coded a 10% random selection of interviews using a subset of key codes, evaluating 526 coded passages. An additional coder reviewed inconsistencies between primary coders and check coders to resolve inconsistencies; primary coders’ were accurate in 89.3% of the 526 passages. A “best cases” case-study approach was used to illustrate successful experiences from participating agencies, then report related cross-agency findings from our thematic analyses. Cases were collaboratively chosen by the research team based on exemplary implementation of one or more of the five NIATx principles. For cross-agency analyses, thematic content was extracted from interview summaries by reviewing queries generated using ATLAS.ti 5.0 software.³⁵

The case study approach is designed to elicit details from the viewpoint of the participants by “developing an in-depth analysis of a single case or multiple cases”.³⁶ The method allows researchers to explore individuals or organizations, simple or complex interventions, relationships, communities, or programs (Yin, 2003). When the approach is applied correctly, it becomes a valuable method for health science research to develop theory, evaluate programs, and develop interventions. These methods have become important tools in the study of health care delivery.³⁷ Amongst many others, organizational improvement case studies within the behavioral health care field include a benchmark study of the successful application of total quality management (TQM) to the mental health field (Sluyter and Barnette, 1995), the use of program specification and accountability evaluation to monitor service delivery (Fleming and Bolig, 1998), a multi-site intervention designed to improve access to care for chronic drug users (McCoy, Messiah, and Zhao, 2002), and a multiple case study of how staff in child out-of-home care programs used data to make data-driven decisions to inform program decisions (Patel and Riley, 2007.

Results

Results are organized around the five NIATx principles. Within this framework, data were reviewed and sites were selected for in-depth analyses based on documented efforts to address the respective principles, as well as the thoroughness and consistency of those efforts. Note that, although each case study is used as an exemplar for a single principle, most case example include applications of some of the other principles, providing evidence for the value of simultaneously using all multiple principles. Pseudonyms are used in the following report of results to protect anonymity of agencies and individuals.

Principle 1: Understand and Involve the Customer

Esperanza House (pseudonym) provides examples of how attention to client needs and receptivity to their feedback can improve care agency-wide. Esperanza House is located in an urban, southern city and serves about 100 homeless and indigent clients each year. The agency offers a full continuum of care for substance users and their families, including outpatient counseling, residential drug treatment and prevention, and community education. Esperanza House used NIATx principles to apply customer-focused changes after walk-throughs suggested they were not meeting client needs.

The change leader asked clients to brainstorm ideas about quality improvement and invited them to serve on PDSA change teams:

The customer focus aspect has been a big part of our change projects and change teams. All the change teams have had customers as part of their group - either current or former clients.

Many Esperanza House PDSA cycles were “driven by clients wanting better customer service as soon as they [walked] through the door.” For example, one recommendation by the change team concerned the reorganization of support staff to make the waiting room and intake process more “customer-friendly.” Early PDSA cycles included: (1) training back office staff to answer client questions and moving them into the building lobby, and (2) training intake coordinators to focus on client-related needs such as making follow up appointments and reducing barriers to returning for appointments. Lastly, a billing staff person was trained to engage clients and perform individual financial consultations and payment planning. According to the change leader, this latter change had some unintended but pleasing results:

Results of this change cycle have been very positive: Client self-pay fees have increased from 22% per month to around 90%. Support staff are less burned out, and the reorganization has allowed [Esperanza House] (at support staff request) to reduce their hours…which increases our revenue. We were just looking for better service - we didn’t anticipate the increase in fee collections.

When asked how all of these customer-focused organizational changes generally affected the agency, the change leader reported:

Overall it’s been very positive. Clients come in and feel a little more welcomed. We have reduced the paperwork so it’s not as frustrating as it was. We do a better job at explaining the process to them and what the end results may be. Clients seem to be more willing to work together in the group. There’s a different tone. It’s a general atmosphere, a different feeling.

She also indicated that “client[s] may be better drivers [for change] than we are.” As a result of their PDSA cycles, this agency adjusted walk-in times to be more inclusive, offered additional treatment modalities and techniques such as biofeedback, and restructured the treatment orientation process to include linkage with an existing client to provide sober support. They also trained their staff to be more alert to client disengagement and implement motivational techniques to prevent early drop out.

Esperanza House provides an example of implementing strong consumer involvement in process improvement through changes that involved minimal staff time and resources. Other NIATx agencies varied significantly on this dimension. Some made efforts to solicit client feedback and understand their needs, but most did little to involve customers directly in organizational changes or PDSA cycles. For example, clients were infrequently included as members of change teams or invited to provide input when changes were planned. The most common approach to involving the customer tended to be collecting client response after a PDSA cycle was initiated, although this was rarely institutionalized and often tended to be impromptu and informal. Additionally, when client surveys were implemented, staff reported that they often “did not ask the right questions” and resulted in general information that could be helpful but yet did not directly address the organizational changes that had been made or allow clients to provide targeted responses for further improvements. Esperanza House provides an example of the many benefits agencies can achieve if they implement Principle 1 comprehensively.

Principle 2: Fix Key Problems

New Life (pseudonym) is a mid-size, urban, multi-modality clinic. The agency’s clients are generally indigent and unable to pay for treatment. In 2005, New Life’s facility was devastated by flooding. The organization had difficulty re-opening due to limited financial resources and a severe staff shortage. These key problems were compounded by the fact that staff who remained at the agency after the flood were compelled to focus on non-treatment-related problems such as cleaning the facility, organizing volunteers, recruiting replacement staff, and addressing general infrastructure needs such as restoring water and power. In addition, as clients began to return to the facility, staff discovered that the clientele had new and substantial mental health needs as a result of their experiences during and after the flood. The change leader reported:

The clients we have [now] are… suffering from Post-Traumatic Stress Disorder. And they are maybe just blocking the whole…experience out. You can be having a conversation and they would just out of context say, “I can’t believe what I saw…dead bodies.”

Additionally, New Life had experienced a dramatic reduction in resources, including many staff who were displaced and did not return to the organization. The impact was disorienting to the clients:

I am having to explain to people who want to come back that we are not the [New Life] we used to be. They are coming back with the expectation of routine and normalcy like [pre-flood] and when they come they are confused and upset.

As a result of these instabilities, clients were leaving treatment prematurely. To address this issue, the change leader and CEO worked with staff and clients on the change team to implement a PDSA cycle designed to facilitate a major organizational and cultural shift. The clinical director felt that, before the flood, too much staff time had been consumed with “babysitting” and conducting “detective” work to root out banned activities. To address this problem, the change cycle called for a discontinuation of the “policing” of clients and instead required clients to be “responsible for their surroundings and each other.” As part of the change in organizational policy, and consistent with Principle 1 (Involve the Customer), the clients were given the responsibility of addressing and resolving these kinds of concerns. For instance, if a client was behaving disruptively, the other clients were responsible for calling a community meeting so that all clients could discuss and resolve the problem. This resulted in clients being empowered and accountable to each other for upholding community procedures and rules. The goal of this change was to encourage engagement in treatment, increase retention in the client community, and reduce the amount of staff time devoted to creating and enforcing rules. A change team, composed mostly of clients, was charged with the task of fostering engagement in newly admitted clients. The change leader reported on one of their change activities:

[This group] came up with a greetings system of their own and an orientation twice a week to orient the new clients. Of the 50 people who were here last Friday, 40 of the 50 had been here for more than 30 days. That shows great success on their part….

Staff also gave residents responsibility for creating agency guidelines about recreation time, personal devices allowed in the facility, and even how to address drug use among clients. The change leader explained:

If the community agrees, they can have a cell phone. CD players, the community now decides. You must report when you think it’s a problem. Is this person still a productive member? Even if we [staff] may think it not the best thing, if the community gets together and recognizes value, they take ownership of it. Many of the things that people use as excuses to leave, like ‘I have no freedom I’m getting the hell out’; a lot of those barriers are removed. The group also decides what to do when an individual relapses.

A support staff person agreed that this shift had been very successful and was pleased with the reduction in time spent on non-clinical interventions:

Some of the things we used to constantly battle over [were things such as] TVs and cell phones. We have waved the white flag and surrendered. We are now spending time on things that are more effective. The community polices itself on those issues. There is less bickering among the clients themselves. There is overall more general cooperation. That is echoed by a lot of clients who have returned. I have talked with many of them and they think the whole atmosphere is a lot different.

To quantitatively monitor the results of this change, continuation rates were collected and analyzed, showing significant improvements:

When we started to involve the clients more in the change, there was a dramatic jump in continuation [retention in treatment]. We have also looked at our retention numbers for November [baseline] and December [2005] and they were 60 and 69% respectively. About 75% have now [February 2006] been here more than 30 days. I think it is a big by-product of the previous changes.

New Life provides an example of an organization using a crisis to overhaul their approach and organizational structure and address key problems. While participating agencies were generally encouraged to begin process improvements slowly, maintaining focus on small, incremental changes while building change team infrastructure, New Life took a host of issues head on, all at the same time, and was successful. Some participating organizations did not progress beyond the stage of small changes, while others tended to focus on what appear, in the final examination, to be less important changes. For example, initial change efforts often targeted aesthetic or tangential issues such as alterations in food or beverages provided to clients during group sessions or minor cosmetic upgrades to waiting rooms. While these changes may be helpful, more critical issues often went unaddressed.

Pick a Powerful Change Leader

Beth (pseudonym) is director of program development and quality at a large, urban, multi-site treatment organization, “Sober Living.” She illustrates some of the most important qualities of an effective change leader. In particular, although Beth took the lead on ideas for change initiatives, over time she encouraged change team members to select their own targets. She remained actively involved in decision-making as the team took on more responsibility, while maintaining an even-handed approach to chairing the change team. This section describes the key themes found in analyzing the approaches of effective change leaders illustrated by Beth’s example. Her approach included the following: a) attention to collecting and analyzing data, b) ability to communicate clearly about change cycles, c) support from the agency’s CEO, d) unfailing enthusiasm, and e) encouraging others’ ideas.

Beth routinely incorporated the collection and analysis of data into the change team’s process improvement strategy, reporting that data were not just useful but “essential [to their change efforts] and always surprising.” She maintained the change team’s focus on specific outcomes of interest, avoiding much of the confusion other change leaders struggled with over identifying appropriate methods. These data-driven process improvements were based not only on quantitative data collected for each change cycle, but also face-to-face interviews with clients (Principle 1). One change team member explained how they had continuously interviewed clients about how changes had affected their personal experiences. For example, when conducting a change cycle at a men’s residential program, the change leader and her team interviewed residents individually. Beth reported that, in addition to addressing specific change cycle outcomes, face-to-face interviews with clients “energized the change team and the residents,” and were vital to making good decisions:

Asking the client what matters to them helps staff get around prejudices about what is important to the client. Staff tend to only change or look at what fits in with their own perception of what is important.

As with most agencies involved in NIATx, many staff at Sober Living resisted the idea of making changes to the organizational processes with which they were comfortable. This reluctance to change enhanced the need for effective leadership. Beth encountered resistance agency-wide, but found it to be particularly strong within the agency’s detoxification unit. During a focus group interview with staff, they reported that when they first started working on the change initiatives in the detoxification program, they resented Beth, who was seen as coming from another part of the program and therefore a staff person who did not understand their needs. They described how resistant the unit had initially been to Beth, given her outsider status, with comments like: “Who is she coming in here and talking about possible changes? She doesn’t even work in this unit.” Beth gained acceptance and buy-in through effective communication with staff about the change process. She engaged detoxification staff in conversation and with e-mail correspondence, thoroughly explaining the PDSA process and how the change team determined what changes should be implemented. She elicited and addressed their concerns, encouraging two-way communication. As a result, Beth was able to help the unit 1) improve the quantity and quality of information provided to clients about the intake process and treatment; 2) streamline intake paperwork so that it was less burdensome to staff; and 3) cross-train staff to address all components of the intake process in order to avoid delays.

In addition, Beth followed through with middle managers in the detoxification unit to ensure that they were communicating effectively with unit staff and clients about process improvements. Rather than being passively receptive to feedback, she routinely interviewed staff about how change efforts were going and regularly disseminated results of the change cycles by e-mail. Staff buy-in and support for the projects increased as they saw evidence that their efforts were producing real improvements, and they became more invested in the change process. Increasingly, ideas “bubbled up” from them for additional change cycles.

In addition to Beth’s effective leadership style, she also had strong support from the agency’s CEO. Without this type of foundation, change leaders may feel that change efforts are not given adequate priority over day-to-day operational activities. In Beth’s case, the CEO “always provided [her] with the support she needed as he talked to [her] almost daily.” This support was noted by other staff members who “felt that the change leader was committed 110% to the project and that she had the support of the executive director.”

This support from the CEO was largely due to Beth’s enthusiasm about using change cycles to improve the organization. Her energy was also critical to gaining the trust and buy-in of staff members. The interviewer noted that she was generally “very excited” about the change process and “really put a lot of effort into it.” In order to accomplish her new duties as change leader, approximately 50% of her time was devoted to agency quality improvement. Her belief in, and dedication to, process improvement in turn infused staff with enthusiasm and resulted in near universal acceptance. “She has won everyone over and is much respected,” reported a change team member.

Beth’s experience shows how effective a strong change leader with excellent support from upper management can be. Across the other NIATx agencies, interviews with support staff indicated a wide range of leadership methods and styles which were met with varying levels of success. Primary pitfalls included unclear expectations for staff, lack of or confusing communication with change team members, turnover within the change leader position that created instability, lack of autonomy, and lack of adequate authority.

Principle 4: Get Ideas from Outside the Organization

Two participating agencies illustrate how organizations can leverage knowledge gained from other institutions. These agencies were from the same state but participated in different NIATx cohorts. Step Ahead, the agency funded in the first cohort, had been engaged in process improvements for a longer period of time than Choices (in the second cohort of grantees. Initially, the flow of information was designed to be unidirectional—from the more experienced Round 1 agency, to the Round 2 site. Soon after their initial interactions, however, staff discovered that the transmission of knowledge between these agencies was bidirectional and that much could be learned by the “more-experienced” agency from the “less-experienced” one.

The idea for engaging in the sharing process was initiated by Choices’ CEO and change leader. Choices’ first PDSA activity was to improve their procedures for assessing walk-in clients. During preliminary discussions between their respective CEOs, Choices learned that Step Ahead had already undertaken this as a change project. The Choices change leader explained:

We knew of [Step Ahead] because of the NIATx grant and that that was one of their changes… we thought, ‘Hey this is an idea that has already been tried so we might as well learn from them and see how it works’.

Step Ahead was receptive to sharing its change cycle process and hosting an information session. According to their change leader:

[Choices] decided to call us when they found out that we were [in the] first round and we… agreed…and arranged a site visit. [The CEO of Choices] came over with 6 or 7 of his staff including counselors, intake staff, and a few data people.

The site visit was scheduled over an entire day and followed a structured agenda. During the visit, Step Ahead provided specific advice about how they changed their assessment procedures for walk-ins, as well as general information about the agency, its data systems and details about other NIATx change cycles they had implemented. After a period of initial introductions in the morning, staff divided into four groups: 1) the counselors discussed clinical interventions; 2) the executive sponsors compared business case plan development, diffusion of ideas, and sustainability; 3) the change leaders met and discussed how to complete change cycle forms, choose change team members and conduct change team meetings; and 4) the data managers met and discussed the quickest and most effective ways to provide feedback to change teams and other staff members.

After the group sessions, all participants came together for general discussion and networking. Step Ahead then conducted a tour of several of its facilities. Choices staff were able to observe a client orientation group, a new process that had arisen from a Step Ahead PDSA cycle (previous orientations were completed with individuals rather than in groups). This exchange also resulted in learning about unexpected commonalities between the programs:

We mainly went to [Step Ahead] to learn about the walk-ins but we learned that we have the same computer system and our computer people were talking to their computer people to get ideas about how to use it more effectively. There was a lot of information sharing there, because they were using it differently.

One of the primary tools discussed at the site visit was Step Ahead’s rapid data collection, analysis, and dissemination of information from PDSA cycles. Although methodical reporting of change cycle data was intrinsic to the global NIATx evaluation, Step Ahead found that the monthly accounting system was inadequate for their needs and that change cycles were unnecessarily long as a result. Step Ahead shared their system of accelerated process-improvement monitoring as well as creative ways to disseminate the results using PowerPoint presentations and e-mail.

At the close of the day, the group returned to Step Ahead’s corporate offices for a question-and-answer session. Participants then “decided on a phone call agreement that if [Choices] needed anything they could call [Step Ahead] or vice versa.” Step Ahead’s change leader reported that the relationship between the two agencies began to feel like a team:

It became a collaboration. I could call [Choices’ data manager] and say, ‘I’m running into this’ or ask, ‘What did you do for this?’ I could tell her, ‘I can’t seem to get these numbers—do you have an easier way to run the program?’ She is really amazing.

After the site visit, Choices staff reflected on what they had learned and how to incorporate Step Ahead’s recommendations. They reported that the site visit had been extremely valuable and they had gained a tremendous amount of useful “first hand” information. And, after reflection by the group, Choices staff concluded that their organization could benefit from almost all of the lessons learned at Step Ahead by adapting their methods to suit Choices’ particular set of clients and services:

We decided at that point that we did not want to [manage walk-in clients] the same way that they did. But it was very helpful information and helped us make that decision. Shortly thereafter, we piloted our walk-in [procedures].

Likewise, Step Ahead’s change leader reported:

We both exchanged ideas – they got more from us because we were in the process for over a year, but we got ideas from them as well as far as efficiency. We loved having them. Later, I heard [Choice’s executive sponsor] speak of how much they were able to do so quickly because they were able to observe us and get peer to peer support; it helped them grasp the [NIATx] concept quicker.

Step Ahead and Choices continued to collaborate and share information with one another over the course of the intervention. Our cross-agency analyses of all NIATx agencies, however, suggest that “getting ideas from the outside” was the least incorporated of NIATx’s five principles. This appeared to be due in large part to lack of time, staff shortages, and the concentration of effort on change projects within the agencies. Moreover, there was little opportunity for in-person agency-to-agency collaboration due to the geographical separation of most participants. Exceptions to this were informational exchanges and networking that took place at NIATx Learning Collaboratives (yearly NIATx conferences, regularly attended by key agency staff such as change leaders and CEOs, as well as other staff involved in the change process). These collaboratives provided structured opportunities to learn from other agencies’ efforts via presentations as well as informal opportunities for participants to talk with each other about their experiences. Attendance at these collaboratives was highly valued:

Learning from others in the NIATx network is very helpful, especially the learning collaborative. They helped us learn about PDSAs and aims and about process improvement. The topics were good--especially sessions about increasing admissions and marketing. Also, there were opportunities to learn and talk with other agencies. We can learn a lot from others’ experiences.

Principle 5: Use PDSA Rapid Cycle Change Projects

Transformations (pseudonym) treats several hundred patients a year in a northeastern city. It provides inpatient and outpatient treatment, methadone-maintenance, and detoxification services to a mostly publicly funded population. Transformations provides an excellent example of the use of a series of linked PDSA cycles designed to improve retention in the agency’s outpatient and residential treatment facilities. To begin, the change team brainstormed about how to get clients to engage in and complete outpatient treatment. Baseline data showed that early treatment drop-out was a serious problem in their outpatient services: 50% of the clients that completed intake appointments did not return for the first treatment appointment. The change team considered three questions before designing their change projects:

1. “What are we trying to accomplish?” The change team decided the goal of the first change was to increase early retention (defined as four appointments within 30 days) among outpatient clients.

2. “How will we know a change is an improvement?” The team decided that changes would be deemed successful if they increased the percent of admitted clients per month who attended treatment appointments following their intakes.

3. “What changes can we test that may result in an improvement?” The change team decided that clients may not have a proper understanding of the attendance policy and that the change should focus on increasing comprehension of that policy.

The team brainstormed and decided to try a short procedure to have all clients read the attendance policy out loud during the intake session (Plan). Any points that needed clarification were addressed to ensure each client’s comprehension of the agency’s attendance expectations. The change was implemented (Do), but after tracking the data, they found no change in retention rates in the month following implementation (Study), thus abandoned that change in favor of a different method of improving early retention (Act).

The second PDSA cycle was more complex: The team decided that they would train intake staff in Motivational Interviewing (MI) ³⁸ and ask staff to implement MI techniques during the intake appointment (Plan). After staff were trained, the intervention was implemented (Do) and data were analyzed (Study). Over the following month, retention (over four treatment sessions) after intake increased from 50% to 90%. At that time, the change team agreed on an appropriate action (Act): They would adopt MI training for all intake staff and make MI processes a part of the agency’s standard procedures.

Following this PDSA cycle, staff were so pleased with the improvement in retention that the same PDSA cycle was initiated for their residential facilities. In June, 2005, the baseline residential retention rate (the proportion of admitted clients who completed one week of participation in treatment) was 80%. Anticipating an improvement, the change team planned motivational interviewing training for residential intake staff and implemented it during the residential intake process. By October, 2005, continuation rates had risen to 100% for that month. The change was considered a success and adopted as standard procedure in the residential unit.

To increase staff interest and buy-in for the changes, the change team posted week-by-week tallies of change-related data assessments so all staff could see the results of the process improvement efforts. In addition, the change team regularly solicited ideas for change cycles from all staff, and were so excited and motivated by staff ideas that they implemented a number of changes at once. Their NIATx coach observed that because so many changes were going on at the same time, it was impossible to truly detect which changes were having positive effects. He challenged them to slow down and complete existing change cycles before implementing new ideas.

The progress seen through these initial successes increased staff attention to customer needs and inspired new change cycles designed to further increase retention in care, including (a) formally introducing new residential clients to staff and providing a tour of the facilities; (b) creating a ‘health management group” for clients on psychiatric medications to provide information on specific conditions and needs; (c) changing medication administration times to allow clients an extra hour of sleep; and (d) instituting an “open door policy” to allow clients to express any concern or need to clinical staff or supervisors. A counselor reported that the agency’s efforts made it easier for clients to become comfortable enough to succeed in treatment:

A value of initiating the [“open door policy”] change and documenting it is that it teaches each person to take ownership and do their part to make a difference. We practice eliminating the reasons clients want to leave by being flexible in what we can do. We eliminate these reasons until we get to the core of what’s going on. We give them a little time for it to get better, and after 30 days, they’re more comfortable. It’s intimidating to come work here with this many clients, much less to come in to treatment here.

In addition to the beneficial effects on client retention, Transformations’ change leader reported that the NIATx framework provided an approach to process improvement activities and PDSA cycles that made it easier for staff in all the agency’s units to embrace process improvements:

[The PDSA cycles] have made a big difference in how we approach change. It’s provided a structure for us to make changes and improvements, in a way that’s measurable and in a way that’s easy for people to use. The changes feel less overwhelming because of the process.

In cross-agency analyses, a wide variety of responses to implementing PDSA cycles were observed. Some agencies had difficulty cycles, while others adopted the process quickly. Challenges included (a) linking change cycles to specific principles and identifying appropriate outcome measures [planning], (b) accurately tracking pre- and post-change activities [doing], (c) accurately interpreting differences in pre-post measures [studying], and (d) determining appropriate next steps [acting].

Discussion

The agencies highlighted in these five case studies illustrate the value of NIATx principles when fully and carefully implemented in substance abuse treatment settings. Esperanza House’s customer-oriented changes increased client engagement and improved the agency’s financial status. New Life’s shift to a community model with greater client control significantly increased retention in care. Sober Living’s dynamic change leader, with her attention to rigorous data collection and clear communication about change cycles, illustrates the value, effectiveness, and qualities of strong change leaders.

Step Ahead and Choices capitalized on each others’ organizational improvement lessons by sharing experiences, including clinical interventions, business case development, how to create change teams and conduct change team meetings, and how to provide feedback to staff about the effects of change efforts. Lastly, Transitions used a series of linked PDSA cycles to improve retention, increasing the percent of clients who attended a first treatment session after intake from 50% to 90% in their outpatient unit and from 80% to 100% in their residential unit. These agencies’ experiences elucidate how simple strategies, implemented at different organizational levels, can produce valuable improvements in the delivery of substance abuse treatment.

Implications for Behavioral Health

As demand for treatment services grows beyond current service capacity and resources, process improvement becomes even more important for patients and staff in behavioral healthcare settings. The primary limitation of this study is that the findings are confined to NIATx agencies - agencies that had to recognize the value of process improvement to consider applying for grant funding from the RWJF or CSAT, agencies that successfully completed a walk-through exercise, and also submitted grant applications. Thus, they likely represent agencies more amenable to process improvement strategies from the outset. At the same time, the commonalities of experiences reported across NIATx-participating organizations and as part of NIATx funding applications ⁹ suggest that the lessons learned from these case studies and our cross-agency analyses may have broader applicability. Finally, although these exemplary cases are not themselves representative, they provide valuable opportunities to learn from successful change efforts. By improving administrative and clinical functions, agencies increase the likelihood that they will improve client access to and retention in treatment, thus improving outcomes. With resources such as NIATx, process improvement strategies are accessible to organizations that could not otherwise afford to undertake these efforts.