Lessons Learned about Implementing Research Evidence into Clinical Practice: Experiences from VA QUERI

Abstract

The mission of the Veterans Health Administration's (VHA) quality enhancement research initiative (QUERI) is to enhance the quality of VHA health care by implementing clinical research findings into routine care. This paper presents lessons that QUERI investigators have learned through their initial attempts to pursue the QUERI mission. The lessons in this paper represent those that were common across multiple QUERI projects and were mutually agreed on as having substantial impact on the success of implementation. While the lessons are consistent with commonly recognized ingredients of successful implementation efforts, the examples highlight the fact that, even with a thorough knowledge of the literature and thoughtful planning, unexpected circumstances arise during implementation efforts that require flexibility and adaptability. The findings stress the importance of utilizing formative evaluation techniques to identify barriers to successful implementation and strategies to address these barriers.

The mission of the Veterans Health Administration's (VHA) quality enhancement research initiative (QUERI) is to enhance the quality and outcomes of VHA health care by systematically implementing clinical research findings and evidence-based recommendations into routine clinical practice.^1,2 The main lessons that QUERI investigators have learned from their initial attempts to pursue the QUERI mission is that even with the best laid plans, unexpected events occur, and if something can go wrong, it usually will. Given that all contingencies cannot be foreseen in advance, it is vital to have a method to assess implementation progress on an ongoing basis in order to identify barriers and correct for them quickly. The collection of this type of information is referred to as formative evaluation. For a comprehensive overview of methods of formative evaluation, see Stetler et al.³

Some early QUERI projects included plans to collect such information on an ongoing basis and found the information very useful for addressing unexpected events and for refining initial intervention plans to address local needs and barriers. Other QUERI projects did not include such plans, and often investigators were faced with outcomes that were not easily understood (e.g., why an intervention was successful at 1 site and not at another). Utilizing post hoc interviews and assessments to try to understand the results of their efforts often led to discoveries after the fact of ways that they could have improved the chances of success for their intervention. These early successes with formative evaluation and missed opportunities have led QUERI investigators to a great appreciation of the importance of formative evaluation in practice improvement efforts. This paper presents lessons learned through formative evaluation or post hoc assessments that allowed QUERI investigators to improve their intervention planning, either for active projects or for future efforts. A more extensive list of lessons learned, including those presented in this paper, is available online along with additional information and examples from QUERI projects in the “QUERI Guide to Implementation Research,” at http://www.hsrd.research.va.gov/queri/implementation/. The specific examples that illustrate the lessons described in this paper are drawn from the lead author's experience with the substance use disorders (SUD) QUERI's efforts to improve the quality of opiate agonist therapy in the VHA. However, it is important to note that these lessons were also encountered during other QUERI projects. Details and results from individual QUERI projects are available elsewhere for the interested reader.^4–15

Lessons Learned

Customize the Intervention to Local Conditions

The SUD QUERI developed 4 best practice recommendations for improving the quality of care in opioid agonist therapy clinics and a toolkit for assessing current practices and implementing change strategies. The SUD QUERI then used these recommendations and tools to work with clinical teams at 8 VA opioid agonist therapy clinics. When working with clinical teams, the implementation team planned to assess baseline performance, provide recommendations for areas in need of improvement, direct clinical teams to appropriate strategies for change available in the toolkit, and assist them in implementation of a change plan. The implementation team recognized that each clinic would have different baseline compliance with each recommendation and, therefore, would identify different QI goals and strategies. However, the team did not anticipate the impact of each clinic's readiness to change on how the intervention would proceed.¹⁶

Formative evaluation procedures included a research associate separate from the implementation team who conducted monthly interviews with the clinical team leader to document the clinical team's perception of the implementation efforts. Through data collected from these interviews, the implementation team learned that the intervention model worked very well for clinics that were in the “action” stage, e.g., those that had room for improvement, recognized the need for improvement, and were ready and willing to take steps to make this improvement happen. However, some of the clinics enrolled in the initiative were in the “precontemplation” or “contemplation” stages, either not agreeing that they had areas that needed improvement or not committed to taking action toward improvement. The intervention was faltering in these clinics as the implementation team kept pushing action and the clinic team kept passively resisting any change, agreeing to action plans but not making any progress.

The information gained through the monthly interviews helped the implementation team to conceptualize each clinics' readiness to change using the transtheoretical model of behavior change,¹⁶ allowing the implementation team to tailor their intervention approach to the clinic's stage of change. With the stalled clinics, this helped the implementation team to recognize the need to focus more time and energy on articulating a compelling need to change and on describing how modified processes would enhance attainment of organizational goals. One successful approach was to solicit buy in for an “experiment” in which the clinical team would change their practices for a specified period of time and use their outcome data to assess whether or not they would choose to maintain the change in practice. This strategy seemed to be less threatening than recommending a permanent change in practice, although most changes were sustained beyond the “experiment.” Spending more time up front to assess the clinical team's agreement with the practice recommendations so interventions could be tailored to their stage of change would have avoided several months of stalled efforts.

Data collection and feedback schedules may also need to be adjusted based on local conditions. While the SUD QUERI team originally required sites to submit monthly practice data, 1 clinic was guideline compliant at baseline. Understandably, staff members were not pleased with putting in the time to submit data on a monthly basis when their feedback consistently confirmed their outstanding performance. This clinic's data submission schedule was reduced from monthly to quarterly and the quarterly reports were used to “monitor the maintenance of best-practices.” Another clinic was working on shifting the philosophy of the program from detoxification to long-term maintenance on opioid agonist therapy. Unlike goals such as increasing patients' agonist doses or frequency of counseling, shifting the philosophy of the program would not be expected to rapidly impact the outcomes for individual patients. Continuing to receive feedback on a regular basis became discouraging to staff who were not seeing an impact of their efforts from month to month. Therefore, this clinic was also shifted to quarterly data submission, which decreased the workload for staff and increased moral as they could see positive trends each time they received a feedback report.

Develop Clear Benchmarks for Performance that are Clinically Meaningful and Locally Acceptable

The SUD QUERI team originally stated to clinics that methadone doses should be greater than 60 mg unless there was a clinically appropriate reason for a lower dose. Feedback to clinics showed the percentage of patients receiving 60 mg or more and the percentage of patients receiving less than 60 mg. The monthly evaluation interviews with clinic leadership revealed dissatisfaction with the recommendation and the feedback. The interviews also provided indications for how the feedback could be revised to be more credible and have a greater impact. Staff felt that they were being penalized for instances where care appropriately deviated from guideline recommendations as all low-dose patients were lumped into 1 category regardless of whether there was an appropriate reason for their low dose or not. They wanted to have a definitive benchmark (e.g., a specified percentage of patients whose doses should be over 60 mg) that took into account the minority of patients that are maintained on lower doses for clinically appropriate reasons. Some clinicians also were not satisfied that the recommendation was relevant to their clinical population. Frequently heard arguments, which are valid to a degree, included: “These recommendations are based on research participants in controlled research settings. Those participants are very different from the patients I see in my clinic,” and, “Those recommendations are not based on VHA patients and VHA patients differ in many ways from patients in other hospital systems.”

The implementation team dealt with these issues in 2 ways. First, a dose review was conducted in all participating clinics to determine the percentage of patients that were being maintained on low doses for clinically appropriate reasons. This allowed the implementation team to state to the clinics that, within VHA opioid agonist therapy clinics, approximately 20% of patients are appropriately maintained on low doses. Therefore, a reasonable benchmark to aim for is that a minimum of 80% of patients should be on doses above 60 mg. Second, in the monthly feedback reports sent to clinics, the implementation team added a graph showing the percentage of patients on 60 mg or higher doses for every participating clinic. For clinics with larger percentage of patients on low doses of medications, this negated the argument that the recommendation was not relevant to VHA patients. All clinics could compare their standing with several of their “peer” clinics. Supplying the clinics with these 2 additional pieces of information improved the acceptability and meaningfulness of the recommendation and spurred more rapid implementation of strategies to improve dosing practices.

Know Your Audience: Identify and Utilize Local Leaders and Involve Relevant Stakeholders

One mistake made by the SUD QUERI team was to recruit clinics into the initiative by talking with the medical director of the program only. When the implementation facilitator arrived at the clinics for site visits, it became obvious that different recruitment strategies could have aided implementation in several ways. First, at some clinics, other clinic staff knew nothing about the initiative until the implementation facilitator arrived for the site visit. This, understandably, led to feelings of resentment among staff who felt “signed on” to do “extra work” that they had not been consulted about. Even in clinics where leadership was fully supportive of the initiative, resentful staff members who were not enthusiastic about implementation efforts were able to undermine the efforts of the implementation team.

Also, it became clear during the site visit by the implementation team that the medical director was not necessarily the “leader” of the clinic. Some medical directors had very little of their work time dedicated to the team and were stretched thin by other commitments. They had little influence over the day-to-day operations of the clinics. In these instances, a nurse, social worker, or addiction therapist often filled the role of program coordinator and was truly in charge of other staff and clinic practices on a day-to-day basis. In some clinics, the implementation team was able to recruit this more influential staff member into a leadership role for the initiative with great success. In other instances, implementation floundered because the team's local contact did not have either the time or the power to lead the initiative at the local level.

Finally, the initiative ran into trouble at 1 site because the intervention focused on the buy-in of the clinical team and did not consider the impact of not recruiting buy-in of the hospital administration. One of the goals of the initiative was to promote a philosophy of treatment that encouraged long-term maintenance in treatment and discouraged rapid discharge of patients who continued to use illicit substances. While the implementation team provided extensive education to the clinical teams regarding the benefits of this philosophy to long-term patient outcomes, hospital administration was not targeted with this educational material. The intervention was successful at influencing the clinic to shift to a more maintenance-oriented philosophy, which resulted in an increase in illicit drug use in the population because patients who continued to use illicit drugs were no longer rapidly discharged. This result led to a conflict between the clinic and hospital administrators who, without having been exposed to the rationale for the change in philosophy and prepared in advance for how the change could effect measures of illicit drug use, interpreted this result to indicate declining outcomes for the clinic.

In summary, the problems caused by not thoroughly assessing the target audience of the intervention represented a “missed opportunity” for formative evaluation techniques. Soliciting input from clinic staff earlier on, perhaps through a conference call, interviewing clinic staff to identify the person(s) who has the most impact on clinic practices and policies on a day-to-day basis and recruiting this person as a local champion for the initiative, and having a broader view in determining stakeholders that would have an investment in the results of the intervention would have prevented many problems that stalled progress with implementation.

Create Networking Opportunities to Enhance Stakeholder Interactions and Maintain Enthusiasm

Early on in the SUD QUERI intervention, feedback from the monthly evaluation interviews indicated that clinical teams felt isolated and were interested in developing opportunities for networking with other opiate agonist therapy clinics. They were interested in sharing information not only about implementing best practices but about other practice concerns. The implementation team responded to this information by adding a monthly conference call and an e-mail group for the clinical teams to the intervention. These 2 networking opportunities served the purpose of the clinical teams by diminishing their sense of isolation and allowing them a venue to ask questions and request information from other participating clinics. The networking opportunities also served the purpose of the implementation team by facilitating the sharing of QI goals, barriers, and strategies for overcoming barriers across clinics. In addition, the networking opportunities served to promote peer norms for best practices in that lower performing clinics began to receive some pressure from high-performing clinics to conform to best practice standards. Finally, the networking opportunities assisted in maintaining momentum and enthusiasm for the initiative by creating a sense of community.

Conclusions

The lessons presented in this paper are consistent with commonly recognized ingredients of successful implementation efforts. For example, in a recent systematic review of factors affecting the success of efforts to diffuse and disseminate innovations in health service delivery, Greenhalgh found that some of the factors influencing adoption of innovations include: (1) the adaptability of the innovation to the needs of the adopter, (2) the feasibility, workability, and ease of use of the innovation, (3) the harnessing of the influence of true opinion leaders, and (4) the support of informal interorganizational networks for the adoption of the innovation.¹⁷ While 1 purpose of this paper is to increase awareness of these factors when planning an implementation effort, the examples provided highlight the fact that even with careful intervention planning, all possible circumstances cannot be anticipated in advance. The examples highlight the value of formative evaluation as a means to identify problems with intervention tools and strategies as soon as possible and to gather input from stakeholders regarding effective remedies for local implementation barriers. It is our hope that the lessons learned in VHA QUERI will assist others interested in implementing research evidence into clinical practice by alerting them to common pitfalls of implementation research as well as to strategies that may improve their chances for success.