Choosing Wisely: An idea worth sustaining

Abstract

Objectives

To evaluate the sustainability potential of Choosing Wisely (CW) to address unnecessary medical care at Ontario community hospitals.

Data Sources/Study Setting

Ontario community hospitals and their affiliated family health teams (FHTs).

Study Design

A mixed‐methods study involving the administration of a validated sustainability survey to CW implementation teams followed by their participation in focus groups.

Data Collection/Extraction Methods

Survey data were collected using an Excel file with an embedded, automated scoring system. We collated individual survey scores and generated aggregate team scores. We also performed descriptive statistics for quantitative data (frequencies, means). Qualitative data were triangulated with quantitative assessments to support data interpretations using the meta‐matrix method.

Principal Findings

Fifteen CW implementation teams across four Ontario community hospitals and six affiliated primary care FHTs participated. CW priority areas investigated were de‐prescribing of proton pump inhibitors (PPIs) and reducing Pre‐Op testing and BUN/Urea lab testing. Survey results showed steady improvements in sustainability scores from baseline to final follow‐up among most implementation teams: 10% increase for PPI de‐prescribing (six FHTs) and 2% increase (three hospital teams); 18% increase in BUN/Urea lab testing (three hospital teams). Regardless of site or CW priority area, common facilitators were fit with existing processes and workflows, leadership support, and optimized team communication; common challenges were lack of awareness and buy‐in, leadership engagement or a champion, and lack of fit with existing workflow and culture. All teams identified at least one challenge for which they co‐designed and implemented a plan to maximize the sustainability potential of their CW initiative.

Conclusions

Evaluating the sustainability potential of an innovation such as Choosing Wisely is critical to ensuring that they have the best potential for impact. Our work highlights that implementation teams can be empowered to influence implementation efforts and to realize positive outcomes for their health care services and patients.

What is known on this topic

• Choosing Wisely is a de‐implementation innovation aimed at reducing unnecessary tests and low‐value care that are unlikely to benefit patients or which may even cause harm.

• Evaluating the long‐term sustainability of innovations, such as Choosing Wisely, is often lacking, and may lead to implementation failure, wasted resources, and poor patient outcomes.

• The success of ongoing implementation of innovations is dependent on assessing their sustainability potential, which has the best potential to support sustained improvements in health care services and patient outcomes over time.

What this study adds

• Understanding the implementation context of target knowledge users, their environment, resources, and mechanisms of the implementation process are necessary for optimized sustainability of innovations.

• Implementation teams can influence and optimize their implementation efforts through solutions‐focused discussions whereby implementation challenges can be identified and addressed.

• Sustainability assessments can be strengthened by using a sustainability model as a platform for change, using a repeated measurement strategy to allow for optimization of sustainability over time, and to include a maintenance strategy for long‐term sustainability or until the change becomes part of routine care.

1. BACKGROUND

It is estimated that 30% of health care services, tests, treatments, and procedures are unnecessary and unlikely to benefit patients ¹ , ² , ³ and can also lead to patient harm, wasted resources, and longer wait times for care. ³ , ⁴ , ⁵ , ⁶ , ⁷ Choosing Wisely (CW) is a physician‐initiated, campaign aimed at reducing overuse by addressing low‐value care that is unlikely to benefit patients or even cause harm. ⁸ , ⁹ , ¹⁰ The campaign has now spread internationally to over 20 countries ¹¹ with the goal of providing evidence‐based recommendations, resources, and clinical guidance to ensure high‐quality care and to facilitate the communication between physicians and patients in their decision making about unnecessary tests, treatments, and procedures. ⁹ , ¹⁰ , ¹¹

The successful implementation and spread of evidence‐based innovations such as CW and their potential to improve care can only be achieved if they are sustainable. ¹² , ¹³ This is important for CW because, despite the overwhelming worldwide commitment to its adoption, its effect on decreasing unnecessary care has been slow. ¹⁴ , ¹⁵ , ¹⁶ This is not surprising given that the dissemination of CW recommendations tends to happen nationally while implementation takes place at local health system levels, where these efforts tend to be the most variable. ¹⁴ Challenges to the uptake of CW recommendations can also occur at the individual provider and patient levels. For example, clinicians may agree with a CW recommendation but may be reluctant to follow it if it has implications on the way they practice (e.g., lack of time, malpractice concerns) or may not be accepted by patients. ¹⁷ While this is not surprising to some extent given the wide range of CW priority areas, settings, and dissemination efforts, we need to better understand the factors that influence “de‐implementation” (i.e., “reducing or stopping the use or delivery of services or practices that are ineffective, unproven, harmful, overused or inappropriate” ¹⁸ ) of care services and to identify the factors that influence behavior change in de‐implementation, particularly when it is part of usual care or routine use. ¹⁸ Other drivers of low uptake of CW campaigns include a lack of understanding of how or why low‐value services were implemented in the first place, the determinants of de‐implementation (facilitators and barriers) and how it operates in what contexts, for whom and why; and the lack of robust evaluation of the broader effects of CW campaigns including the consideration of their intended and unintended consequences. ¹⁴

Sustainability is a key implementation outcome and has been defined as: “the degree to which an innovation continues to be used after initial efforts to secure adoption are completed” ¹⁹ or becomes a routine part of practice to support continuous care delivery. ¹² , ²⁰ , ²¹ Ensuring sustainability capacity is also needed to scale up innovations. ²² , ²³ , ²⁴ However, sustainability is seldom considered or not considered early enough in implementation or de‐implementation processes, ²⁵ , ²⁶ so it is not surprising that up to 40% of all new programs or interventions do not last beyond 1–2 years. ²⁷ , ²⁸ Another challenge is that investigation of sustainability is largely lacking, ²¹ , ²⁵ , ²⁹ , ³⁰ focusing mainly on short‐term outputs and neglecting the long‐term, decision‐informing outcomes needed for practice and policy domains. ²⁶ , ³¹ These deficiencies can lead to implementation failure and wasted resources ⁴ , ⁵ , ⁶ , ³² , ³³ , ³⁴ and worsen quality of care. ⁶ , ²⁵ , ³⁵ , ³⁶ Increasing “awareness” about low‐value care is not enough to achieve sustainable health impacts. We need to demonstrate long‐term benefits of effectiveness and uptake of innovations through behavior and practice change, increased patient knowledge and acceptance of new innovations, and to show value to the system through decreased health care utilization. ⁹ , ¹⁰ Assessing the sustainability potential of innovations and directly responding to identified challenges has the best potential to optimize implementation efforts and to support sustained improvements in health care services and patient outcomes over time. ¹³ The objectives of our study were as follows: (i) assess the sustainability potential of CW across Ontario community hospitals and their affiliated primary care sites; (ii) to work with CW implementation teams to optimize their CW implementation efforts; and (iii) to evaluate our overall research process to advance the science of sustainability.

2. METHODS

2.1. Design overview

Our objectives were addressed using a 12‐month, sequential mixed‐methods design ³⁷ , ³⁸ (see study flow in Appendix A of Supporting information) involving three iterative phases of investigation across three time points: T1 (baseline), T2 (3‐month follow‐up), and T3 (final follow‐up). Research and ethics board (REB) approval was attained through Clinical Trials Ontario (CTO) for NYGH, Markham Stouffville Hospital (MSH), St. Joseph's Health Centre (SJHC) and Southlake Regional Health Centre (SLRHC). REB approval for Michael Garron Hospital (MGH) was attained through their REB office. The reporting of our study was guided by the Standards for Reporting Implementation Studies checklist. ³⁹

2.2. Setting

In Ontario, NYGH was an early adopter of CW and began implementing the initiative in 2014 and has demonstrated significant reductions in unnecessary testing. In 2016, the hospital received funding to spread CW to a network of 7 community hospitals and 13 affiliated primary care centers in Ontario called “Joint Centres for Transformative Health Care Innovation”. Joint Centres is a partnership between NYGH and 6 large community hospitals and 13 affiliated primary care family health teams (FHTs) in Ontario. As a platform for innovation and collaboration, the combined goal of the Joint Centres is to share evidence‐based, effective innovations to improve the quality, safety, performance, and value in health care, and to share innovative practices such as to support the implementation of CW campaigns and recommendations. The spread project has included the identification of physician champions and selection of the top low‐value tests and practices to be targeted for reduction by each Joint Centres organization. Our project to assess the sustainability potential of CW was a sub‐study of this spread effort.

2.3. Recruitment of CW teams

Leveraging the governance structure of the Joint Centres CW advisory board consisting of Joint Centres hospital and primary care clinic and administrative leads, we used a non‐probabilistic, purposive sampling strategy to identify teams that were involved in the implementation of any CW initiative that was deemed priority at their sites between 2017 and 2018. We used this sampling strategy as we sought to identify information‐rich, homogeneous cases to reduce variation, simplify analysis, and to facilitate group interviewing. ⁴⁰ Implementation team members could be frontline health care providers (i.e., nurses, pharmacists, physicians, trainees), decision makers (i.e., chiefs, directors, managers), or researchers. Our sampling frame was 24 potentially eligible CW implementation teams from five Ontario Joint Centres community hospitals (NYGH, MSH, SJHC, MGH, and SLRHC) and six FHTs affiliated with these hospitals. All 24 teams were invited to attend an information session to introduce the study and its processes, and to seek their consent to participate. Fifteen teams attended 12 information sessions. Each hospital selected CW campaigns according to what was introduced by CW Canada during the study period as well as what topics were prioritized by their hospital leadership team: de‐prescribing proton pump inhibitors (PPIs) and reducing preoperative (Pre‐Op), blood, urea, nitrogen and urea (BUN/Urea), and IP echocardiogram (IP Echo) testing.

2.4. Phase 1: Baseline assessment of sustainability (T1)

We used a two‐step process to assess the baseline sustainability potential of each team's implementation of their CW initiative.

• Step 1: NHS sustainability survey: With the help of site administrative and clinic leads, we emailed a validated sustainability survey (in an Excel file) developed by the UK National Health Service (NHS) Institute for Innovation and Improvement Program ⁴¹ to all consenting CW implementation teams (n = 15). This survey is based on a sustainability model representing 10 factors across three domains that are considered important in sustaining change in health care. ⁴¹ We selected this model because it provides guidance on how to achieve sustainability of an innovation through operationalized steps and processes. ⁴¹ The model and survey are designed to help teams identify the factors that support or hinder the sustainability of their innovation including how to address identified challenges (and successes) associated with the practice chance. Teams were given 2 weeks to complete the survey with follow‐up email reminders to nonresponders at 2 and 4 weeks after the due date. ⁴² Analysis: Individual survey scores were combined to derive an overall mean team score. The top sustainability challenges and successes were identified by calculating the largest/least difference between actual scores and maximum potential score (a prespecified maximum score built into the NHS survey). ⁴¹ An overall sustainability score of >55% is considered above the threshold of what is considered a potentially sustainable innovation. ⁴¹ Outcomes: Overall mean team sustainability scores and mean scores for each of the 10 sustainability factors of the NHS sustainability survey.

• Step 2: Action planning focus groups: Two weeks after completing the survey, teams were invited to participate in 1‐h focus groups to obtain a more in‐depth understanding of their CW implementation including its sustainability potential and to co‐design an action plan that would directly respond to identified challenges. Teams were given a summary of their baseline survey results, which was used to start a conversation about how they might address identified challenges. Teams were also encouraged to select at least one challenge for which to co‐design an action plan that could be implemented within 3 months. Discussion questions were driven by the survey scores as well as guiding questions embedded within the NHS sustainability model. ⁴¹ These questions help to identify sustainability challenges at the early stages of implementation. One week after the focus group, each team received a summary of their plan, which included their team scores, top challenges and successes, and the action plan they co‐designed to be implemented within 3 months. Analysis: Sessions were audio‐recorded and transcribed verbatim. Two reviewer pairs independently read transcripts, created an initial list of codes and a codebook of themes through consensus‐based discussions using thematic analysis. ⁴³

2.5. Phase 2: 3‐month follow‐up (T2)

Three months after the baseline assessment (Phase 1), the sustainability survey was administered to all teams once again to determine any changes in the sustainability potential of their CW initiative. Teams were also invited to participate in a second focus group to discuss scores and whether their action plan was implemented. The action plan items were discussed in the context of how they were applied within the previous 3 months, and if further changes needed to be made to address any new challenges. Each team received a summary report of any changes in survey scores and actions they undertook in response to identified challenges from baseline.

2.6. Phase 3: 6‐month follow‐up (T3) and process evaluation

Three months after Phase 2, the sustainability survey was administered one final time to identify any changes in sustainability scores from Phase 2. In addition, we included an additional survey to evaluate our study procedures and processes to advance the practice of sustainability. Questions consisted of assessing the value and usefulness of the information sessions; the usability of the NHS sustainability survey, focus group sessions, action plan summary; the feasibility of participating in the series of surveys and focus groups; and their overall satisfaction with the study. Analysis: Descriptive statistics for quantitative data (frequencies, means) and content analysis ⁴⁴ for qualitative data. We used a meta‐matrix method ⁴⁵ to triangulate survey and focus group data to facilitate our interpretation of findings. Survey scores across the three time points were used to create line graphs to show changes in sustainability scores over time (see Appendix B of Supporting information). Teams also received a report summarizing their survey scores, top successes, challenges, and action plans over time.

3. RESULTS

3.1. Site and participant characteristics

Fifteen teams representing 91 individuals consented to participate across four Ontario Joint Centres hospitals and six affiliated FHTs. The site and participant characteristics of teams are shown in Appendices C and D of Supporting information, respectively. A total of 173 surveys and 29 focus groups were completed by participants respectively at T1 (n = 67; n = 15), T2 (n = 59; n = 14), and T3 (n = 47). Teams consisted of health care professionals, chiefs/directors/managers, and clinic and administrative staff: clinicians (nurses, physicians, pharmacists), trainees (residents, pharmacy students), and health information data managers.

3.2. NHS sustainability survey

Table 1 shows the mean team scores across all sites and teams for each time point. The overall mean sustainability score across teams at baseline ranged 65%–87%, which means that their CW initiative was above the threshold of what is considered a potentially sustainable (i.e., ≥55%). ⁴¹

TABLE 1.

Mean NHS sustainability survey scores (range) across Ontario Joint Centres hospital and primary care sites and Choosing Wisely priority areas from baseline (T1) to 3‐ (T2) and 6‐month (T3) follow‐up

Setting (number of sites)	CW priority area	Mean NHS sustainability score (range); number of participants			Change in mean scores between T1 and T3
		T1 (n = 67)	T2 (n = 59)	T3 (n = 47)
Six primary care teams	PPI	68% (53%–74%), n = 26	73% (66%–80%), n = 24	78% (70%–83%), n = 22	10% increase*
Team 12		53%	68%	70%	17% increase*
Team 4		67%	66%	72%	6% increase*
Team 6		68%	73%	83%	5% increase*
Team 7		70%	80%	80%	10% increase*
Team 2		73%	75%	78%	15% increase*
Team 10		74%	78%	83%	9% increase*
Three hospital teams	PPI	65% (59%–72%), n = 17	67% (64%–69%), n = 16	67% (65%–69%), n = 10	2% increase*
Team 5		63%	64%	65%	2% increase*
Team 11		72%	67%	66%	4% decrease**
Team 9		59%	69%	69%	10% increase*
Three hospital teams	BUN/Urea lab testing	70% (61%–89%), n = 11	72% (58%–85%), n = 12	88% (72%–91%), n = 9	18% increase*
Team 14		61%	58%	72%	11% increase*
Team 8		61%	72%	88%	27% increase*
Team 3		89%	85%	91%	2% increase*
Two hospital teams	Pre‐Op testing	87% (82%–92%), n = 11	83% (68%–97%), n = 7	77% (57%–97%), n = 6	10% decrease**
Team 13		82%	68%	57%	25% decrease**
Team 1		92%	97%	97%	5% increase*
One hospital team (Team 15)	IP Echo testing	79% (73%–85%), n = 2	Did not complete		Not applicable

Abbreviations: BUN, blood, urine, nitrogen; IP Echo, Internet protocol echocardiogram; NHS, National Health Service; PPI, proton pump inhibitor; Pre‐Op, preoperative.

*Represented an increase scores over time.

**Represented an decrease in scores over time.

3.2.1. Sustainability scores by CW priority area

PPI de‐prescribing: Of the six PPI de‐prescribing primary care teams, there was a 10% increase in sustainability survey scores over time among the 26 (T1), 24 (T2), and 22 (T3) survey respondents, and a 2% increase among the 17 (T1), 16 (T2), and 10 (T3) survey respondents of three PPI hospital teams. BUN/Urea lab testing: Of three BUN/Urea lab testing hospital teams, there was an 18% increase in sustainability survey scores over time among 11 (T1), 12 (T2), and 9 (T3) survey respondents. Pre‐Op testing: Of two Pre‐Op testing hospital teams, one had a 5% increase in sustainability scores while the second had a 25% decrease over time: 82% (T1) to 68% (T2) to 57% (T3). IP Echo: One Echo imaging testing hospital team comprising two members completed the sustainability survey at T1 (mean sustainability score range 73%–85%) and attended the first focus group but did not complete the follow‐up surveys or final focus group.

3.3. Qualitative focus groups

Sustainability facilitators: Analysis of 29 focus groups revealed three themes related to sustainability facilitators that were common across hospitals and primary care teams and CW initiatives (Appendix E of Supporting information; Figure 1):

1. Leadership support: Success factors among primary care teams were the prioritization of PPI de‐prescribing as a patient safety issue by clinical leadership, having a lead physician acting as an ambassador for the initiative, and having a pharmacist involved in the process. Among hospital teams, having support from the organization and the well‐recognized nature of CW were perceived as reasons for success. Other identified facilitators were clinical leadership and senior leadership support from the department chiefs and the corporate level for reducing Pre‐Op and BUN/Urea lab testing.

2. Fit with existing processes and workflows: Facilitators included staff awareness of CW initiatives, ability of electronic medical record (EMR) systems to help monitor de‐prescribing activities and progress, the well‐supported nature of PPI de‐prescribing by CW (i.e., algorithm, tapering schedule) and having a team with EMR experience. Teams involved in reducing Pre‐Op and BUN/Urea testing found their implementation straightforward (i.e., removal from the order set), which resulted in less paperwork for administrative staff, less workload for nurses, and less time spent for patients for tests that they do not need.

3. Optimized team communication and collaboration: Success was attributed to opportunities for teams to present the CW initiative to their peers to help support an environment to encourage staff involvement, enhanced team collaboration, and existing communication channels for feedback. CW publications and pamphlets were perceived as helpful communication tools to support changes in the process (i.e., de‐prescribing algorithm) as well as the use of patient posters by pharmacists for shared decision making and education.

FIGURE 1.

Common and unique facilitators across sites (primary care and hospital) and Choosing Wisely priority areas: PPI de‐prescribing; Blood, BUN testing; Pre‐Op testing. BUN, blood, urea, nitrogen; PPI, proton pump inhibitor; Pre‐Op, preoperative [Color figure can be viewed at wileyonlinelibrary.com]

Sustainability challenges: Three themes related to challenges were common across sites and CW priority areas (Appendix F of Supporting information; Figure 2):

1. Lack of awareness and buy‐in: To overcome these challenges, teams used newsletters, meetings, and emails to share information; created simple, one‐page infographics; incorporated PPI de‐prescription as part of resident training; held educational events and meetings with staff; made posters and handouts for patients; created publications and posters, and presented at operations committees, rounds, and conferences. To increase buy‐in, one FHT sent a mass email about PPIs, which led to patient visits about de‐prescribing, and another encouraged those who regularly engage in CW activities to meet with those who do not. Other tactics were undertaking a trial to evaluate the impact of CW implementation, engaging in conversations with frontline staff, and performing an “audit and feedback” exercise.

2. Lack of leadership engagement or a champion: To overcome these challenges, teams rotated clinical leadership, engaged the quality improvement specialist to gather data to help support engagement efforts, and set up an ongoing dialogue to engage site leadership prior to the final changes in order sets.

3. Lack of fit with existing workflow and culture: To overcome these challenges, teams made the initiative become a habit (similar to taking blood pressure) by providing education, increasing communication, monitoring with audits and feedback, supporting research projects by residents, and to provide CW to nonacademic sites.

FIGURE 2.

Common and unique challenges across sites (primary care and hospital) and Choosing Wisely priority areas: PPI de‐prescribing; Blood, BUN testing; BUN, blood, urea, nitrogen; Pre‐Op testing. PPI, proton pump inhibitor; Pre‐Op, preoperative [Color figure can be viewed at wileyonlinelibrary.com]

Overlap: Figure 3 shows that the largest proportion of implementation teams perceived having leadership support as a sustainability facilitator regardless of setting (primary care, hospital) or CW initiative. Figure 4 shows that the largest proportion of teams perceived lack awareness and buy‐in, lack of leadership engagement or a champion, and lack of fit with existing workflow and culture as top challenges. There was a 78% overlap in challenges between primary care and hospital teams that implemented the PPI de‐prescribing priority. Among primary care teams, the majority (>50%) identified lack of leadership engagement or a champion, frequent staff turnover, volume of quality improvement initiatives, and the time‐consuming and resource‐intensive nature of PPI de‐prescribing as challenges, while two‐thirds of hospital teams found lack of awareness and buy‐in and training as top challenges.

FIGURE 3.

Proportion of teams who identified overlapping and unique sustainability facilitators during focus groups by Choosing Wisely priority area (PPI de‐prescribing, BUN testing, Pre‐Op testing) and site (primary care, hospital). BUN, blood, urea, nitrogen; PPI, proton pump inhibitor; Pre‐Op, preoperative [Color figure can be viewed at wileyonlinelibrary.com]

FIGURE 4.

Proportion of teams who identified overlapping and unique sustainability challenges during focus groups by Choosing Wisely priority area (PPI de‐prescribing, BUN testing, Pre‐Op testing) and site (primary care, hospital). BUN, blood, urea, nitrogen; Pre‐Op testing. PPI, proton pump inhibitor; Pre‐Op, preoperative [Color figure can be viewed at wileyonlinelibrary.com]

3.4. Evaluation of our process

Forty‐five of the 47 individuals across 14 teams who participated in the final follow‐up (T3) completed the evaluation survey (response rate 96%). Appendix G of Supporting information shows the dispersion of mean scores across the evaluation survey. Respondents perceived their participation in the study as feasible, helped strengthen the implementation of their CW initiative, increased their knowledge about sustainability, and were overall satisfied (mean score range 4.0–4.3 out of 5). Teams perceived the information session as effective for introducing the study and clarifying its objectives and found it useful overall (mean score range 4.1–4.3). The NHS sustainability survey was perceived as easy to complete and understand, a reasonable amount of time to complete, and to learn about the study (mean score range 4.0–4.3). Participants perceived the action planning focus groups as helpful, well organized with effective facilitators that helped them to understand the survey results and identify sustainability challenges (mean score range 4.0–4.3). Team reports were perceived as easy to read and well organized (mean score 4.1).

4. DISCUSSION

Overall, our study showed that all but one implementation team had steady improvements in sustainability scores over time, which is an indication that PPI de‐prescribing, BUN/Urea testing, and Pre‐Op testing CW initiatives are sustainable in community hospital and primary care settings. The team that saw a decrease in their sustainability scores may be explained by the following: (i) the complexity of their implementation given that this team had a total of 55 surgeon's offices, not all of which are located at their hospital; (ii) the hospital was in the midst of a merger with two other hospitals; (iii) the site is not fully electronic/automated; and (iv) the patient care manager (who acted as the point person to send out surveys and coordinate focus groups) went on maternity leave midway through the study (T2). This highlights the importance of understanding the implementation environment of the knowledge user team, including the determinants of usual practice and how the new way of doing things (i.e., CW) may influence improvement efforts.

Another important finding was that the sustainability scores at baseline were about 5%–22% lower for teams that implemented the PPI de‐prescribing initiative (regardless of hospital or primary care setting) compared with other CW initiatives that are focused on reducing test ordering (i.e., BUN/Urea, Pre‐Op, IP Echo). Our focus group data support this observation as the majority of challenges were clustered around the PPI de‐prescribing initiative (78% overlap). Figure 2 highlights this clustering effect, which may in part be due to the complexity of PPI de‐prescribing. Our focus groups revealed that de‐prescribing PPIs requires behavior change from both providers and patients to be successful. Patients have a large influence on prescribing decisions, in part because they may not be aware of or understand the benefits of stopping PPIs, and providers feel pressure to re‐prescribe PPIs when patients' gastrointestinal symptoms return after stopping PPIs. Providers may also lack awareness of the evidence or the buy‐in for the de‐prescribing effort, or they may not see the immediate benefits of stopping PPIs. These complexities are in contrast to CW initiatives that involve the removal of tests or procedures from electronic order sets, requiring little behavior change to implement successfully. Therefore, we need to better understand not only the mechanisms of the de‐implementation processes but the perceptions of all knowledge users involved, and to design interventions that best match or mitigate identified de‐implementation challenges. It also highlights the importance of engaging patients in implementation/de‐implementation efforts. None of the participating teams engaged patients, which underscores the need to increase knowledge and skills in the practice of implementation and de‐implementation to optimize quality improvement efforts such as CW undertaken by hospitals and primary care.

An important consideration for de‐implementing low‐value care strategies is equity, diversity, and inclusion (EDI), which has been largely ignored in CW campaigns. ⁴⁶ For example, inappropriate prescribing of antibiotics and unnecessary urine testing are not available by gender, ethnicity, or other equity stratifiers ⁴⁷ even though many ethnic and racial minorities are more likely to receive low‐value care ⁴⁸ (e.g., higher use of PPIs among adult White women ⁴⁹ ; less effective care among minority groups for Pre‐Op testing prior to cataract surgery and cardiac testing prior to noncardiac surgery ⁴⁸ ). Implementing health care decision making in a culturally safe way has the potential to address inequities among vulnerable and underserved populations by facilitating participation in health care that better meets their needs. ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² CW campaigns are not involved in implementation or direct measurement, so there is an assumption that as clinicians reduce low‐value care, high‐value care can be prioritized for those who need it most. However, this is not the case, so CW campaigns represent an opportunity to empower underserved populations to achieve optimized and equitable care. ⁴⁸ For example, to better support care teams and increase consumer education using targeted messaging to both patients and clinicians using culturally relevant messages and channels. ⁵¹

We found that several aspects of our study were essential for measuring and encouraging sustainability that can be considered by others adopting CW campaigns: Information sessions with CW implementation teams helped to ensure that all relevant knowledge users were involved, and an opportunity to introduce the study and how to complete the NHS sustainability survey. Platform for change: The process of convening CW teams in focus groups after completing the sustainability survey provided a platform for teams to co‐design and implement an action plan to address their identified sustainability challenges, thereby strengthening the implementation of their CW initiative. Repeat measurements of sustainability empowered teams to respond to their implementation and sustainability challenges and to see the impact of their efforts over time. Leveraging health equity in addition to patient safety should be taken into consideration to enhance the buy‐in and awareness of CW campaigns. In particular, future studies should focus on investigating the role of leadership and how they respond to patient experiences of CW, and to strengthen leadership engagement in the context of safety, diversity, equity, and inclusivity. Generalizability of our findings: We embedded a step to facilitate the maintenance of each team's CW initiative through the provision of an Implementation and Sustainability Guide (Appendix H of Supporting information), which was designed to help further support their existing or new CW implementation efforts. Our future work will involve validating this guide so it can be applied by other implementation teams and sites.

4.1. Strengths and limitations

Our study had many strengths. To our knowledge, we are the first to evaluate the sustainability potential of implemented CW campaigns at hospital and primary care settings. Most CW evaluations are focused on measuring the impact of campaigns to reduce low‐value care, ⁹ , ¹⁵ , ⁵³ while others have assessed the awareness and perceptions of CW by providers. ⁵³ We evaluated the sustainability potential of CW campaigns across four Ontario community hospitals and their affiliated FHTs, which contributes to advancing sustainability science. We created a strategy to ensure full engagement of teams through action planning focus groups, using discussion questions informed by the NHS sustainability survey. Our involvement (i.e., mobilizing teams and providing opportunities for solutions‐focused discussions) may have strengthened the implementation of CW by empowering teams to identify and respond to their challenges and to leverage their strengths. Most teams who engaged in our process have adopted CW into their practice and have made it part of their routine care—this is the very definition of sustainability. ¹² , ²⁰ , ²¹ Additionally, all activities to engage teams during the 12‐month study period (i.e., sustainability surveys at three time points and focus groups at two time points) may have reinforced the CW implementation. This highlights the need for a formalized process to engage teams, so they have a forum to discuss and work through their implementation and sustainability challenges. Another aspect that strengthened our methods was the involvement of experienced moderators during our focus group sessions. Evidence shows that facilitation is an important aspect of implementation science, and has potential to support innovations for optimized impact. ²¹ , ⁵⁴ This was demonstrated in our work as facilitators helped team participants to understand the underlying concepts of sustainability and the survey results, to stimulate discussion and prompt ideas, and to encourage teams to apply their new knowledge into practice.

Lastly, ensuring sustainability capacity is needed to scale up innovations. ²¹ One of our outputs was a conceptual implementation and sustainability guide (Appendix H of Supporting information), which after further validation will have the potential to successfully scale up CW campaigns.

Our study had some limitations. Not all teams were at the same implementation stage of their CW initiative, which may have reduced the reliability of our trend analysis; mid/late implementers had different experiences from early implementers. However, we used focus group data to explain divergent trends including the decreased implementation scores we observed for some of the teams. We observed variable rates of attrition and participation both within and across teams, which may have diminished the representativeness of implementation teams. However, frequent staff changeover (e.g., revolving residents and students, maternity leaves) is a normal part of clinic team operations, so our participation rates are a reflection of that reality. Not surprisingly, hospital teams had a higher attrition rate than FHTs (39% vs. 15%) in part because the nine hospital teams had variable implementation schedules, whereas all six FHTs had fully implemented their CW initiative at the start of our study. Additionally, the governance and operational structure of FHTs were more streamlined; all six FHT teams were governed by one clinic lead and focused on a single CW initiative (i.e., PPI de‐prescribing) compared with the nine hospital teams that implemented four different initiatives (PPI, BUN/Urea, Pre‐Op, IP Echo) with each hospital having their own clinic/administrative leads. The NHS sustainability framework represents an implementation construct, while CW campaigns are about de‐implementing low‐value care. We recognize and acknowledge that these two concepts are not necessarily the same. Implementation is about adopting new, evidence‐based practices, while de‐implementation is about “abandoning practices that are not evidence‐based (low‐value care) and are unlikely to benefit patients given the harms, cost, available alternatives or preferences of the patient.” ⁵⁵ There is some overlap in factors that influence de‐implementation and implementation (i.e., characteristics of the context, intervention, patient, health professional, and organization). However, abandoning familiar and established practices and behaviors is a more complex process and more challenging than adopting a new practice, ⁵⁶ , ⁵⁷ and the way each of the influencing factors operate is likely very different. ⁵⁷ Additionally, de‐implementation has different types of determinants than implementation. ⁵⁶ , ⁵⁸ For example, patient factors driving low‐value care may include determinants that are also important in implementation (i.e., attitudes, knowledge, and skills), while patient fear, anxiety, inaccurate perceptions about the intervention, or lack of trust in health care are determinants unique to de‐implementation. ⁵⁹

5. CONCLUSIONS

Evaluating the sustainability potential of an innovation such as Choosing Wisely is critical to ensuring that they have the best potential for impact. Our work highlights that implementation teams can be empowered to influence their implementation efforts, and to realize positive and sustainable outcomes for their health care services and patients.

AUTHORS' CONTRIBUTIONS

Monika Kastner and Donna McRitchie conceived the idea and the design of the study; all authors participated in the execution of the study, and data analysis and interpretation. Monika Kastner, Julie Makarski, Kathryn Mossman, Kegan Harris, and Leigh Hayden developed the evidence tables and supporting graphs representing the survey and focus group data. Monika Kastner and Julie Makarski drafted the manuscript, and all authors read and approved the final manuscript.

Supporting information

Appendix S1. Supporting information.

Kastner M, Makarski J, Mossman K, et al. Choosing Wisely: An idea worth sustaining. Health Serv Res. 2022;57(3):568-578. doi: 10.1111/1475-6773.13917