Adapting the Consolidated Framework for Implementation Research to Create Organizational Readiness and Implementation Tools for Project ECHO

Abstract

The Project Extension for Community Healthcare Outcomes (ECHO) model expands primary care provider (PCP) capacity to manage complex diseases by sharing knowledge, disseminating best practices, and building a community of practice. The model has expanded rapidly, with over 140 ECHO projects currently established globally. We have used validated implementation frameworks, such as Damschroder's (2009) Consolidated Framework for Implementation Research (CFIR) and Proctor's (2011) taxonomy of implementation outcomes, combined with implementation experience to (1) create a set of questions to assess organizational readiness and suitability of the ECHO model and (2) provide those who have determined ECHO is the correct model with a checklist to support successful implementation. A set of considerations was created, which adapted and consolidated CFIR constructs to create ECHO-specific organizational readiness questions, as well as a process guide for implementation. Each consideration was mapped onto Proctor's (2011) implementation outcomes, and questions relating to the constructs were developed and reviewed for clarity. The Preimplementation list included 20 questions; most questions fall within Proctor's (2001) implementation outcome domains of ”Appropriateness“ and ”Acceptability.“ The Process Checklist is a 26-item checklist to help launch an ECHO project; items map onto the constructs of Planning, Engaging, Executing, Reflecting, and Evaluating. Given that fidelity to the ECHO model is associated with robust outcomes, effective implementation is critical. These tools will enable programs to work through key considerations to implement a successful Project ECHO. Next steps will include validation with a diverse sample of ECHO projects.

Project Extension for Community Healthcare Outcomes (ECHO) is a telementoring and capacity-building intervention developed at the University of New Mexico in response to disparities in access to and the quality of hepatitis C care across the state.¹ Project ECHO uses a “Hub” and “Spoke” model to promote knowledge exchange between health care specialists typically located at academic centers (“the hub”) and primary care providers (PCPs) at the front line of community health care (the “spokes”). Through regularly scheduled virtual educational clinics, Project ECHO creates a supportive community network where PCPs can connect with specialists and with other PCPs practicing in similar settings via multipoint video technology to discuss best practices in care and complex cases managed within their practice. This multifaceted knowledge and capacity-building intervention includes two fundamental components: a didactic lecture delivered by a member of the hub team (based on curriculum developed from guidelines, best practices, and/or a needs assessment); and recommendations for case management (telementoring) offered by the community in response to anonymized clinical cases presented by spoke sites.

When implementing a complex intervention such as Project ECHO, it is important to understand organizational readiness relating to factors involved in the intervention. Significant evidence in implementation science suggests that assessing organizational readiness before implementing any major intervention within an organization is crucial to minimize risk that the intervention will not be accepted or fail, reducing wasted time and financial investment.^2–5 Identifying whether organizational readiness exists is an important practice that will allow an organization to understand whether they are best equipped to implement a new project or to understand whether and why it is likely to be rejected.²

This article will outline how ECHO Ontario Mental Health (ECHO-ONMH) adapted and integrated Damschroder's (2009) Consolidated Framework for Implementation Research framework to create a readiness tool for the implementation of ECHO and a set of implementation guidelines. While determining what knowledge translation and capacity-building model would best support primary care providers looking to build capacity to treat clients with mental health and addiction concerns, our team was introduced to the ECHO model, which had been replicated globally with positive outcomes. Our team understood that a significant investment of time, money, and effort was required to set up an ECHO project, so we endeavored to assess whether we had sufficient organizational and leadership traits to succeed in the implementation of ECHO. We received advice from peers who had launched a similar project and sought replication support from the ECHO Institute in New Mexico; however, we believe that we would have benefited from a tool to assess organizational readiness for an ECHO project. To address this resource gap, we turned to implementation science, more specifically, to validated frameworks such as the CFIR⁶ and proposed taxonomy of implementation outcomes by Proctor et al.^7,8

In the seminal Project ECHO study by Arora et al, rural primary care physicians supported by Project ECHO were able to achieve comparable treatment outcomes for hepatitis C compared with outcomes in a specialist-run hepatitis C clinic located in an academic center (nonassociated with ECHO), demonstrating ECHO's ability to support nonspecialists to use best practices and treatment algorithms to improve health outcomes.¹ In addition, our recent systematic review of the evidence for Project ECHO assessed a total of 39 studies, which addressed 17 different medical conditions and found that these studies support the ECHO model's effectiveness in improving the knowledge (n = 4) and self-efficacy of participating PCPs (n = 8), as well as changes in provider behavior (n = 1), improved patient outcomes (n = 7), and cost-effectiveness (n = 2) across a number of disease conditions and medical specialties.⁹ In addition, in a recent assessment of our ECHO Ontario Mental Health project outcomes, our team found that there was a significant improvement in pre-post knowledge of over 12% for participating primary care providers (P < .001, d = 1.13).¹⁰

The ECHO model uses both situational and social cognitive learning theories and enables participating PCPs to identify learning gaps (cognitive dissonance) and reflect critically on their learning process.^11–14 Project ECHO allows for problem-centered learning to occur in the clinical context where new knowledge will be applied; promotes interprofessional collaboration among participants; models best-practice care; supports learners to feel that they are benefiting and improving self-efficacy; and allows participants to receive positive feedback and reinforcement from clinical opinion leaders.^13–15

The Project ECHO model has rapidly expanded in recent years, with over 140 international ECHO projects covering chronic diseases such as hepatitis C, HIV/AIDS prevention, addictions and psychiatry, chronic pain management,¹⁶ asthma,¹⁷ and dementia¹⁸ among others. In addition, large ECHO projects have been implemented across the United States of America through the Veterans Health Administration that spans numerous conditions and states.^19,20 Enthusiasm for and expansion of the ECHO model has been further supported by the introduction and approval of the ECHO Act in 2016 in the USA, which will allow for further evaluation of this model as a capacity-building tool.²¹ According to the ECHO Institute, 13 of 142 ECHO projects are no longer active (about 9%). Although this number is currently low, the costs associated with implementing an intervention such as ECHO make it important to reduce the potential of funding an ECHO that is not equipped to succeed. Although the data assessing sustainability of the ECHO model are still new and relatively unknown, it is a critical metric to be able to understand the long-term impact of this educational intervention.

As this model expands, it is increasingly important to ensure that emerging programs maintain fidelity to the model to allow across-program evaluation and to ensure a strong return on investment. The aim of the model is to offer flexibility while ensuring that new projects are in alignment with the following key principles:

1. The project must use technology to leverage scarce resources.

2. The project must use case-based learning to master complexity-learning loops.

3. The project shares best practices to improve knowledge (to increase desired outcomes).

4. The project uses a web-based database (iECHO) to monitor outcomes.

ECHO projects are required to sign an agreement with the ECHO Institute in New Mexico before obtaining training, assuring that they will implement their project following the four aforementioned key principles. Currently, according to the ECHO Institute, there are a total of 223 potential ECHO sites that have either signed contracts but not joined a replication training at the ECHO Institute in New Mexico (N = 83) or trained but have not launched yet (N = 140). For the latter group, the average time in this prelaunch state is 230.3 days. Our hope is that these tools can help sites implement quickly and effectively.

Damschroder et al⁶ created the meta-theoretic CFIR framework, which has provided implementation scientists with a standard set of constructs such as intervention characteristics, internal and external settings, characteristics of individuals, and processes. We selected this framework because it is a comprehensive set of constructs that integrates 19 different well-known implementation models, including Promoting Action on Research Implementation in Health Services (the PARiHS Framework), Technology Implementation Process Model, Replicating Effective Programs framework, and a Practical, Robust Implementation and Sustainability Model (PRISM) among others.⁶ The CFIR framework has been shown to be effective for guiding successful implementation across numerous health domains, such as weight management,²² health records systems,²³ internet patient-provider communication,²⁴ and cancer screening.²⁵ Although the CFIR list of constructs is comprehensive, the framework does not specify interactions between the constructs⁶; some constructs may be linked and need to be considered concurrently. Likewise, the implementation outcome measures put forth by Proctor et al^7,8 especially acceptability, adoption, appropriateness, cost, feasibility, penetration, sustainability, and fidelity (which are further defined in Table 1 below) are also regarded as integral to ensuring implementation success.^7,8 Similarly, we selected this framework, as it provided us with a comprehensive list of implementation outcomes by aggregating common taxonomy proposed through important implementation outcome evaluation frameworks (eg, Roger's Theory of Diffusion of Innovation²⁶ and the RE-AIM Framework).^8,27 Our goal for using these two frameworks was to ensure that we have considered and addressed all important factors, based on well-validated constructs, which are essential to preparing new ECHOs to assess their organizational readiness and guide an effective implementation.

TABLE 1.

Key Preimplementation Considerations

Using theoretical concepts from the CFIR⁶ and implementation outcomes by Proctor et al^7,8 combined with practical experience from the launch of ECHO-ONMH, this study aimed to: (1) build on already validated implementation tools, such as the CFIR, by adapting and consolidating frameworks to create a set of assessment questions to help organizations assess their readiness and capacity to support an ECHO project; (2) provide those who have determined ECHO is the correct model for their organization with a checklist to support a successful implementation process that maintains fidelity to the ECHO model based on our experience and implementation science. These tools will support an approach to implementing the ECHO model globally in a standardized manner and will offer further opportunity for validation of the questions (and CFIR constructs) and harmonized implementation outcomes in the future.

METHODOLOGY

Checklist development was divided into two phases: phase one that focused on the development of key considerations before adopting the ECHO model and phase two that focused on practical, high-level processes to guide implementation.

Preimplementation Organizational Readiness Questions

During phase one, a comprehensive list of Damschroder's CFIR Constructs (intervention characteristics, internal and external settings, characteristics of individuals, and processes) and their associated descriptions were reviewed and discussed by the ECHO-ONMH team, each with intimate familiarity with the ECHO implementation process, including the initial project conceptualization and proposal process. ECHO-related tasks were then mapped onto each CFIR construct and subconstructs, creating a third column of ECHO-specific implementation constructs, linking the ECHO practical steps with CFIR theoretical considerations. Subsequently, questions corresponding to each of the ECHO-centric implementation construct were developed that could be used to assess whether important implementation factors and steps were feasible within the organization and externally. These questions were geared toward individuals/organizations who were thinking about adopting the ECHO model. Questions were organized into a table with three columns: (1) the Damschroder construct and corresponding definition; (2) the ECHO adaptation of the construct; and (3) a key question related to ECHO that addressed the initial Damschroder implementation construct. These key questions, based on each construct and ECHO-ONMH implementation experience, were discussed by a team of four to confirm that the developed question effectively addressed key factors within the construct. The questions were reviewed for clarity by all members of the implementation team, and a subset of the questions was piloted for clarity and applicability by potential end-users at workshops for the Council of Psychiatric Continuing Education (ie, directors of continuing education in Psychiatry from Canadian universities) and the American Psychiatric Association. Finally, to link all considerations to support the eventual evaluation of ECHO project implementation, questions were organized according to four of Proctor's implementation outcomes (appropriateness, acceptability, feasibility, and cost). Only four of the eight implementation outcomes proposed by Proctor et al^7,8 were used in this analysis because the remaining outcomes were better linked with postimplementation assessment (adoption, penetration, sustainability, and fidelity). At the conclusion of this process, overlapping questions were discarded (Fig. 1). Figure 1 outlines how we have linked Proctor's implementation outcomes to the CFIR constructs, with sample questions in each domain.

FIGURE 1.

Approach to organizational readiness questions

Process List

Phase two sought to develop a detailed implementation process (the fifth CFIR domain), which included tasks and procedures to implement an ECHO. ECHO-ONMH project plans, which were developed by the ECHO manager and team, were extensively reviewed with all tasks listed. The ECHO Institute's repository for implementation and project plan tools was also reviewed for any additional implementation steps that were not included in ECHO-ONMH's plan. The resulting set of implementation tasks and procedures were linked with the four subcategories of the CFIR process construct, especially: (1) planning, (2) engaging, (3) executing, and (4) reflecting and evaluating. Practical implementation steps were discussed by the implementation team and categorized. A list was developed that proposed high-level implementation steps to support the implementation of the ECHO project based on practical experience linked with theoretical implementation approaches (Fig. 2).

FIGURE 2.

Implementation process

RESULTS

Table 1 outlines a list of 20 key questions for consideration by individuals and/or organizations interested in developing an ECHO program. The 20 questions reflect a robust set of considerations meant to help guide the decision to adopt and plan an ECHO project. These questions are not meant to be sequential; rather, each section can occur concurrently. As can be seen in the table, most questions developed fall within the “Appropriateness” and “Acceptability” categories, this reflects the focus on evaluating whether the ECHO model is the best tool to meet end-user need before commencing implementation. This tool is not meant to provide a ranking for the level of preparedness or likelihood to succeed; rather, the goal of these questions is to provide additional clarity around some of the complexities of implementing Project ECHO, which can equip new projects with a broad understanding of both internal and external factors that might affect the successful implementation of ECHO.

Table 2 provides a checklist for implementation processes and steps required to implement an ECHO. Some of the articulated procedural steps overlap two sections but after discussion and consensus from the ECHO implementation team were placed in one section to reduce duplication. Although a customized, detailed project plan with a timeline is important for all newly developing ECHOs, these are the key aspects necessary to launch an ECHO project and to support an ongoing quality-improvement process. The four sections have steps that will occur concurrently. The steps include the following: (1) Planning, for example, identifying end-user needs, hiring and training hub and operations team, securing funding, and developing a project and communication plan. (2) Engaging internal and external stakeholders (through meetings or focus groups) and identifying champions for the model. (3) Executing to implement ECHO through four key areas, including operations; curriculum development; end-user recruitment/registration, and communications. (4) Reflecting and evaluating by ensuring that the program has a clear evaluation framework that enables a robust assessment of how well the intervention meets its stated aims, such as improving knowledge, or achieving health outcomes. Certain continuing medical education evaluation models, for example Moore's Framework for Continuing Medical Education²⁸, are effective frameworks to assess not only how ECHO supports participant change in knowledge and self-efficacy but also how this knowledge change translates into performance-, patient-, and population-level improvements. In addition, using the Implementation Outcome framework by Proctor et al^7,8 , key implementation outcomes should also be assessed to understand whether a project was successfully implemented as expected.

TABLE 2.

Process Checklist

DISCUSSION

As suggested by Proctor et al,^7,8 it is important to implement an intervention effectively and complete intermediate assessments of implementation to make assertions about the effect of the intervention on provider learning and patient health outcomes. The growing evidence for the ECHO model, in combination with the recent introduction of the ECHO Act,²¹ has led to a significant growth in ECHO projects globally and lends increasing importance to the understanding of key implementation success and organizational readiness factors. As may be expected, given the implementation of the ECHO model across various disciplines, each of the preimplementation considerations put forth in the table above may not be applicable for certain individuals/organizations; however, it provides an overview of important factors for an ECHO project. Of additional note, many of the questions and processes highlighted in the tables may occur simultaneously and as articulated by Damschroder et al⁶ does not describe the relationship between each construct and/or outcome. Although this set of questions does not generate a score about readiness to start an ECHO, it can be used to facilitate a discussion and assess gaps that should be deliberated when considering this project model. Although it is not yet clear whether there is a threshold that would identify optimal readiness to implement based on these questions, new projects should aim to address as many of these questions as possible during implementation.

A limitation is that these questions have been adapted and tested within the context of one ECHO project. The next steps for our team are to test these questions with approximately 10 ECHO projects in varying countries among different disease areas and complete a full validation through factor analysis to confirm construct validity. Although the tools developed in this framework will provide an important first step to supporting organizational readiness assessments for ECHO, some analysis that correlates which of these constructs are consistently present in successful ECHOs and which are lacking in ECHOs that do not successfully implement might help build a more predictive model for implementation. This could help determine which constructs are key drivers of implementation success and which are just beneficial. In addition, identifying benchmarks to support optimal implementation (ie, a certain threshold of participants, fidelity to the project, etc) would help projects determine whether implementation has been successful.

CONCLUSION

These tools, developed based on implementation science theories, are a preliminary step to supporting an evidence-informed approach to the implementation of the ECHO model globally. They will offer further opportunities for implementation research and validation of constructs and harmonized implementation outcomes in the field. Effective implementation of ECHO, an educational model that has key facets that require high fidelity, is crucial to attribute whether observed outcomes (positive or negative) are an effect of the intervention as delivered with high fidelity. In addition, these tools have utility beyond the ECHO model to help organizations determine readiness for new continuing medical education models delivered within primary care. The ECHO model is increasing in reach and scale, with growing evidence for its potential to improve knowledge and capacity in primary care. Although the ECHO Institute in New Mexico provides replication support to newly developing ECHO projects, these tools are complimentary and can support the rapid expansion of Project ECHO throughout the world. This set of foundational questions, rooted in validated implementation science frameworks, can help ensure organizational fit and readiness to start an ECHO project and reduce wasted cost to funders by avoiding financing projects that are not prepared to implement an ECHO successfully.

Lessons for Practice

Lessons learned include the importance of assuring organizational readiness to implement a continuing medical education model, such as Project ECHO, to ensure fidelity. Effective implementation is paramount to confirm that practice change and health outcomes can appropriately be attributed to the educational intervention model. Using evidence-based theoretical frameworks such as the CFIR to standardize an approach to assessing organizational readiness and implementation of the ECHO model will help improve the ability to assess this intervention globally. These lessons can also support implementation of other similar interventions in the rapidly growing field of virtual continuing medical education and capacity building.