Research and Scholarly Methods: Implementation Science Studies

Abstract

Traditional research focuses on efficacy or effectiveness of interventions but lacks evaluation of strategies needed for equitable uptake, scalable implementation, and sustainable evidence-based practice transformation. The purpose of this introductory review is to describe key implementation science (IS) concepts as they apply to medication management and pharmacy practice, and to provide guidance on literature review with an IS lens. There are five key ingredients of IS, including: (1) evidence-based intervention; (2) implementation strategies; (3) IS theory, model, or framework; (4) IS outcomes and measures; and (5) stakeholder engagement, which is key to a successful implementation. These key ingredients apply across the three stages of IS research: (1) pre-implementation; (2) implementation; and (3) sustainment. A case example using a combination of IS models, PRISM (Practical, Robust Implementation and Sustainability model) and RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance), is included to describe how an IS study is designed and conducted. This case is a cluster randomized trial comparing two clinical decision support tools to improve guideline-concordant prescribing for patients with heart failure and reduced ejection fraction. The review also includes information on the Standards for Reporting Implementation Studies (StaRI), which is used for literature review and reporting of IS studies,as well as IS-related learning resources.

Traditional biomedical and health services research address questions related to efficacy (i.e., testing under optimal conditions focused on internal validity) or effectiveness (i.e., testing under real-world conditions with a focus on external validity) of interventions being studied. In these studies, less focus is placed on understanding factors and strategies needed for the equitable uptake, implementation, and sustained use of the intervention as it relates to practice change and evidence-based practice transformation. Even more broadly generalizable effectiveness trials, such as pragmatic clinical trials, provide little guarantee of public health impact.^1,2 Additionally, it takes an average of 17 to 20 years for 14% of evidence-based innovations to be integrated into routine clinical practice.^1,3,4 Implementation science (IS) is intended to address these gaps and accelerate change.⁵ These gaps also exist within pharmacy. Accordingly, some have called for the use of IS to transform pharmacy practice and curricula.^6–9 There are numerous opportunities to accelerate the implementation of evidence-based interventions in pharmacy. Potential targets for the application of IS models and methods to advance pharmacy practice and care delivery include iterative design and evaluation of different models for pharmacist-led chronic disease management in outpatient clinics and transitions of care services, de-prescribing of inappropriate medications in older adults, antibiotic stewardship, applying clinically relevant pharmacogenomic recommendations that promote health equity, or improving population health using value-based reimbursement metrics.^10–12 A recent review of factors influencing the implementation of innovations in community pharmacies identified important barriers to nationwide implementation, including pharmacy staff engagement with the innovation, resources needed for successful operationalization of the innovation, and external engagement (e.g., the perceptions of patients and other health care professionals and their relationship with the community pharmacy) with implemented innovations.¹³

IS is defined as “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services”.^5,14–16 Research projects using IS frameworks incorporate health care perspectives from multiple stakeholders across multiple levels including the patient, provider, organization, and policy. Its goals are to optimize the equitable uptake and scalable sustainment of innovative interventions into routine settings by identifying contextual barriers and facilitators, and developing and applying strategies to overcome these barriers and enhance facilitators.^17–23

IS studies are different from efficacy, effectiveness, quality improvement, program evaluation, and dissemination studies. Efficacy and effectiveness studies evaluate the efficacy and effectiveness of interventions, whereas IS usually seeks to evaluate implementation outcomes of increasing uptake and sustainability of interventions with established efficacy or effectiveness.^17,24 IS often employs effectiveness-implementation hybrid study designs in which implementation and effectiveness outcomes of an intervention are evaluated simultaneously.²⁴ Quality improvement usually starts with a specific problem identified from multiple stakeholders in a specific practice to find strategies to improve that specific problem, whereas IS typically begins with an under-utilized evidence-based practice to addresses quality gaps at multiple stakeholder levels.^1,17 Additionally, IS findings lead to generalizable knowledge that can be widely applied beyond the specific practice site under study.^1,17 Program evaluation may overlap but is not the same as IS.²⁵ Program evaluation is the science of improvement with a focus on changes in outcomes while IS integrates intervention into a variety of settings, with a focus on methods and measures and the impact of implementation strategies.²⁵ IS may overlap with dissemination science, but is different in that dissemination science uses communication and education strategies to spread information.^15,17,26 For example, the pharmaceutical industry effectively employs communications and marketing strategies such as direct-to-consumer advertising to disseminate drug information, but they seldom study the implementation impact of practice change or sustainability from drug products.^27,28

Translational research investigates “the scientific and operational principles underlying each step of the translational process.”²⁹ Within the spectrum of the five-phased National Institute of Health’s roadmap for translational research continuum spanning from basic research (T0) through population health research (T4), IS fits within the third (T3) and the fourth (T4) translational steps.^5,30,31 The third step (T3) is translation to practice and identifies strategies that move evidence-based interventions or clinical guidelines into practice, and the fourth step (T4) is translation to community and involves stakeholders to evaluate implementation strategies and sustainability of interventions. IS bridges the “evidence-practice gap” – the gap between scientific discoveries or evidence and their translation into clinical practice.⁵ Most IS research in health care is conducted in disciplines other than in pharmacy. The purpose of this introductory paper is to describe key IS concepts as they apply to medication management and pharmacy practice, and provide guidance on literature review with an IS lens.

IMPLEMENTATION SCIENCE CONCEPTS

The key ingredients (or core elements) of IS studies include: (1) evidence-based intervention; (2) implementation strategies; (3) IS theory, model, or framework (TMF); (4) IS outcomes and measures; and (5) stakeholder engagement that are integrated within and informs all of the former components (Figure 1). Stakeholder engagement is at the core of IS and no IS work can be successfully completed without the engagement of diverse stakeholders. These key ingredients apply across the three stages of IS research, which include: 1) pre-implementation planning or study design; 2) implementation or study conduct; and 3) sustainment or study evaluation. To guide our review of IS, we propose a visual that has been successfully used in prior training and introductory presentations and includes the five key ingredients of IS. The idea of the “3 stages” of IS has been published in multiple works, but most prominently through the Veterans Administration Quality Enhancement Research Initiative (VA QUERI) Implementation Roadmap.^32,33

Figure 1.

Key ingredients/core elements of implementation science (IS) studies.

This section explores each of the key ingredients and provides guidance on including these across the three stages of IS research. Further, to illustrate how these ingredients are applied at each stage of research (Figures 2 and 3), we will refer to a case example describing the implementation of a clinical decision support (CDS) tool to improve guideline-concordant prescribing for patients with heart failure and reduced ejection fraction (HFrEF). Instead of including multiple examples on a broad scope, we selected one well-developed example with details to guide the reader through the process of designing an IS study. This study, conducted by a co-author (KT), aimed to compare the effectiveness of two different CDS tools to improve guideline-concordant prescribing of beta blocker medications for patients with HFrEF: a commercial CDS versus an enhanced CDS.^34,35 Both CDSs alerted primary care clinicians within the electronic health record (EHR) during an office visit and recommended initiation of a beta-blocker for patients with HFrEF. However, the enhanced CDS was customized (user-centered) to be contextually relevant for the local health setting, whereas the commercially available (not user-centered) CDS was provided by the EHR vendor and created for the average health system. As depicted in Figure 2, IS was used to inform the overarching study design (pre-implementation stage), monitor and evaluate adaptations (implementation stage), and develop (pre-implementation stage) and evaluate (sustainment stage) outcome measures. IS was also used to inform the user-centered and contextually relevant design of the enhanced CDS tool (implementation stage), but not the design of the commercially available CDS tool. Figure 3 is adapted from Smith and colleagues’ Implementation Research Logic Model and illustrates how the PRISM (Practical, Robust Implementation and Sustainability model) / RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) IS framework was applied to this study.³⁶

Figure 2.

Stages of implementation science (IS). Example illustrating how IS and the five key ingredients (bolded) were applied to a cluster randomized trial comparing two clinical decision support (CDS) tools to improve guideline-concordant prescribing for heart failure.

Figure 3.

Description of how the PRISM/RE-AIM IS framework was used to inform the design of the enhanced CDS intervention and outcomes evaluated. CDS = clinical decision support; IS = implementation science; PRISM= Practical, Robust Implementation and Sustainability model; RE-AIM = Reach, Effectiveness, Adoption, Implementation, Maintenance.

Evidence-Based Intervention

Evidence-based interventions can be broadly defined in the context of IS research and can include what Brown and colleagues referred to as the 7Ps: programs, practices, principles, procedures, products, pills, and policies.^25,37 Deciding on what is ready for broader dissemination and implementation (i.e., when there is enough evidence to justify practice change) can differ depending on the standards of a given field or industry.^38,39 The type and strength/amount of evidence expected for a new medication before broad dissemination will be very different from the evidence required for the implementation of a behavioral program to address smoking cessation. Because there is not one standard to determine the readiness of a program for broader implementation, it is important to discuss the type of evidence that exists and why the researcher or practitioner believes that the available evidence justifies spread.

In the heart failure example, the evidence-based intervention is a CDS tool that makes recommendations to improve adherence to guidelines (Figures 2 and 3). In this study, the CDS tool was designated as the intervention’ based on the research question, whereas another research question may have appropriately designated the CDS tool as an IS strategy. The potential for ambiguity between the intervention and strategy reinforces the importance of clearly making these designations in research. The guideline recommendations are supported by numerous, high-quality studies and are highly regarded by the clinical community as important to HFrEF management,^40,41 and CDS tools are considered to be an evidence-based method for supporting decision-making that are required by federal regulations;^42,43 thus, the CDS tools are ready for spread.

Implementation Strategies

The ingredient that distinguishes IS most from more traditional efficacy and effectiveness research is the focus on implementation strategies. Implementation strategies are processes or activities that are used to support the equitable uptake, implementation, and sustained use of an intervention. They can be classified as discrete (individual activity) or multicomponent implementation strategies; they can act at different levels, such as the patient or student (sometimes also called supportive interventions), clinician or professor (i.e., pharmacist, physician, faculty), setting (i.e., pharmacy, clinic, classroom), or leadership (i.e., network manager, clinic director, Dean).^44,45 To facilitate the distinction between interventions and strategies, Curran and colleagues described the intervention as the what or ‘the thing’ that is being implemented, and the implementation strategy as ‘how’ you will implement ‘the thing.’⁴⁶ Powell and colleagues developed a taxonomy for implementation strategies, the Expert Recommendations for Implementing Change (ERIC). The ERIC taxonomy provides the listing of 73 distinct strategies organized around nine key domains,^44,45 as described in Figure 4.⁴⁷

Figure 4.

Expert recommendations for implementing change (ERIC): nine domains and examples of implementation strategies (used with permission from the University of Washington Implementation Science Resource Hub).⁴⁷

Multiple publications and resources have been developed to support the selection of implementation strategies.^48–52 Briefly, implementation strategies can be linked to the anticipated or previously identified contextual determinants (barriers, facilitators) of the implementation of the intervention. It is important to sufficiently specify what the implementation strategy entails including what it is called, what activities are involved and why those were selected, when and by whom it is delivered to whom, and when and the target outcomes it is intended to change. Figure 5 provides guidance from Proctor and colleagues on specifying implementation strategies^47,53 and Figure 6 depicts how this guidance was applied to the workflow integration as an IS strategy that was used in the heart failure case.

Figure 5.

Recommendations to specifying implementation strategies (used with permission from the University of Washington Implementation Science Resource Hub).^{47, 53}

Figure 6.

Applying the recommendations for specifying implementation strategies. Description of how the workflow integration IS strategy used in the heart failure example was specified.

The IS TMF (see more details about this ‘ingredient’ in the next section) can be helpful in identifying implementation strategies as it is shown by the ERIC- Consolidated Framework for Implementation Research (CFIR) matching tool, which links specific CFIR constructs with relevant strategies from the ERIC compilation.⁵²

In the heart failure example, implementation strategies were designed to address contextual determinants. For example, incentive strategies were used to address clinician time constraints, such that clinicians saved time by using the CDS intervention. Figure 3 illustrates how implementation strategies can be identified.

IS Theory, Model, and Framework (TMF)

Most studies in IS are guided by specialized theories, models, and frameworks (TMFs).^54–56 The purpose of IS TMFs is to provide a general structure for IS studies where IS relevant and connected constructs can inform the: formulation of research questions; identification of key contextual determinants (barriers, facilitators) and related strategies; selection of study design, measurement plan (e.g., specific implementation and outcome measures), and analytic plan; and interpretation and presentation of findings. Nilsen provides a classification of TMFs as ones that can guide the implementation of evidence-based interventions into practice (i.e., process models), models that help us better understand and explain influencers of implementation outcomes (i.e., determinant frameworks, classic theories, and implementation theories), and models that guide evaluation (i.e., evaluation frameworks).⁵⁷ A TMF can fulfill multiple of these purposes but could evolve overtime; for example, the RE-AIM framework can be used as a planning, implementation, and evaluation TMF⁵⁸ and CFIR 2.0 can be used as a determinant and evaluation framework.⁵⁹

For full benefit, IS TMFs should be consistently used iteratively throughout the lifetime of a project from early conceptualization through implementation to sharing of findings. TMFs should also be used with a health equity lens to promote equity and representativeness of outcomes.^18,20,23 There are a large number of IS TMFs available to choose from to guide planning, implementation, and evaluation activities for IS studies.^55,60,61 The most recent review by Strifler and colleagues identified 150 IS TMFs.⁶² The most commonly used TMFs include the Diffusion of Innovations theory,⁶³ the CFIR,^13,64 the RE-AIM framework⁵⁸ and its contextually-expanded version of the Practical, Robust Implementation and Sustainability model (PRISM),⁶⁵ the Exploration, Preparation, Implementation, Sustainment model (EPIS),⁶⁶ and the Promoting Action on Research Implementation in Health Services (PARiHS) framework.⁶⁷ An extensive listing of IS TMFs and guidance on how to plan, select, combine, adapt, use, and measure IS TMFs has been created by co-author Rabin and colleagues in the form of a publicly available interactive webtool, the Dissemination and Implementation Models in Health webtool (http://www.dissemination-implementation.org). In addition, TCast (an implementation Theory Comparison and selection tool) can be used to compare different IS TMFs.⁶⁸

Because a large number of IS TMFs exist, it is unlikely that the creation of new TMFs is warranted. We encourage researchers and practitioners to identify a TMF that has been used in their situation or with the population they are concerned with. In some cases, the combination of multiple TMFs can provide a more comprehensive reflection of the research or practice need. Often the selection is guided by the availability of expertise for a given TMF. Because the use of TMF requires some familiarity of how to operationalize them, it is best to work with a colleague who has prior experience and expertise in using the given TMF. Many of the models we listed earlier have similar or comparable constructs and can often be used interchangeably. As a general rule, a desirable IS TMF will include multiple contextual levels (e.g., student, teacher, dean), links contextual domains with implementation and effectiveness outcomes, and has measures linked to its constructs. For those newer to IS, it can be helpful to have examples of the use of the given TMF in their own area of interest.

The heart failure case example illustrates how PRISM/RE-AIM can be used to inform a pharmacy-relevant study (Figures 2 and 3). PRISM is a good fit for this case example as it includes multilevel contextual domains (Intervention characteristics as perceived by patients and clinicians; Internal recipients of patients, clinicians, and leaders; Implementation and sustainability infrastructure; and External environment) that interact dynamically, and directly connect with a set of implementation outcomes (Reach, Adoption, Implementation, and Maintenance at the organizational level) and clinical outcomes (Effectiveness, Maintenance at the individual level) as described in Figure 3. To learn more about PRISM/RE-AIM, a set of tools are freely available at re-aim.org. The next section will provide more specifics about IS outcomes and measures.

IS Outcomes and Measures

Along with implementation strategies, another unique aspect of IS studies is the inclusion of pragmatic implementation outcomes along with more traditional clinical outcomes that consider key issues such as representativeness. In fact, often the primary focus of IS studies is on implementation outcomes because the effectiveness of the intervention has already been established. In the previous section, we listed a set of implementation outcomes included in the RE-AIM framework (i.e, Reach, Adoption, Implementation, and organizational Maintenance). Specific definitions from a more academic and pragmatic perspective are provided by Estabrooks and Glasgow and through the numerous resources of the RE-AIM.org webtool.⁶⁹

Another commonly utilized guide for identifying outcomes of IS studies is the Implementation Outcomes Framework (IOF).⁷⁰ The IOF distinguishes implementation outcomes, service outcomes, and client outcomes. The key implementation outcomes in IOF include acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability. The service outcomes are organized by relevant standards of care as defined by the organization or setting; examples include efficiency, safety, effectiveness, equity, and patient-centeredness.⁷⁰ Finally, client outcomes align more closely with outcomes from traditional efficacy and effectiveness studies and can include measures of various clinical outcomes as well as satisfaction and quality of life. As it might be noted by the reader, the implementation outcomes defined by the RE-AIM framework and the IOF are similar. A recent publication by Reilly and colleagues provides definitions and a crosswalk between the two frameworks.⁷¹ There are multiple publications providing guidance for key measurement considerations for IS.^72–76 In this paper, we highlight only one and encourage the reader to explore additional guidance from other publications. For successful and meaningful evaluation of IS studies, it is critical to develop outcomes measures from multiple stakeholder perspectives (e.g., what matters to leaders versus clinicians versus patients) that consider representativeness and to collect a combination of quantitative and qualitative data.^77–80 Finally, there are resources and repositories that can support identification of IS measures. These include the SIRC Instrument repository,^81,82 the Grid-enabled Measures IS workspace,^83,84 and the IS policy measures database by Washington University in St. Louis.⁸⁵

The heart failure case example relies on the RE-AIM outcomes and provides an overview of how the individual constructs of RE-AIM are operationalized (Figures 2 and 3). To assess reach, data were collected from the EHR and a ratio of patients who were reached by the CDS intervention was divided by all patients with a HFrEF diagnosis. In this situation, the ideal denominator for reach would be those eligible for a beta-blocker medication, but due to technical constraints of data availability, we used a less precise denominator. Similarly, adoption was evaluated using EHR data and defined as the proportion of clinicians who saw a CDS alert and used it. For the comprehensive exploration of implementation success, a combination of EHR data and information gained from interviews with clinicians and leaders was used.

Stakeholder Engagement

While all four prior ingredients are important for IS research, the meaningful engagement of diverse stakeholders throughout the planning, implementation, and sustainment of the intervention can serve as one of the most important predictors of success.^86–88 When the diverse stakeholders are identified and included as partners throughout the entire IS research process, the likelihood of the equitable uptake, implementation, and sustained use of an intervention will be substantially increased. All IS studies should identify key stakeholders and devote strategies and resources to engaging them in the research process. Stakeholder engagement is considered on a spectrum from minimal engagement (inform) to complete partnership and co-creation of products (empowerment).^89,90 While not all research projects are resourced appropriately to include all stakeholders as equal partners, there are always ways to improve engagement practices⁸⁷ and increase representation in order to improve representative and equitable implementation.^18,20 Regardless of the extent of stakeholder engagement, IS cannot be successfully implemented without stakeholder engagement.

The heart failure example describes how diverse patients, clinicians, and leadership were engaged at each stage of the study, which included focus groups and interviews (Figures 2 and 3).

REPORTING ON IMPLEMENTATION SCIENCES STUDIES

Similar to how the Consolidated Standards of Reporting Trials (CONSORT) was developed to address concerns of inconsistent reporting of randomized controlled trials,⁹¹ the Standards for Reporting Implementation Studies (StaRI) was developed to provide transparent and accurate reporting of IS studies.^92,93 StaRI was developed under the guidance of the EQUATOR Network (www.equator-network.org). The StaRI Checklist comprises 27 items, with 3 overarching components: (1) logic pathway of an explicit hypothesis for both IS strategy and intervention; (2) balance between fidelity (“the degree of adherence to the described IS strategy and intervention”) and adaptation (“the degree to which users modify the strategy and intervention during implementation to suit the local needs”); and (3) context of successful implementation in the planned, facilitated process. As suggested by the first overarching component, the intervention and the implementation strategy both need to be considered when reviewing IS studies. As discussed in an earlier section of this paper, and as in most IS studies, the primary focus is on the implementation strategy, while also considering the evidence about the impact of the intervention on the priority population. StaRI was designed agnostically and applies to a wide range of study designs including randomized controlled trials, case studies, cohorts, time series, pre-post, and mixed methods, and should be used in combination with the design specific checklists.

In the heart failure example, the logic pathway used was the PRISM/RE-AIM framework and the hypothesis was that a user-centered and contextually-relevant CDS tool would lead to greater reach, adoption, and effectiveness of prescribing evidence-based medications for HFrEF; fidelity and adaptation were balanced by preemptively defining core components or functions of the CDS tool (e.g., interruptive CDS tool for primary care clinicians within the EHR workflow, CDS recommendations were evidence-based) and planning for proactive evaluation and documentation of any unanticipated adaptations based on their alignment with the core components/function with a description of contextual factors; and the context of the local setting and population (leadership structure, clinicians, patients) was described with enough detail that others could evaluate the relevance of the findings to their situation.

RESOURCES

In this introductory review, we have described key IS concepts using an example applicable to medication management and provided guidance on reviewing the literature with an IS lens. We look forward to seeing more IS studies in the field of pharmacy practice, medication management, and health professional education. For those interested in finding resources or training related to IS, additional information is available from the National Cancer Institute’s TIDIRC program – Training Institute for Dissemination and Implementation Research in Cancer (https://cancercontrol.cancer.gov/is/training-education/training-in-cancer/TIDIRC-open-access), the Academy Health (https://academyhealth.org/evidence/methods/translation-dissemination-implementation), the National Institutes of Health (NIH) National Center for Advancing Translational Sciences (https://ncats.nih.gov/), or the Agency for Healthcare Research and Quality (AHRQ) (https://www.ahrq.gov/ncepcr/tools/transform-qi/index.html).

Table 1.

Key Considerations When Reporting on or Reading an Implementation Science (IS) Study

• Compliance with the StaRI Standards (Guidance of the EQUATOR Network)92,93

• Three overarching components: (1) logic pathway; (2) balance between fidelity and adaptation; and (3) context

• Five key ingredients: (1) evidence-based intervention; (2) IS strategy (primary focus); (3) TMF; (4) IS outcomes and measures; and (5) stakeholder engagement

• Combine other standards as needed (e.g., CONSORT, STROBE)91

Note: StaRI – the Standards for Reporting Implementation Studies; EQUATOR – Enhancing the QUAlity and Transparency Of health Research; TMF – Theory, Model, and Framework; CONSORT – Consolidated Standards of Reporting Trials; STROBE – Strengthening the Reporting of Observational Studies in Epidemiology.