Combining Improvement and Implementation Sciences and Practices for the Post COVID-19 Era

Abstract

Health services made many changes quickly in response to the SARS-CoV-2 pandemic. Many more are being made. Some changes were already evaluated, and there are rigorous research methods and frameworks for evaluating their local implementation and effectiveness. But how useful are these methods for evaluating changes where evidence of effectiveness is uncertain, or which need adaptation in a rapidly changing situation? Has implementation science provided implementers with tools for effective implementation of changes that need to be made quickly in response to the demands of the pandemic? This perspectives article describes how parts of the research and practitioner communities can use and develop a combination of implementation and improvement to enable faster and more effective change in the future, especially where evidence of local effectiveness is limited. We draw on previous reviews about the advantages and disadvantages of combining these two domains of knowledge and practice. We describe a generic digitally assisted rapid cycle testing (DA-RCT) approach that combines elements of each in order to better describe a change, monitor outcomes, and make adjustments to the change when implemented in a dynamic environment.

INTRODUCTION

In response to the SARS-CoV-2 pandemic, health services introduced numerous changes including clinical practice, service delivery, and assessment methods, among others. There are well-developed research methods and frameworks for rigorous evaluation research into the effectiveness and implementation of these changes. But, do we have the knowledge and tools to provide research-informed help to evaluate changes when the evidence is questionable, and while the change is being implemented? For changes where there is good evidence, do researchers and practitioners have the methods and tools to adapt these effectively to a rapidly changing situation?

Given the fluid nature of the pandemic and the health system response, it is likely that many more changes will be made. We propose that combining and developing improvement and implementation methods and tools can help services respond more effectively to the pandemic and its consequences and to other emergencies or evolving crises. We highlight elements of implementation and improvement that can be combined for this purpose, and provide a clearer understanding of the relevant features of the implementation and improvement in what can be a confusing field with many different terms and models.

Before the pandemic, we conducted a narrative review of the overlap and differences between improvement and implementation sciences and practices.^{1, 2} In this article, we suggested combining improvement and implementation science in ways which would be fruitful for developing the applied knowledge and practice needed in this era of change and uncertainty. Combining these sciences can help clinicians, managers, and policymakers to bring a range of changes into routine practice more quickly to effectively respond to the demands of the pandemic. This combined approach can help to adapt proven changes to reduce disparities for vulnerable populations. In addition, combining these sciences in this way provides a method to rapidly give feedback to adjust the changes so as to reduce harm and make the change more effective. An appropriate combination of these two applied sciences could be more effective than each individually for reducing suffering, mortality, health disparities, and waste.

WHAT IS IMPROVEMENT AND IMPLEMENTATION SCIENCE AND PRACTICE?

The improvement and implementation sciences are relatively new and generally refer to knowledge about change in organization and individual practice. As such they draw on and overlap with the knowledge domains of organization behavior, systems science, operational research, and the psychology of behavior change, among others. From a sociology of knowledge perspective, they are self-proclaimed “sciences,” involving researchers competing with other knowledge fields for funding and legitimacy, and changing definitions to suit their purpose and interests. Related to both domains are a growing occupation of practitioners who draw on and sometime contribute to this body of knowledge. This includes clinical and management practitioners, as well as specialist practitioners such as quality improvement facilitators and implementation specialists who form growing occupational groups with professional societies.

IS THERE A DIFFERENCE?

Traditionally, implementation is about establishing “evidence-based practices” in everyday service delivery.³ Improvement is about a project team devising and testing changes locally. The change may be one developed by the project team using a set of quality improvement tools,⁴ or it may be a change found effective elsewhere. Both sciences seek to enable change in practice and organization and an understanding of which changes are effective. While exchanges between the domains are beginning, one hindrance is the different definitions of terms for similar concepts, due to the separate historical development of each field. The purpose of this article is not to provide a scholarly presentation of the different understandings and definitions of the two fields—these are described in our reviews and summarized in key overviews.^{1, 2, 5–7} Researchers should define the terms they are using in presentations and papers to improve communication, accumulation, and dissemination of knowledge. Table 1 identifies seven common terms that cross implementation and improvement sciences, and the definitions that we have used in this article (Fig. 1). We use a pragmatic definition of science as “knowledge published in a peer-reviewed academic journal with a high-impact factor rating.”

Table 1.

Common Terms That Cross Implementation and Improvement Sciences (Our Definitions)

Evidence-based practice (EBP):	A change that has been proven to be effective in a research study published in a credible peer-reviewed scientific journal.
Intervention:	A change in practice or organization intended to achieve a particular outcome (sometimes called the “new better way” or the “what” to be implemented, or the “change content.” It can be an EBP or it can be a change that is not proven but is thought to achieve a particular outcome (e.g., many health services reforms)). Typically, it is an assessment or treatment method, service delivery model, public health intervention, or other work practices (e.g., hand hygiene practice) undertaken by a practitioner or patient).
Implementation:	The activities and arrangements used to enable people and organizations to establish the intervention/EBP and take up this “new better way” in their everyday working (e.g., providing training to perform the “new better way” and defining responsibilities and arrangements for the training).
Sustainment:	The activities and arrangements used to maintain an intervention after initial implementation, often because other surrounding changes threaten to de-implement the intervention.
Implementation research:	An activity undertaken to describe, understand, explain, and evaluate how healthcare “takes up” a proven intervention/EBP in one unit, service, or in many. Implementation research may or may not evaluate ultimate outcomes for patients, but usually evaluates “how much” the intervention/EBP is “taken up” by a service (e.g., number of times the new practice is performed/opportunities to perform it).
Improvement research:	The activity undertaken by workers in healthcare, using quality improvement methods such as process analysis, to plan and gather data about an “improvement-change.” The change is often developed locally to address a quality or safety problem: sometimes it is not based on a tested EBP. The term also covers also studies by researchers to understand, explain, and evaluate quality improvements and quality methods carried out in healthcare. In this article, both these types of “research” are viewed as contributing to “improvement science.”
Improve-mentation integration:	Combining elements of both improvement and implementation science and practice in ways appropriate to the objectives of the research or improvement project.

Figure 1.

Common terms that cross implementation and improvement sciences (our definitions).

A common assumption of implementation has been that “exactly copying” a proven intervention, with fidelity to the original evaluated intervention, will achieve similar results in other settings. More recent implementation research has considered intentional—as well as unintentional—adaptations to both the intervention and the implementation strategy, especially in scale-up programs. For interventions with evidence of effectiveness, such as many quality breakthrough collaboratives, fidelity to the intervention may be important.³³ But improvement practitioners do not always start with a change already evaluated in research: sometimes improvement teams formulate their own change to address a quality problem, for example, by using a method such as process-flow analysis or failure mode analysis.⁴ Another significant difference is that improvement practitioners perform iterative tests of change (e.g., plan-do-study-act cycles) while implementation practitioners generally do not.

Many researchers would not consider these iterative tests of change to be sufficiently rigorous to contribute to science. However, practitioner testing using rigorous statistical process control and annotated time series designs can provide critical information about implementation that is not available through traditional health services research designs. Traditional health services research designs can also further constrain the process of continuous intervention improvement.⁸ Reflecting different understandings, institutional ethics review boards in one institution will classify a quality project as research and in another as improvement.⁹

Recent implementation studies have considered adaptation of an intervention. For example, where one component of the intervention is educational materials, a small adaptation is translation to Spanish for certain practitioners or patients, and this may or may not be sensitive to cultural meanings. Could implementation researchers test a range of types of adaptations, and if so, can they work closely with practitioners to give timely feedback about the local effectiveness of the adaptations? There are examples of researchers testing adaptations made to implementation strategies to help units by showing limited fidelity implementation of evidence-based practices. In one study, some sites showed limited implementation so the study added more facilitation assistance to help to exactly copy a new evidence-based practice for patients with mental health challenges.¹⁰

Some methods, like evidence-based quality improvement, specifically target and shape a delivery system’s use of QI methods for adapting implementation of evidence-based practices, such as guidelines, to specific settings.^{11, 12} However, a more common situation is where adaptation of the evidence-based improvement change is required to enable local implementation. There are examples of implementation research documenting adaptations to the improvement change that practitioners make, rather than to the implementation methods. In a few cases, the implementation research evaluated the adaptations, but did not make use of improvement methods that could have strengthened the study.^{13, 14}

Other differences between implementation and improvement science relates to the range of sectors and fields where the research is undertaken. To date, most improvement research has been undertaken in hospitals and has focused on less complex interventions than those studied in implementation research. In contrast, implementation research is carried out across a broad range of welfare and education services as well as in healthcare, and often involves cross-sector and multi-level complex programs.¹⁵ Central to implementation research is the theory that outcomes are due to context–intervention interactions: context is viewed not as the “background” as is often the case in traditional biomedical and health services research, but as an “actor in the play.” This leads to some implementation research using observational research designs and mixed methods approaches to learn more about implementation of interventions within specific contexts.¹⁶

Similarities and Overlaps

Yet there are also similarities that are often not sufficiently recognized by those working in each field (Table 2 and Fig. 2). Both types of science are concerned with changing clinical practices or changing how services are organized. Also, more recently, both implementation and improvement scientists have learned that complex interventions require adaptation to context. Both fields are concerned with the question of whether the adaptations are more or less effective than the originally tested version, leading to guidance about how best to adapt tested interventions.¹⁴

Table 2.

Summary: We Observe These Differences and Similarities—Does This Reflect Your Experience? (Based on Our Two Reviews of Research^{1, 2})

Differences	Similarities
Different definitions: of implementation science and of the science of improvement	Change: both improvement and implementation researcher study change in clinical practice and the organization of service delivery.
Different terms: and taxonomies used in the two fields	Researcher’s role in change: both fields include researchers who observe and study change, but also include researchers who are more involved in making changes in different ways.
Application: Improvement science is mostly applied in healthcare and manufacturing industries whereas implementation research is carried out in a range of welfare services.	Applied research with practical aims: both fields of research aim to produce knowledge that is actionable in practice.
Continuum of “research”: improvement research is carried out both by researchers and by staff working in services. Most implementation research is by full-time researchers not practitioners.	Under-use of costing and return on investment analysis: neither of the fields pays sufficient attention to estimating the costs of the intervention and its implementation to different parties, and the savings or extra costs to different parties at different time points. (We have found that this information is essential for decision-makers and for tracking pilot studies in order to decide scale-up, and one reason why many practitioners find the research not relevant to their concerns.)
Research designs and questions: Implementation research tends to use a broader range of research designs for a broader range of research questions, and makes more use of observational research designs.	Innovation in research methods: compared to a number of other fields, there has been limited innovation in research practice and methods, especially regarding the use of digital data and digital technologies.
Previous knowledge and theoretical background: much practice-based improvement research draws on theories about work processes, systems, and statistics. Implementation research generally draws on a range of social and behavioral science research theories and clinical methods.	Systems theory: researchers in both fields have sought to make use of systems theory.
Attention to context: improvement research using experimental designs has tended not to consider the role of contextual factors in a systematic way. Implementation science tends to consider contextual factors as important explanatory variables for implementation outcomes.
Adaptation of intervention: practice-based improvement research often adapts an intervention after running a small-scale test of the change; implementation research in the past has tended to focus on whether or not the intervention is copied exactly (fidelity).
Generalizable knowledge: implementation research aims to produce generalizable knowledge, often by testing hypotheses, and is required by most academic journals to be approved by ethics review boards. Improvement projects are often classified by institutional ethics review boards as not intended to be generalized and as therefore non-research. (Academic research into improvement is different and aims to produce generalizable knowledge.)

Figure 2.

Summary: we observe these differences and similarities—does this reflect your experience? (based on our two reviews of research¹²

EXCHANGE BETWEEN THE FIELDS AND COMBINATIONS

The recent pandemic and the continuing changes have motivated development and innovation in the practice and science of both implementation and improvement. We suggest four areas for greater exchange and mutual development, so as to be more relevant to the demands posed by the pandemic and its consequences for both short-term investigations and other types of research and practice.

Defining Terms and Using Taxonomies

The development of each science is hindered by different uses of terms,⁷ and sometimes these differences are not recognized, which causes more confusion. A recent review of 72 Cochrane reviews of improvement strategies concluded that “researchers reported neither fidelity definitions nor conceptual frameworks for fidelity in any articles”.¹⁷ To address these issues, taxonomies from both fields could be drawn upon to describe an intervention. Implementation science includes taxonomies of 73 different implementation strategies¹⁸ and 93 behavior change techniques,¹⁹ both produced from systematic reviews of research. Improvement science, likewise, provides taxonomies of types of quality improvement,^{20, 21} and also ways to distinguish improvement approaches from clinical interventions such as a tool for assessing QI-specific features of QI publications.²² In the new COVID-19 era, researchers and practitioners can use these taxonomies to widen their choice of changes, and design a strategy more suited to the improvement change or context. We do note, however, that a recent study found that there was no consensus between experts about which strategies would address different contextual and other challenges of implementing different interventions.²³

As regards guidance for choosing an implementation strategy for a particular improvement change in a particular context, we endorse the conclusion of this recent study, of the “need for a more detailed evaluation of the underlying determinants of barriers and how these determinants are addressed by strategies as part of the implementation planning process”.²³ There are particular problems with the growing number of “readiness assessment tools” and “capacity for change tools” (e.g.,^24–26). There is uncertainty about whether these assessments measure all, or some, of the context of an intervention, and which types of intervention each might be most suited for assessing readiness or capacity to change. Researchers from both fields together could collect and assess these tools and provide practical improvers with a guide about which to use, and for which purposes.

Using Research Theories and Frameworks from “the Other Field”

Implementation science provides evidence-based models about context factors that influence implementation.¹⁶ Related to this are theories about the interaction between context implementation and the intervention, and how this affects outcomes. Improvers can use these frameworks in several ways:

• Before a study: To identify data to gather to describe existing care practices and organization as well as barriers and facilitators to change.

• To develop a theory or model of how the intervention might work, which can later be refined using data from the study, which can test the theory of the mechanisms hypothesized to be at work (e.g.^27–29)

• To help to design an intervention based on earlier theories of effective interventions for the purpose²⁹

• During a study: To decide which data to gather to describe the changes actually made and then to be able to explain outcomes.²³

• After a study: Frameworks and theories produced by the study can help address some challenges of generalization by giving guidance to implementers for repeating the intervention elsewhere.³⁰

Implementation science and other research recommends that intervention designs be theory-based,^{30, 31} but such “program theories” or “logic models” are rarely used. Lack of attention to the theory underlying the intervention can impair the accumulation of evidence and reduces the help that research can give to practical innovators hoping to design effective change.

Improvement science can contribute theories and frameworks to assist research and practice in implementation. This includes theories of scale-up or spread and the collaborative break-through approach.^{32, 33} Many other frameworks are described in,⁴ but the evidence-basis for each varies greatly, and the scope of interventions and contexts for which they are relevant are not well-delineated.

Using Research Methods from Other Fields and the DA-RCT Generic Approach

Pragmatic experimentation or iterative testing is a generic approach used in many fields to test a change on real people or organizational processes. The plan-do-study-act testing cycle is central to quality improvement projects.^{4, 8, 34} Combined with statistical process control, and the choice of the right outcome data to track, this method is under-used in change projects and research and is one method of choice in the COVID-19 era. Some implementation research has used versions of this iterative testing model to give rapid feedback of the effects of implementation actions.¹¹

We propose a generic model for iterative testing of changes where controlled trials are not possible, and where feedback during implementation can be used quickly to adjust a change to make it more effective. For neutral language that is easily understandable by colleagues in both fields, we have found “digitally assisted rapid cycle testing” (DA-RCT) to be a useful generic approach. The approach is to choose outcomes that can be attributed to the change intervention and that can be easily collected using digital technology, or where digital data already exists (e.g., data on prescribing of antibiotics as an outcome of more appropriate prescribing that is part-attributable to a change intervention). Rapid cycle testing can be performed on small samples using statistical methods and time series or other comparative designs.⁸ This approach can be used to provide rapid feedback to implementers about the effects of COVID-19 responses, or adaptations. These include adaptions to patient assessment or treatment methods, patient information or self-help interventions, staff daily work practices, organization flow and patient pathway arrangements, supplies logistics changes, IT changes, public health interventions, the effects of policy implementation strategies, and changes to financing systems.

In addition, we propose that improvement researchers can increase the range of research designs they use by employing non-experimental or naturalistic methods commonly used in research into public health, social work, and education implementation research.^{15, 16}

Finally, more researchers from both fields could develop closer partnerships with practitioners, to learn about the change in the researcher role and the new methods needed for practice-engaged research. By exchanging lessons, for example, about when and how to get ethics review board approval, more might be gained by researchers coming together to consider ways forward for practice-based and partnership research.³⁴

“Improve-mentation” Research Teams, or Centers

One combination is to establish new “improve-mentation” research centers with staff having background in both fields, with the explicit objective to bring together improvement and implementation research for faster-impact improvements. An alternative would be to reform existing centers for this purpose, although there is the possibility that “carry-over” of existing projects and staff may dilute the focus, or give a bias towards either improvement or implementation research rather than the combination. Another option is to form research teams that combine researchers with skills and knowledge in the two fields for specific studies. The rapid impact research into Stockholm Healthcare response to the pandemic used this approach.³⁵ This option could be assisted with a searchable improve-mentation database of researchers’ skills, which could also be used by researchers quickly to assemble project teams for preparing or undertaking COVID-19 and other proposals or projects.

CONCLUSIONS

Our purpose was to describe how research and practitioner communities can use and develop a combination of implementation and improvement approaches to enable faster and more effective change in the future, especially where evidence of local effectiveness is limited. We provided a simple overview of two diverse and sometime confusing fields. More clearly defining terms and measures in scientific articles and presentations could help exchange, give consistent and less confusing terminology, and also help to develop the fundamental concepts of both fields. We presented definitions that we found allowed shared communication in workshops and conferences involving both implementers and improvers, and have been particularly useful for our work in the first part of 2020 to give rapid assistance for the COVID-19 changes and adaptations and for addressing disparities.

Both research and practical change programs could use frameworks, methods, and designs from the other field to speed improvement and help develop the sciences. We gave our understanding of the differences and some of the consequences of what we see is now an unnecessary separation of the domains, and one not justifiable in the COVID-19 era. Different “improve-mentation” combinations range from minimal exchange over specific subjects, to full integration of the fields, including forming combined science and development units. One aim of the article was to raise for debate the value and costs of combining the sciences and methods of improvement and implementation, both in research and in practical improvement. We look forward to hearing others’ views about whether they perceive an unnecessary separation between the domains, and other ideas for drawing on both fields, as well as how a third domain which we have not considered—“knowledge translation”—may speed and make improvements more effective.

Funding

This article draws on a project funded in part by the US Department of Veterans Affairs, Veterans Health Administration Quality Enhancement Research Initiative, through core funding to the Center for Implementation Practice and Research Support (Project # TRA 08-379), as well as funding from the Department of Medical Management, Karolinska Institutet, Stockholm, Sweden. Additional support was provided by the Quality Enhancement Research Initiative through its Care Coordination QUERI program (Project #QUE 15-276).

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Disclaimer

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the U.S. Department of Veterans Affairs or the Karolinska Institutet.