Research versus practice in quality improvement? Understanding how we can bridge the gap

Abstract

The gap between implementers and researchers of quality improvement (QI) has hampered the degree and speed of change needed to reduce avoidable suffering and harm in health care. Underlying causes of this gap include differences in goals and incentives, preferred methodologies, level and types of evidence prioritized and targeted audiences. The Salzburg Global Seminar on ‘Better Health Care: How do we learn about improvement?’ brought together researchers, policy makers, funders, implementers, evaluators from low-, middle- and high-income countries to explore how to increase the impact of QI. In this paper, we describe some of the reasons for this gap and offer suggestions to better bridge the chasm between researchers and implementers. Effectively bridging this gap can increase the generalizability of QI interventions, accelerate the spread of effective approaches while also strengthening the local work of implementers. Increasing the effectiveness of research and work in the field will support the knowledge translation needed to achieve quality Universal Health Coverage and the Sustainable Development Goals.

Introduction

After mixed results from the Millennium Development Goals (MDGs) strategy, the global agenda recognized the critical role of ensuring not just access but quality of health care delivery. As a result, quality and improvement have become a core focus within the Universal Health Coverage movement to achieve the goal of better population health and Sustainable Development Goals (SDGs)[1–3]. In low- and middle-income countries, quality improvement (QI) is used to identify performance gaps and implement improvement interventions to address these problems at the local, sub national and national levels. Methods used by these improvement interventions range from process improvements using incremental, cyclically implemented changes appropriate to the local context, to system-level interventions and policies to improve and sustain quality. Regardless of the scope of improvement efforts and methods employed, the impact and spread of QI has often fallen short. Causes of these lost opportunities include how decisions about improvement interventions are made, the methodology for measuring the effectiveness of the intervention, what data are collected and used and how the information on both the implementation and the intervention is communicated to drive spread and knowledge translation [4, 5]. Practitioners engaged in improvement in their organizations are frustrated by research reviews which often show a lack of conclusiveness about the effectiveness of QI when many of them see the local benefits from their work. Researchers complain about the lack of rigor in the application of QI methods in practice sittings and about poor documentation of the implementation process [6].

There is a growing realization of the need for common ground between implementers and researchers that promotes use of more systematic and rigorous methods to assess the improvement intervention effectiveness when appropriate but does not demand that all QI implementations be subject to the experimental methods commonly considered to be the gold standard of evidence. To explore the causes of this gap and address how to bridge the gap and better engage the targeted consumers of generated knowledge, including communities, governments and funders, a session ‘Better Health Care: How do we learn about improvement?’ was organized by Salzburg Global Seminar (SGS) [7]. The session brought together experts from a range of fields and organizations, including researchers, improvement implementers from the field, policy makers, and representatives from countries and international organizations.

For a partnership between researchers and implementers to become more consistent in improvement projects and studies, the incentives and priorities of each of these groups need to be better aligned in QI work and its evaluation. In this paper, we build on the Salzburg discussions, existing literature, and our own experience to explore the barriers to collaboration and offer suggestions on how to start to address these barriers. In the spirit of quality improvement, we hope that these recommendations are adopted and tried by groups interested in advancing the research and the practice of QI.

Why the gap exists

Both groups use data to evaluate whether improvements have taken place and are interested in the question of ‘did it work’. However, these gaps have occurred in part because of differences in goals, evidence needs and methods used and incentives for results and dissemination.

Goals

As we consider the major differences between researchers and implementers, we should recognize that there is not a clearly defined dichotomy between these two groups. Rather, those who are focused on in improvement are part of a continuum and are driven by a range of goals from driving and demonstrating local improvements to a focus on attributing these improvements to QI methods that can be generalized and spread, as illustrated in Table 1, which also describes differences in incentives, discussed further below. Organization-based implementers focus on quality improvement projects, where the primary goal is driving change to a local problem to improve care. Policy and decision makers' goals are broader improvement, needing evidence for current and future decision on what methods and implementation strategies to use. Researchers have a goal of developing new and generalizable knowledge about the effectiveness of QI methods.

Table 1.

Selected participants and stakeholders in quality improvement work and research and their incentives and goals

	Goals	Incentives
QI team members and institutional champions	Implement effective QI projects and promote and support change in their institutions through good improvement practice	Local improvement and disseminate the best local knowledge about what works
Policy makers whose goals are	Prioritization to invest in improvement projects based on best available evidence from academic research and practical wisdom	Make effective, yet timely and practical decisions given constraints on time and knowledge to choose and spread efficient, effective and sustainable improvement
Embedded (practice-based) researchers, QI implementers engaged in research	Drive improvement in their own setting, advance the best improvement methods in their own settings and create generalizable knowledge to make a plausible case linking the QI activities to observed outcomes for broader dissemination	Create practical yet generalizable knowledge linking improvement activities to observed outcomes for dissemination to both practice and research audiences
Academic and other researchers	Establish strong causal relationships between QI and outcomes, promoting more rigorous experimental research in QI	Use of rigorous science that can be published in peer-reviewed journals and establish objective standards of evidence

Incentives for results and dissemination

The differences in goals and evidence are related to often competing incentives. Implementers are incentivized to improve quality and meet the demands of stakeholders, whether local communities, government or funders. Researchers are rewarded through dissemination of evidence in high-impact peer-reviewed journals, research grants and academic promotions. Policy makers are rewarded by timely response to gaps with broad visible changes in their populations. Timeframes of these incentives are also often different, with the most rigorous studies taking years to measure impact, followed by careful analysis and dissemination. Implementers and policy makers, however, are often under pressure to show short-term change and respond to new and emerging issues even as they continue with existing improvement work.

The goals of documentation and dissemination of projects can also differ between researchers and implementers and their stakeholders. There is a strong recognition that the evidence generated by even the best QI efforts is not effectively translated into further spread and adoption [8]. This is because implementers working on QI interventions in their organizations are incentivized by improvement and do not usually have a demand to document their work beyond communication with organizational leaders. While there are growing venues for sharing of case reports through learning collaboratives and local meetings designed to facilitate peer learning, this documentation typically involves a description of the process of implementation, but not at a level of detail or rigor of value to researchers and the broader community. There are a number of disincentives for implementers to increase the rigor and detail of their local work including competing demands to deliver services and ongoing improvement, and the paucity of journals interested in publishing even well- documented local results because they prioritize rigorous results of evaluations with strong designs involving carefully constructed QI research studies. Researchers are incentivized by more academic dissemination through these peer-reviewed journals and presentation at conferences. This nonalignment results in practitioners being deprived of access to broader venues to disseminate their work and researchers losing rich contextual data that is critically important to evaluate the effectiveness of QI.

Evidence needed and methods prioritized

The differences in the goals and incentives of different stakeholders lead to differences in the amount of evidence that is considered adequate and the methods used to generate this evidence. Implementers are interested in the evidence of change in their local projects, with less emphasis on transferring or generalizing what they did for use in other settings. They may rely on a combination of pre-and-post intervention data, QI statistical methods such as run charts and tacit organizational knowledge to assess the evidence of change in their projects. Policy makers have an interest in evidence which is robust enough from the QI to inform resource allocation, but may still have a focus on a specific geography rather than generalizability at scale. They are interested in generalizable knowledge about successful QI methods, but are sensitive to the burden and costs and time of requiring rigorous research methods on implementing groups.

Researchers aim for evidence which is robust enough to provide globally relevant conclusions with limited threats to internal validity. This group is most supportive of the use of rigorous experimental research designs to generate the highest possible standards of evidence. Traditionally, this had been limited to a small set of rigid experimental designs with appropriate controls or comparison groups driven in part by research funders and academic standards to be able to attribute change to the improvement interventions. This set of designs has been expanding in the past few years as better understanding of the value of quasi-experimental methods has emerged. [9, 10]

Why better alignment is needed

QI interventions differ from many fixed clinical or public health interventions [11]. In this supplement, Ramaswamy and others describe QI interventions as complex (multi-pronged and context-specific) interventions in complex systems (non-linear pathways and emergent behaviors). For better learning from QI, implementers, policy makers and researchers both need to know not just effectiveness (the focus of local measurement, outcomes research and impact evaluation) but also 'how and why' the change happened (implementation), cost and sustainability ensuring that the evidence produced will be more relevant to the stakeholders at the local and broader level. Therefore, finding a common ground through ‘development of a culture of partnership’ [12] to co-identify appropriate methods and data collection to understand and disseminate implementation strategies is critical to inform how to how to create the different knowledge products: generalizable evidence for dissemination (researchers), insights into how to scale (policy makers) and how to sustain the improvements (implementers) [13]. A well-known and commonly cited example is the Surgical Safety Checklist, which was found to improve adherence to evidence-based practices and save lives across a range of settings [14]. However, attempts to replicate these successes were not always effective since capturing generalizable knowledge on how to introduce and support the implementation of this intervention with fidelity was not part of the original research dissemination, [15] a lesson understood by the original researchers and addressed through accompanying toolkits [16].

Another important area where collaboration between implementers and researchers is needed to improve learning from QI in understanding the impact of different contextual factors to identify which aspects of an improvement intervention are generalizable, which are context-specific and which are critical to address when planning replication. During the seminar, a study of antenatal corticosteroids (ANCS), an intervention found in higher income settings to reduce death among premature infants, was discussed to identify how contextual factors can be better addressed through local knowledge to inform implementation [17]. The randomized controlled trial showed that implementation of ANCS in low-resource settings resulted in increased mortality among some of the infants who were given steroids; the published conclusion was that ANCS was not a recommended improvement intervention in these settings. The group identified the gap in the translation of ANCS use from resource richer settings did not consider the different contextual factors which required adaption such as the lack of capacity to accurately determine prematurity needed to determine eligibility for the steroids.

Starting the work to bridge the gap

Based on the reasons for the gaps identified above, we recommend a number of initial steps to better bridge the gap between researchers and implementers:

• Aligning project goals and joint planning: Before QI projects get launched, the initial work must start with implementers and researchers discussing and agreeing on the goals and objectives of the work including and beyond local improvement. In addition to alignment of improvement goals, all stakeholders must be engaged at the start of the QI project to agree on the purposes and uses of the results, local learning or broader dissemination or both. This work needs to happen at the design phase and continue with ongoing planned communication throughout the work. This will ensure that all stakeholders are jointly engaged in identifying the most appropriate research questions and the most appropriate methods to answer them.

• Choosing the right research design. The joint framing of goals and research questions can lead to a selection of evaluation and research designs at an appropriate, mutually agreed upon level of rigor including right research methodology for success [18]. This balancing of rigor versus flexibility, described in the meeting as a ‘bamboo scaffold that bends in the wind’ can only be accomplished when there is an open discussion of trade-offs between investments in data collection for research and data collection for demonstrating local improvements. Detailed documentation of implementation approaches is time consuming and resource intensive, and cannot be routinely expected for every project. On the other hand, some improvement in documentation as part of routine practice will benefit practitioners by providing important insights about local sustainability, and can be used by researchers to assess generalizability, attribution and scale.

  The need to understand both process and context in the evaluation and study of QI interventions also cannot be accomplished without engaging both researchers and practitioners in the process [13]. The knowledge about how the project was implemented, and what was relevant to the context often resides with those responsible for implementation. However, as mentioned previously, the implementers often have neither the incentives nor the support to systematically document and disseminate this knowledge in a way that makes it available for general use. Researchers can play a key role in influencing the QI research integration by supporting systematic documentation of the implementation process in addition to an evaluation of outcomes and by partnering with implementers to make this happen. Introduction of adaptive designs such as SMART trials into improvement research may also offer a common ground where improvement implementers and researchers can collaborate introducing use of data to make mid-course changes to the implementation design.

• Building implementer research capacity. Building capacity of implementers as potential producers of and better consumers of research and evaluation results in another important approach to bridge the gap. For example, empowerment evaluation is designed to increase the likelihood that programs will achieve results by increasing the capacity of program stakeholders to plan, implement and evaluate their own program [19]. Building capacity within implementing organizations through technical support provided by researchers for interested implementers can establish a viable infrastructure for practitioners and researchers to work together more effectively. For example, multi-year research practice partnerships in facilities in Kenya has led to sustainable QI programs with dissemination of methods and results through co-authored peer-reviewed journals and conference presentations [20] Similar results were seen for research capacity building targeting implementers in the Africa Health Initiative in five countries in Africa [21]. Support for practice-based researchers to build their capacity in QI and in process evaluation using implementation science methods can also increase the potential of improvement projects to produce the knowledge needed about the implementation to spread learning within and beyond their organization.

• Aligning incentives to drive collaboration: Creating areas of shared incentives will require initiatives from funders and universities to appreciate the higher value of co-produced research, reward capacity building of researchers in the field and fund innovative models of embedded research where researchers are part of or embedded into the implementing organization [22]. In addition, offering opportunities for meaningful participation in research and building capacity for this work among implementers has also been associated with better improvement and dissemination [23].

• Simplifying documentation for dissemination of learning: As mentioned earlier, it is useful for both implementers and researchers if documenting the implementation of QI programs becomes part of routine practice. However, this will not happen without simplifying documentation standards. SQUIRE and TiDieR guidelines are very helpful for academic publications. However, they are not always a good fit for projects whose primary purpose is not research but who have the potential to add to the knowledge needed to improve QI [24, 25]. Researchers could partner with implementers to develop simpler, practice-based research guidelines and to create other venues such as through existing organizations focused on quality and improvement where methods and results could be posted using these guidelines without a formal peer-review process. Templates and examples could be provided to improve the quality of documentation as well as editorial staff to assist with structure and formatting. The incentive for implementers is to get their stories told, and at the same time provide an opportunity for researchers to get data on where to focus further research. In addition, there are growing options to share knowledge and research findings such as the WHO Global Learning Lab for Quality UHC which provides a forum for implementers to disseminate work available to broader community [26].

Conclusion

To improve learning from and effectiveness of QI work requires involvement and collaboration between both researchers and practitioners. Researchers can advance the field by creating generalizable knowledge on the effectiveness of interventions and on implementation strategies and practitioners improve outcomes on the ground by implementing QI interventions. By increasing the collaboration, more systematic evaluations of interventions in local contexts and better design of research will result in production of the generalizable knowledge needed to increase the impact of QI. In order for this to take place, there needs to be an intentional effort to address the gaps that challenge researchers and practitioners working together. This can occur by aligning incentives, increasing the value and utility of produced research to implementers, and as a shared community developing new guidance to bring these different groups to more effective collaboration. The growing experience in QI and improvement science offers many opportunities for better collaboration between researchers and implementers to increase the value of this partnership to accelerating progress toward quality Universal Health Coverage and the Sustainable Development Goals.

Funding

M.D. received financial support from SGS to attend this seminar.