Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol

Abstract

Introduction

Engaging policymakers, healthcare managers and policy analysts in the conduct of knowledge synthesis can help increase its impact. This is particularly important for knowledge synthesis studies commissioned by decision-makers with limited timelines, as well as reviews of health policy and systems research. A scoping review will be conducted to assess barriers, facilitators, strategies and outcomes of engaging these individuals in the knowledge synthesis process.

Methods and analysis

We will follow the Joanna Briggs Institute guidance for scoping reviews. Literature searches of electronic databases (eg, MEDLINE, EMBASE, Cochrane Library, ERIC, PsycINFO) will be conducted from inception onwards. The electronic search will be supplemented by searching for sources that index unpublished/difficult to locate studies (eg, GreyNet International database), as well as through scanning of reference lists of reviews on related topics. All study designs using either qualitative or quantitative methodologies will be eligible if there is a description of the strategies, barriers or facilitators, and outcomes of engaging policymakers, healthcare managers and policy analysts in the knowledge synthesis process. Screening and data abstraction will be conducted by 2 team members independently after a calibration exercise across the team. A third team member will resolve all discrepancies. We will conduct frequency analysis and thematic analysis to chart and characterise the literature, identifying data gaps and opportunities for future research, as well as implications for policy.

Ethics and dissemination

This project was commissioned by the Alliance for Health Policy and Systems Research, WHO. The results will be used by Alliance Review Centers of health policy and systems research in low-income and middle-income countries that are conducting knowledge synthesis to inform health policymaking and decision-making. Our results will also be disseminated through conference presentations, train-the-trainer events, peer-reviewed publication and a 1-page policy brief that will be posted on the authors' websites.

Strengths and limitations of this study.

• We will conduct a comprehensive literature search of multiple electronic databases and sources for difficult to locate and unpublished studies (or grey literature).

• Our scoping review will conform to the methodologically rigorous methods manual by the Joanna Briggs Institute.

• To increase the feasibility of our scoping review, we will limit to English, papers disseminated in the past 20 years, and have one data abstractor and one verifier.

• To focus our scoping review, we will not include patients or healthcare providers as knowledge users of the knowledge synthesis process.

Introduction

Engaging knowledge users (ie, individuals who can use research results to make decisions¹), such as patients, healthcare providers, healthcare managers and policymakers is imperative to ensure broad impact of health research and is an essential component for effective knowledge translation (KT; also referred to as research usage or implementation).² Traditionally, the engagement of knowledge users has been at the end of research activities, whereby researchers disseminate their results passively to potential knowledge users who may be involved with further dissemination of results.^{3 4} However, the limitations of such end-of-grant KT activities are increasingly recognised, both in terms of policy-relevant processes and potential impact on health policymaking.⁵

Recent research focusing on the engagement of policymakers suggested that iterative and ongoing collaboration can facilitate real-world policymaking.⁶ A qualitative study based on 106 interviews of potential knowledge users suggested that an integrated KT approach may increase the use of research findings.⁷ In integrated KT, knowledge users are included as part of the research team and may participate in all aspects of the research conduct and dissemination.⁸

A recent scoping review synthesised 13 empirical studies on engaging policymakers in health research using integrated KT.⁴ In this review, Gagliardi et al⁴ identified 9 barriers (eg, differing needs and priorities between researchers and policymakers, lack of understanding of the process) and 15 facilitators (eg, opportunities for interaction, strong commitment, skill, experience) for integrated KT in health research. They noted that none of the included studies used a theory or framework (such as the framework for effective engagement in comparative effectiveness research)⁹ to inform the engagement activities, and the specific details regarding the engagement pathways were rarely reported. In addition, they were unable to determine contextual factors that may facilitate or hinder the integrated KT process.

Knowledge synthesis can be used to comprehensively summarise a body of research and package research results in an accessible way for decision-makers, creating a unique opportunity for knowledge user engagement. Keown et al¹⁰ identified five opportunities to engage knowledge users in the knowledge synthesis process, including topic consultation, input into the protocol, participation as a review team member, meeting to discuss results and dissemination (eg, lay summary of review findings, presentations to stakeholders, publications).

To examine stakeholder engagement further, Oliver and Dickson¹¹ conducted interviews of policymakers and researchers. They identified four models to increase engagement of policymakers in the conduct of knowledge synthesis for policymaking. For example, one model focuses on conducting rapid reviews specifically to address political priorities. They concluded that circumstances and needs of policymakers need to be matched with the appropriate knowledge synthesis method. Other efforts have also examined ways to increase the uptake of knowledge synthesis for policy decision-making.^12–14 However, the optimal strategies for engaging policymakers, healthcare managers and policy analysts in the knowledge synthesis process and how these vary by context remain unclear. Furthermore, there is a paucity of research on engagement strategies to support policy-relevant knowledge synthesis in low-income and middle-income countries (LMICs).

As such, we aim to conduct a scoping review to assess barriers, facilitators, strategies and outcomes of engaging policymakers, healthcare managers, and policy analysts in the knowledge synthesis process. The specific research questions are:

1. What are the barriers and facilitators in engaging policymakers, healthcare managers and policy analysts in the knowledge synthesis process?

2. What strategies exist to engage policymakers, healthcare managers and policy analysts in the knowledge synthesis process?

3. In studies describing strategies for engaging policymakers, healthcare managers and policy analysts, what outcomes do they measure to evaluate engagement mechanisms (eg, attitudes, beliefs, knowledge) and what are the results (eg, benefits, unintended consequences)?

4. What contextual factors were reported in the included studies (eg, health system setting, high-income countries vs LMICs)?

Methods

Commissioning agency

This project was commissioned by the Alliance for Health Policy and Systems Research (HPSR), WHO (hereafter the Alliance) with a 5-month timeline. We were requested to conduct this scoping review to inform the Alliance's portfolio of work on HPSR, including support to Review Centers of HPSR in LMICs (hereafter Alliance Review Centers) that are conducting rapid reviews and systematic reviews to inform health policy decision-making.

Study design

This study will be completed using the scoping review method¹⁵ following the methods outlined by the Joanna Briggs Institute Methods Manual for scoping reviews.¹⁶ A scoping review is a type of knowledge synthesis approach used to map the concepts underpinning a research area and the main sources and types of evidence available.^{15 17}

Protocol

We used the Preferred Reporting Items for Systematic Reviews and Meta-analysis for Protocols (PRISMA-P)¹⁸ to draft this protocol (see online supplementary appendix A). The draft protocol has been reviewed by the research team members, and revised as required. The final protocol was registered prospectively with the Open Science Framework on 6 September 2016 (https://osf.io/4dy53/).

Framework

The framework for effective engagement in comparative effectiveness research⁹ was used as a starting point for conceptualising this research. This framework was selected because it covers all aspects of stakeholder engagement, including inputs (eg, values, research, experience), methods (eg, questionnaires, facilitated workshops) and outputs (eg, decisions, outcomes, perspectives). Stakeholder engagement is defined as “an iterative process of actively soliciting the knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interests in a particular issue, for the dual purposes of: creating a shared understanding [and] making relevant, transparent and effective decisions”.⁹ As well, this framework includes policymakers as well as other key stakeholders (eg, patients and consumers, clinicians, healthcare providers) and defines policymakers as “Individuals and organizations that create, monitor, and oversee policies or regulations of healthcare-related issues, such as federal, state, and local government agencies”.

Eligibility criteria

The Population, Intervention, Comparator, Outcome and Study design eligibility criteria are described below,¹⁹ and presented in figure 1.

Figure 1.

PICO concept map: the Population, Intervention, Comparator, Outcome and Study design eligibility criteria. KT, knowledge translation.

Population: Eligible studies will include any type of health policymaker, healthcare manager or policy analyst. These individuals will be defined as follows:¹⁴

1. Policymakers: “individuals [non-elected] at some level of government; they may have some responsibility for making recommendations to others and may include regional and federal representation.”

2. Healthcare managers: “individuals in a managerial or supervisory role in a healthcare organization with management or supervisory mandates, including public health officials.”

3. Policy analysts: “individuals [non-elected] at some level of government; they may have some responsibility for analyzing data and may include regional and federal representation.”

Non-elected policymakers and analysts will be the focus because elected officials are not the typical target audience for knowledge synthesis who would be using the results to make decisions. We will not include other types of knowledge users, such as patients or clinicians. This is because the scoping review focuses on health system stakeholders, and the Alliance will use the findings to support evidence-informed policymaking. In order to be included, the policymaker, healthcare manager and policy analyst will need to be making decisions related to health systems, health policy, social determinants of health and health-related disciplines, following the definition of health put forth by the WHO.²⁰

Intervention: Studies that describe an active engagement strategy for knowledge users (ie, policymaker, healthcare managers and policy analysts) in the knowledge synthesis process will be included. Engagement will be defined using the framework established by Keown et al¹⁰ and will include planning of a systematic review, participation in all the steps of the review, involvement with developing key messages from the results and relevant dissemination strategies, and engagement towards integration of review findings into policy. If possible, engagement will also be categorised according to the steps in the policymaking cycle using the Howlett and Ramesh framework,²¹ including agenda setting, policy formulation, decision-making, policy implementation and policy evaluation.²²

In order to be included, knowledge users will have to be an active part of the knowledge synthesis process (eg, embedded approach or integrated KT) or of strategies fostering the use of knowledge syntheses (eg, knowledge brokers, policy dialogues). Passive KT activities focusing on dissemination (eg, presenting results at conferences, assessing potential user need to seek out information, publications, optimising presentation formats of knowledge syntheses), or more interactive knowledge exchange activities that focus on understanding how policymakers use knowledge syntheses (such as interviewing policymakers about the usefulness and relevance of knowledge synthesis in their decision-making)^{12 23} will not be eligible. We will define knowledge synthesis as any type of evidence synthesis including (but not limited to): systematic reviews (Cochrane handbook), scoping reviews,^{16 17} rapid reviews²⁴ and other types of emerging review methods,^25–29 such as realist reviews, meta-ethnography, meta-narrative and meta-synthesis. Since health technology assessment and comparative effectiveness research includes knowledge synthesis activities, these types of research are eligible for inclusion, unless it is explicit that a knowledge synthesis was not part of the engagement activities with policymakers, healthcare managers and policy analysts. The development of clinical practice guidelines will not be eligible unless it is explicit that policymakers, healthcare managers or policy analysts were engaged in the knowledge synthesis that was used to develop the guideline.

Comparators: Any comparator is relevant for inclusion (eg, studies comparing one form of engagement to another or vs no engagement). In addition, studies without a comparator are eligible for inclusion (eg, studies examining barriers or facilitators of engaging knowledge users in knowledge synthesis are eligible).

Outcomes: Studies that identify strategies, barriers, facilitators or outcomes and contextual factors for engaging health policymakers, healthcare managers, or policy analysts in the conduct and use of knowledge synthesis will be included. Papers refining or developing theory, conceptual models and frameworks will be excluded, unless they also describe barriers, facilitators and strategies or outcomes (eg, attitudes, beliefs, knowledge, benefits, unintended consequences) related to engaging policymakers, healthcare providers or policy analysts.

Study designs: All study designs using qualitative or quantitative methodologies will be eligible for inclusion, except for case reports. Specifically, we will include experimental (randomised controlled trials, quasi-randomised controlled trials, non-randomised clinical trials), quasi-experimental (interrupted time series, controlled before–after studies), observational (cohort, case–control, cross-sectional, case series) and qualitative studies (interviews, open-ended questionnaires, focus groups).

Time periods: To increase feasibility, we will restrict inclusion to the past 20 years.

Setting: Studies in any setting or country will be considered for inclusion.

Other: Owing to time limitations (ie, 5 months in total), only papers written in English will be considered for inclusion. We will search for the full-text articles for conference abstracts and study protocols that fulfil our eligibility criteria.

Information sources and search strategy

An experienced librarian will search the following electronic databases from inception onwards: MEDLINE, EMBASE, ERIC, PsycINFO, Joanna Briggs, The Cochrane Library, EBM Reviews, The Campbell Library and Social Work abstracts. Another experienced librarian peer reviewed the main (MEDLINE) search using the Peer Review of Electronic Search Strategies (PRESS) Statement.³⁰ The final literature search for MEDLINE can be found in online supplementary appendix B. Search results will be imported and stored in an Endnote (EndNote [computer program]. Version X7: Thomson Reuters) library by an information specialist for reference management.

The electronic search will be supplemented by searching study for unpublished and difficult to locate studies through sources that may include GreyNet International (http://www.greylit.org), registries, conference abstracts and dissertations. Additional articles will be sought by searching references from relevant review articles and contacting experts in the field (eg, Alliance network). No language restrictions will be imposed; however, we will limit to papers published from 1996 onwards.

Study selection process

The results of the search will be uploaded to our online Synthesi.SR software (Synthesi.SR. Knowledge Translation Program. St. Michael's Hospital, 2012. http://www.breakthroughkt.ca/login.php) which will be used for study selection (ie, title/abstract screening and full-text article screening) using the eligibility criteria outlined above.

To ensure high inter-rater reliability, a training exercise will be conducted on the screening tool prior to starting the screening process and full screening of citations and potentially relevant full-text articles will proceed only with >75% agreement across the team. If poor agreement is observed, the eligibility criteria will be clarified and the training will be repeated. Subsequently, all levels of screening and data abstraction will be conducted by two reviewers, independently. Conflicts will be resolved by a third reviewer.

Data items and data abstraction process

It is anticipated that data will be extracted on study characteristics (eg, study design, country of corresponding author, funding source, journal discipline), population characteristics (eg, number of participants, type of knowledge user and level within the health system, country and economy status (using the World Bank categories)),³¹ health system settings, intervention characteristics (eg, format and content features of the review, objective of engagement activity, timing and level of engagement activity, resourced required, theory or framework³² used to inform the intervention) and outcomes (eg, barriers, facilitators, strategies, as well as outcomes from intervention papers—such as attitudes, beliefs, knowledge, benefits, unintended consequences). We will also categorise research usage according to instrumental (concrete use), conceptual (use to change attitudes) or symbolic (persuasive use).³³

Data abstraction will be conducted using a standardised Excel form. To conduct data abstraction, a training exercise will occur among the team using a random sample of five included studies. Full data abstraction will proceed only with >75% agreement across the team. If poor agreement is found, the data abstraction form will be clarified and the training exercise will be repeated. Subsequently, two reviewers will abstract data, independently, with conflicts resolved by a third reviewer. If a large number of studies is identified (>25), we will conduct data abstraction with one reviewer and one verifier.

Risk of bias assessment

Appraisal for risk of bias will not be performed because this is a scoping review. This is consistent with the Joanna Briggs Institute Manual¹⁶ and scoping reviews on health-related topics.¹⁷

Synthesis of results

Results will be summarised quantitatively (using frequencies) and qualitatively (drawing on the descriptive analytical method) to identify barriers, facilitators, strategies and outcomes of engaging policymakers, healthcare managers and policy analysts in knowledge synthesis. If possible, we will stratify results by type of economy country (eg, low-income, middle-income or high-income economy country). Meta-analysis will not be performed.

The main focus of the studies will be ‘charted’¹⁵ by one reviewer and verified by a second reviewer using taxonomies of barriers and facilitators to engaging policymakers in health research that were previously established.^{4 34} New categories will be added, as required, as the preliminary framework was restricted to integrated KT activities for any type of research, and the focus of this scoping review is on any type of engagement specifically for knowledge synthesis. This taxonomy will be further revised to include contextual factors that influence policymakers, such as institutions, interests and external events, as well as whether the factor influences attitudes, norms or behaviours. We will use our previously established methods¹⁴ to conduct thematic analysis for this step. Specifically, using the pre-existing categories, team members will code all studies using NVivo software, (NVivo qualitative data analysis Software [computer program]. Version 10. Australia: QSR International Pty Ltd; 2012) which will be verified by a second reviewer, independently. The coding guide will be updated iteratively and discussed regularly by the team and the preliminary coding scheme will be aggregated by themes; themes will be ordered by the number of studies in which they were identified. We will also chart whether a theory was used to inform engagement and the type of theory used, type of knowledge synthesis examined (eg, rapid reviews for policymaking),³⁵ funding status and source of funding, health domain, review step, policymaking stage, as well as the impact, usefulness, benefits and unintended consequences of engagement.

Discussion

Implications

Findings from this scoping review will be used by the Alliance Review Centers of HPSR in LMICs that are conducting knowledge synthesis to inform health policy decision-making. As well, the Alliance and its partners within and outside WHO will share the results to other knowledge synthesis centres globally, including centres supported by the Cochrane Collaboration and the Global Evidence Synthesis Initiative (GESI).³⁶ This will allow institutions with expertise in knowledge synthesis the opportunity to optimally engage policymakers, healthcare managers and policy analysts in their research.

The anticipated output will be an understanding of all barriers, facilitators, strategies and outcomes of engaging policymakers, healthcare managers and policy analysts in knowledge synthesis. For example, an anticipated output might be that we would identify ‘x’ articles that assessed barriers such as ‘y’ (eg, lack of a trusting relationship between the centre and government officials) and of these, ‘z’ focus on attitudes that affect public health officials in LMICs.

Our review will provide a better understanding of engagement determinants and mechanisms within various health system settings. As previous evidence suggests the potential benefits of engagement strategies,¹⁴ we expect our review will provide useful insight to support the conduct and uptake of policy-relevant knowledge synthesis worldwide.

Dissemination

This project is part of a research programme on rapid reviews commissioned by the Alliance. The other two components comprise writing a Guide to Rapid Reviews of HPSR, as well as providing mentorship and capacity building for Alliance Review Centers in Ethiopia, South Africa and Lebanon that are conducting knowledge synthesis for health policy decision-making agencies.

The idea for this scoping review emerged from the Alliance, to inform the portfolios of work on HPSR and use of evidence in health policymaking. The review findings will be shared with the Alliance-funded centres that are engaging with policymakers, healthcare managers and policy analysts in different LMIC settings. However, our results are relevant to any knowledge synthesis centre conducting reviews for these knowledge users. As such, our results will also be disseminated widely through conference presentations and workshops, train-the-trainer events with the Alliance Review Centers, peer-reviewed publications and a one-page policy brief that will be posted on the authors' websites.