Translating evidence, including evidence from systematic reviews, into policy and practice is a major challenge for evidence producers.1–3 Stakeholder involvement in how reviews are prioritized, produced, and disseminated may improve their relevance and translation into policy and practice.2,4 Stakeholder involvement is continuing to grow, with a 2018 scoping review identifying nearly 300 examples in the field of systematic reviews.5 However, many reported too few details for replication, thus limiting evidence to inform recommendations for future practice.5
In Cochrane Reviews, as in other types of health research, stakeholder involvement has taken different forms and involved a range of methods.6,7 Levels of involvement within and across reviews can also vary, as categorized by the ACTIVE (Authors and Consumers Together Impacting on eVidencE) framework.7 According to ACTIVE, involvement may fall on a continuum (Table 1).
TABLE 1—
Case Study Examples Using the ACTIVE Continuum of Stakeholder Involvement
| Level of Involvement | Description | Case Study Examples |
| Leading | Lead responsibility for conducting and completing the review; initiating the review | Deciding on the scope of the project (case study 1) |
| Coauthoring the protocol (case study 2) | ||
| Controlling | Making decisions for 1 or more aspects of the review process | Deciding on the study design (case study 2) |
| Deciding which interventions would be included (case study 3) | ||
| Deciding whether individual studies are included in a review (case study 2) | ||
| Influencing | Providing information or input that directly influences the review process but without direct control over decisions | Participating in a workshop to prioritize review topics (case study 1) |
| Providing views about aspects of the PICO question (case study 3) | ||
| Providing feedback on a template for an evidence summary for policymakers (case study 4) | ||
| Contributing | Providing views or information that may indirectly influence the review process (e.g., participants in a survey or focus group) | Contributing views to an online survey (case study 1) |
| Receiving | Receiving the results of a review or other information about a review | Receiving review information in a tailored form (case study 4) |
Note. ACTIVE = Authors and Consumers Together Impacting on eVidencE; PICO is a mnemonic for Population/Patient/Problem, Intervention, Comparison, Outcome.
Source. Pollock et al.7
We share the stakeholder involvement experiences of three Cochrane editorial groups (Consumers and Communication, Effective Practice and Organization of Care, and Public Health) that form part of the Cochrane Public Health and Health Systems Network.8 The network is responsible for wide-ranging evidence syntheses published in the Cochrane Library.
We present the lessons of our collective experiences from four major research activities in systematic review prioritization, production, and dissemination. In describing key elements of stakeholder involvement in each project, we have used the ACTIVE framework as a transparent way of categorizing how stakeholders were engaged (Table 1).
The lessons from each case study help to draw out potential facilitators and barriers to stakeholder involvement in future research activities. Given the state of knowledge and evidence in this field, future measurement of the effect of stakeholder involvement on the uptake of systematic reviews into health policy and practice should incorporate an examination of barriers and facilitators to involvement.
We include the following groups as stakeholders: policymakers, health decision-makers, health professionals, consumers (meaning patients, caregivers, their representatives, and the public), guideline developers, and research funders.
Stakeholder involvement is defined as “any role or contribution of stakeholders toward the development of a review protocol, completion of any of the stages of a systematic review, or dissemination of the findings of a review.”7(p246)
CASE STUDY 1
Case study 1 describes stakeholder involvement in setting systematic review priorities.
Description
In this project, the goal of Cochrane Consumers and Communication was to set priorities for future Cochrane Reviews in the areas of health communication and participation.9 The aim of involving stakeholders was to ensure that future end users of Cochrane Reviews had the evidence they needed for decision-making. Reflecting this aim, and to ensure best practice in stakeholder involvement, we worked in partnership with stakeholders throughout the project.
This project consisted of four stages. First, we recruited an 11-member steering group to oversee the project via our extensive consumer, health service, and policymaker networks. The steering group provided a leading level of input because they were responsible for making key decisions related to the methods and execution of the review. For example, they determined how broad the range of topics should be that were included in the priority-setting exercise.
In the second stage, we identified the research priorities of consumers and other stakeholders through an international online survey. We invited participation from anyone interested in health care communication and participation.9 The survey yielded 151 responses and comprised a combination of consumers’ (30%), health professionals’ (50%), and others’ (20%) views. The stakeholders who participated in the survey provided a contributing level of input because they shared their views or opinions relevant to the topic but were not involved in how their input would shape the final priorities.
In the third stage, we held a face-to-face workshop for stakeholders to further prioritize and convert the topics identified in the survey into systematic review questions.10 The 28 workshop participants engaged in group discussions, a voting round, and then small group work to explore the top 12 research priorities in depth. They provided an influencing level of input because their participation directly shaped the final research priorities.
Finally, the researchers mapped prioritized questions against existing Cochrane Consumers and Communication Cochrane Reviews and editorial criteria (i.e., feasibility) to shortlist five priority questions for future reviews of key importance to a wide range of end users.10
Key Lessons
As this project involved consumers and other stakeholder groups, we learned that skills and techniques to proactively lessen power differences between stakeholder groups were required. In the workshop, for example, we ensured that consumers would constitute at least half of the participants. We also hired an independent workshop facilitator who was skilled in actively seeking consumers’ opinions. Previous research supports the use of these strategies to reduce power differences.11
Financial support was also vital to this project. Through the funding body and small grants, we were able to employ two part-time researchers, pay workshop costs (i.e., food, participant payment, facilitator payment), and produce a professional report in partnership with our stakeholders.
CASE STUDY 2
Case study 2 describes stakeholder involvement in a qualitative evidence synthesis.
Description
Commencing in 2018, the Cochrane Consumers and Communication group has been working with a stakeholder advisory panel to undertake a qualitative evidence synthesis (QES) on one of the priority questions identified in case study 1.12 The scope of the synthesis is to identify the evidence to support best practices for partnerships between health providers and consumers at the governance level to improve person-centered health services.
Before the commencement of the project, we undertook Cochrane’s online training modules about involving stakeholders in reviews.13 These modules provided us with ideas about how to involve stakeholders at different stages of the review, including topic selection, protocol development, developing the search, and data collection and analysis.
We recruited an 18-member stakeholder panel to provide guidance throughout the review. We recruited consumer, health provider, and policymaker stakeholders through the researchers’ professional networks and through the mailing list of an Australian national health organization. Panel members reflected key end users of the review. We have used various modes of involvement throughout the project so far, including teleconferences and face-to-face meetings.
The stakeholder panel has predominantly been involved at a controlling level in this project by making decisions about aspects of the review process. For example, the panel voted on the type of review that should be conducted: an effectiveness review or a QES. The majority voted for a QES, and hence this was commenced.12
When the protocol was under development, we sought the stakeholder panel’s advice via in-depth discussion during teleconferences to ensure that the protocol draft was addressing the right questions. Stakeholders critiqued the protocol, for example, requesting further justification for the exclusion of certain types of articles. All stakeholder panel members were invited to be coauthors of the protocol. Many accepted and thereby took lead responsibility for the publication.
During the selection of studies for the review, stakeholders were engaged in controlling, rather than leading, roles. This was because of the technical nature of the task and limited time available for both researchers and stakeholders to select studies. During a face-to-face meeting, we grouped the stakeholders into teams with the researchers and asked them to apply the selection criteria to a range of qualitative studies. The stakeholders thus had control over decisions about whether a selection of studies was included in the review, but researchers retained overall responsibility for the data collection phase.
Key Lessons
Consistent with previous research, we found that flexibility was vital when asking stakeholders to make key decisions. When stakeholders chose between a QES and an effectiveness review, the researchers needed to be willing to change direction and flexible enough to be skilled in both review types. The funders also needed to be flexible in accepting either a QES or an effectiveness review as the final product.14
Undertaking training on working with stakeholders facilitates stakeholder involvement.15 For us, it was particularly useful in the initial setup of the project when outlining the different levels of stakeholder involvement and different methods for decision-making (e.g., voting). Once the project started though, we ultimately determined methods of involvement through negotiation with the stakeholder panel, rather than leaving them fixed. Listening to stakeholder needs is key to engagement.16
We also learned that we needed more time than originally anticipated to involve stakeholders in different tasks. For example, after publication of the protocol we realized that several of the stakeholders were still unsure of the process for conducting a Cochrane Review. Involving stakeholders in study selection also required the development of a specialized training package before our face-to-face meeting. Next time, we would factor in more time and resources to help stakeholders engage in different review tasks.
CASE STUDY 3
Case study 3 describes stakeholder involvement in Cochrane Reviews for a guideline process.
Description
In case study 3, researchers at Cochrane Effective Practice and Organization of Care worked with stakeholders and the World Health Organization (WHO) to prioritize questions for Cochrane Reviews for a guideline process and to scope those reviews. The guideline aimed to develop recommendations on digital interventions for health systems strengthening.17
We invited representatives of key stakeholder groups and people with relevant specialist knowledge and programmatic experience of digital health programs in a range of settings to join the guideline development group. This group partnered with the WHO and researchers to scope the guideline and design the research. For example, the WHO and the guideline development group had control of the final decision about the interventions to be included in the guideline. The guideline development group also influenced the design of the Cochrane Reviews that would inform the recommendations. They provided their views about the design of the PICO questions (PICO is a mnemonic for Population/Patient/Problem, Intervention, Comparison, Outcome), which stakeholder groups’ views needed to be captured in these syntheses, and which comparison groups and outcome measures were the most relevant.
The WHO also carried out surveys across relevant global and regional networks to further prioritize the questions and outcome measures and received responses from more than 300 people working with digital health and health systems. We incorporated these views into the final protocols. For instance, for the guideline question on telemedicine for communication between health care providers, the stakeholders agreed that the QES should focus on health care providers’ and managers’ perspectives on acceptability and feasibility. However, for guideline questions, such as targeted messaging to service users via mobile phone, the stakeholders found that service user acceptability and feasibility were equally important to consider.
Key Lessons
The WHO routinely incorporates stakeholder feedback into their guideline processes and makes stakeholder involvement feasible through a number of practical measures.18 For instance, expenses related to stakeholder feedback, such as the reimbursement of stakeholders’ travel expenses, were incorporated into the guideline budget. In addition, the different stages of stakeholder feedback were incorporated into the guideline’s timeline, helping to ensure that the researchers had sufficient time to properly consider this feedback and incorporate it into the development of the reviews. These types of practical measures are key to facilitating stakeholder involvement.4,6,14
CASE STUDY 4
Case study 4 describes supporting stakeholders to use review findings.
Description
The final case study demonstrates an approach to working with stakeholders to broaden the access and use of Cochrane Review evidence in public health policy and practice.
Members of Cochrane Public Health Europe (CPHE) developed a German language summary format for Cochrane Reviews in collaboration with public health decision-makers. The format was based on scientific literature and user tested with public health decision-makers in Austria, Germany, and Switzerland. The decision-makers played an influencing role in this project by providing feedback on the draft format, which the researchers incorporated into the final summary format. Users welcomed the final summary format as a useful and credible source of information.19
Additionally, CPHE translated the plain language summaries, abstracts, and press releases of the most relevant, recently published CPH reviews into German. German was chosen because CPHE currently consists of institutions in Germany, Austria, and Switzerland, and summaries in English were less accessible to the targeted stakeholders. CPHE developed a tailored message about the review and compiled stakeholder lists for each country. Subsequently, CPHE members sent personalized e-mails with links to the translated review to stakeholders via an automated mailing platform. As the reviews being disseminated had already been published, stakeholders could not play an active role in shaping the review process, thus they were in receiving roles. However, providing them with tailored dissemination information facilitated their decisions about the applicability and implementation of the results in their own contexts. This project is currently being evaluated through monitoring stakeholders’ response rates as well as a stakeholder survey about the usage and relevance of the information provided. Revisions and adaptations to the project will be implemented based on the results of the evaluation.
Key Lessons
Although summarizing, translating, and tailoring dissemination messages for individual reviews is time consuming, CPHE found that these forms of targeted dissemination contribute to reaching new audiences beyond the research community (e.g., federal and national ministries beyond the health sector, nongovernment organizations, and private companies). This is consistent with previous research demonstrating that stakeholders’ access to Cochrane Review findings is facilitated by providing a brief summary of findings, without jargon, translated into their preferred language.19
IMPACT OF STAKEHOLDER INVOLVEMENT
Although various studies have proposed instrumental impacts of stakeholder involvement at the beginning of and during the research process, few empirical studies have tested these impacts so far.20 The integration of systematic review results into policy and practice is a complex, nonlinear process.2,3 Consequently, it may be more beneficial and enhance the relevance of research to focus on understanding the process of stakeholder involvement and impact.21 Our case studies suggest that future assessments of stakeholder involvement should include an analysis of the attitudinal and practical factors that facilitate or impede stakeholder involvement in reviews, such as availability of time, researchers’ willingness to be flexible, the commitment of the funding body to stakeholder involvement, and researchers’ skills and knowledge.
Notably, there is evidence of instrumental impact for involving stakeholders at the final, knowledge dissemination, stage (as shown in case study 4). A Cochrane Review of interventions to improve the use of systematic reviews in decision making found that targeted, tailored messages based on systematic reviews delivered to health care professionals may improve evidence-based practice.22 This suggests that even when stakeholders are not involved in decision making during the process of the review itself, they can benefit from being involved after its completion.
CONCLUSIONS
Stakeholder involvement in systematic reviews has the potential to enhance relevance and impact on policy and practice. Cochrane case studies across different facets of public health illustrate that different methods, involving varying levels of stakeholder input, can be used to generate and support the use of relevant evidence. Our case studies have shown that factors such as time, researcher flexibility, and researcher skills and training can facilitate or hinder stakeholder involvement. Evidence of the effects of stakeholder involvement are currently limited, and future studies should include an evaluation of the facilitators and barriers to the process of stakeholder involvement.
ACKNOWLEDGMENTS
Case study 1 was funded by the Victorian Government Department of Health and Human Services and Safer Care Victoria, the Australian National Health and Medical Research Council, and the La Trobe University Research Focus Area funding scheme. Case study 2 was funded through the La Trobe University Research Platform and Research Focus Areas funding schemes. Case study 3 was funded in part by the World Health Organization. In addition, the Norwegian Satellite of the Effective Practice and Organisation of Care Group receives funding from the Norwegian Agency for Development Co‐operation via the Norwegian Institute of Public Health. S. Lewin also receives funding from the South African Medical Research Council. D. O’Connor is supported by a National Health and Medical Research Council Translating Research into Practice fellowship (grant APP1168749).
We wish to acknowledge the stakeholders involved in each of the case studies. We also thank Lisa Bero, Professor of Medicine and Public Health, University of Colorado, and senior editor, Cochrane Public Health and Health Systems Network, for her editorial oversight of this publication and commentary series.
CONFLICTS OF INTEREST
The authors have no conflicts of interest to declare.
HUMAN PARTICIPANT PROTECTION
We obtained ethics approval for research described in the case studies when required by the home institution where the research occurred.
REFERENCES
- 1.Fox DM. Evidence and health policy: using and regulating systematic reviews. Am J Public Health. 2017;107(1):88–92. doi: 10.2105/AJPH.2016.303485. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Bero L. Systematic review: a method at risk for being corrupted. Am J Public Health. 2017;107(1):93–96. doi: 10.2105/AJPH.2016.303518. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Greenhalgh T, Malterud K. Systematic reviews for policymaking: muddling through. Am J Public Health. 2017;107(1):97–99. doi: 10.2105/AJPH.2016.303557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Concannon TW, Grant S, Welch V et al. Practical guidance for involving stakeholders in health research. J Gen Intern Med. 2019;34(3):458–463. doi: 10.1007/s11606-018-4738-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Pollock A, Campbell P, Struthers C et al. Stakeholder involvement in systematic reviews: a scoping review. Syst Rev. 2018;7(1):208. doi: 10.1186/s13643-018-0852-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Synnot AJ, Hill S. Public involvement in health research. In: Saks M, Allsop J, editors. Researching Health: Qualitative, Quantitative and Mixed Methods. 3rd ed. London, England: SAGE; 2019. pp. 427–452. [Google Scholar]
- 7.Pollock A, Campbell P, Struthers C et al. Development of the ACTIVE framework to describe stakeholder involvement in systematic reviews. J Health Serv Res Policy. 2019;24(4):245–255. doi: 10.1177/1355819619841647. [DOI] [PubMed] [Google Scholar]
- 8.Bero LA. Improving the quality of systematic reviews in public health: introduction to the series. Am J Public Health. 2020;110(11):1601–1602. doi: 10.2105/AJPH.2020.305914. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Synnot A, Bragge P, Lowe D et al. Research priorities in health communication and participation: international survey of consumers and other stakeholders. BMJ Open. 2018;8(5):e019481. doi: 10.1136/bmjopen-2017-019481. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Synnot AJ, Tong A, Bragge P et al. Selecting, refining and identifying priority Cochrane Reviews in health communication and participation in partnership with consumers and other stakeholders. Health Res Policy Syst. 2019;17(1):45. doi: 10.1186/s12961-019-0444-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Boivin A, L’Espérance A, Gauvin FP et al. Patient and public engagement in research and health system decision making: a systematic review of evaluation tools. Health Expect. 2018;21(6):1075–1084. doi: 10.1111/hex.12804. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Merner B, Hill S, Colombo C Consumers and health providers working in partnership for the promotion of person‐centred health services: a co‐produced qualitative evidence synthesis (protocol) 2019. Available at: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013274/full#. Accessed April 18, 2021.
- 13.Pollock A, Morley R, Watts C. Involving people: a learning resource for systematic review authors. Available at: https://training.cochrane.org/involving-people. Accessed April 18, 2021.
- 14.Boaz A, Hanney S, Borst R, O’Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Syst. 2018;16(1):60. doi: 10.1186/s12961-018-0337-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Brunton G, Thomas J, O’Mara-Eves A, Jamal F, Oliver S, Kavanagh J. Narratives of community engagement: a systematic review-derived conceptual framework for public health interventions. BMC Public Health. 2017;17(1):944. doi: 10.1186/s12889-017-4958-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Walker E, Shaw E, Nunns M, Moore D, Thompson Coon J. Health Expect. 2020. No evidence synthesis about me without me: involving young people in the conduct and dissemination of a complex evidence synthesis. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.World Health Organization. WHO guideline: recommendations on digital interventions for health system strengthening. 2019. Available at: https://www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en. Accessed April 18, 2021. [PubMed]
- 18.World Health Organization. Handbook for Guideline Development . 2014. Available at: https://www.who.int/publications/guidelines/handbook_2nd_ed.pdf?ua=1. Accessed April 18, 2021.
- 19.Busert L, Mütsch M, Kien C et al. Facilitating evidence uptake: development and user testing of a systematic review summary format to inform public health decision-making in German-speaking countries. Health Res Policy Syst. 2018;16(1):59. doi: 10.1186/s12961-018-0307-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–145. doi: 10.2217/cer.14.79. [DOI] [PubMed] [Google Scholar]
- 21.Synnot A, Tong A, Ryan R, Hill S. Evaluation of the Cochrane Consumers and Communication Group’s systematic review priority setting project. Health Res Policy Syst. 2020;18(1):98. doi: 10.1186/s12961-020-00604-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Murthy L, Shepperd S, Clarke MJ et al. Interventions to improve the use of systematic reviews in decision-making by health system managers, policy makers and clinicians. Cochrane Database Syst Rev. 2012;9:CD009401. doi: 10.1002/14651858.CD009401.pub2. [DOI] [PubMed] [Google Scholar]