Characteristics of stakeholder involvement in systematic and rapid reviews: a methodological review in the area of health services research

Abstract

Objective

Engaging stakeholders in reviews is considered to generate more relevant evidence and to facilitate dissemination and use. As little is known about stakeholder involvement, we assessed the characteristics of their engagement in systematic and rapid reviews and the methodological quality of included studies. Stakeholders were people with a particular interest in the research topic.

Design

Methodological review.

Search strategy

Four databases (Medline, Embase, Cochrane database of systematic reviews, databases of the University of York, Center for Reviews and Dissemination (CRD)) were searched based on an a priori protocol. Four types of reviews (Cochrane and non-Cochrane systematic reviews, rapid and CRD rapid reviews) were retrieved between January 2011 and October 2015, pooled by potential review type and duplicates excluded. Articles were randomly ordered and screened for inclusion and exclusion criteria until 30 reviews per group were reached. Their methodological quality was assessed using AMSTAR and stakeholder characteristics were collected.

Results

In total, 57 822 deduplicated citations were detected with potential non-Cochrane systematic reviews being the biggest group (56 986 records). We found stakeholder involvement in 13% (4/30) of Cochrane, 20% (6/30) of non-Cochrane, 43% (13/30) of rapid and 93% (28/30) of CRD reviews. Overall, 33% (17/51) of the responding contact authors mentioned positive effects of stakeholder involvement. A conflict of interest statement remained unmentioned in 40% (12/30) of non-Cochrane and in 27% (8/30) of rapid reviews, but not in Cochrane or CRD reviews. At most, half of non-Cochrane and rapid reviews mentioned an a priori study protocol in contrast to all Cochrane reviews.

Conclusion

Stakeholder engagement was not general practice, except for CRD reviews, although it was more common in rapid reviews. Reporting factors, such as including an a priori study protocol and a conflict of interest statement should be considered in conjunction with involving stakeholders.

Strengths and limitations of this study.

• A systematic approach to assess stakeholder engagement in systematic and rapid reviews was used to fill the gap and to provide an overview of the current situation.

• The different review types were able to inform on stakeholder involvement and pointed to potential improvements.

• With the scope of this study being restricted to health services research we intended to focus on a best-case approach where the context is considered most important and therefore, stakeholder involvement might be most regularly, but the overall findings might not be applicable beyond this domain.

• The majority of studies were from high-income countries limiting the generalisability beyond this setting.

• The perception and the reporting of stakeholder involvement might be influenced by the lack of a uniform definition of stakeholders and depend on the criteria used.

Introduction

Evidence synthesis remains a rapidly growing field. There are different methodological approaches and formats depending on the research question and the intended use of the review, such as scoping, systematic or rapid reviews as well as realist syntheses, policy briefs and health technology assessments (HTAs). We will focus on full systematic reviews as a well-established review type in healthcare and on rapid reviews as an emerging one. A traditional full systematic review is a review that ‘attempts to collate all empirical evidence and that fits prespecified eligibility criteria in order to answer a specific, usually narrow research question or intervention’.¹ Within systematic reviews, Cochrane full systematic reviews were established as ‘gold standard’ in knowledge synthesis. A couple of studies compared Cochrane and non-Cochrane systematic reviews and found that the reporting of Cochrane systematic reviews was the most complete one,² that they were less prone to bias due to greater transparency in reporting as well as due to the quality criteria used, such as the risk of bias assessment.³ Furthermore, Hopewell et al described that the inclusion of grey literature routinely performed in Cochrane systematic reviews limited publication bias and provided more conservative treatment effects compared with non-Cochrane systematic reviews without including grey literature.⁴

Rapid reviews are characterised by a less complex research question and aim to synthesise evidence within a shorter time period, with time frames ranging from 1 week to 9 months.^{5 6} They might, therefore, be prone to be of lower validity as a consequence of accelerating and streamlining the review process.⁵ However, in response to an increasing demand from stakeholders, rapid reviews are being performed more frequently than before.⁶ Only few formal definitions of different rapid reviews exist and few studies have examined their methodology.⁷ To cover the variety of rapid reviews we included rapid reviews listed in medical databases as well as other rapid reviews from the databases of the University of York, Center for Reviews and Dissemination (www.crd.york.ac.uk/CRDWeb), such as the Dare reviews (Database of Abstracts of Reviews of Effects) or HTAs (CRD rapid reviews). Rapid reviews target specific audiences including government policymakers, healthcare institutions, health professionals and patient associations and research questions are often tailored to these stakeholders.^{8 9}

Stakeholder as a very broad term may include anyone affected by an issue and/or anyone who can provide input on the topic.^{10 11} Hence, not only decision makers, health professionals and their organisations are targeted, but also citizen or patients, other researchers and the media. Involving stakeholders in health services research might be possible at all stages of systematic or rapid reviews and this is often referred to as co-production. Thus, stakeholder engagement aims participatory research and might direct the research question, define the scope and context of the review as well as contribute to the literature search, the evidence synthesis and interpretation and might facilitate dissemination and use. Therefore, it enhances the relevance of findings for policy and practice and contributes to the sustainability of health systems.^12–14 We especially focused on stakeholder involvement among health services research. This embraced 'the multidisciplinary field of scientific studies regarding social, financing and personal factors, organisational structures and processes, health technologies, and how these factors affect access to, the quality and cost of healthcare, and ultimately, our health and well-being’.¹⁵ Here, we considered the context of conducted studies and dissemination issues as most important, and we expected health services research to be the research field where stakeholders were involved most regularly.

Study aim

To date, little is known about the extent of stakeholder engagement in systematic and rapid reviews and there have been few efforts to directly report the specific effects regarding their involvement. We consider this information to be relevant for integrated knowledge translation, the dissemination and acceptance of systematic and rapid reviews in policy and practice. We focused on systematic and rapid reviews as they represent different types of evidence synthesis and are well-established in research and practice. In addition, reviews needed to belong to the area of health services research as an established field for systematic and rapid reviews as well as a best-case sample with a considerable extent of stakeholder involvement. We aimed to assess the extent and characteristics of stakeholder engagement in published systematic and rapid reviews and to specifically determine reporting characteristics.

Methods

Study design

Based on the prespecified protocol this methodological review was performed to assess characteristics and reporting of stakeholder engagement in random samples of systematic and rapid reviews in the field of health services research (online supplementary file S1). A total of four types of reviews, two groups of each, systematic and rapid reviews, were assessed as they were considered to exhibit potentially differences in stakeholder involvement. This included Cochrane systematic reviews, non-Cochrane systematic reviews, rapid reviews and rapid reviews of the databases of the University of York, CRD rapid reviews. For the reporting of this study the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist was used (online supplementary file S2).

Search strategy and screening

For systematic reviews, we searched the Cochrane database of systematic reviews for articles including the term ‘systematic review’ in their title or abstract (Cochrane systematic reviews) and EBSCO Medline and Embase with excluding the term ‘Cochrane’ in title or abstract (non-Cochrane systematic reviews). Rapid reviews were searched in EBSCO Medline and Embase and in the databases of the University of York, CRD rapid reviews (www.crd.york.ac.uk/CRDWeb). Sample search strategies are depicted in online supplementary files S3a and S3b. We included studies published between January 2011 and December 2015 without language restriction. Our search was completed on 22 October 2015 and therefore, the search results for 2015 did not cover the whole year. All results were pooled by review type and duplicates were excluded based on authors, journal and publication year. For each of the four lists of potential study types, each reference was given a unique random number using the sample() function available in R.¹⁶ Each list was then sorted by the random number, and the articles were screened in order for inclusion and exclusion criteria by two reviewers until a total of 30 studies were reached per group (JF and MM tested their consistency in assessing the inclusion and exclusion criteria for about 10 different studies and MM checked the assignment for about half of the study sample). Full text screening was performed to ensure the decision. One review author (JF) performed data extraction, supervised and spot-checked by a second researcher (MM). Disagreements were resolved by consensus.

Sample size and selection of review groups

As rapid reviews were often targeted to specific audiences,^{8 9} they were expected to involve stakeholders to a greater extent. We estimated a proportion of stakeholder involvement of 0.70 for rapid reviews and of 0.25 for systematic reviews, which resulted in a minimal expected difference of 0.45. For a two group-comparison, rapid and systematic reviews, with n=25 per group there will be an estimated power of 0.88 and an alpha of 0.023. To account for a potential subgroup analysis, we included a sample of n=30 reviews per group, with four review groups (Cochrane and non-Cochrane systematic reviews, rapid and CRD rapid reviews). This resulted in an overall sample of 120 studies.

Inclusion and exclusion criteria

We included reviews targeting health services research as defined by Lohr¹⁵: “Health services research is the multidisciplinary field of scientific investigation that studies how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and ultimately, our health and well-being. Its research domains are individuals, families, organizations, institutions, communities, and populations”. Therefore, we included HTAs (eg, rapid reviews as part of HTAs) targeting effectiveness as well as meta-analyses and systematic reviews for effectiveness and utility, reviews of pharmaceutical trials under everyday conditions, reviews of basic research in care-related fields, reviews of quality research or of methodological developments in the field of health services research, reviews about the development and application of new technologies (ehealth) and the implementation of knowledge into clinical practice. We excluded reviews that focused exclusively on economic or cost analyses, performed a narrative review, an overview of reviews or a protocol of a systematic review in order not to mix so different methodological approaches. The following study types were also excluded: reviews of efficacy studies without assessing the quality of life, reviews including clinical efficacy trials phase I–III, epidemiological reviews to assess determinants or risk factors, non-human studies, establishment of databases or registries, reviews with unclear design or description of the intervention.

Data extraction

Stakeholders were defined as people with a particular interest in the research topic (but were not members of the primary research team).^{10 17} We evaluated any kind of stakeholder involvement and recorded different groups of stakeholders: institutional healthcare providers, representatives of hospitals or community services, patients/consumers, participants of government agencies and healthcare policymakers at Federal, State and local levels, associations of health professionals and researchers (if not members of the primary study team). If available, we identified their fields and stages of involvement as well as their contribution to the study. This included all review steps, such as formulating the research question(s), determining study characteristics, contributing to the writing of the protocol, participating within the review process with searching, screening, data extraction, synthesising, interpreting the study results and/or establishing recommendations. In addition, we extracted the following data: institution where the review was performed, contact details of the corresponding author, the type of intervention, study setting, characteristics of the population (sample size, age range, sex), funding, declared conflict of interest, year of publication. The methodological quality of included articles was assessed using the AMSTAR tool, a measurement tool for the assessment of the methodological quality of systematic reviews.¹⁸ Where available, our rating was compared with an already existing one from the ‘health evidence’ or ‘health systems evidence’ platforms.^{19 20} Both quality ratings were then categorised as strong (≥8), moderate (4–7) or weak (<4). In the absence of a quality assessment tool for rapid reviews, we used AMSTAR also for rapid reviews, being aware that this might result in lower rating scores due to abbreviated procedures. In addition, the use of the PRISMA checklist was recorded when mentioned in the articles.²¹ AMSTAR ratings are presented as online supplementary file S4, study characteristics as online supplementary file S5.

To ascertain stakeholder engagement as well as to assess the extent or stages of their involvement, preformulated questions were sent by email to the contact authors of the included articles. We asked them, if there had been any stakeholder involvement (yes or no) and if so, to specify the number of stakeholders, the stages of involvement and their self-assessed estimation of the effect of stakeholder engagement on the review outcome. In case of missing responses, one reminder was sent.

Patient and Public involvement

None involved.

Results

Our search identified 57 822 citations remaining after exclusion of duplicates. Although, not all of these records will fulfil the inclusion criteria, such as for example, being designed as a systematic review, we assigned them as potential review group. For the screening step, the counts of the articles that needed to be screened to reach the final set of studies are depicted in figure 1.^22–141 The excluded studies are presented in online supplementary table S1.

Figure 1.

Flow diagram of study selection. CRD, databases of the University of York, Centre for Reviews and Dissemination; RR, rapid review; SR, systematic review.

Table 1 presents the characteristics of the included studies. Cochrane systematic reviews generally focused on the prevention and treatment of specific conditions, whereas 30% of non-Cochrane systematic reviews were categorised with a community-based topic, with a wide range of participants and possible outcomes and a broader range of study settings. Of note, 23% of each, rapid reviews and CRD rapid reviews, were categorised as health system intervention, because they focused on methodological questions or health quality research.

Table 1.

Characteristics of included reviews by review type

	Cochrane SR (%)	Non-Cochrane SR (%)	RR (%)	CRD RR (%)	Total (%)
Random sample (% of total per group)	30 (6)	30 (0.1)	30 (14)	30 (32)	120 (0.2)
Geographic location of the corresponding author (% of random sample)
Africa	2 (7)	0	0	0	2 (2)
Asia	5 (17)	3 (10)	1 (3)	0	9 (8)
Australia	3 (10)	5 (17)	2 (7)	0	10 (8)
Europe	11 (37)	13 (43)	13 (43)	2 (7)	39 (33)
North America	7 (23)	8 (27)	14 (47)	28 (93)	57 (48)
South America	2 (7)	1 (3)	0	0	3 (3)
Thematic focus
Prevention	11 (37)	5 (17)	4 (13)	7 (23)	27 (23)
Treatment	18 (60)	14 (47)	15 (50)	16 (53)	63 (53)
Health system	1 (3)	2 (7)	7 (23)	7 (23)	17 (14)
Community-based*	0	9 (30)	4 (13)	0	13 (11)
Funding
Governmental, institutional or WHO	30 (100)	30 (100)	30 (100)	30 (100)	120 (100)
Additional funding by company	0	0	1 (3)	0	1 (1)
Conflict of interest (COI)
Declared none	21 (70)	15 (50)	19 (63)	29 (97)	84 (70)
COI declared	9 (30)	3 (10)	3 (10)	0	15 (13)
Unmentioned	0	12 (40)	8 (27)	1 (3)	21 (18)
Study protocol
Yes, mentioned in article	30 (100)	5 (17)	4 (13)	1 (3)	40 (33)
Yes, mentioned in correspondence	0	8 (27)	9 (30)	0	17 (14)
No, confirmed by correspondence	0	2 (7)	4 (13)	0	6 (5)
Unmentioned	0	15 (50)	13 (43)	29 (97)	57 (48)
AMSTAR rating
Median (range)	11 (7, 11)	7 (3, 11)	4 (2, 10)	8 (5, 11)	8 (2, 11)

AMSTAR: measurement tool for the ‘assessment of the methodological quality of systematic reviews’, not applicable questions were not counted. Higher AMSTAR scores indicate higher quality.

*With exclusion of treatment or preventive interventions.

CRD, databases of the University of York, Center for Reviews and Dissemination; RR, rapid review;SR, systematic review.

We detected noticeable differences when comparing the median AMSTAR scores: Cochrane systematic reviews and CRD rapid reviews showed higher median scores than non-Cochrane systematic reviews and rapid reviews. Where available, we also collected existing AMSTAR ratings from the ‘health evidence’ or ‘health systems evidence’ platforms. For 68% (13/19 study ratings) we found a congruent classification as strong, moderate or weak, respectively (online supplementary file S4). In non-Cochrane reviews information about methodological specifications of included studies were often lacking. As an example, all 30 Cochrane systematic reviews (100%) mentioned a pre-existing review protocol, whereas a high amount of non-Cochrane systematic reviews (59%) and of rapid reviews (43%) did not clarify whether there had been a protocol or not. Similarly, all 30 Cochrane systematic reviews and all but one CRD rapid reviews (97%) included a conflict of interest statement, whereas 40% of non-Cochrane systematic reviews and 27% of rapid reviews lacked such a paragraph. All reviews were funded by national or international governmental or institutional sources. Only one rapid review reported the acceptance of additional funding from a pharmaceutical company.

Characteristics of the study population by type of included review are presented in online supplementary table S2.

Any stakeholder involvement was mentioned in 13% of Cochrane systematic reviews, 20% of non-Cochrane systematic reviews, 43% of rapid reviews and 93% of CRD rapid reviews. Except for CRD rapid reviews, where the involvement of stakeholders was routinely reported in the methods or appendix section, about half of the stakeholder involvement was reported in the articles and the other half was confirmed via email by contact authors and remained unmentioned in the reviews. With 67% the email response rate was highest for rapid reviews. When comparing the amount of stakeholder involvement in different review topics, there were notable differences: the proportion of stakeholder involvement in reviews focusing on prevention or treatment of specific conditions was lower than in those targeting health system interventions. Interestingly, the proportion of stakeholder involvement has been quite constant between the years 2012 (5/15, 33%) and 2014 (12/34, 34%) even though the amount of rapid reviews has increased substantially (table 2). There was, however, a remarkable increase in the proportion of stakeholder involvement for the year 2015 (22/32, 69%), but this was mostly due to the fact that a majority of CRD rapid reviews, which presented the highest proportion of stakeholder involvement, was indexed in 2015.

Table 2.

General characteristics of stakeholder involvement (SI) by review type

	Cochrane SR (%) (n=30)	Non-Cochrane SR (%) (n=30)	RR (%) (n=30)	CRD RR (%) (n=30)	Total (%) (N=120)
SI mentioned in article	2 (7)	2 (7)	6 (20)	28 (93)	38 (32)
SI only mentioned in correspondence	2 (7)	4 (13)	7 (23)	0	13 (11)
Total SI	4 (13)	6 (20)	13 (43)	28 (93)	51 (43)
SI per year
2011	0/5 (0)	0/1 (0)	0/3 (0)	0/0 (0)	0/9 (0)
2012	3/7 (43)	1/5 (20)	1/3 (33)	0/0 (0)	5/15 (33)
2013	0/9 (0)	0/6 (0)	3/5 (60)	9/10 (90)	12/30 (40)
2014	1/7 (14)	3/9 (30)	6/15 (40)	2/3 (67)	12/34 (35)
2015	0/2 (0)	2/9 (22)	3/4 (75)	17/17 (100)	22/32 (69)
Number of stakeholders involved per review (% of SI)
1	2 (50)	2 (33)	2 (15)	0	6 (12)
2–4	1 (25)	1 (17)	4 (31)	2 (7)	8 (16)
>4	1 (25)	3 (50)	4 (31)	26 (93)	34 (67)
Unspecified	0	0	3 (23)	0	3 (6)
Types of stakeholders (multiple roles possible) (% of SI)
Patients/consumers	2 (50)	0	2 (15)	5 (18)	9 (18)
Professional organisations	1 (25)	0	0	5 (18)	6 (12)
Caregivers	2 (50)	0	3 (23)	23 (82)	28 (55)
Researchers	1 (25)	3 (50)	2 (15)	0	6 (12)
Policymakers	0	3 (50)	4 (31)	3 (11)	10 (20)
Unspecified	0	0	2 (15)	4 (14)	6 (12)
Funding source (% of SI)
Governmental or institutional funding	4 (100)	6 (100)	13 (100)	28 (100)	51 (100)
Funding source involved as stakeholder	0	1 (17)	2 (15)	0	3 (6)
Thematic focus (% per review type per topic)
Prevention	0/11 (0)	0/5 (0)	1/4 (25)	6/7 (86)	7/27 (26)
Treatment	4/18 (22)	3/14 (21)	6/15 (40)	16/16 (100)	29/63 (46)
Health system	0/1 (0)	1/2 (50)	5/7 (71)	6/7 (86)	12/17 (71)
Community-based	0/0 (0)	2/9 (22)	1/4 (25)	0/0 (0)	3/13 (23)

CRD, databases of the University of York, Center for Reviews and Dissemination; RR, rapid review; SR, systematic review.

CRD rapid reviews turned out to have by far the highest proportion of reported stakeholder involvement. The types of stakeholders engaged were listed in the appendix of 26/30 articles, but it was not specified what they had specifically contributed to the review, for example, if their contribution had affected the final results and conclusions of the reviews (table 3). One author mentioned in the review that the reason for involving stakeholders was to understand the clinical perspective.

Table 3.

Engagement characteristics of stakeholder involvement (SI) by review type

	Cochrane SR (%) (n=30)	Non-Cochrane SR (%) (n=30)	RR (%) (n=30)	CRD RR (%) (n=30)	Total (%) (N=120)
Stages of SI (multiple stages possible) (% of SI)
Research question	1 (25)	1 (17)	4 (31)	0	6 (12)
Study characteristics	0	0	6 (46)	0	6 (12)
Protocol writing	0	1 (17)	2 (15)	0	3 (6)
Review process	3 (75)	1 (17)	5 (38)	0	9 (18)
Interpretation of findings	0	1 (17)	1 (8)	0	2 (4)
Result synthesis	0	0	2 (15)	0	2 (4)
Multiple or all stages	1 (25)	2 (33)	1 (8)	28 (100)	32 (63)
Unspecified	0	2 (33)	1 (8)	0	3 (6)
Reasons mentioned for SI (% of SI)
Stakeholder commissioned/requested review	0	1 (17)	4 (31)	0	5 (10)
To get consumer feedback	1 (25)	0	0	0	1 (2)
To refine research question	1 (25)	0	0	0	1 (2)
To make the review more relevant for target audience	2 (50)	0	8 (62)	0	10 (20)
To understand clinical perspective	0	1 (17)	0	1 (4)	2 (4)
Unspecified	0	4 (67)	1 (7)	27 (96)	32 (63)
Effect of SI mentioned by authors (% of SI)
Substantial or positive effect	2 (50)	4 (67)	11 (85)	0	17 (33)
Negative effect	0	0	0	0	0
No effect mentioned	2 (50)	2 (33)	2 (15)	28 (100)	34 (67)
Explicit encouragement for involving stakeholders in further studies (% of reviews)	0	1 (3)	1 (3)	1 (3)	3 (2)

CRD, databases of the University of York, Center for Reviews and Dissemination; RR, rapid review; SR, systematic review.

We detected stakeholder involvement in 43% (13/30) of rapid reviews. In contrast to CRD rapid reviews, there was usually a small number of stakeholders engaged. They were involved at different phases, such as determining study characteristics, formulating the research question, within the review process in general, and less commonly in protocol writing, result synthesis and the interpretation of findings. One article involved stakeholder throughout all stages, in one case the phase of involvement was not specified. In total, 85% of the authors confirmed, that the stakeholders had significantly contributed to the review. Of these, 62% mentioned that involving stakeholders led to an increased impact of the review and enabled to focus on the needs of target groups, which made the review more relevant to for example, patients or policymakers. Stakeholders had commissioned one third of the rapid reviews or had asked for evidence. Only two of 13 authors (15%) did not mention any substantial effect. In two cases one of the funding sources was involved as stakeholder.

We identified stakeholder involvement in 20% (6/30) of non-Cochrane systematic reviews. In four cases the stakeholder involvement was confirmed via email, two articles mentioned the involvement of stakeholders. Only two types of stakeholders were reported: policymakers and researchers. If mentioned, all phases of involvement occurred about equally. A total 67% (4/6) study authors reported significant benefit from the stakeholders’ contributions. One author mentioned that stakeholder supported the researchers to understand different perspectives of the problem, and 33% (2/6) reported no effect. In one case, the stakeholder was involved as funding source.

Cochrane systematic reviews turned out to be the group with the smallest amount of stakeholder involvement. All stakeholder engagement was confirmed and specified by email. The types of stakeholders engaged were patients, caregivers, professional organisations and researchers. They were mostly involved for providing feedback during the review process, in one case they contributed in formulating the review question. One review involved stakeholders at all stages. A total of 50% (2/4) of the authors reported substantial benefit, the other 50% (2/4) reported no significant effect on study results or conclusions. Two authors had involved stakeholders to make the review more relevant for its target audience, one author claimed that stakeholders helped him to refine the research question. One author involved stakeholders to get direct consumer feedback before publishing the review. No stakeholders were involved in funding.

Discussion

The involvement of stakeholders varies by review type. Our findings suggest that rapid reviews tend to involve stakeholders more than twice as frequently than systematic reviews. On average, they also involved a greater number of stakeholders per review. In addition, we detected considerable differences in the phases in which stakeholders were involved. Rapid reviews involved them at very early stages of the review process, such as determining the intentional study characteristics or formulating the research question. Furthermore, it seemed to be much more common for policymakers, who were the most frequent group of stakeholders involved in rapid reviews, or other stakeholder groups, to substantially contribute to rapid reviews than to systematic reviews.

Of note, the majority of rapid and non-Cochrane systematic review authors reported that the involved stakeholders had positive and considerable effects on the study results, making the review more relevant for the targeted audience. However, the minority of rapid and non-Cochrane systematic review authors reporting no effect of stakeholder involvement on the review process, may illustrate that the stakeholders’ contribution to non-Cochrane systematic reviews was seen as an additional and welcomed feature to the review, but not as a substantial part. This shows the importance that within and between review types an information and experiences exchange between researchers could benefit the stakeholder engagement.

A strength of this study is to provide an overview with its rather low amount of stakeholder engagement in the assessed review types despite known benefits regarding the use of evidence in policy and practice. With respect to the different procedures used by review types, the reported experiences of stakeholder engagement, for example, in rapid reviews, could further benefit and facilitate stakeholder engagement in other review types, for example in systematic reviews.

One limitation of this study is the low response rate to our emails by contact authors. Given the fact that nearly half of our emails to study authors remained unanswered or could not be sent, the rate of unmentioned stakeholder involvement might still be higher than our numbers suggest. Of note, there was no considerable difference between systematic reviews and rapid reviews in the percentage of unmentioned stakeholder involvement.

Although our search is not very recent this article highlights the current situation and there is a call for action. Of course, an updated assessment might be needed in the years following.

Although when using a broad definition of the term ‘stakeholder’ including everyone with a particular interest in the research topic (but who are not members of the primary research team), this term was not consistently used by contact authors. In their emails, some authors listed peer reviewers as stakeholders or one author erroneously mentioned members of the research team, who performed literature research and data extraction as stakeholders. In addition, interdisciplinary knowledge exchange is an important part of evidence-based research, but there is a difference whether experts from other fields were included as researchers in the study team or whether they were considered as stakeholders. We did not count any experts or peer reviewers as stakeholders. Furthermore, discussion is needed, whether the funding body of a review might contribute as a stakeholder and how a potentially associated conflict of interest could be avoided.

The reported involvement of stakeholders corresponded with Cottrell et al who mentioned that stakeholders might contribute to different study types and evidence phases.¹⁴² Based on, Keown et al¹⁴³ concluded, that stakeholder involvement led to an increased relevance and depth of review findings, more clarity in defining research questions, broader dissemination of their results and increased awareness of target groups. Although they mentioned that this engagement process required flexibility and might be resource-intensive and time-intensive, they nevertheless concluded that involving stakeholders facilitated implementation and should be indispensable for future research.

Although the overall proportion of stakeholder involvement has not yet increased prominently in the past few years, there have still been considerable efforts in creating standardised procedures for involving stakeholders in evidence synthesis. Keown et al identified five opportunities in the systematic review process, where potential stakeholders could be engaged on a regular base.¹⁴³ The CRD rapid reviews included were mainly performed by Health Quality Ontario (Canada) and routinely held expert panels, including physicians, caregivers and sometimes, consumer representatives and professional organisations. The fact that they included stakeholders on a regular basis might confirm, what Keown et al¹⁴³ had already suggested: the engagement of audience members interested in or affected by the investigated topic definitively resulted in more benefits than limitations. We, therefore, suggest that future researcher involve stakeholders more broadly in the process of evidence synthesis, to increase the relevance and acceptance of the knowledge transfer. One example of an organisation ensuring that stakeholders are involved in research is the James Lind Alliance (http://www.jla.nihr.ac.uk).

In future, the reporting of stakeholder involvement should be improved and its effects better evaluated and communicated. Stakeholder engagement could also be included in reporting checklists of all review types.