Update to the PRISMA guidelines for network meta-analyses and scoping reviews and development of guidelines for rapid reviews: a scoping review protocol

Abstract

Objective:

The objective of this scoping review is to develop a list of items for potential inclusion in the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) reporting guidelines for network meta-analysis (NMA), scoping reviews (ScRs), and rapid reviews (RRs).

Introduction:

The PRISMA extensions for NMA and ScRs were published in 2015 and 2018. However, since then, their methodologies and innovations, including automation, have evolved. There is no reporting guideline for RRs. In 2020, an updated PRISMA statement was published, reflecting advances in the conduct and reporting of systematic reviews. These advances are not yet incorporated into these PRISMA extensions. We will update our previous methods for scoping reviews to inform the update of PRISMA-NMA and PRISMA-ScR as well as the development of the PRISMA-RR reporting guidelines.

Inclusion criteria:

This review will include any study design evaluating the completeness of reporting, offering reporting guidance, or assessing methods relevant to NMA, ScRs, or RRs. Editorial guidelines and tutorials that describe items related to reporting completeness will also be eligible.

Methods:

We will follow the JBI guidance for scoping reviews. For each PRISMA extension, we will i) search multiple electronic databases from inception to present, ii) search for unpublished studies, and iii) scan the reference lists of included studies. There will be no language limitations. Screening and data extraction will be conducted by 2 researchers independently. A third researcher will resolve discrepancies. We will conduct frequency analyses of the identified items. The final list of items will be considered for potential inclusion in the relevant PRISMA reporting guidelines.

Review registration:

NMA protocol (OSF: osf.io/7bkwy); ScR protocol (OSF: osf.io/7bkwy); RR protocol (OSF: osf.io/3jcpe); EQUATOR registration link: https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/

Introduction

Systematic reviews are pivotal underpinnings of evidence-informed practice and policy,1 and hence should be accurately and completely reported. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) reporting guideline provides a minimum set of recommended items to promote clear, transparent, and reproducible descriptions of what was done and what was found in a systematic review.2 Lack of transparency in systematic reviews reduces their quality, validity, and applicability. Inadequate reporting hampers proper quality assessment, potentially leading to erroneous health recommendations and negative impacts on patient care and policy.3,4 Based on our experience, systematic reviews with network meta-analysis (NMA), scoping reviews (ScRs), and rapid reviews (RRs)5 are commonly requested by decision-makers.6,7

The original PRISMA statement, published in 2009, was developed to increase transparency and reproducibility of systematic reviews with meta-analyses of health care interventions.8 Multiple extensions of PRISMA have been developed for other research synthesis methodologies.9

The application of NMAs has rapidly increased during the past decade across a range of health research disciplines.10,11 NMA is now a commonly used statistical method applied when systematic reviews aim to assess the comparative effectiveness of multiple interventions.12-16 The increased use of NMA is perhaps unsurprising because the method (compared with pairwise meta-analysis) addresses more “complex” questions more closely aligned to those asked in clinical decision-making.

An ScR is designed to answer an entirely different question. ScRs systematically identify and map the nature and breadth of evidence on a particular topic, field, concept, or issue, often irrespective of source (ie, primary research, reviews, non-empirical evidence) within or across particular contexts.5,17,18 ScRs use less in-depth analysis and typically include no quality or risk of bias assessment. ScRs often guide future research and can serve as a starting point for systematic reviews.

RRs are expedited systematic reviews whereby authors modify or omit processes to speed up completion of the review, which is crucial for timely decision-making.19-21 The COVID-19 pandemic led to an increase in RRs20,22-28 due to the rapid decision-making that was needed. This highlighted that systematic reviews, which take 1–2 years to complete,29 could not meet the urgent needs of decision-makers and society.

Overall, there has been a steep increase in the number of NMAs, ScRs, and RRs in the past 5 years. A PubMed search in the years 2018–2023 (using the search terms “network meta-analysis [ti],” “scoping review [ti],” and “rapid review [ti],” respectively, with a search date of November 15, 2023) showed 6388 articles related to NMAs, 18,769 related to ScRs, and 1012 related to RRs (likely encompassing a combination of these research syntheses, review protocols, and methods articles on these topics). This is compared to 1954 articles relevant to NMAs, 2321 to ScRs, and 202 to RRs published up until 2018.

Evidence shows that the PRISMA guidelines improve reporting completeness.30 However, important advancements in the relevant methodologies have occurred since the PRISMA extensions for NMA (2015)10 and ScRs (2018).31 Several pressing reasons necessitate a significant update to these PRISMA extensions.

First, we recently found that some elements were incompletely reported when assessing at a granular and comprehensive level (eg, authors did not report both the terms “systematic review” and “NMA” [or related forms of meta-analysis] in the title).30 This research suggests that additional items or modification of the present items may be needed to reflect important aspects of NMAs not covered in the 2015 NMA extension.

Second, since publication of the PRISMA-NMA extension, there have been many methodological advances, including modelling of complex interventions,32,33 modelling dose effects,34 dealing with and assessing missing data,35,36 assessing transitivity37,38 (ie, similarity of the distribution of effect modifiers across treatment comparisons), and assessing certainty of evidence (eg, CINeMA39 [Confidence in Network Meta-Analysis] and GRADE40 [Grading of Recommendations, Assessment, Development and Evaluation]), for which reporting items may be necessary. Similarly, PRISMA-ScR does not include important aspects on reporting methods for extracting data,41 synthesizing evidence,42,43 use of automation tools,43,44 and consideration on how ScRs differ from mapping reviews and evidence gap maps.43,45 JBI updated their methods manual on ScRs in 202046 and these new developments have yet to be incorporated into PRISMA-ScR. Furthermore, since 2020, the JBI ScR methodology group has worked on several advancements in ScRs, such as engaging knowledge users in ScRs,47 providing a formal definition of ScRs,17 writing ScR protocols,42 challenges and solutions for ScRs,43 and data extraction in ScRs,48 among others. Finally, to date there is no reporting guideline for RRs. A PhD project (Stevens) initiated work on an extension of PRISMA for RR, including a review of RR literature, an empirical evaluation of the completeness of reporting of RR literature,49 and a survey of knowledge users50; however, this work is outdated and newer developments in RR methods are now available.51-54 Recently, interim guidance on reporting RRs was published, but this has not been extensive and additional work on developing the PRISMA-RR is necessary.55

Third, in 2020, the PRISMA statement was updated to reflect advances in the conduct and reporting of systematic reviews. PRISMA 2020 uses a new structure of broad items, called elements. Updating these PRISMA extensions to ensure consistency will facilitate their inclusion in a web application that generates a reporting template and checklist customized to the characteristics and methods of the particular review.56 Prior research assessing the impact of PRISMA guidance on the completeness of systematic review reporting has demonstrated considerable improvement in reporting over time.30,57-61 However, it is logical to assume that if guidance does not reflect current methodological standards, health care recommendations and evidence-based decision-making may be adversely affected.

Finally, the original PRISMA extensions (NMA and ScR) do not include patients and the public as research partners, and thus, these valuable perspectives are omitted. Inclusion of these perspectives will allow input and guidance into aspects of reporting that are important (particularly for consumers of systematic reviews with NMA, ScRs, and RRs), as well as into components of the explanation and elaboration documents, and, finally, dissemination through patient and public networks. Our multi-sectoral team involves journal editors, clinicians, policymakers, statisticians, methodologists, and patients along with members of the public.

In this protocol, we outline our planned methods for identifying items to be used in the update of the PRISMA extensions for NMA, ScR, and the development of PRISMA-RR. We will conduct an ScR to identify items for potential inclusion in the PRISMA reporting guidelines for NMA, ScR, and RR.

Review question

What are the items that should be reported in systematic reviews with NMA, ScRs, and RRs in order to be consistent with current best evidence?

Inclusion criteria

Concept

This review will consider studies that explore one of the following in the context of human health (including the psychology, education, and sociology disciplines) or philosophy, using any study design that:

• provides guidance, a tutorial, or a reporting guideline relevant to reporting NMAs, ScRs, or RRs; these may include a checklist, flow diagram, or text to guide authors in NMA, ScR, or RR reporting

• evaluates the completeness of reporting in NMAs, ScRs, or RRs

• evaluates reporting quality (as defined by the authors) in NMAs, ScRs, or RRs

• evaluates sources of bias in NMAs, ScRs, or RRs

• evaluates the risk of bias in NMAs, ScRs, or RRs

• evaluates the methodological quality (as defined by the authors) in NMAs, ScRs, or RRs.

Types of sources

The proposed scoping review will consider quantitative, qualitative, and mixed methods study designs for inclusion. Systematic reviews and text and opinion papers will also be considered for inclusion.

Methods

We will update 3 previous scoping reviews21,49,62 conducted by members of the research team in parallel to identify additional, more recent studies pertaining to evaluations of reporting completeness and other key resources to inform the NMA, ScR, and RR extensions. We followed the PRISMA-P reporting guidelines for this protocol,63 while the JBI guidelines for scoping reviews will be used to guide the methods of this scoping review.42,46 Reporting of the final findings will follow the PRISMA-ScR guidelines.31 The methods for this study were drafted using input from research synthesis experts and knowledge users, including patient and public partners. We registered the reporting guideline updates with the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network website and uploaded the protocols to Open Science Framework on January 5, April 3, and June 17, 2024.64-67

Eligibility criteria

We will include all study designs that offer:

• reporting guidance or evaluate completeness of reporting NMAs, ScRs, and RRs; these may include a checklist, flow diagram, or text to guide authors in RR reporting

• studies assessing methodological quality relevant to NMAs, ScRs, or RRs

• editorial guidelines or tutorials that describe items related to reporting completeness for NMAs, ScRs, and RRs (eg, in the World Association of Medical Editors [WAME], International Committee of Medical Journal Editors [ICMJE], and Committee on Publication Ethics [COPE]).

If duplicate sources are identified, the most recent one will be selected. We will exclude commentaries, manuscript formatting publications, and journal author guidelines.

Search strategy

We will update our previously developed literature searches21,49,62 based on feedback from the team. The literature search strategies will be developed by an experienced librarian (JM), and will be peer-reviewed by another librarian using the Peer Review of Electronic Search Strategies (PRESS) checklist.68 We will search multiple electronic databases, including MEDLINE (1946–present), Embase (1947–present), the Cochrane Library, and ERIC (1965–present).69 The final literature searches for MEDLINE (Ovid) can be found in Appendix I. We will search for unpublished literature based on guidance from the Canadian Agency for Drugs and Technology in Health (CADTH) and Grey Matters70; for example, we will search Google Scholar and organizational websites (EQUATOR, PRISMA, CIHR, Agency for Healthcare Research and Quality [AHRQ], JBI Evidence Synthesis, Cochrane, UK National Institute for Health and Care Excellence,23 Guidelines International Network, and IQWiG [Institute for Quality and Efficiency in Health Care] in Germany). The literature search will be supplemented by reviewing reference lists from included articles using the citationchaser tool (Zenodo, Geneva, Switzerland).71,72 The search strategies will not be limited by publication status, study design, or language.

Study selection

To ensure reliability, all reviewers will pilot-screen 50 citations at level 1 (titles and abstracts) and 25 articles at level 2 (full-text papers) before screening all sources independently. Pilot tests will be repeated up until high percent agreement (> 75%) is achieved across the team at both levels. Two team members will work independently to screen for inclusion. Conflicts will be resolved through discussion with a third investigator. We will use the research synthesis software platform, Synthesi.SR73 (Knowledge Translation Program, Toronto, Canada) for screening. Team members will translate non-English articles using DeepL Translate (DeepL, Cologne, Germany) and Crowdsourcing.74 We will document the search and selection process with the PRISMA 2020 flow diagram.2

Data extraction

Two team members will independently extract data using a standardized form co-created by the reviewers. The categories from which items will be extracted will be as follows: goal of the study (eg, reporting completeness, reporting guidelines, methodological quality elements); study characteristics (eg, first author, year of publication, journal, study type [eg, survey, guideline]); key findings (eg, items relevant to reporting, completeness of reporting results as indicated in the relevant study); methods used, such as agreement activities used to develop reporting guidance (eg, Delphi exercise, face-to-face meetings); and progress of the study (ie, if there have been any updates). Prior to data extraction, we will conduct a calibration exercise on a sample of 10 included articles and modify the form as required. Data extraction will begin when sufficient percent agreement is observed (ie, >75%). Discrepancies will be resolved through discussion or by consulting a third team member, if needed.

Risk of bias appraisal and assessment for reporting bias

Methodological appraisal is generally not applicable to scoping reviews, and will not be conducted.31

Data analysis and presentation

Two researchers will categorize the study results into broader concepts independently using content analysis, as defined in the broader categories of PRISMA 2020 (such as title, abstract, introduction, methods, results, and discussion). The items extracted from each paper will be discussed between the extractors and the leads of each PRISMA extension (PRISMA-NMA: AAV and BH; PRISMA-ScR: ACT; PRISMA-RR: AS), with the possibility of refining the wording of these items for clarity, aiming to generate a set of consensus items from each paper. Then, once items from the included papers have been grouped by concept (eg, synthesis methods), in addition to deleting duplicate items, rewording of items will be considered to capture the content of all similar items. The final list of items deemed unique will be retained for discussion with the team of each PRISMA extension, who will assess for potential relevance to each research synthesis. We will present the number of studies identifying each of the unique items and relevant characteristics in tables and figures.

Next steps

We will update (or develop) the 3 PRISMA extensions to reflect current evidence and ensure engagement of all team members to co-develop a knowledge translation and dissemination strategy. This strategy will increase awareness and enable knowledge users—including authors, journal editors, peer reviewers, patients and the public, clinicians, and health care agencies—to use the updated reporting guidance. We will follow the “guidance for developers of health research reporting guidelines”74,75 for updating the PRISMA extensions to NMA, ScR, and creating the PRISMA extension to RR. Overall, we will adopt the PRISMA 2020 structure of broad elements in developing the PRISMA extensions.2 In particular, we will revise the elements relevant to each research synthesis and will only add, remove, or revise the checklist elements, where necessary.

Patient involvement and dissemination

To ensure patient and public perspectives are fully integrated into this work, 3 patient partners (MS, JT, SL) were involved from project conception and helped to refine the research question. They will also advise on patient/public engagement, interpret findings, and plan dissemination. Patient/public partners will be involved in conducting the research, from protocol development, data collection, interpretation of results, and writing of the article. The results will be disseminated to lay audiences through press releases, social media, Strategy for Patient-Oriented Research (SPOR) Evidence Alliance (EA) website/newsletter, and presentations. We will financially compensate patient/public partners by applying principles outlined by the SPOR-EA policy, which was co-produced with patient partners.76

Acknowledgments

Douglas M. Salzwedel, MLIS (Information Specialist, Therapeutics Initiative, University of British Columbia), for peer reviewing the MEDLINE search strategy; Neal Haddaway and Carole Lunny for providing feedback on our original proposal; and Brahmleen Kaur for formatting the paper.

Authors contributions

All authors declare that they meet the ICMJE conditions for authorship. AAV, ACT, BH, and AS conceived and designed the study. AAV wrote the first draft of the manuscript. All authors edited the manuscript and contributed to its revisions. All authors read and approved the final version of the manuscript.

Funding

This work was funded by a Canadian Institutes of Health Research (CIHR) Project Grant (No 190036). The funders will have no role in the conceptualization, design, data collection, analysis, decision to publish, or preparation of the manuscript.

SES is funded by a Tier 1 Canada Research Chair in Knowledge Translation and Quality of Care, the Mary Trimmer Chair in Geriatric Medicine, and a Foundation Grant (CIHR). ACT holds a Tier 1 Canada Research Chair in Knowledge Synthesis for Knowledge Users. ZM is supported by an NHMRC grant APP1195676. LH is supported by a Tier 1 Canada Research Chair in Knowledge Synthesis and Translation.

Appendix I: Search strategy

MEDLINE(R) ALL (Ovid) <1946 to November 21, 2023>

1. Network meta-analysis/ (5486)

2. ((network* or network-based or “mixed treatment “or mixed-treatment or “multiple treatment comparison” or mtc) adj2 (meta-analys#s or metaanalys#s or meta analys#s or” meta regression” or meta-regression)).tw,kf. (10,051)

3. ((Indirect comparison* or indirect treatment* or bayesian) adj2 (meta-analys#s or metaanalys#s or meta analys#s or “meta regression” or meta-regression)).tw,kf. (2738)

4. (Indirect comparison* or indirect treatment* or mixed-treatment or mixed treatment or bayesian).tw,kf. and (Review Literature as Topic/ or meta-analysis as topic/ or systematic review as topic/ or *Matched-Pair Analysis/ or Technology Assessment, Biomedical/) (878)

5. ((multiparamet* adj2 evidence adj2 synthesis) or (multi-paramet* adj2 evidence adj2 synthesis)).tw,kf. (30)

6. or/1-5 (11,749)

7. report*.ab. /freq = 3 or report*.kf. (459,722)

8. Publishing/ or Open Access Publishing/ or Periodicals as Topic/ or exp checklist/ or Publication Bias/ (89 201)

9. Research Design/ and (mt or st).fs. (50,941)

10. ((journal or periodical or publication or publish* or presentation) adj2 (report* or bias* or requirement* or adherence or compliance or guideline* or recommendation* or standard* or guidance or instruction* or checklist* or check list* or evaluat*)).tw,kf. (77,693)

11. ((clear* or fully or adequately or inadequately or completely or incompletely or poor* or transparent* or method* or quality or element* or requirement* or guideline* or recommendation* or standard* or guidance or instruction* or assess* or apprais* or bias* or characteristic* or criteri* or critiqu* or evaluat* or quality or checklist* or check list* or score$1 or scoring or adherence or compliance or approach* or item* or measure or measures) adj2 (report* or conduct)).tw,kf. (228,485)

12. or/7-11 (811,148)

13. 6 and 12 (1497)—NMA

14. (scoping adj (review or reviews or study or studies or exercise* or project or projects or report or reports or meta-review*)).tw,kf. (24,365)

15. (systematic scoping review or systematic scoping reviews or mapping Review or mapping Reviews or literature map* or evidence map*).tw,kf. (2605)

16. 14 or 15 (25,272)

17. 16 and 12 (5810)—Scoping

18. (rapid adj2 (review or reviews or assessment* or synthes#s)).tw,kf. (12,035)

19. ((expedited or accelerated or rapid) adj systematic review*).tw,kf. (340)

20. (brief review* or rapid evidence review* or Evidence Summar* or quick review* or Rapid Advice Guideline* or Rapid Evidence-Based Literature Review* or Rapid Interim Review* or Rapid Structured Literature Review* or Rapid Synthes#s).tw,kf. (22,645)

21. or/18-20 (32,743)

22. 21 and 12 (1852) RR