How can we address the surge of low-quality systematic reviews and their impact on high journal rejection rates?

Abstract

Journals have experienced a significant rise in submissions of systematic reviews and other types of reviews that often fall short of acceptable quality standards. These shortcomings typically stem from insufficient rigor in their methodology, reporting, or critical appraisal. As a result, these submissions are frequently rejected raising concerns about the standards authors are following when preparing such work. This growing trend of low-quality reviews not only places a burden on editorial teams but also poses a risk to the scientific community by potentially disseminating flawed or unreliable conclusions.

Ensuring that articles maintain high standards is crucial for preserving the integrity of the scientific literature and facilitating evidence-based decision-making. In an effort to address this problem, this viewpoint editorial aims to offer concepts and recommendations on available tools for future authors to improve the quality of their reviews, as well as to guide readers and potential journal reviewers on how to critically interpret these articles.

The problem

The number of systematic reviews (SRs) and meta-analyses has increased dramatically in recent years, with growth rates for instance of 67% and 132%, respectively between 2010 and 2014, compared to just a 27% rise for all items indexed in PubMed for the same period ¹. Some researchers attribute this rise to the perception that these studies offer a quicker and more cost-effective route to publication, often used to boost academic rankings. Following this line of reasoning, we could then assert that there is also an increase in submissions, but not all of them are high quality; only about 30 to 50% meet rigorous standards ². Many low-quality submissions suffer from methodological flaws, incomplete reporting, or insufficient critical evaluation. Even when using self-assessment checklists (e.g. CASP), original articles frequently contain issues such as bias, confounding or other limitations that were not identified by the authors of the review.

SRs play a vital role in synthesizing evidence; however, the influx of substandard submissions has led to alarmingly high rejection rates in academic journals, with some disciplines and high-impact journals rejecting as many as 90% to 95% of submissions ³. Beyond rejection, these low-quality reviews waste valuable resources, strain the peer review system, and risk propagating flawed conclusions into the scientific community. This trend imposes significant burdens on authors, reviewers, and editorial boards, delaying the publication of high-quality research. Factors contributing to this phenomenon include the pressure to publish, insufficient training in SR methodology, and an over-reliance or misuse of automated tools.

In an effort to address this problem, this viewpoint editorial aims to offer some concepts and recommendations on available frameworks and tools for future authors to improve the quality of their reviews. Additionally, it aims to equip readers and potential reviewers with tools to critically appraise SRs, ultimately fostering better practices and enhancing the quality of published research.

Differences between types of reviews within the evidence ecosystem

As a starting point, and to provide context, Table 1 below highlights some of the main organizations and initiatives involved in evidence synthesis, which play a critical role in shaping and promoting standardized approaches to reviews.

Table 1. Organizations and initiatives in Evidence Synthesis.

Organization Name	Website
Campbell Collaboration	https://www.campbellcollaboration.org
Critical Appraisal Skills Programme (CASP) UK	https://casp-uk.net
Cochrane	https://www.cochrane.org
Evidence Synthesis International (ESI)	https://evidencesynthesis.org
EPPI-Centre (Evidence for Policy and Practice Information and Co-ordinating Centre)	https://eppi.ioe.ac.uk
Joanna Briggs Institute (JBI)	https://jbi.global
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)	https://www.prisma-statement.org

First it is essential to present and clarify two key concepts regarding evidence-based research: a) evidence synthesis and b) the various types of reviews, as these are often misunderstood and misapplied by many authors in submissions. This challenge is not new; in 2009, Grant and Booth ⁴⁾ identified 14 distinct types of reviews, and by 2016, Tricco et al. ⁽⁵, had documented 25 different methods for knowledge synthesis.

Therefore, evidence synthesis, as a form of secondary research, consolidates information from various sources to guide health decision-making. It aims to rigorously and systematically review existing research to support evidence-based practice. Over time, different types of reviews have been developed within evidence synthesis to address the substantial challenge of summarizing the available literature comprehensively. According to the JBI Manual for Evidence Synthesis, various types of reviews coexist, each one with different roles in a broader “Evidence ecosystem” ⁶:

A SR uses rigorous methods to minimize bias in the searching, filtering, collection, synthesis, presentation, interpretation, and reporting of evidence ⁷. SRs are widely regarded as the most reliable source of evidence for decision-making because they use predefined scientific methods to synthesize all available information on a specific research question. Their importance in guiding decisions and developing guidelines is well recognized by consumers, researchers, patients, and clinicians alike. Its key characteristics include clearly defined objectives, pre-established eligibility criteria, explicit and reproducible methodology, a systematic literature search, an assessment of the validity of included studies, and a systematic synthesis of findings. In contrast, non-systematic (or narrative) reviews are typically invited contributions from experts that provide an overview of a field but are more prone to bias due to the lack of clear methodology and the potential selection of evidence to support a pre-existing view. In the past, narrative reviews were commonly authored by expert opinion leaders using non-systematic methods, often relying solely on research familiar to them rather than considering the full scope of available studies. Today, we recognize that evidence comes in many forms and that policy and practice are shaped by diverse perspectives and sources of evidence, encompassing feasibility, appropriateness, meaningfulness, and effectiveness. The Joanna Briggs Institute (JBI) reflects this understanding by including "textual evidence" (narrative, expert opinion, and policy) in their critical appraisal tools, adopting a more inclusive approach to evidence evaluation. While JBI doesn't explicitly state it, these types of evidence are generally understood to carry a higher risk of bias due to subjectivity, lack of experimental control, difficulty in replication or potential publication bias. By incorporating these categories, JBI acknowledges the valuable insights such evidence can provide, particularly in areas where quantitative evidence is limited. However, their separate categorization and specific checklist questions imply a recognition that additional scrutiny is necessary when evaluating this type of evidence. Within narrative reviews, we find a more structured subtype: the “integrative review”, which is widely used in nursing and health research, also referred to as a "comprehensive review" or a "critical overview" ⁸. This approach accommodates a wide range of study designs, including experimental and non-experimental studies, as well as theoretical and empirical literature. However, its methodology is not always clearly defined, which can introduce variability in its application.

Scoping reviews (ScR), also called “mapping reviews” or “scoping studies” (revisión panorámica, revisión de alcance o mapeo in Spanish) are often used to determine if embarking on one SR is worthwhile ^9-11. Researchers often begin with a ScR to gain a clearer understanding of the available evidence on a specific topic. ScR are exploratory in nature, typically addressing broad questions. Often use a Participant, Context, Concept (PCC) format to frame their research questions. In 2018, the Preferred Reporting Items for Systematic Reviews (PRISMA) Statement, a set of guidelines aimed at enhancing the transparency and quality of reporting in SR and meta-analyses, was extended to Scoping Reviews - the PRISMA-ScR ¹².

Umbrella reviews (revisión de revisiones o meta revisiones in Spanish) systematically compile and evaluate multiple SRs and meta-analyses (SRMAs) on a specific research topic, making them particularly valuable when numerous SRMAs exist, as they provide a broad overview of the evidence ^13,14. Their primary objective is to summarize findings across different reviews, highlighting consistencies and discrepancies. Additionally, umbrella reviews aim to identify gaps in the literature and offer insights to inform clinical practice and policy decisions.

Second, it is crucial to clarify the distinction between SRs and meta-analyses. Meta-analysis is a statistical technique that generates an overall estimate, along with its confidence interval, and can be applied to both interventional (experimental) and observational studies. In interventional studies (such as randomized controlled trials), meta-analysis helps determine the overall effectiveness of a treatment or intervention compared to a control or alternative. In observational studies (such as cohort, case-control, or cross-sectional studies), meta-analysis can be used to assess associations between exposures and outcomes, estimate risk factors, or evaluate trends in populations ¹⁵. Some authors use the terms, SR and meta-analysis, interchangeably, but this is incorrect. Meta-analyses are typically conducted as part of a SR, but not all SRs include a meta-analysis.

Finally, and importantly, in the post-pandemic era, two types of SR have gained significant prominence for decision-making: rapid reviews and living evidence syntheses ¹⁶. "Living systematic review" (LSR) are systematic reviews that are continually updated, incorporating relevant new evidence as it becomes available. LSRs may be particularly important in fields where research evidence is emerging rapidly, current evidence is uncertain, and new research may change policy or practice decisions ¹⁷. Rapid reviews (RRs) are a form of knowledge synthesis in which SR methods are simplified and processes are expedited to produce the review in a shorter time frame ¹⁸. Unlike ScR, which aim for breadth and a general overview, RRs prioritize speed and synthesis for decision-making. Both approaches, LSR and RRs, have become increasingly important due to the need for timely and up-to-date evidence in rapidly evolving situations.

Thus, addressing concerns about the quality of published work and seeking to understand and improve its underlying causes, we could argue that the misinterpretation of the differences between review types is an unavoidable factor that often contributes to substandard submissions. For instance, authors may want to conduct a SR but they perform just a narrative review, failing to apply the rigorous methods and structured approach required for a SR. Similarly, confusion between SR and meta-analyses can lead to inadequate analyses, where authors may claim to have conducted a SR but omit key elements such as data synthesis or quality assessment. Other common flaws in these types of submissions are for instance the poorly defined research questions, which make it difficult to establish a clear and coherent focus for the study. Additionally, incomplete literature searches are often observed, limiting the theoretical context and depth of analysis. Lastly, conclusions are often weak because they lack sufficient support from the data or fail to adequately address the research objectives.

The landscape of evidence synthesis is evolving, with new and ongoing challenges driving debate, research, and innovation. One major issue is the lengthy time required to complete a full review, often exceeding two years, which poses difficulties for both authors and decision-makers who need quicker access to evidence. Efforts to increase efficiency, including rapid reviews and the use of technological solutions like automation and crowdsourcing, are gaining momentum. The Cochrane Collaboration, for instance, has embraced rapid reviews, especially during the COVID-19 pandemic, when critical science and public health policy issues where often framed as false dichotomies ¹⁹ evidence-based, flexible policies, and continues to explore methods for keeping reviews up-to-date through "living" SR.

Additionally, Cochrane is expanding its methods to include non-randomized studies, big data, and complex methodologies to address emerging areas like public health and environmental exposure. The organization remains committed to advancing methods and improving the utility of reviews for decision-makers ¹⁵.

Recommendations, frameworks and tools

Returning to the question of this viewpoint, to address the surge of low-quality SRs and its contribution to high journal rejection rates, we must move toward practical considerations in order to tackle this issue. In this regard, authors (and also critical readers and potential reviewers) can leverage many of the best available tools for improving review quality ². They should consider adhering to established guidelines, such as PRISMA ²⁰, or Methodological Expectations of Cochrane Intervention Reviews (MECIR) standards, while ensuring rigorous methodology and transparent reporting. Providing comprehensive training for reviewers and promoting the use of evidence-based tools like AMSTAR-2 ²¹, a 16-item checklist focusing on the methodological quality of SR of healthcare interventions, or ROBIS ²² can significantly enhance the quality of both submissions and evaluations. A key consideration is the role of AI as we mentioned previously. Automated tools are valuable for tasks such as literature screening, data extraction, and bias assessment, but their misuse can lead to errors and hinder thorough evaluation. For example, algorithms may misidentify relevant studies due to variations in terminology or incomplete indexing. Automated RoB assessments may overlook methodological flaws, and data extraction tools can introduce inaccuracies if not manually verified. Furthermore, relying too heavily on AI-driven writing assistants for drafting reviews can result in generic or misleading interpretations, as AI lacks the critical thinking and contextual understanding essential for high-quality reviews. Additionally, encouraging collaboration with experts and emphasizing the importance of addressing relevant, well-defined research questions can further elevate the standards of SR ²³. It is also worth noting the concern regarding the acceptance of low-quality SR in many journals, particularly in smaller journals in Latin America or those relying exclusively on the APC model.

Table 2 summarizes some of the most relevant domains in evidence synthesis, gives some details, and provides tools and frameworks with references to many of them.

Table 2. Recommendations for authors conducting evidence synthesis.

Domain	Recommendations	Frameworks / Tools
1. Question	Define a well-formulated and clear research question.	PICO, PCC, PICO(T/S), PEO, PICOC, SPIDER, PEO 24. FINER 25, SMART 26, TOPICS+M criteria 27. Right Review tool 23
2. Search strategy	Develop a robust and comprehensive search strategy.	CitationChaser 28, AntConc, PubReMiner, MeSH on Demand, YALE MeSH Analyzer, Carrot2, VOSviewer 29
3. Eligibility criteria	Establish clear and stringent inclusion and exclusion criteria.	Cochrane Handbook 15, JBI Manual 6
4. Protocol	Register your protocol in advance 30.	PROSPERO, OSF, Figshare, Zenodo, protocols.io, INPLASY, Cochrane Library, PRISMA-P 31,
5. Critical appraisal tools	Utilize standardized tools to assess the quality and the risk of bias in the included studies.	RoB 2 32, ROBINS-I 33, ROBIS 22, AMSTAR-2 21, CASP tools , GRADE 34, JBI critical appraisal tools, QUADAS-2 35, MMAT 36, EPHPP 37,38
6. Screening and data extraction	Ensure transparency in data management using reference manager software.	EndNote, Zotero, Mendeley, Rayyan (free), Covidence (free for Cochrane Reviews), DistillerSR, RobotReviewer, CADIMA (free), Systematic Review Accelerator (SRA), TERA
7. Synthesis and analysis	Choose the appropriate synthesis method according to the type of data: quantitative data may warrant a meta-analysis or narrative synthesis, while qualitative data might be better suited to a narrative synthesis.	RevMan (Cochrane) 39, CMA.V4 40, MetaEasy (Excel add-in), R (Metafor and Meta Packages), Narrative Synthesis Guidelines 41, Dataparty, NVivo/Atlas.ti, Eppi-Reviewer, MetaInsight 42
8. Reporting	Adhere to appropriate reporting guidelines based on the type of evidence synthesis product (e.g. PRISMA for reporting SR and meta-analyses, or PRISMA-ScR for ScR). Create evidence profiles and summaries of findings tables, commonly used in conjunction with GRADE.	PRISMA 20, PRISMA-ScR 12, QUOROM Statement 43, MOOSE guidelines 44 GRADE 45, GRADEpro 46, CINeMA, MAGICapp SRDR+ (Systematic Review Data Repository Plus)

On the other hand, it is important to emphasize that journals should enforce stricter adherence to the methodological and reporting standards we previously mentioned. Additionally, peer reviewers should receive training in the critical appraisal of SRs and other evidence synthesis products. In this regard, academic institutions should offer comprehensive training in SRs methodology, and funding bodies should require adherence to established guidelines for SRs as a way to foster improvements in study quality. Encouraging collaboration between methodologists and subject matter experts would also be highly beneficial.

Conclusions

It is crucial to emphasize the importance of maintaining high standards of quality for SRs. The growing concern over the prevalence of low-quality SRs has had a significant impact on high journal rejection rates and the overall quality of the scientific literature when accepted by journals. To address this, it is essential to uphold rigorous methodological and reporting standards for SRs to ensure their reliability and validity. Ongoing education and awareness for both authors and readers are key to improving the quality of future submissions. Authors must commit to adhering to best practices to produce high-quality reviews, while journals have a responsibility to continue promoting quality through clear guidelines and a thorough critical review process. By doing so, we can improve the credibility and usefulness of SRs in advancing scientific knowledge.