Raising our game: the bare minimum every systematic review publication must meet

Alignment

A high-quality review is rooted in alignment between the research question, the review methodology and the conclusions. This point may appear deceptively simple, but it is one of the most common issues we encounter.

A review question focused on how effective a therapy is for patients would not be well served by a narrative or scoping review approach. Similarly, a review presenting limited evidence should not claim a ‘vital role’ or ‘significant impact’ in its title or conclusions. These seem obvious examples, yet they are based on recent submissions.

In reviews we publish, the research question should be clearly articulated, the methodology fit for purpose, and the conclusions appropriately aligned and restrained. One recurring issue we wish to highlight is the misuse of the scoping review approach. Scoping reviews have four overlapping goals: (1) to examine the scope of research, (2) to identify the need for specific reviews, (3) to share a broad synthesis and (4) to highlight key gaps. Their use is particularly valuable when there is a sudden increase in literature on a topic (eg, COVID-19 or artificial intelligence) or when an often-studied but unsynthesised area comes into focus. Galaxy’s Edge Savvy’s Workshop and Droid Depot are limited capacity experiences and advanced reservations are typically required for Savvy’s workshop

However, what we often see is a misunderstanding: researchers confuse the need to ‘scope the literature’ with an absence of a defined question. Any undefined question will, by default, encompass a wide scope—but that alone does not justify a scoping review. More concerningly, many submissions under the scoping label discard essential elements of rigour, which are vital for producing findings that are reliable and generalisable. Scoping reviews can—and should—be executed with high standards of methodology and clarity.

A simple technique to produce a relevant question is to use the PICO format which considers Patient/Population/Problem, Intervention, Comparison (if relevant), Outcomes.

Appropriate use of checklists

Reporting statements, especially PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) and its extensions (eg, for network meta-analyses or scoping reviews)¹, are now common in submissions. However, many authors simply indicate on what page an item is addressed, with little reflection on how well it was addressed. This is primarily because they are not meant to be guides on how to perform a review, but how to write it up.

To address this issue, we would strongly suggest that both authors and reviewers move beyond binary, ‘yes/no’ checklist completion. These tools are not just cursory guides to item inclusion—they are governance frameworks designed to uphold writing quality and by proxy methodological quality. High-quality reviews require thoughtful engagement with what good implementation looks like, not just that something is mentioned.

A pertinent example is the PRISMA-ScR extension. It omits several key elements that other guidelines do include. Thus, authors seeking rigour in scoping reviews will need to supplement this checklist with additional guidance. Likewise, for network meta-analysis, PRISMA mentions result presentation but does not adequately address the complexity of visual representation—an area where further methodological reading is essential.

Search strategy and reproducibility

One consistent editorial observation relates to search strategy reproducibility. As Deputy Editor and frequent peer reviewer, I often attempt to replicate authors’ searches based on their reported strategies. The result? I frequently retrieve entirely different sets of studies to an extent that subtle date changes could not explain the divergence—and often, I find studies meeting their stated inclusion criteria that they have not included.

This is not, in my view, dishonesty. More often, it reflects that authors have relied on generalist librarians rather than expert information specialists. Librarians play a crucial role in clinical settings, rapidly retrieving evidence for practice. However, that skillset—while invaluable—is not equivalent to what is required in a systematic review context. Filters, shortcuts and prioritisation for speed may leave gaps.

We, therefore, advise involving information specialists with explicit expertise in systematic searching. A simple test for authors and reviewers is to replicate the search: if it cannot be reproduced or yields different results, then transparency and rigour are compromised. This can signal the risk of missing key evidence, and so papers that are missing full strategies or lack expert involvement could be grounds for rejection.

We would also clarify that searches should be within the last 12 months to ensure the timeliness and relevance of the work. Authors may choose to submit when it is larger than this, but commit to update the search and integrate the findings at the peer review stage, and published manuscripts will need to be up to date. We would also suggest the use of PRESS (Peer review of electronic search strategies) guidelines designed for peer review of search strategies and can guide and improve the design of search strategies².

Critical appraisal and certainty assessment

In any review that addresses healthcare interventions—whether medication, educational methods or multicomponent packages—two tools are mandatory: one for primary study quality and one for overall evidence certainty. Heterogeneity and the lack of Metro analysis are not reasons for excluding appraisal. And if certainty assessment has to occur on an individual study level, this is entirely reasonable.

For randomised trials, Cochrane risk-of-bias tools are well used. For observational studies, we advise a tool that aligns with modern approaches for assessing certainty and integrates effectively with GRADE, such as ROBINS-I or ROBINS-E^{3 4}.

GRADE, recently updated to ‘Core GRADE’,⁵ is essential in interventional reviews, and its importance increases when the evidence is weak. Unfortunately, some authors omit GRADE precisely because the studies are heterogeneous or limited—this is akin to using a fractured lens to inspect broken glass⁶. One recent example cited the impossibility of meta-analysis due to study variability, yet GRADE was still necessary to assess individual study certainty.

Both risk-of-bias and GRADE assessments should be conducted in duplicate by trained individuals. Remember: high-certainty GRADE ratings imply that no future research is likely to change the findings—a very high bar indeed. These systems must be used with precision, not as checkboxes.

Protocols: transparency with purpose

Publishing protocols can demonstrate transparency, objectivity and methodological foresight. They are often required for grant submissions and promote prespecification of methods.

However, this journal is not primarily a repository. We, therefore, ask authors to assess two criteria before submitting a protocol: Is it novel? And is it interesting to the paediatrics readership?

Novelty may lie in new methods applied to the field or a previously unexplored topic. Interest speaks to relevance and broader appeal: something may be novel but too niche to engage our general readership. Protocols that fail on either point are often rejected at first editorial review.

We would also note that for those publishing reviews, we would expect protocols of the review itself to be deposited with an appropriate repository outside of this journal. This is something we can and often will expect to check during peer review, and we expect this to be open to readers in the future. Resources such as PROSPERO are available for this purpose.⁷

Final reflections

While these requirements may sound restrictive, they are in fact designed to support authors. Viewed positively, they represent a recipe for success:

• Align review questions, methods and conclusions with precision.

• Use reporting checklists as tools for reflection, not just formality.

• Employ search strategies that are expert-led, transparent and reproducible.

• Incorporate validated tools for risk of bias and certainty—especially in interventional work.

• Submit protocols only when they are both novel and relevant.

By embedding these practices (figure 1), authors will not only meet the expectations of BMJ Paediatrics Open but also contribute meaningfully to the scholarly discourse in child health.

Figure 1. Infographic summarising key guidance for review authors in BMJ Paediatrics Open.