Open Medicine endorses PROSPERO

’Tis as impossible that he’s undrown’d / as he that sleeps here swims.

The Tempest, 2.2

Systematic reviews, which synthesize data from individual studies, are considered the highest level of evidence for evaluating the effectiveness of health care interventions. They inform clinical practice guidelines and are used by health policy-makers to guide key decision-making. They can point to gaps in knowledge that need to be filled by new research, and identify areas where, because of a saturation of evidence about effectiveness, new research is not needed and might even be unethical.¹ On 22 February 2011, the Centre for Reviews and Dissemination at the University of York launched PROSPERO, an international prospective register of systematic review protocols. The registry was developed to address the excessive duplication of systematic reviews, improve transparency and minimize reporting biases. Open Medicine endorses systematic review protocol registration at PROSPERO and encourages prospective authors to register their review protocols on health care interventions.

One recent study estimated that 11 new systematic reviews are published every day.² There is currently substantive duplication of specific topics among reviews, each of which might have slightly varied methods and quality.³ Although replication of research is an important aspect of the scientific method, unnecessary duplication may result in the waste of academic resources without bringing us closer to the truth. This duplication is also an ineffective use of taxpayer dollars, particularly when such reviews are funded by public agencies. If adopted widely, PROSPERO will allow review authors to quickly familiarize themselves with ongoing reviews so that they can avoid duplication. Policy-makers will be better able to identify those questions regarding health care effectiveness that require urgent investigation, as distinct from those that are already under way. Funders can also use the register to help identify instances of excessive duplication. PROSPERO may serve to increase the visibility of a particular review and potentially foster collaboration among various groups. Such collaboration could also expand a review group’s ability to assess evidence published in languages other than English.

The plethora of reviews being produced is known to be of variable quality.⁴ In 2009, a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement called for improved transparency in the reporting of systematic reviews through the registration of systematic review protocols: such registration would reduce the probability of selective reporting biases.⁵ Synthesis methods such as systematic review and meta-analysis were originally developed to reduce bias in assessing the treatment effect of a particular health care intervention. These approaches work on the condition that acceptable methods of searching and appraising the evidence are undertaken.

However, a number of studies have found that publication bias and selective reporting biases threaten the validity of some systematic reviews.^6-9 In the context of systematic reviews, selective outcome reporting arises when the specific outcomes that the reviewer chooses to describe do not match those specified in the review protocol, or their level of importance does not match that specified in the protocol (e.g., primary, secondary).⁶ Kirkham and colleagues⁷ compared published Cochrane systematic reviews to their protocols and found that 22% (64/288) had a discrepancy in at least one outcome measure and that 75% of these discrepancies were attributable to changes in the primary outcome. The changes were sometimes made after the results of the individual trials were known.⁷ In fact, other research indicates that selective outcome reporting bias in systematic reviews may be underestimated. Kirkham and colleagues⁸ found that outcome reporting bias was suspected in at least one randomized controlled trial in over one-third (96) of the 283 Cochrane systematic reviews they examined in detail. Among the 81 reviews that included a single meta-analysis of a primary outcome, the sensitivity analysis revealed that the selective outcome reporting bias overestimated the treatment effect estimates by 20% in about 23% of the meta-analyses.⁸ This compounding of bias means that authors of systematic reviews must be scrupulously careful and transparent.

Despite the knowledge that completed reviews frequently deviate from their protocols, to date most systematic reviews do not have publicly accessible protocols. The PROSPERO registry allows editors, peer reviewers and other stakeholders to better assess the methods reported in the systematic review and to examine whether the review deviated from the original protocol and why. One can also review the methods used to minimize these biases, given that the information would be far more transparent.

From the journal editor’s perspective, the data elements for systematic review protocol registration provide enhanced transparency and increase the probability that credible evidence will be published. A competing interest statement would help editors, peer reviewers and readers consider whether conflict of interest might have affected the methods and interpretation of results. A publicly accessible systematic review protocol registry would also improve reproducibility, in that any protocol changes would be documented and a rationale provided. In our current environment of proprietary ownership of health care knowledge, PROSPERO can only be a first, albeit crucial, step in improving the quality of synthesis research. PROSPERO will act as an open repository of data provided by those authors who chose to register, and will be important for all of the reasons described here.

One substantial limitation to PROSPERO’s ability to ensure transparency between the protocol and the final review is the fact that, even if the protocol is accessible, the final review will not be unless it is published, optimally, in an Open Access journal or made freely accessible in the more traditional closed-access journals. Open Medicine will continue to engage the scientific community in committing to the ongoing improvement of transparency in health care research (see Box 1).

Box 1.

Building on PROSPERO

Second, PROSPERO currently supports protocol registration for formal systematic reviews of health care interventions, whereas we know that other knowledge syntheses such as scoping and rapid reviews are also populating the published and grey literature and may equally require scrutiny of their methodological clarity and transparency.^10,11

Finally, PROSPERO cannot act as a watchdog in the systematic review community, and we recognize that well conducted systematic reviews are immensely time-consuming endeavours. Given the importance of systematic reviews in clinical and health policy decision-making, protocol registration should be mandated by funding bodies and, in future, by scholarly journals to enhance the transparent reporting of systematic reviews.

Biography

The authors are on the editorial team of Open Medicine.