Peer review of human studies run amok: a break in the fiduciary relation between scientists and the public

Abstract

Peer review aims to ensure the quality and credibility of research reporting. Conducted by volunteer scientists who receive no guidance or direction, peer review widely varies from fast and facilitative, to unclear and obstructive. Poor quality is an issue because most science research is publicly funded, whereby scientists must make an effort to quickly disseminate their findings back to the public. An unfortunately not uncommon barrier in this process is ineffective peer review. Most scientists agree that when done well, editors and reviewers drive and maintain the high standards of science. At the same time, ineffective peer review can cause great delay with no introduced improvement in final product. These delays and requests interfere with the path of communication between scientist and public, at a great cost to editors, reviewers, authors and those who stand to benefit from application of the results of the studies. We offer a series of concrete recommendations to improve this process.

Peer review is key to assuring the quality and credibility of health research reporting. As with many scientific oversight boards, peer review relies heavily on a team of volunteer scientists to carry this load. This process asks a lot of volunteers who generally undertake these responsibilities without clear instruction or guidelines from the target journal. The result is a peer review process that varies from fast and facilitative (at high-resource journals), to unclear and obstructive (at lower impact, lower resource journals).

This is an issue because, as scientists, it is our job to quickly and effectively share the scientific work that we have been entrusted to conduct. Specifically, when we accept public funds to support our research, this acceptance represents an implicit agreement to conduct the scientific work as planned and to make every effort to efficiently disseminate results on completion. Ineffective peer review waylays these efforts.

Thus, within this perspective, it is our goal as fellow authors, peer reviewers and editors, to provide an overview of the existing strengths and weaknesses of the peer review process. From there, we offer a set of clear and concrete guidelines to encourage appropriate, efficient and attentive peer review.

When the process works

Any scientist can attest to when the peer review process has gone well. In fact, most of us credit our best work to that modified based on editorial and reviewer feedback. In this respect, editors and reviewers can (and do) recommend additional steps that strengthen reporting and clinical interpretations. Examples include encouraging authors to incorporate additional literature, to conduct or evaluate additional analyses (eg, adjustment for multiple testing, reporting absolute effect sizes), attending to alternative clinical or scientific interpretations (eg, placing findings back into the clinical context) and not allowing conclusions to go beyond those allowed by the data. When done well, editors and peer reviewers drive and maintain the high standards of health science reporting.

When it does not

At the same time, the process does not always move smoothly. Many factors contribute to this, including greater competition for publication in prestigious journals, an abundance of review requests for a relatively smaller pool of peer reviewers, mistaken ideas about what the scope of peer review should be and most critically, an absence of explicit guidance about a journal's expectations and requirements in the peer review process. The result is that constructive and timely advice may be forgone in favour of re-creating and dismantling the science. Numerous unnecessary and detailed revisions can result in substantive delays (from months to years from submission to acceptance) without demonstrable improvement in the final product. Worse, these steps can (and do) introduce lower quality science (eg, post hoc analyses) with an inordinate loss of time and effort to the researcher, reviewer and editor and delays for the public. The question is whether these recommendations and the attendant costs, are critical to high-quality reporting and positioning of that scientific work for publication in that journal. Does peer review interfere with the path of communication between scientist and public?

Avenues for improvement

We recommend that editors review manuscripts from the start to judge publication priority (eg, Does the paper advance the science? Does it fit with the mission/priorities of the journal?). Determining whether a paper meets journal priorities can be done prior to the peer review, known as the ‘insta-reject’ (AD Bryan, email communication, October 2009). While this may feel brusque to an author team, it is a scientific saving grace. Letting authors know that the manuscript, as it is currently conceived or developed, is not a good fit for the journal, allows authors to quickly, with no revision, pick a better and more suitable home. Papers sent for peer review should be only those likely to be published if a detailed peer review confirms the editors’ initial impressions. The only papers that should be sent for revision are those that will be published provided the authors are responsive to the revision requests.

We also encourage editors to provide reviewers with clear journal guidelines, requiring that they ‘affirm’ proof of this knowledge prior to review (eg, via checkbox within the journal review site). We challenge journals to require reviewer attention to priority over preference, with revisions only requested in instances where attentive responses will improve the manuscript and position it for eventual publication. To facilitate transparency, we support open review, with reviewer names available to author teams. Finally, we support a mediational role for editors, whereby editors actively synthesise reviewer comments to guide authors about which reviewer comments to respond to, which to ignore and what is ultimately expected from the author team to best position their submission for eventual publication within that journal (eg, whether to respond to each comment, how to manage conflicting queries from different reviewers, whether or not to conduct new analyses). This represents a substantive shift from the convention of providing a templated letter with appended reviewer comments, without any editorial summary or guidance.

Recommendations for improvement

Ultimately, peer review is not perfect. However, it is the best process that we have to ensure the integrity of our science and the public dissemination of our scientific work. We therefore offer these simple recommendations in a format that editors can easily share with peer reviewers (see box 1).

Box 1. Key messages.

Recommendations to Improve Peer Review

For journals

1. Provide peer reviewers with concrete guidelines regarding the journal's mission and priorities.

2. Require that reviewers read and affirm their knowledge of journal requirements.

3. Conduct open review.

4. Recommendations to authors should arrive within 4 weeks of submission. For peer reviewers

5. Review the work before you. Do not request authors to report a different study.

6. Be courteous, clear and concrete.

For Editors

• Encourage the mediational role of editors.

• Promptly alert authors regarding ‘fit’ and reject the paper prior to review or revision requests if the fit is not good or there are fatal flaws.

For the journal:

1. Provide peer reviewers with concrete guidelines about how to review a manuscript, including attending to questions such as (A) originality, (B) importance of the work to general readers and (C) scientific reliability. Examples include: BMJ: http://www.bmj.com/about-bmj/resources-reviewers/guidance-peer-reviewers; Nature: http://www.nature.com/authors/policies/peer_review.html

2. Require that reviewers read the relevant journal's review guidelines before beginning a review for that specific journal. This would be akin to the disclosure and proof of knowledge/ethics form that most journals require for authors in the manuscript submission process.

3. Conduct open review (eg, BMJ), whereby authors will know who has reviewed their work, to open the process of transparency and communication.

4. Peer reviews should be timely. Ideally, recommendations to authors should arrive within 4 weeks of submission.

For the peer review team:

• Review the work before you. Suggesting the authors do a different study is not helpful to the process of improving the current manuscript (unless it is a recommendation for the discussion section). Do not belabour the process by adding steps that reflect reviewer preference rather than scientific integrity.

• Be courteous, clear and concrete. Provide clear and actionable recommendations. If the requested changes cannot be made quickly and efficiently (eg, requiring collection of a new sample), it is not a good fit.

For the editorial team:

• Be prepared to undertake a mediational role, filtering and synthesising reviewer comments so that authors clearly know what steps should be taken to best position their paper for acceptance within that journal.

• Quickly notify authors if the paper does not fit editorial priorities, or if the revisions will not make the paper acceptable for eventual publication in that journal.