Eyes wide open: reader and author responsibility in understanding the limits of peer review

Abstract

‘Medical science can only flourish in a free society and dies under totalitarian repression.’¹

Peer review post-publication is relatively easy to define: when the world decides the importance of publication. Peer review pre-publication is what the scientific community frequently means when using the term ‘peer review’. But what it is it?

Few will agree on an exact definition; generally speaking, it refers to an independent, third party scrutiny of a manuscript by scientific experts (called peers) who advise on its suitability for publication. Peer review is expensive; although reviewers are unpaid, the cost in time is enormous and it is slow. There is often little agreement among reviewers about whether an article should be published and peer review can be a lottery.

Often referred to as a quality assurance process, there are many examples of when peer review failed. Many will be aware of Woo-Suk Hwang’s shocking stem cell research misconduct at Seoul National University.² Science famously published two breakthrough articles that were found subsequently to be completely fabricated and this happened in spite of peer review. Science is not unique in making this error. However, love it or hate it, peer review, for the present time at least, is here to stay.

In this article, Philippa Benson, Managing Editor of Science Advances (the first open access journal of the American Association for the Advancement of Science), discusses the merits of peer review. Dr Benson has extensive experience in the publishing world and was Executive Director of PJB Consulting, a not-for-profit organisation supporting clients on issues related to converting to full electronic publishing workflows as well as challenges working with international authors and publishers. Her clients included the Public Library of Science journals, the American Society for Nutrition and the de Beaumont Foundation. She recently co-authored a book, What Editors Want: An Author’s Guide to Scientific Journal Publishing (University of Chicago Press), which helps readers understand and navigate the publishing process in high impact science and technical journals. Her master’s and doctorate degrees are from Carnegie Mellon University.

JYOTI SHAH

Commissioning Editor

Readers of biomedical science must keep up with developments in scientific peer review to be able to appropriately select which articles from which journals they want to read. Authors need to be similarly informed because understanding the strengths and failings of the peer review system today allows researchers to accurately assess the journals to which they may want to submit their work for review and publication.

Many involved in scientific, technical and medical publishing these days feel that the review system has become so problematic that the traditional system should be abandoned and replaced with mechanisms such as open review. If you assess the pros and cons of peer review, the cons do seem to win out in sheer number. However, in my view and that of many other editors, journal managed peer review remains the best available mechanism of maintaining quality in the progress of science. It certainly remains the only method that is widely accepted in the scientific community.

The full list of the flaws in the peer review system is extensive but a few of the negatives loom large at the top. These are discussed below and summarised in Table 1.

Table 1.

Some of the flaws in the peer review system

Problem	How authors and readers can learn more
Peer review takes time.	Look to see whether the journal posts the submitted, revised and posted date of articles. If available, check over six or more articles to see whether the time between submission and publication varies significantly.
Many journals do not make their peer review process fully transparent. The process of peer review is not standard from journal to journal. The quality of peer review is not standard.	Look to see whether the review process is described in detail on the journal website. If information is not provided, ask the editorial office to provide details about: >how the review process is managed (eg in-house or outsourced) including how the journal verifies identity and qualification of reviewers >how reviewers are selected >whether reviews are open, single, or double blinded >how authors can monitor the status of a submitted manuscript
Reviewers can be biased.	Communicate with editors if you suspect reviews are biased and ask editors for additional review.
Fraudulent reviews are increasing.	Authors should provide sufficient evidence that reviewers they suggest are real and legitimate, and reviewers should include institutional websites and publications lists to verify their identities.

The duration of peer review

Scientific research is a highly competitive professional endeavour, and the groups that publish good work often and quickly are better positioned to get continued funding. At the top of this list of problems, therefore, is that peer review takes time – sometimes a lot of time. Even the swiftest of journals conducting peer review can take weeks to move a paper from submission, through one or more layers of editorial review and to finding willing expert reviewers.

Once in review, additional weeks or months may pass and reviews are often handed in late. In fact, editorial staff spend inordinate amounts of time chasing overdue reviewers. When reviews are in, authors are often asked to revise papers, which can then be sent out for review again. In the meantime, researchers doing work on similar topics may be getting their manuscript through review at another journal and may end up being the first to get new results to press. Time is not on the side of the author in terms of peer review.

The transparency of peer review

Another problem with peer review is that few journals make their manuscript review process fully transparent. The word ‘fully’ is important. While many journals post their general submission-to-review workflow, few provide details about how quickly editors are successful in getting reviewers to assess a manuscript or what percentage of the reviewers for that journal submit their reviews on time. These detailed parameters make a difference to the end game. It is important to note, however, that properly managed peer review systems do provide a clear and documented paper trail of how decisions to accept or reject a paper were made if disputes do arise.

The process of peer review

Once an author submits a paper, transparency remains limited to an author being able to track a paper to see whether it is with an editor, under review or that a decision is ‘in process’. However, the range and median of time at each of those stages is not standard. In fact, it differs widely across journals according to the journal’s peer review and editorial workflow configuration.

Bias in peer review

Some journals use single blind reviews (where reviewers know the names of the authors but not vice versa) while others use double blind reviews (where authors and reviewers are both anonymous). Although the theory is that the knowledge by reviewers of who the authors are should not have an impact on the review itself, the reality is that it often does. When reviewers know something about the authors (or their institutions or laboratories), bias can kick in (whether intentionally or not). This can either speed up or slow down the review process. A plethora of studies have shown that reviews can be biased in a host of ways, including bias against non-native speakers of English, women and country of origin of the research.

The quality of peer review

Closely related to the issue of how much time it takes to get an article through peer review is the reality that the quality of reviews can vary greatly. On the one hand, many reviewers take on the duty of serving as a peer reviewer with pride and seriousness. These experts spend considerable time reviewing a paper and writing up advice to authors about how to improve their work on both conceptual and technical levels. Journal editors highly value such reviewers and the detailed guidance they provide. However, many reviewers hand in superficial reports along with their recommendations (reject, revise or accept) and include only cursory comments for the author. The quality of reviewers a journal can quickly solicit is also a reflection of how well that journal is connected to the community of experts in the field of the article in question.

Fraudulent reviews

A growing item on the list of challenges for quality peer review is the fact that instances of fraudulent reviews are increasing. In March 2015, a major publisher retracted dozens of articles following the discovery that the manuscripts had been reviewed by people using fake names who had been organised to facilitate the publication of papers by others in the group. This kind of highly unethical behaviour is increasing, and it demands the time and attention of editorial staff, who would otherwise be focused on improving the management and efficiency of legitimate peer review.

What do researchers think of peer review?

As an abstract process, peer review is indeed inherently flawed. Nevertheless, when managed properly by a publisher and a board of editors, the process can (and does) provide researchers with extraordinary guidance in refining their work into a clear and meaningful contribution to the progress of science. In fact, despite all the negatives and risks, researchers still highly value scrutiny and feedback from their peers. A report investigating the views of researchers found that peer reviewed journals remain the most preferred and trusted vehicle for research communication, and that feedback through review by peers significantly improved the research.¹ Journals are still seen as the arbiters of quality: the parties most able to objectively manage and maintain the quality of published science.

Researchers and publishers are taking action to address the problems inherent in peer review by continuing to study and address the challenges of the system. Efforts include oversight by organisations and groups such as the Committee on Publication Ethics, the International Congress on Peer Review and Biomedical Publication, the Council of Science Editors and Retraction Watch as well as services such as Peer Review Evaluation and CrossCheck, which were created specifically to support and strengthen the peer review process.

Conclusions

Despite its many limitations and flaws, journal managed peer review will remain the predominant method of evaluating the quality of research en route to publication until a different method is embraced by authors, readers, publishers, and (importantly) institutions that support research and employ researchers. Until such time, all those involved in peer review can improve the system by keeping informed about the inherent risks in the process, being watchdogs of quality and communicating about problems when recognised.