A systematic review of peer review for scientific manuscripts

Abstract

Background

The usefulness of peer review has been expressed as a method to improve the quality of published work. However, there has been a lack of systematic reviews to date to highlight the essential themes of the peer-review process.

Methods

We performed a search of the English language literature published prior to October 2011 using PubMed to identify articles regarding peer review. Inclusion and exclusion criteria were developed a priori. Data were extracted and then analyzed for the prevalence of peer-review themes contained within the literature.

Results

Of the 941 articles found during our original literature search, 37 were selected for review. The majority were commentary/editorial articles. The themes in our search included the structure and process of the peer-review system, the criteria for papers, ethics, and the different forms of the peer-review process.

Conclusions

The criteria for submission will vary, but our systematic review provides a comprehensive overview of what reviewers expect from authors. Our systematic review also highlighted ethical considerations for both authors and reviewers during the peer-review process. Although the topic of peer review is expansive and its process may vary from journal to journal, the understanding of the themes outlined in this paper will help authors recognize how to write a more successful paper. Also, more research must be carried out to establish the efficacy of the different styles of peer review, and it would be presumptuous to draw conclusions until further research is established.

Introduction

Over 300 years ago, the first two scientific journals were conceived and, thus, began the peer-review process. In January of 1665, the Journal Des Scavans (Journal of the Learned) emerged in Paris, followed shortly by Philosophical Transactions of the Royal Society which was published in London in March of the same year [36]. The Royal Society recognized early in the journal’s history that the decision to publish an author’s work would grant a sense of legitimacy to his words. Likewise, approving a questionable manuscript for publication could tarnish the reputation of the society [36]. In response to this fear, the Royal Society developed a system in which papers would be reviewed by the editor and critiqued by qualified society members who were knowledgeable on the paper’s topic [21, 36]. Manuscripts that could stand up to this scrutiny were determined to be reputable enough for publication and, thus, endorsed by the society. The French Academie peer-review policy was similar, except that Academie members would include “Academician” after their name to indicate that the published materials had been reviewed and approved by an official committee [36]. In the cases that a manuscript was determined to be both logical and reasonable, but the validity of its content could not be verified, the paper would be published with the note sit penes authorum fides (let the author take responsibility for it) [36]. This was to protect the journal from possible backlash caused by the publication of fraudulent papers. Even today, the critical function of peer review is to prevent the publication of incorrect or inadequate research while improving the accuracy and clarity of published papers [16, 17, 35].

Peer review is necessary to identify scientific manuscripts worthy of publication and to improve the quality of published research. The term “research” covers a wide gamut from medicine to science to marketing. Despite the differences in subject material, all researchers share a common experience—the sometimes arduous peer-review process. Despite its complexity, peer review can act as a filter to help separate out papers with “irrelevant, trivial, weak, misleading, or potentially harmful content” while also “improving the clarity, transparency, accuracy, and utility” of potential papers [17, 35]. The process is time-intensive and takes the effort of a mostly volunteer staff to evaluate papers for publication and construct objective and useful feedback [45]. Additionally, one’s academic achievement and career, as measured by publications and grant funding, is dependent on the judgment of one’s “peers” in a presumed fair appraisal process. Despite the uniform use of the peer-review process, to our knowledge, a systematic review has not been performed to evaluate the critical constructs of this process and its potential pitfalls.

The specific aim of this paper is to help perspective authors and reviewers understand the process of peer review by performing a systematic review of the literature. We will identify the important themes of peer review and distill the essential components of peer review from the literature. Access to this information will help familiarize authors with the process and increase the understanding of how their contributions are being judged. Specifically, we aim to determine themes that are relevant to helping authors navigate the tedious and sometimes confusing peer-review process.

Materials and Methods

We performed a search of the English language literature using PubMed to identify peer-review articles that were published prior to October 10, 2011. Our search criteria were as follows: (“peer review, research/ethics” [mh:noexp] OR “peer review, research/history” [mh:noexp] OR “peer review, research/methods” [mh:noexp]) OR (“peer review” AND “time factors” [mh]). The search used Medical Subject Headings (MeSH) terms because of the sheer volume of published papers on peer review. We decided to use the “peer review, research” MeSH term because we wanted to exclude other forms of peer review that were not relevant to the evaluation of scientific publications. We then narrowed our search further using the subheadings: methods, ethics, and history. Observance of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed during the course of this systematic review.

Exclusion and inclusion criteria were determined a priori and were used to screen the articles by title, abstract, and full text, in that order (Table 1). No restrictions were placed on study methods. Guides to writing a review were excluded because we wanted to focus on the actual process of how a paper is peer reviewed, not the process of how a referee should respond to an article.

Table 1.

Inclusion and exclusion criteria

Inclusion criteria

Papers must focus on manuscripts or articles

Peer-reviewed article, commentary/editorial

Focus on the actual process of peer review

Focus on history, ethics, process/structure, and types of peer review

Exclusion criteria

Case studies

Speech or correspondence

Was not written in English

Prior to searching the materials, we developed themes that we wanted to explore during our review of the literature. From these general ideas, we were able to determine four themes and seven subthemes (Table 2). The articles were individually examined for the presence of these themes.

Table 2.

Peer-review themes

Themes
Structure/process of peer review
Criteria for submitted manuscripts
Different methods of peer review
Single-blinded system
Double-blinded system
Open system
Peer agreement system
Author-suggested referees
Ethics
Author
Referee

Results

Study Retrieval and Characteristics

The search resulted in 941 articles. After screening titles, abstracts, and full text, 37 articles remained. The screening process is illustrated by Fig. 1. Twenty-nine articles were commentaries/editorials (78%), 6 review articles (16%), 1 randomized control trial (3%), and 1 survey (3%) (Table 3).

Fig. 1.

Exclusion process

Table 3.

Articles reviewed

Title	Article type
Working double-blind. Nature 2008; 451: 605–606	Commentary/editorial
Blaustein JD. Fraud: just say no! Endocrinology 2010; 151: 1–3	Commentary/editorial
Budden AE, Tregenza T, Aarssen LW, et al. Double-blind review favours increased representation of female authors. Trends Ecol Evol 2008; 23: 4–6	Review
Christensen NB and Yokomizo A. How to peer review. Int J Urol 2010; 17: 754	Commentary/editorial
Coats AJ. Ethical authorship and publishing. Int J Cardiol 2009; 131: 149–150	Commentary/editorial
Cossu R. Anonymous but fully recognised. Waste Manag 2010; 30: 1–3	Commentary/editorial
Fisher RS and Powers LE. Peer-reviewed publication: a view from inside. Epilepsia 2004; 45: 889–894	Commentary/editorial
Freedman JE. Promoting ethical conduct in the publication of research. Cardiovasc Ther 2008; 26: 89–90	Commentary/editorial
Freshwater D. Editors and publishing: integrity, trust and faith. J Psychiatr Ment Health Nurs 2006; 13: 1–2	Commentary/editorial
Froman RD. Blinded review revisited. Res Nurs Health 2010; 33: 273–275	Commentary/editorial
Froman RD. Hitting the bull’s eye rather than shooting yourself between the eyes. Res Nurs Health 2008; 31: 399–401	Commentary/editorial
Graf C, Wager E, Bowman A, et al. Best Practice Guidelines on Publication Ethics: a publisher’s perspective. Int J Clin Pract Suppl 2007; (152): 1–26	Commentary/editorial
Groves T. Is open peer review the fairest system? Yes. BMJ 2010; 341: c6424	Commentary/editorial
Heddle NM and Ness PM. Reviewing manuscripts: tips and responsibilities. Transfusion 2009; 49: 2265–2268	Review
Hoppin FG, Jr. [How I review an original scientific paper]. Rev Mal Respir 2003; 20: 671–678	Commentary/editorial
Johnson JT, Niparko JK, Levine PA, et al. Standards for ethical publication. Arch Otolaryngol Head Neck Surg 2007; 133: 7–8	Commentary/editorial
Khan K. Is open peer review the fairest system? No. BMJ 2010; 341: c6425	Commentary/editorial
Klein DF and Glick ID. Conflict of interest, journal review, and publication policy. Neuropsychopharmacology 2008; 33: 3023–3026	Commentary/editorial
Lemann J, Jr. Serving as a reviewer. Kidney Int 2002; 62: 1081–1087	Review
Loonen MP. Alternative peer review system: peer agreement system. J Plast Reconstr Aesthet Surg 2010; 63: 1931–1932	Commentary/editorial
Manske PR. Structure and format of peer-reviewed scientific manuscripts. J Hand Surg Am 2006; 31: 1051–1055	Review
McCalmont TH. Crystal clear. J Cutan Pathol 2011; 38: 540–541	Commentary/editorial
Minion D, Sorial E, and Endean E. Ethics of guidelines for reviewers of medical manuscripts. J Vasc Surg 2007; 46: 391–393	Commentary/editorial
Oyesiku NM. The registrar. Neurosurgery 2010; 67: 1165–1166	Commentary/editorial
Pitkin RM. Blinded manuscript review: an idea whose time has come? Obstet Gynecol 1995; 85: 781–782	Commentary/editorial
PLoS Medicine Editors. Making sense of non-financial competing interests. PLoS Med 2008; 5: e199	Commentary/editorial
Pollock RE and Ewer MS. The integrity of authorship: doing the right thing. Cancer 2010; 116: 3986–3987	Commentary/editorial
Provenzale JM and Stanley RJ. A systematic guide to reviewing a manuscript. J Nucl Med Technol 2006; 34: 92–99	Review
Reider B. Fabrication, falsification et al. Am J Sports Med 2010; 38: 445–447	Commentary/editorial
Thomas SP. The long journey to publication: some thoughts on the journal review process. Issues Ment Health Nurs 1998; 19: 415–418	Commentary/editorial
Tierney AJ. Reviewers support blinding in peer review. J Adv Nurs 2008; 64: 113	Commentary/editorial
Triadafilopoulos G. The manuscript review process. Gastrointest Endosc 2006; 64: S23–25	Commentary/editorial
van Rooyen S, Delamothe T, and Evans SJ. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ 2010; 341: c5729	Randomized control trial
Wager E, Fiack S, Graf C, et al. Science journal editors’ views on publication ethics: results of an international survey. J Med Ethics 2009; 35: 348–353	Survey
Wagner AK, Boninger ML, Levy C, et al. Peer review: issues in physical medicine and rehabilitation. Am J Phys Med Rehabil 2003; 82: 790–802	Review
Webb TJ, O’Hara B, and Freckleton RP. Does double-blind review benefit female authors? Trends Ecol Evol 2008; 23: 351–353; author reply 353–354	Commentary/editorial
Yoshida Y. Peer review system: any other choice? Int J Hematol 2006; 83: 191–192	Commentary/editorial

Prevalence of Themes

The most prevalent theme was ethics [1, 4, 7, 8, 10, 11, 18, 20, 25, 26, 28, 30, 31, 33, 40–42, 46]. This theme occurred in 18 (49%) of the articles reviewed. There were two dominant subthemes within ethics: the ethics of the peer reviewer or referee (n = 10; 27%) and the ethics of the authors (n = 8; 22%) (Table 4). For example, it is considered unethical for an author to alter images, include only specific data, use only certain references, and create an experiment to obtain a specific outcome. Fabricating results is ethical misconduct, but authors must also understand that altering data to misrepresent the original findings is also ethically wrong. Reviewers also need to be aware of ethical misconduct, such as bias based on gender or nationality. Financial and nonfinancial conflicts of interest are also areas of ethical concern [11, 20, 31].

Table 4.

Ethical considerations [8, 11, 26, 27, 30, 32, 40, 41, 46]

Author	Reviewer
Plagiarism	Remain unbiased towards
Authors included in paper should reflect who did work	Gender
All possible conflicts of interest should be addressed	Nationality
Financial and nonfinancial	Previously submitted work
No duplicate submissions—original work	Inappropriate promotion of like-minded research
Fabrication	Inappropriate delay or prevention of publication of work with which the reviewer personally disagrees with but has no grounds for such action
Making up results which never occurred	Editors should not have the final publication decision for any paper that they are a coauthor
Falsification
When experimental results have been altered, modified, or edited to the point that the data does not represent the original findings
Other forms of fabrication or falsification
Digitally altering images
Choosing to selectively present specific data
Suppressing data that is harmful to your past research
Using only certain references that bias your paper
Creating an experiment to obtain a specific outcome
Maintain confidentiality when appropriate

Another prevalent theme in the literature was the discussion of different versions of peer review, which was discussed in 12 (32%) of the articles reviewed [2, 10, 13, 19, 23, 28, 29, 39, 41, 44, 46, 47]. These different methods of peer review include the single-blinded system, double-blinded system, open system, peer agreement system, and author suggestion-based system (Table 2). Each of these methods can affect the transparency of the peer-review process. In the single-blinded system, the referee knows the identity of the author as well as the institution that submitted the work, but the author does not know the identity of the reviewer. In a double-blinded system, both the author and the referee identities are hidden. Contrarily, in the open system, both the identity of the author and the referee are exposed. The peer agreement system helps to foster cooperation between the author and the reviewer by having both work to refine the study protocol and complete the manuscript. An author suggestion-based system, in which the author provides a list of potential reviewers, assumes that authors are in the best position to determine who is uniquely qualified to review their research. This method also allows authors to avoid exposing sensitive research to their competitors.

Eight of the articles (22%) describe the criteria that peer reviewers are looking for when they analyze manuscripts [3, 5, 6, 9, 22, 24, 32, 43]. An important aspect to remember is that every journal has different criteria and formatting that must be followed. However, despite each journal’s specific tastes in citation, formatting, and page numbering, there are some basic criteria that reviewers expect within manuscripts. We have compiled a list of helpful criteria which authors should consult when writing (Table 5).

Table 5.

Checklist criteria for the author [3, 5, 22, 24, 31, 42]

When writing a manuscript for publication
General
The study follows the guidelines set forth by whatever publication you are submitting to
The study is original
The study demonstrates new findings
The study asks a question and provides an answer
Does the study follow ethical and special requirements?
Are the results trustworthy?
Is there a clear and concise language?
No typographical errors
Have you expressed any financial or nonfinancial conflicts of interest?
Title
Does the title accurately describe the study?
Abstract
Is the study summarized?
Is a reader able to understand the abstract without reading the manuscript?
Does the manuscript match with the abstract?
Introduction
Does it introduce and provide background for the study?
Does it explain the purpose of the study?
Are any unique words adequately defined?
Methods
Accurately and adequately describe your study design
Are the appropriate variables defined?
Is the design of the study a match for the question being asked?
Do you describe your methods and explain particular methodological choices?
Why did you choose a particular type of imaging?
Why did you choose a particular method of statistical analysis?
Results
Do you clearly explain your results?
Are the results in your tables and graphs consistent with your text?
Was any data excluded from the results?
Discussion
Is this clear and concise?
Are the main results reiterated?
If there was a hypothesis, was it discussed?
If there was a research question, was it discussed?
Are the conclusions supported by the data?
Are the limitations discussed?
Are any unexpected results discussed?
Are any future directions discussed?
Figures, graphs, and tables
Are they all labeled correctly?
Do they show the results in a clear and concise manner?
Does the manuscript need to be read in order to understand the graphs?
Are appropriate, pleasing to the eye colors used?

Lastly, six articles (16%) discussed the process and structure of peer review to give authors insights into a journal’s review methodology [3, 5, 14, 15, 38, 40]. These papers discuss how referees prepare themselves to review an article and how they read the article. Although every referee is different, some will first look at the abstract to understand how the experiment, methods, and results are structured. An examination of the abstract also gives the reviewer insight into what the author feels is important. This initial familiarity with the abstract helps the referee understand what the author is trying to convey in the paper. When reading the paper, reviewers will focus on screening for ethical violations and make sure the science behind an author’s work is presented clearly [15]. Part of the tedious process of peer review is finding referees who are experts and are known to have opposing viewpoints. This will help create a productive debate of opinions, and the feedback will help the author address potential problems that other readers may find [40].

Discussion

From the author and even the reviewer’s perspective, it is important to know the process and structure of peer review. Authors with a clear idea of the peer-review process will be able to write better papers, whereas reviewers can provide constructive and unbiased reviews. Our systematic review revealed six articles (16%) that talked about the process and structure involved with peer review. Understanding the process and structure of peer review gives authors insight into how their manuscript will be evaluated. An author needs to understand that, if their paper does not meet the journal criteria, then it will be removed early in the process without further consideration. Problems with the science, integrity, quality of reasoning, and application of scientific principles are all a part of what reviewers are looking for when they review a paper. Understanding the criteria referees are looking for within submitted manuscripts is a critical part of peer review (Table 5). Some problems noted by editors included ignoring submission routes, length restrictions, deadlines for revision, required materials, and the journal’s audience and goals. Failure to state a clear hypothesis, poor trial design, overgeneralizing the applicability of findings, and obtuse writing are also popular reasons for rejection [6, 9, 14].

The ethical code of conduct for both author and referee was discussed in a substantial number of papers reviewed. The ethical considerations include the disclosure of any financial or nonfinancial conflicts of interest to the readers and editors [11, 20, 31]. A financial conflict of interest arises when a researcher may receive compensation as a reward for specific outcome of their research. Financial conflicts of interest are usually easy to see; however, nonfinancial conflicts of interest are easier to hide. They can be personal, academic, or a number of other factors that influences personal judgment. To curb the addition of noncontributing colleagues, authors must only include coauthors who actually shared the burden of work. Authors should also be aware of plagiarism, redundant publication, fabrication, and falsification [4, 8, 11, 25, 33, 42].

The ethical considerations for referees should also include an unbiased attitude towards an author’s gender, previous work, and nationality [28, 46]. However, reviewers who are experts in a particular field may demonstrate a negative bias towards a competitor’s manuscript from the same discipline. When this is combined with the fact that reviewers hold a certain degree of power, some may be tempted to use this position to their advantage. For example, if an author’s manuscript is reviewed by one of their competitors, it would be possible for the reviewer to produce an unfairly negative critique in order to prevent or stall publication [30, 41]. Allowing authors to pick their own reviewers would provide the opportunity to choose reviewers who they feel are qualified to review their research and allows authors to prevent exposing sensitive research to their competitors [47]. There have been documented cases of reviewers stealing ideas from the manuscripts they are reviewing, and this was the case in the 1980s when an anonymous peer reviewer was found to be stealing ideas from unsuspecting authors [34]. Reviewers may also be more critical towards manuscripts within their area of expertise because they are better equipped to review these topics. The reviewer’s knowledge of the subject will result in a more critical and extensive review. Likewise, if a reviewer receives a manuscript from outside their area of expertise, they may be less capable of formulating deep thought-provoking questions and determining potential problems. In addition to subject knowledge, it was also found that a reviewer’s level of experience with the peer-review process is significantly associated with producing higher quality, more critical, stricter assessments of manuscripts than their less experienced colleagues [27]. Consequently, care should be taken to ensure that reviewers are familiar with the content they are reviewing while also considering whether the reviewer may be in competition with the author.

Seven papers (58%) of the 12 found within the “different methods of peer review” theme specifically discussed the double-blinded peer-review process [2, 10, 28, 29, 39, 44, 46]. A consistent theme throughout these articles centered on the argument that there is inconsistent evidence that the double-blinded review process actually improves peer review and more research must be done before a conclusion can be reached [10, 13, 29, 41, 46]. However, some of these papers state that authors think that the process of a double-blinded review does increase the quality of manuscript review by allowing reviewers to be more objective, and this is reflected by the growing interest in the double-blinded method among researchers [10, 29, 41]. However, in an editorial written by Roy Pitkin, he goes on to say that surveyed authors were more likely to favor a double-blinded review system if they had a paper recently rejected [29]. An interesting point of debate is whether the double-blinded review process may help to increase the number of papers published by female authors [2]. It is true that women are not as well represented in the scientific community as men; however, studies have found no significant effect of double-blinded reviews increasing the number of female-authored papers [44]. Clearly, more studies need to be carried out to properly determine if there is any advantage of the double-blinded peer-review system.

A few of the papers discussed the benefits and downsides to open peer review. Open peer review can encourage reviewers to be more complete and constructive when their identity is not obscured [10]. Under an open peer-review system, referees are able to receive acknowledgment by authors instead of remaining anonymous [13]. As with single-blinded reviews, open peer review makes the author’s identity known to the reviewer. This has the potential to expose an author to biases regarding their previous work, nationality, or gender [28, 46]. However, conflicting studies argue that open review does not bring any real advantages and requires further evaluation [13, 19].

Biases and Limitations

One limitation of our study was narrowing our search in order to work with a manageable number of articles. We accomplished this by choosing specific MeSH terms, using only English language articles, and using only one database—PubMed. This decision was made because the total volume of literature to review would become unmanageable if multiple databases were used. Because we are only focusing on MeSH terms, our accuracy is limited to how well the articles within PubMed were indexed with these specific MeSH terms. Article type may have also limited our results as well. Due to a lack of scientific literature, our systematic review consisted primarily of editorials and commentaries. Authors of these types of manuscripts are motivated to present their opinions. Thus, the conclusions of these papers may not represent the views of participants in the peer-review process as a whole.

Suggested Alternative Forms of Peer Review

Peer review is constantly evolving and alternatives to the classic form of peer review have been developed. A two-level system of peer review is evidence that researchers are constantly trying to improve on the current systems. The first level is a closed review process that determines the suitability of an article for publication on a website. The second level involves a public review process, and if the article passes through both levels, then it will be published in a print journal [12].

A more radical alternative to the peer-review process is to publish everything and let the scientific community decide what information is actually usable [37]. An inherent problem with publishing everything is the sheer abundance of material that is produced. There are no checks and balances to ensure any form of credibility or accuracy before the material is presented. However, this method argues that, even with the increased publications, the scientific community will still be able to discover important research.

The process of peer review could also be changed with the help of the peer agreement system. A recent editorial in the Journal of Plastic, Reconstructive & Aesthetic Surgery outlined this system, in which authors would submit research ideas or protocols rather than finished research manuscript. The referees would evaluate these ideas and protocols for revision, acceptance, or rejection. An accepted research proposal would earn guaranteed publication. In this system, the peer reviewers become coauthors, and the bond between editors and authors are strengthened [23].

Peer review is an important process for all authors to understand. Ultimately, peer review was created to protect scientific integrity and promote the sharing of research with other colleagues. It can help authors discover problems and helps to strengthen the credibility of their research. The extensive amount of published material relating to peer review can be overwhelming for readers to sort through, and this paper provides a relevant guide for authors regarding the peer-review process. The necessity of having quality control measures for published work is important to the scientific community, and without such measures, the quality of published work would not be what it is today.