The peer review process

Abstract

The peer review process provides a foundation for the credibility of scientific findings in medicine. The following article discusses the history of peer review in scientific and medical journals, the process for the selection of peer reviewers, and how journal editors arrive at a decision on submitted manuscripts. To aid authors who are invited to revise their manuscripts for further consideration, we outline steps for considering reviewer comments and provide suggestions for organizing the author's response to reviewers. We also examine ethical issues in peer review and provide recommendations for authors interested in becoming peer reviewers themselves.

Introduction

The review of research articles by peer experts prior to their publication is considered a mainstay of publishing in the medical literature.[1,2] This peer review process serves at least two purposes. For journal editors, peer review is an important tool for evaluating manuscripts submitted for publication. Reviewers assess the novelty and importance of the study, the validity of the methods, including the statistical analysis, the quality of the writing, the presentation of the data, and the connections drawn between the study findings and the existing literature. For authors, peer review is an important source of feedback on scientific writing and study design, and may aid in professionalization of junior researchers still learning the conventions of their field. Nevertheless, peer review can be frustrating, intimidating, or mysterious. This can deter authors from publishing their work or lead them to seek publication in less credible venues that use less rigorous peer review or do not subject manuscripts to peer review at all. In this article, we trace the origins of the scientific peer review system, explain its contemporary workings, and present authors with a brief guide on shepherding their manuscripts through peer review in medical journals.

The History of Scientific Peer Review

The introduction of peer review has been popularly attributed to the Royal Society of Edinburg, which compiled a collection of articles that had undergone peer review in 1731.[2,3] However, this initial process did not meet the criteria of peer review in its modern form, and well into the twentieth century, external and blinded peer review was still far from a requisite for scientific publication. Albert Einstein protested to the editor of an American journal in 1936 that his article was sent out for review, whereas this was not the practice of the German journals to which he had previously contributed.[4] Nevertheless, by the 1960s, the scientific value of peer review was becoming widely accepted, and in recent years, publication in a peer-reviewed journal has become a standard metric of scientific productivity (for the researchers) and validity (for the study).[5,6] In fact, publication in peer-reviewed quality journals is used to evaluate the quality of research during the academic promotion process. Today, peer review continues to evolve with the introduction of open review (reviewer comments posted publicly with the final article), postpublication review (reviews solicited from readers in an open forum after article publication), and journal review networks (where reviews are transferred from one journal to another when an article is rejected).[7,8,9] The constant at the center of this change remains the individual reviewer, who is asked to contribute their expertise to evaluating a manuscript that may or may not ever be shared with a wider scientific audience.

Reviewer Selection

The opacity of the peer review process is due, in part, to the anonymity of the reviewers and authors' lack of familiarity with how reviewers are selected. Typically, reviewers are selected by an editor of the journal, although depending on the size and organization of the journal, this may be the Editor-in-Chief, an Associate Editor, a Managing Editor, or an Editorial Assistant. Some journals permit authors to suggest their own reviewers, although the extent to which editors use these suggestions is variable. Authors may also be asked specifically or allowed to oppose reviewers, if they feel that certain scholars cannot grant their manuscript an unbiased hearing. Again, it is at the editors' discretion whether these requests are heeded. It has been suggested that these “opposed” reviewers may even be deliberately selected to ensure critical evaluation of a controversial manuscript. Alternatively, for very specific and narrow subject areas, there may be a limited number of appropriately qualified reviewers.

In general, reviewers may be of any academic rank and from a wide range of medical disciplines. A reviewer may be selected for their expertise in the topic of the study, but also for their general methodological expertise, or because they have been a reliable reviewer for the journal in the past. Qualified reviewers may not be invited if they cannot be reached by the editorial team, if they tend to submit late or uninformative reviews, or if they are too closely connected with the manuscript authors (e.g., colleagues at the same institution) and therefore may not provide an unbiased review. The reviewers initially selected by the editors may decline the invitation to review, mandating that the editors seek other reviewers. Unfortunately, this process of waiting for a response from the initial invitation to review (aside from the time taken to review) is one of the more common causes resulting in a delay in getting a response from the journal when a manuscript is submitted. The invited reviewer may pass the review on to a junior faculty member to allow them to participate and experience the academic peer review process. This may be performed with the permission of the editor, and noted after the review is submitted to the editor when the invited reviewer identifies that another person has participated in the process.

The initially received reviews may conflict with one another, leading the editors to cast a wider net for experts who will agree to review a submission. Because many factors may delay the completion of the review process, editors may proactively invite more reviews than they require and decide on the manuscript after a minimum number of reviews have been completed. The use of email and the internet has greatly facilitated communication for the review process, which used to be accomplished via telephone and postal mail. In most instances, an initial email is sent to the reviewer inquiring regarding their availability and interest. They are then asked to agree to review, at which time, a secondary email with a link to the journal site, the manuscript, and the review forms is sent.

How Reviewers Assess a Manuscript

From the reviewer's perspective, participation in the review process begins with an invitation from the journal editors to consider reviewing a submitted manuscript. If they accept, the reviewers will be able to access the submitted manuscript files, and sometimes the authors' cover letter, and other article metadata (e.g., the authors' list of preferred reviewers, figures, tables, etc.). Some journals ask reviewers to complete a structured questionnaire regarding the manuscript, rating its attributes on a numeric scale, or answering specific questions about each article section. All journals permit the submission of free-response evaluations. It is these evaluations that typically carry the greatest weight in the editors' final decision. The free-text reviewer reports also give the authors specific instructions about revising their manuscript and responding to the concerns that are raised. Reviewers may also submit confidential free-response comments to the editors (not seen by the authors) and indicate to the editors if they would be willing to review a revised version of the manuscript. In the end, the reviewer is asked to indicate their final recommendation to accept the manuscript without changes, accept after minor revisions, reconsider after major revisions, or reject. Some journals may offer additional variations on these recommendations, such as “reject but allow resubmission,” discussed below.

Regardless of the requested format for reviews, reviewers will typically evaluate several key aspects of submitted manuscripts. For original research studies, these will include the importance of the research question, the rigor of the methods, the completeness, accuracy, and novelty of the study and its results, and the validity of conclusions drawn from the data. The presentation of the manuscript, including the writing style, structure, grammar, and syntax also determine how the manuscript is received by the reviewers. Although the study design and results may be valid, these findings may be lost if the presentation is not precise or if there are grammar and spelling errors.

Reviewers also consider whether the study adds to existing knowledge in the field, whether it was ethically conducted, and whether it may be subject to any conflicts of interest. The editor and the reviewers also evaluate the study content and decide whether it is valuable and relevant to the readers of the journal. Although the study may be valid and well performed, it may be decided that the subject matter fits more appropriately in a journal of a different specialty. Along those lines, there may be overlap in the interests and fit of journals in different specialties, so that common topics in anesthesiology research may be of interest to journals from surgical specialties, pain medicine, or healthcare quality and patient safety, depending on the article content.

Some reviewers may submit their comments in paragraph form, building a narrative of the study's strengths and weaknesses section by section, whereas others may submit a short summary of the study followed by a list of criticisms or suggested corrections. Less commonly, reviewers may annotate the original manuscript with specific changes and questions or using the track-changes function of the word-processing software. Although the reviewers may recommend a specific editorial decision (e.g., recommend accepting an article with revisions, recommend rejection) in their comments to authors, this is generally discouraged by most journals and does not override the final decision reached by the editorial team. The ultimate decision generally resides with the section editor or the editor-in-chief, once they have seen and evaluated the comments of the reviewers. Depending on the format of the journal, the manuscript may be reviewed by one to five individuals. When there are specific statistical questions or advanced methods used, a separate review of the analytic methods may be required. For high-profile journals with high Impact Factors, a recommendation to accept may be required from all reviewers to receive a favorable editorial decision. At times, if there is a split decision, an additional reviewer or member of the editorial board may be asked to evaluate the manuscript to break the tie.

Almost all journals practice blinded review, where the reviewers' identities are not revealed to the authors. Double-blind review, where authors' identities are concealed from reviewers, although previously uncommon in medical journals, has been increasingly used. The editors communicate their decision and reviewers' evaluations to the authors in a decision letter (e-mail), informing of manuscript acceptance or rejection.

Reviews and the Editorial Decision

The comments submitted by external reviewers are collected by the editorial team and considered when determining the overall decision on the submitted article. The reviews may be read directly by the Editor-in-Chief, or by one or more Associate or Section Editors. The first editor reading the reviews might provide a recommendation that is then considered by the more senior editor; or the editors may convene to discuss the reviews and reach a decision as a group. In some journals, editors may write their own summary of the reviewers' criticism (sometimes adding their own) or may point out the critiques they consider most important to their decision. In other journals, editors weigh the number of positive and negative reviews or may reject an article unless all reviewers endorse its acceptance or revision.

Based on the external reviews and their own reading of the manuscript, the editors will reach one of several options regarding the manuscript. Unconditional acceptance of an article on its first submission to a journal (without any requested revisions) is very rare. Sometimes, articles will be conditionally accepted or accepted with minor revisions, meaning that the editors wish the authors to make changes to their manuscript based on the reviewers' comments but will not send the revised manuscript for a further round of external review. Rather, if the comments are generally minor, the editor will ensure that the comments are appropriately addressed in the authors' revision. The more common decision is “major revision,” where editors are willing to consider a revised version of the article but will subject it to further external review, by the original reviewers, a new set of reviewers, or a combination of both. Some journals also use a “reject and resubmit” decision, indicating lower enthusiasm for a resubmitted version of the article but still permitting resubmission, perhaps in an alternative format (e.g., brief report or letter to the editor, vs. full article) or with extensive revisions. For this latter decision, a full review will be accomplished as the revised manuscript is handled in much the same way as a new submission.

If the editors feel an article is a poor fit for their journal or falls too far below its standards, they may reject submissions outright without sending the manuscript for external review. This “desk reject” should not be confused with articles being “unsubmitted” by a managing editor or editorial assistant. The latter can happen due to style or formatting issues with the initial submission, which the author is asked to correct before the manuscript proceeds to review. Having a manuscript “unsubmitted” does not preclude resubmission of a corrected manuscript and is unlikely to affect reviewer assessment and, eventually, editorial decision.

Revising the Manuscript

When the initial editorial decision is positive, but not an unconditional acceptance, authors may elect to revise their manuscript and resubmit it to the same journal with a point-by-point response to the reviewers (discussed in the next section). The primary aim of the authors for this revision should be to address the criticisms and concerns raised during the initial review. Yet, this may be easier said than done when faced with conflicting recommendations, hostile reviews, or simply a large number of suggestions to be accommodated within a strict manuscript word limit. To streamline the process of responding to reviews, we offer the following roadmap as a suggestion.

Address the “fatal flaws”

Reviewers or editors may point out critical weaknesses of the study that prevent it from drawing the intended conclusions or even any conclusions at all. For example, an inaccuracy in the data, a bias in patient recruitment, a limitation of sample size, or a lack of follow-up may be so severe that the manuscript cannot provide credible evidence on the treatment or exposure it is meant to study. In particular, a lack of appropriate ethical approval would disqualify a study from publication, no matter how methodologically rigorous it may have been. In systematic reviews and analyses of existing databases, prior publication of a near-identical paper by a different group may also fundamentally preclude a paper from acceptance. On the rare occasions when the paper's central conclusions are found invalid and cannot be corrected through new analysis or a different framing of the authors' argument, reconceiving the study may be a better approach than attempting to revise and resubmit. At other times, some of these issues may be approached and the editor and reviewers satisfied by adding text to the discussion outlining the limitations of the current study. This may allow authors to acknowledge the concerns expressed by the reviewers and yet not redo their study from the beginning.

Amend the data analysis

More commonly, reviewers ask for changes to the data analysis without implying that these requests invalidate the entire study. We recommend making these changes before any further edits to the manuscript, because the intent is often to see if the paper's original findings are robust. In the best case scenario, any additional analysis will only confirm and strengthen the central conclusions. However, additional analyses sometimes reveal contradicting findings, which the authors should frankly address in the revised manuscript, by pointing out the contradiction and speculating about why different analyses of their data may have reached different conclusions. Especially when the study design was prospectively registered, the authors should explain in the manuscript which analyses were planned a priori and which were added post hoc. In these studies, authors should also avoid changing the pre-specified primary outcome, which would have been used for any a priori power or sample size calculation.

Decline infeasible or inappropriate suggestions

Some requests may not be feasible, for example, when requested data were not collected for a prospective study, or when collecting the data would mean starting chart review from scratch for a retrospective study. At other times, it may not be feasible to comply with the reviewers' requests if they disagree with the study type, the study cohort, or make other requests that would require a new or different study to address. Reviewers could also request changes to the statistical analysis that are not appropriate for the data at hand or for the study aims. In these cases, authors have the choice of rebutting the reviewers' comments while making no change in their manuscript, but an argumentative revision that leans too heavily on this option may be received poorly on re-review, resulting in rejection of the manuscript. In our experience, authors may be successful in responding to the reviews while rebutting one or two of the reviewers' suggestions, but a legitimate argument must be made for the rebuttal, and the reasons clearly stated.

Explain the study rationale and methods

Having completed the revision of the data analysis, authors should check that their methods section includes a complete and correct explanation of how the data were collected and explains how the analysis was performed. It may be appropriate to end the introduction by stating the hypothesis of the study. In the methods section, reviewers will often ask about the ethical committee approval of the study, the site(s) where the study was conducted, patient inclusion and exclusion criteria, the consent process, the procedures involved and the protocol for anesthetic management, and the specific data points that were collected during the study. For prospective clinical studies, authors should also indicate whether the study was submitted to a trial registry (such as ClinicalTrials.gov), and whether this was done before or after study enrollment had started. Clearly stated ethical approval and trial registration information must be provided for all submissions. Explanations may be sought if the editors and reviewers believe that the study did not meet standards for ethical approval, patient consent, or trial registration. Other requests related to methods may ask to clarify how the primary and secondary aims outlined in the introduction were addressed in the analysis, and how the sample size was determined, whether based on a statistical power analysis or logistical considerations (e.g., how many patients could be recruited with available resources). When a statistical power analysis is performed, reviewers may ask for more detail about the assumptions of this analysis and any supporting data from pilot studies or previous publications.

Check the conclusions and limitations

Having revised the introduction, methods, and results, the authors should revise the discussion to make any changes to the conclusions required by new or different study findings. We recommend that authors start the discussion with a review of what the study found, and then discuss how the study findings relate to similar work that has been previously published. An excessively long discussion does not ensure that a study will be published and, in fact, may detract from the quality of the manuscript. For a scientific study (retrospective or prospective), the discussion should not read like a comprehensive review of the literature. Typically, the discussion of study limitations will be expanded in the revised manuscript to include additional study weaknesses pointed out by reviewers, acknowledge suggested changes that could not be made to the study methods, and mention other suggestions for future studies that would build on the current results or answer questions left unanswered by the current study. Reviewers may ask that the conclusions be more specific in addressing the primary aim or hypothesis of the study (stated in the introduction), but they may also encourage authors to go further afield in their discussion, connecting their findings to results from previous publications and describing how their findings support or challenge current clinical practice.

Writing the Response to Reviewers

As seen above, manuscript revision can require more writing and (re)analysis than even the initial submission. Therefore, the aim of the revision memo (response to reviewers) is to summarize for the editors and reviewers how each change addresses the concerns raised on the initial review. This document is handled differently by different journals; some require it to be uploaded as a separate file, others require that the revision memo be entered in a text-box during the online submission process, and still others require that the response to review be included in the cover letter for the resubmitted manuscript. Therefore, authors should pay close attention to the decision letter and its instructions as to how they should submit their response to reviewers and how they should refer to manuscript edits in the revision memo (e.g., by page number, by line number, or copying sections of the revised manuscript into the memo).

Typically, the reviewers' comments should be copied and entered in the response memo so that each comment is numbered and the response clearly listed after it, in a different font style or color. It is equally important to determine how the journal would like the changes tracked in the revised manuscript. Some journals will ask that the authors use the track-changes mode in the word processing software, whereas others may ask for changes to be highlighted or be added in a different color font. Deleted manuscript text may need to be shown in strike-through font or simply removed from the revised submission, depending on the journal. Journals may ask for two copies of the revised manuscript: one showing the changes and one in a clean format that is ready for copyediting.

A typical revision memo will include a short paragraph acknowledging the editorial decision and reviewer comments and briefly summarizing key changes made to the manuscript. This would be followed by a numbered list of comments from the editors and reviewers (as received in the decision letter), with the authors' response to each one. Although not all reviewers and editors submit their comments as a numbered list, the authors may want to break up long sections or paragraphs of the reviews into shorter, numbered comments, to separately describe how each one was addressed in the revision. The authors' responses need not be excessively ingratiating but should respect the reviewers' effort in evaluating the manuscript, and concisely explain what was changed or why a change was not or could not be made. Different reviewers may have conflicting recommendations for revision. This may be as simple as one asking for a more concise definition of a method while another asking for a more detailed explanation. With conflicting reviews, the authors may consider taking the recommendation that is endorsed in the editor's comments (if this is provided), the one that is best aligned with the study aims, or the one that best matches the methods and writing style used in other contemporary papers in the field; and explaining this rationale when responding to the reviewers.

What to Do with a Rejected Manuscript

Based on reviewer reports or their own judgment, editors may reject a manuscript with no option to resubmit. It is essential to read the decision letter closely as some journals will state that they cannot publish a manuscript in its current form but offer to consider a new submission of a substantially revised manuscript (“reject and resubmit,” as mentioned above, in contrast to “revise and resubmit”). When the manuscript is rejected with no option of resubmission, authors may appeal this decision, but this option is rarely exercised and may not change the editors' decision. Appeals are also generally only successful when made by experienced and recognized scholars in the field.

Unless the study is discovered to be so flawed as to preclude publication in any venue, authors will usually consider submitting it to another journal after the initial rejection. Taking a single rejection and tabling a manuscript without further submission is rarely a good option. It is possible that multiple rejections will precede an eventual acceptance for valuable work. Given the amount of time taken to devise, implement, and up a study, we encourage authors to consider resubmission to a new journal, if the study is well conceived and addresses an important problem or question. In this case, the criticisms in the initial review are not binding, but still worth the authors' consideration. Particularly, authors should address any major flaws in the study's approach and conclusions (distinct from reviewers' preferences for additional data analysis unrelated to the primary aims), and correct any factual, spelling, or grammatical errors prior to resubmission. Adding recommended secondary analyses could sometimes strengthen the next submission, although just as often, the reviewers at the next journal may find these additional analyses superfluous, and will have their own set of analyses to recommend.

Becoming a Reviewer

Like any complex skill, navigating the peer review process is best learned through repetition. Becoming a peer reviewer for scientific journals is an important way to hone this skill, as well as providing a service to the scientific community, and adding to one's academic credentials as an expert whose opinion is sought by journal editors. The most common entry point to becoming a reviewer is through scientific publication; the authors of published articles can be contacted by another journal to provide a review on a related submission. One's expertise in a specific area may be noted by the editor who performs a topic search of key words when looking for reviewers. Alternatively, editors and associate editors may call on colleagues who they know are recognized experts in a particular field. Academic mentorship is also important, as mentors may ask junior colleagues and faculty to help them with reviewing article submissions, or may pass their name along to journal editors to be considered for inclusion in the reviewer pool. Once one has successfully reviewed for a journal, they are frequently called upon to review other submissions, especially if their review was returned in a timely manner. Many journals will give a specific timeframe within which the review is to be completed, while others will not. In most cases, a response within 2–4 weeks is considered acceptable. Some journals have now started editorial fellowships that aim to provide an immersive experience in the peer review and publishing process for early-career scientists. Lastly, researchers wishing to become peer reviewers may contact journal editors themselves, or register reviewer accounts in journal online submission systems. Although the general structure of peer review reports is described above, more specific guidance on performing peer review is available in other publications.[10,11]

Peer Review Ethics

Authors, reviewers, and editors have a shared responsibility for the ethical conduct of peer review. This is necessary to sustain the professional and public trust in peer review, as a system of evaluation that is accurate, constructive, and free from bias. Recently reported ethics violations have included authors misrepresenting the identity of suggested reviewers, reviewers plagiarizing a manuscript sent to them for review or recommending its rejection and then conducting a similar study, and editors inappropriately pressuring authors to cite articles published in their journal.[12,13,14] Some journals and publishers have also been criticized for circumventing the peer review process for submitted manuscripts.[15] For reviewers, it is most important that they be unbiased and not have any hidden agendas or personal vendettas to settle. For authors, ethical conduct in peer review includes disclosing the study's ethics committee approval, trial registration, and consent process; disclosing any related or overlapping prior publications; disclosing any actual or potential conflicts of interest; and submitting the manuscript only to one journal. These requirements are typically stated in the journal's guidelines for authors, and may need to be acknowledged in the cover letter accompanying the manuscript. In responding to reviews, authors should also carefully consider whether their revisions still fall within the scope of the ethics committee approval for the study and the informed consent that was obtained, and whether the revised manuscript remains faithful to the aims and study design of any pre-registered trial protocol.

Conclusion

Scientific research is not complete until it is published, but not all research can or should be published. It falls to peer-review to determine the difference. By engaging with the process of peer review, authors can improve the quality of their work as well as gain confidence that it is published in a reputable medium. Furthermore, the fact that a study has been peer reviewed will increase its stature and potential for recognition. However, the peer review process does not assure this. Although responding to reviews can be challenging, we hope that the suggestions sketched out in this article will help authors plan their approach to manuscript revision and resubmission. We also encourage authors to participate in this process as reviewers, so that the labor of peer review is properly shared among the community of scientists.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.