The Peer Review Process

Abstract

The purpose of peer review is to evaluate the scientific merit of the submitted work and to assess suitability for publication. This process is intended to provide an unbiased, independent critique to ensure publication of high-quality manuscripts that demonstrate validity and reliability. Reviewers are subject-matter experts who volunteer their time to participate in peer review. A proper review provides constructive and helpful feedback in a timely manner that authors can use to improve both current and future work. When given the opportunity to revise, authors should carefully consider all comments and adequately address all concerns. This paper provides guidance to clinicians for both aspects of the peer review process: participating as a reviewer and responding to reviewer feedback.

Introduction

Scientific manuscripts submitted for publication undergo peer review prior to acceptance. The peer review process is multifaceted and involves several stakeholders including the authors, the journal, the editors, and the reviewers. The purpose of peer review is to evaluate the scientific merit of the work and to assess suitability for publication.¹ Peer review is intended to provide an unbiased, independent assessment and critique of the submitted manuscript.² The process helps ensure publication of high-quality manuscripts with sound study design, valid methods, reliable results, and implications for clinical practice.

The quality of the peer review critique is an important part of the process for all stakeholders. Journals frequently rely on the expertise and knowledge of reviewers to help guide editorial decisions. A properly done review not only evaluates study rigor but also provides authors with valuable feedback that can be used to improve their work. Authors should carefully consider all suggestions and recommendations and take advantage of the opportunity to revise the manuscript when presented the option.

Although most journals offer instructions for completing peer review, the peer reviewer role is not standardized, and processes vary among journals.³ It is often presumed that author experience alone provides the basic skills needed to review a manuscript.³ However, formal instruction in peer review is generally not included in training programs and is consequently a learned skill.^3-5 Similarly on the author side of the process, there are few resources available to assist authors with responding to peer review. The purpose of this paper is to provide guidance to clinicians for the author and reviewer aspects of the peer review process: participating as a reviewer and responding to reviewer feedback.

Models of Peer Review

There are 2 main models used for the peer review process: open or closed. Author and reviewer identities are known to one another for open review whereas in closed review at least one group, usually the authors, is blinded to the identity of reviewers. The peer review process is not without challenges, and there are advantages and disadvantages to both models. Nonetheless, it is a valuable process that acts as a filter to ensure scientific rigor, validity, and reliable results of published manuscripts.

Open review has the advantage of transparency and may improve accountability. Reviewers are more likely to be respectful and courteous with open review and less inclined to be rude or leave scathing comments. However, the disadvantage is that some reviewers might be hesitant to provide honest and critical feedback that could result in low-value reviews.^6,7 This is especially true if the paper is written by someone with influence in the field, grant funding, or the professional organization. Conversely, novice authors may make unnecessary changes based on comments from well-known reviewers solely due to reputation and not the quality of the review. The open review model has also been associated with decreased reviewer willingness to participate in the peer review process.^7,8

Closed peer review is either single or double blind. For a double-blind review, the identities of both authors and reviewers are concealed from one another. This process can be helpful in preventing bias against author sex, ethnicity, or geographic location. However, in many cases reviewers have still been able to identify authors based on content, writing style, or use of self-citation.⁷ Single-blind review is the most common model of peer review used by journals, including Respiratory Care. In the single-blind model, reviewers remain anonymous to the authors while author identities are revealed to reviewers. A disadvantage of blinded peer review is the potential for unprofessional or rude comments.

Studies that have compared single and double-blind peer-reviewed manuscripts demonstrated no significant differences in terms of review quality, time to complete review, or reviewer recommendations.^9-11 Compared to open peer review, closed review may result in a more honest critique of the work since reviewer identity is masked to authors. Consequently, this could also potentially lead to impolite or harsh reviews that are overly critical or not beneficial in helping authors improve their work. Some journals publish peer review comments and allow reviewers to reveal their identity to authors, if desired. However, comments should not be published without permission from both the reviewer and author, and reviewer identity should not be revealed without reviewer consent.²

Peer Review Process

The process of peer review begins with authors submitting their work to a journal. The editor screens the manuscript to determine if it will undergo peer review as some manuscripts are rejected without review. Manuscripts that do not go through peer review are often those deemed not a good fit due to a topic outside the journal’s scope or not within the editorial priorities of the journal, not of general interest to the core readers, or found to have obvious fatal flaws.¹² If the manuscript proceeds to peer review, the editor invites reviewers by email to evaluate the work. The number of invited reviewers varies by journal but ranges from 2–5.^6,8,13 Oftentimes, additional reviewers are invited since some reviewers will decline or are unavailable at the time of the invitation.

Reviewers are typically subject-matter experts with different backgrounds who volunteer their time, expertise, and knowledge to contribute to the peer review process. It is often perceived as a thankless task since compensation is usually not provided, but good reviewers are genuinely appreciated. Many journals acknowledge reviewer contributions through annual recognition or offer services to track peer review activity such as Web of Science. Participating in peer review could promote career development as it demonstrates professional commitment, and it may generate ideas for new research proposals.⁷

There are several methods for identifying reviewers. Journals typically maintain a database of potential reviewers that includes contact information and key words to identify areas of expertise. Authors who have published within the journal are often included in the database and may be invited to review manuscripts with similar subject matter. Many journals have an online system where prospective reviewers can register and select categories of interest to review. Additionally, potential reviewers may be identified through a literature search of related topics¹⁴ or recommended by another reviewer or colleague.

Reviewers accepting the invitation evaluate the manuscript, submit a critique, and provide a recommendation for publication if requested. However, not all journals ask reviewers to provide a recommendation regarding publication.¹⁵ Following receipt of completed reviews, the editor considers all feedback and recommendations and uses the information to make a decision on the manuscript. In cases where the reviews are of low quality or contradictory, editors may also reach out to select editorial board members or colleagues with content expertise. Decisions range from accept, reject, or revision. Figure 1 provides a general overview of the peer review process.

Fig. 1.

Overview of the peer review process. *Multiple revisions may occur.

It is rare for a manuscript to be accepted on the first submission. There are usually at least a few changes required. Revision can be either major or minor, and both offer authors the opportunity to make the requested changes and resubmit for reconsideration of publication. A decision of minor revision indicates there are minimal issues that need to be addressed and suggests the manuscript is essentially ready for publication. Oftentimes, if the requested changes are made, the manuscript will likely be accepted immediately and may not require additional peer review.

Major revision signifies there are considerable issues requiring attention. However, the manuscript could potentially be accepted if all identified concerns are adequately addressed. Re-submission following major revision typically results in additional peer review to allow reviewers the opportunity to determine if the authors sufficiently dealt with the raised concerns or questions. It is common for multiple revisions to occur if not all items are satisfactorily managed or additional questions are identified following revision. It is also important to note that the offer of a major revision does not guarantee acceptance of the manuscript.

Some journals have an option for reviewers to recommend reject and resubmit. This selection indicates the manuscript requires substantial revision equivalent to that of a new submission and that the potential for acceptance is uncertain. A decision of reject indicates the paper cannot be accepted regardless of revision. Some of the most common reasons manuscripts are rejected after peer review are flaws in methodology, improper statistical analysis, insufficient sample size, failure to contribute to what is already known, or ethical concerns.¹⁶ Table 1 includes a summary of the types of decisions.

Table 1.

Editorial Decisions

Once the editorial decision is made, the editor sends the decision letter to the submitting author, usually by e-mail. The letter generally includes comments from the editor and reviewers for recommended changes, clarifications, questions, or the reasons for rejection. Reviewers should be informed of the editorial decision for the manuscript they reviewed.² Reviewers are often blind copied on the decision letter for journals that utilize single-blind peer review, but in some cases, notification may occur in separate communication. Reading comments from other reviewers can serve as a learning tool for novice reviewers to improve the quality of future submissions and to better understand the rationale for the editorial decision.¹⁷

The duration of the peer review process from author submission to editorial decision varies from 4–8 weeks or more and is dependent upon several factors.^12,18 Authors can introduce delays in peer review for failure to follow submission instructions.¹ Editors may encounter challenges identifying reviewers due to no response or multiple declined invitations to review, with the latter potentially reflecting a lack of interest in the topic. The process can also be delayed by reviewers submitting late evaluations or failing to complete the review after agreeing to do so. Lastly, decisions may be delayed due to the editor having a large number of manuscripts awaiting decision or personal or professional obligations.

Reviewer Ethics and Responsibility

Invited reviewers should only agree to review manuscripts for which they have at least some experience on the topic.¹⁷ Potential reviewers are usually provided with a copy of the abstract to help determine if the paper is within their expertise or of interest. Reviewers should also respond to the request in a timely manner so as to not delay the peer review process.^2,17 When unable to review, it is suggested that reviewers propose alternative potential reviewers. Reviewing a paper requires dedicated time and effort, and the amount of time will vary depending on reviewer experience and complexity of the paper. Reviewers should expect a minimum time commitment of 2–6 h to complete a review,⁷ although this does not have to be done all in one sitting. Dedicated time is required to carefully read the paper, consider the work, compose comments, and submit the critique.

The time frame allotted to complete a review varies by journal and may range from 2–4 weeks. Automated reminders are commonly generated in efforts to prompt reviewers when the review has not been received. Reviewers should notify the editor when unable to complete the review within the designated time as circumstances may have changed that impacted the ability to evaluate the manuscript. Some editors are willing to offer an extension if the reviewer still expects to fulfill the request but may be delayed. Reviewers should not accept an invitation to review a manuscript for which there is not intent to submit a timely review or to purposely delay review due to their own similar work that may be under consideration for publication.¹⁷ The review process is confidential, and therefore the manuscript should not be shared or discussed with others without permission from the editor, and copies of the paper should not be retained for personal use after completing the review.^2,13,15,17 It is not ethical for reviewers to take information or knowledge from an unpublished manuscript for personal gain or to further their own interests.^2,15,17

General Considerations

The intent of peer review is to evaluate the scientific merit of the work and not to judge author characteristics. Therefore, personal comments about author credentials, qualifications, or ability to perform the research are unnecessary. In general, reviewers are expected to be courteous, honest, and professional and avoid being rude, belittling, or condescending toward the authors or the work. Constructive feedback provides an opportunity to make a positive contribution to the work of others.¹⁹ Feedback should be actionable. Instead of stating “The paper needs statistical review,” specific comments are desired such as, “Data in Table 1 are ordinal and should be analyzed using nonparametric methods.” Table 2 provides additional examples for providing actionable comments. Helpful comments can assist authors to improve not only the current manuscript but also future work so that similar errors are avoided in later submissions. Reviewers should avoid being overly focused on grammar and copyediting and should not try to rewrite the manuscript.¹⁷

Table 2.

Generic Versus Actionable Comments

Many journals evaluate reviewers with a scoring system or similar method to assess the overall quality of the review.^14,15,20 The score is usually based on content and timeliness, and the results are often used to determine the status of future invitation. Peer reviewers with good scores may be asked to review more frequently. Reviewers who consistently provide high-quality critiques could potentially be invited to serve on an editorial board.^14,19

Conflict of Interest

Reviewers are obligated to disclose all conflicts of interest, both actual and percieved.^2,13,17 Financial conflicts, personal relationships, affiliations, or other activities that have potential to bias assessment of the manuscript are considered conflicts of interest. Examples include but are not limited to employment related to the manufacturing, sale, or distribution of a product or competitor product; stock or equity interests; intellectual property rights; consultant roles; honoraria; and research funding. Personal relationships with the authors should also be taken into account; for instance, relatives, close friends, mentors/mentees, recent collaborators, or employment within the same institution represent a few of the possible scenarios where it may be unethical for a reviewer to accept an invitation.¹⁷

Typically only relationships that have occurred within the past 2–3 y are required to be disclosed.¹⁷ Disclosure does not infer wrongdoing, but failure to declare conflicts that exist is unethical and is a form of misconduct that impacts the integrity of scientific research and publication.^2,15 When it is unclear if a relationship constitutes disclosure, the reviewer should contact the editor to determine if the association meets criteria as a conflict per the journal’s policy.

How to Perform Peer Review

Review requirements and processes differ among journals, so it is important to be familiar with the particular journal’s instructions. Some journals utilize a scoring rubric, a set of defined questions, other tools to structure the review, or an open format for submitting the report. The review process should begin with reading the entire manuscript at least once before starting the critique. This helps to form an initial impression of the quality of the writing and the flow and organization of the manuscript before focusing on more specific details. Some recommendations have suggested reading the paper, thinking about if for at least a day, and then re-reading before starting the review.⁷

Review Elements

Each section of the manuscript should be evaluated from the title, to the abstract, the body of the paper, tables, figures, and the references. One of the main considerations for reviewing individual sections is inclusion of adequate details to answer the research question. The title of the manuscript should be a concise description that reflects the content of the paper. The abstract should effectively summarize the work and include the main findings. Background information for what is known about the topic and the hypothesis or study rationale should be contained in the introduction.^21,22 The methods section is one of the most important parts of the manuscript because it is used to evaluate validity of the study and to determine creditability of the results.²³ There are multiple components to consider for the methods including study design, ethics statement, materials, procedures, outcome measures, and data analysis. The methods should not contain results.

Results should include the findings of the study without data interpretation. The results section should be clearly presented and report on each outcome identified in the methods but should not include any methods. Tables and figures should be relevant to the study and complement the text rather than present the exact same details. Supplementary files are intended for information that may be too detailed for the main paper. The purpose of the discussion section is to interpret and explain the results.²⁴ It often begins with a summary of the main findings followed by supporting and contrasting literature. Study limitations are also included in the discussion. The discussion should not contain data that were not presented in the results section. The conclusions should be supported by the results and convey the key findings. References should be relevant and up to date.²⁵ Figure 2 includes considerations for reviewing each individual section of a manuscript.

Fig. 2.

Considerations for reviewing each individual section of the manuscript.

Considerations for Review Articles

Review studies have additional considerations for completing peer review depending on the review type. Narrative, scoping, and systematic reviews represent different forms of knowledge synthesis for which each has a specific purpose and utilizes different methodology.^26,27 Narrative reviews are usually written by subject-matter experts and include a comprehensive review of the literature on a given topic that may or may not identify the literature search methods. Scoping reviews are similar to narrative reviews but differ in that formal search methods are used to map and describe the literature for a specific topic.

Primary factors to take into account for narrative and scoping reviews are that the paper presents a balanced view of current evidence and includes all relevant literature. Systematic reviews are considered the standard for synthesizing evidence through rigorous search methods to critically appraise the literature and answer a specific research question.²⁸ Review considerations include adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, clearly described and detailed search methods of relevant databases, risk of bias assessment, and identification of enough evidence to warrant a systematic review.

Review Report Format

Most review platforms include designated sections to provide detailed comments to the authors and separate confidential remarks to the editor for overall evaluation and recommendation. This may be in addition to a scoring rubric or set of questions. General guidance for formatting the review recommends starting the written critique with a short summary of the work followed by a list of major and minor concerns regarding each section of the manuscript.⁶

Comments for authors should not be left blank or merely include a one-sentence recommendation without evaluation of the manuscript. This type of response is not helpful to the editor or the authors. Detailed feedback provides authors with information that can be used to adequately address concerns. Vague statements should be avoided as they are difficult for authors to interpret and formulate a response. Comments and questions should be specific to the identified issues or concerns in the manuscript. Recommendations for revisions should be justified and include the rationale for requested change, and the review details should support the recommendation for publication.

Some reviewers prefer to provide detailed comments for authors in an attached document or annotated copy of the manuscript. When this option is available, it should not be merely a marked-up copy with only grammatical corrections. These types of issues are generally handled and addressed during the copyediting process upon acceptance. Writing styles vary among authors, and reviewers should not attempt to rewrite to a preferred style, but it is reasonable to suggest changes to clarify or better explain a concept.¹⁷ Although reviewers should not be overly focused on grammar or typographical-related errors, it is acceptable to respectfully recommend authors to seek assistance from an experienced science writer who is fluent in the language of the journal. This should only be done if readability is challenging due to language issues and is politely phrased.

Confidential comments to the editor are not shared with authors and can provide further insight regarding quality and relevance of the work or the reasoning behind the recommendation for publication.⁶ This section can also be used to discuss ethical concerns or other potentially sensitive issues regarding the work.²⁹ However, it is not meant to be a platform for disparaging the authors or making unfounded or false accusations since the authors will not have access to comments to the editor.¹⁷

Author Response to Peer Review

Authors have invested substantial time and effort in the manuscript and are often passionate about their work. The decision letter can elicit a variety of emotions ranging from excitement upon manuscript acceptance to apprehension and frustration if the outcome is thought to be unfavorable or the paper was rejected. Authors are cautioned to not react impulsively due to dissatisfaction with the decision and to refrain from responding in a manner that would be regretted later. Instead, authors are encouraged to consider the comprehend, answer, list, and mindful (CALM) approach to responding to peer review.³⁰

CALM Approach

After reading the decision letter for a paper that was rejected or requires major revision, authors should resist the urge to immediately respond. Time should be taken to process the information and allow emotions to subside to be able to better comprehend what the decision actually means. It has been suggested to wait at least one day to gather thoughts on the feedback before proceeding with a response.^12,30 Whether major or minor, revision should be regarded as a positive outcome as it is an opportunity to improve and resubmit the work.¹⁸ It is rare for a manuscript to be accepted on the initial submission. Note that 100% of manuscripts with an invitation to revise that are not revised and resubmitted are not published.

Major revision may require extensive effort, possibly even a complete rewrite of the manuscript, but the potential for acceptance exists if authors are able to make the requested improvements. A rejected manuscript from one journal is not necessarily a terminal outcome. Authors can use the feedback to their advantage to make improvements and submit the work to a different journal.¹² If there are fatal flaws that cannot be resolved, all is still not lost if the experience is regarded as a learning opportunity to not repeat the same mistakes. Occasionally, authors attempt to appeal rejection decisions. Before pursuing an appeal, authors should first determine if the journal has an appeals policy as all decisions are deemed final by some journals whereas others may have a formal process.¹²

After initial sentiments have passed, authors should then consider how to address the identified concerns to formulate a response to the issues and questions raised. The process of revision requires careful contemplation of editor and reviewer comments. Just as reviewers are expected to be kind and courteous, authors should be polite and respectful in the response to reviewers. Refrain from being defensive and argumentative even if the comments seem harsh or inaccurate. Concepts that seem clear to the authors may need clarification for readers to fully understand the intent. Authors should carefully contemplate points of disagreement and choose wisely whether to dispute the issue. If the requested change will not diminish the quality of the manuscript or alter the intended meaning, it is usually best for authors to proceed with the change.^12,18 There may be valid reasons to disagree with a reviewer, but the response must be justified with supporting evidence and approached in a well-mannered fashion.

In addition to making changes to the manuscript, a response to each identified issue is required that includes an explanation for how it was addressed. This is usually accomplished by providing the detailed information either in a document included with the re-submission or entered directly in a designated section on the journal submission site. Creating a list of all comments with an individual response to each issue is recommended to ensure that all points are dealt with in an organized manner and none are overlooked. It is important that authors are clear as to how each concern was addressed and should avoid appearing arrogant or lazy with one-word responses such as, “corrected” or “answered in manuscript.”³⁰ When making the list of comments and the author response, comments from each reviewer and any editor remarks that require response should be copied exactly to facilitate review of the revised manuscript.¹²

The opportunity for revision is usually offered for a limited time frame. Therefore, authors should be mindful of deadlines to submit the revised manuscript. For example, when revisions are offered for papers submitted to Respiratory Care, there is a 6-month period before the original submission is automatically archived in the system. After this window has passed, a new submission might be required if the revision was not received in time. The decision letter usually includes details and instructions for submitting a revision.

The re-submission should include a clean copy and a tracked-changes version of the revised manuscript. This helps make review of the revised manuscript more efficient by showing the changes made and reading the revision without the interruption of the modifications. Some guidance has advocated to include a cover letter with the re-submission that includes a brief statement thanking the editor and reviewers for the opportunity to improve the manuscript.³⁰ If determined the manuscript will be submitted to a different journal, authors should not submit the same version that was rejected but rather take the opportunity to improve the work based on the peer review feedback.¹² Figure 3 includes a summary of considerations for authors responding to peer review.

Fig. 3.

Author considerations for responding to peer reviewer comments.

Summary

Authors, editors, reviewers, journals, and readers are all stakeholders in the peer review process. Editors rely on quality reviewer feedback to guide the editorial decision. Reviewers should evaluate each section of the manuscript and provide constructive comments that authors can use to improve the work. Authors should carefully consider all of the identified concerns and adequately address each individual issue before submitting a revision. A decision of revision should be viewed positively, regardless of if it is major or minor. When the peer review process functions as intended, the outcome is publication of high-quality manuscripts that have been evaluated for scientific rigor and demonstrate validity and reliability.