How to write an effective journal peer review using a staged writing approach: a best-practice guide for early-career researchers

Abstract

Journal peer review is a gatekeeper in the scientific process, determining which papers are published in academic journals. It also supports authors in improving their papers before they go to press. Training for early-career researchers on how to conduct a high-quality peer review is scarce, however, and there are concerns about the quality of peer review in the health sciences. Standardized training and guidance may help reviewers to improve the quality of their feedback. In this paper, we approach peer review as a staged writing activity and apply writing process best practices to help early-career researchers and others learn to create a comprehensive and respectful peer-review report. The writing stages of reading, planning and composing are reflected in our three-step peer-review process. The first step involves reading the entire manuscript to get a sense of the paper as a whole. The second step is to comprehensive evaluate the paper. The third step, of writing the review, emphasizes a respectful tone, providing feedback that motivates revision as well as balance in pointing out strengths and making suggestions. Detailed checklists that are provided in the Supplementary material (available as Supplementary data at IJE online) aid in the paper evaluation process and examples demonstrate points about writing an effective review.

Key Messages.

• Peer review is essential to the scientific process but there is a gap in the training of early-career researchers to create a high-quality peer-review report.

• We present a three-step peer-review process: (1) reading the entire manuscript without commenting to get a sense of the paper as a whole; (2) comprehensively evaluating the paper, aided by our checklists and strategies; (3) writing the review with clarity, a respectful tone and balance in pointing out strengths and making suggestions.

• Our process incorporates critical stages of the writing process: reading, planning and composing. Dedicated attention to each stage will help reviewers to fully engage in creating a more thoughtful, rigorous report.

Introduction

Journal peer review is a critical part of the scientific process. Peer reviewers judge which papers merit publication based on the soundness of the methods and other criteria, such as novelty and significance.¹ For papers that do advance to publication, peer review provides a final opportunity for improvements, such as the addition of key details to help readers to understand how a study was conducted or the acknowledgement of limitations that provide context for interpreting their findings.¹ Peer review also offers an opportunity for reviewers to enforce or challenge norms in the field (e.g. standardization of terminology, use of person-first language). On the other hand, poor-quality reviews can create frustration, reduce confidence in the scientific system, encourage stagnation through resistance to new ideas, allow errors or unreproducible or otherwise poor-quality research into the literature and generate barriers for historically marginalized groups, all of which contribute to growing concerns about the quality of peer review in the health sciences.^2–4

Unfortunately, graduate programmes in the health sciences typically include little to no training on how to write a high-quality peer-review report^5–7 and early-career research epidemiologists are often without the skills that are needed to write a high-quality peer-review report. Additionally, although some studies have suggested that previous interventions did not meaningfully improve the quality or completeness of peer-review reports,⁸ others indicate that training, standardized guidance or checklists may help reviewers to improve the quality of their feedback.^5,9–12 In this article, we aim to help early-career researchers—and others who are interested in improving their peer-review practice—to learn to create a comprehensive and respectful peer review by framing this endeavour fundamentally as a ‘writing activity’ and leveraging best practices in the staged writing process. Specifically, we streamline the writing process into three stages, each of which requires specific attention: reading, content planning and composing.^13–16 In the context of peer review, we argue that the three stages are: (1) reading the complete manuscript to understand the paper as a whole; (2) evaluating the paper comprehensively, aided by our checklists and strategies; and (3) writing the review with clarity, a respectful tone and balance in pointing out strengths and making suggestions. Our approach is of particular relevance to early-career researchers in the sciences because explicit, high-quality writing instruction is not always available.¹⁷ In particular, there may be a lack of emphasis on the role of reading and planning in writing.^18,19

Learning to write an effective peer review offers many benefits. Alongside contributions to the scientific community, peer review helps to build reviewers’ own careers by enhancing their professional skills and fulfilling required service obligations.²⁰ Importantly, peer review helps reviewers to improve their own writing skills.^10,21,22 In fact, some research suggests that reviewers actually learn more from writing a peer-review report than from receiving feedback on their own work.^21,22 Finally, learning a sound journal peer-review approach will also help to guide other types of peer review, such as informal peer reviews for colleagues and classmates, and even grant application reviews.

Several new models of peer review have emerged in recent years, some in response to flaws in the traditional peer-review system.^3,5 Newer models include review of preprints through a preprint service such as bioRxiv²³ and post-publication peer review (including e.g. PubPeer²⁴ and F1000Research²⁵) in which readers provide comments on published articles.²⁶ We developed the approach that is presented here primarily for traditional peer review, including single anonymized (the reviewer’s identity is not revealed), double anonymized (neither the reviewers’ nor the authors’ identities are revealed to one another) and open (published publicly) journal peer review. However, the process that we present will also be helpful for other models of review.

Prior to agreeing to review an article, it is critical to carefully evaluate the journal that has issued the invitation to avoid engaging with predatory practices. We provide a link to an open-access resource²⁷ at the end of the Supplementary material (available as Supplementary data at IJE online) to guide this investigation. Additionally, you should consider whether the review invitation is a good match for your expertise, both methodologically and in terms of content area. Some journals may designate a statistical reviewer for a dedicated evaluation of the methodological approach. Consider, too, whether you have the capacity to complete the review within the requested time frame (reported average time spent on reviews typically range from 5 to 8 h²⁸). Early-career researchers may also consider whether the journal will allow you to discuss the paper with a mentor. Once you have accepted a reviewer assignment, we recommend that you should approach your review by following these steps.

Step 1: Read the entire paper to understand it as a whole

It can be tempting to start writing a review during our initial read, as we encounter and react to sentences, paragraphs and individual tables or figures. However, it is important to start by reviewing an entire piece from beginning to end, including a scan of the tables and figures, before making any assessments. Completing this step before moving to Step 2 will help you to understand the paper as a whole and identify patterns that need to be addressed. Taking notes as you read may help you to keep track of your initial reactions and can help you to remember important concerns as you move to the next step.

Step 2: Comprehensively evaluate the paper before writing your review

Once you have a sense of the paper in its entirety, you will be poised to comprehensively evaluate it. During this stage, maintain your focus on evaluation rather than formulating suggestions. Start with any section but review the entire paper in detail, including the introduction, methods, results, discussion, abstract and title as well as references cited throughout the paper. Use the checklists provided in Supplementary Tables S1–S6 (available as Supplementary data at IJE online) to guide your evaluation. Each table provides a checklist and descriptions of key components for a given section of the paper, in addition to questions to guide your evaluation.

Supplementary Table S1 (available as Supplementary data at IJE online) provides a checklist of components to evaluate for the introduction section, including whether the authors describe the problem that the research seeks to address and what is known and not known (the research gap) about the topic under study, in addition to the objectives of the study. Supplementary Table S2 (available as Supplementary data at IJE online) offers a checklist of method components, guiding you to review descriptions of the study design, recruitment, data collection, ethical approval, analytic variables and laboratory and data analysis, and addressing confounding and bias. A results checklist is displayed in Supplementary Table S3 (available as Supplementary data at IJE online), prompting consideration of the results of recruitment efforts, participant characteristics, main text results in addition to references to tables and figures. Consideration of the tables and figures is supported by Supplementary Table S4 (available as Supplementary data at IJE online), including an evaluation of the necessity for each and the need for each to be understandable without having to refer to the text. Supplementary Table S5 (available as Supplementary data at IJE online) guides evaluation of the discussion section components, including a summary of the main findings, a description of the significance of the results, mechanisms of action, and strengths and limitations. Finally, key points for good titles and abstracts are identified in Supplementary Table S6 (available as Supplementary data at IJE online).

Although the checklists are geared towards original research studies with quantitative outcomes, they can be adjusted for other study types and for studies with a qualitative component. Given the health sciences focus, the checklists include attention to epidemiologic considerations such as the appropriateness of the study design, identification of potential biases and the treatment of potential confounders. Add your own criteria as needed and consider areas of strength in addition to your critiques. We also recommend that you consult the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network’s open resource reporting guidelines.²⁹ Observational studies should follow the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines, including specific versions for cohort, cross-sectional and case–control studies. Additionally, if the journal provides reporting guidelines or checklists (not all do), review them at this stage.^30,31

Step 3: Craft a respectful review with actionable comments

In this step, write your review based on your assessment from Step 2 and translate your appraisals into actionable feedback. The below subsections offer strategies for completing this step.

Familiarize yourself with the journal’s peer-review guidelines and required format

Start by identifying the format that the journal requires for peer reviews. Most use an unstructured format (in which a peer reviewer creates their own document with numbered comments) but some journals ask for structured responses or feedback on specific prompts. It can be frustrating to invest time in writing an unstructured review only to realize that the journal requires a specific format.

Start with a summary

Start by summarizing the paper briefly. Although some may skip this step because they feel it is unnecessary or time-consuming, summarizing the paper helps you process and crystallize what you consider to be the main message of the paper instead of starting with the details. Your summary signals to the editor that you understand the paper well enough to provide a high-quality review and it helps the authors because it conveys your understanding as a reader. If your understanding does not match the authors’ intentions, then that is helpful information for them. The summary is also a good place in which to highlight the overall strengths of the paper, which might otherwise get lost. If you think that it is helpful to include your positionality (e.g. your general research area or methodological expertise) to provide context for your comments, particularly for journals with broad audiences, you can do so here. Box 1 shows good examples of initial summaries, including the purpose of the study, the design and sample size, in addition to the main findings. Note that these summaries need only be a few sentences long.

Box 1.

Three good examples of reviewer summaries of a paper^a

1. This paper describes an analysis of the risk factors for severe disease in a retrospective cohort of 928 children <5 years of age in Vietnam who tested positive for SARS-CoV-2. The study design and data are strong. The abstract, introduction and most of the methods are clear and, in these sections, I found that most of my questions were answered within a few lines. However, there are a few weaknesses in the statistical analysis. I offer the following comments in the spirit of seeing the best version of this paper. For context, I have no specific expertise in SARS-CoV-2 but I have run similar studies for gastrointestinal infections.

2. This manuscript presents an analysis of the initiation rates of cigarettes and e-cigarettes among a cohort of 12 408 youth and young adults in the UK. Understanding initiation rates in this age group is important and the UK context is particularly significant given the approach that the country has taken to regulating e-cigarettes compared with the USA and several other high-income countries. The paper is well written. However, there are some current weaknesses, particularly in the analysis plan, that I think could be addressed.

3. The authors describe a pilot-scale (N = 115), mixed-methods study that describes a text-message-based health-behaviour intervention to reduce drug use relapse in people who have recently completed a residential drug treatment programme. The authors have convincingly articulated a clear need for an intervention in this population and the methods are strong. However, I have reservations about the significance of this study because of the low effect size and lack of clear generalizability. I also recommend copy editing to improve the clarity of the text.

^aThese examples were invented by the authors to demonstrate the points that we make in the paper.

Prioritize your suggestions

Organizing your review into major and minor comments helps the authors to quickly understand the scope and nature of your concerns and where to start with their revisions. (This approach also helps the editor to quickly get a sense of the major issues that need to be addressed.) Authors may become aware of additional issues after considering a major comment. For example, if the sample size is not reported consistently, then the authors may identify additional issues through the process of investigating the discrepancy. Additionally, changes in response to a major comment may render minor comments moot. Additionally, a review may feel less overwhelming when the bigger issues are presented separately rather than encountering a long list that mixes smaller issues with larger concerns. Box 2 shows examples of review excerpts that prioritize major and minor comments for the author.

Box 2.

Three good example overviews of major and minor comments^a

1. I have three major concerns, described in more detail below, regarding (1) the analysis plan, (2) the potential for unaccounted for confounding and (3) the appropriateness of the conclusions given the results. I also have a few minor comments to help to clarify some aspects of the text.

2. Despite the strength of the study design, I felt that the research question could have been better motivated—I did not have a strong understanding of why this specific study was needed to advance the field. Below, I discuss my concern in greater detail and offer some possible ways in which the authors could respond. Below that, I include minor suggestions, including for an additional analysis. In the last section, I list several typos that the authors do not need to respond to individually.

3. It seems to me that the authors have adjusted for a collider (in the epidemiologic sense) in their multivariable statistical analysis. Disease severity (a covariate that was included in the model) is causally downstream of both the patient’s race (the exposure of interest) and by the biomarker-based disease phenotype (the outcome of interest). Accordingly, their effect estimates will be biased. That adjustment is a major flaw in this analysis. Fortunately, this concern is addressable by removing that covariate from the model. I am not going to provide detailed comments on the discussion until the model is rerun because the conclusions are likely to change Additionally, I have included some minor line edits in the introduction and methods, and some suggestions on the presentation of results in the figures.

^aThese examples were invented by the authors to demonstrate the points that we make in the paper.

Use a commenting approach that is tailored to the type of comment and the outcome that you want from the author

Write your review with intentionality and language that helps to motivate positive changes. There are many ways to frame a comment and one framing approach may be more effective than others for certain situations. Consider what a successful response to your comment might look like and frame your comment to elicit the changes that you would like to see.

For example, in the case in which information is missing, a direct suggestion may be best, such as:

To make your methods more complete, describe your participant recruitment approach, including the recruitment location, a description of those who recruited participants, and the circumstances under which participants were recruited. Explain how potential participants were approached and any incentives that were provided.

If a passage is not clear, then it may not be possible to make a specific suggestion. Instead, you may need to respond as a reader³²: ‘I am not clear on what you are saying in this sentence’, or to ask a question: ‘Did all 63 participants complete this part of the study?’

In other cases, it is important to explain why you are making a query or suggestion. For example, you might say:

Your introduction does not point out why it is important for post-partum women to know the different options for family planning. Although the value of such knowledge might seem obvious, research shows that there is often a disconnect between health knowledge and behaviours. In light of this, I suggest that you explain why knowledge is valuable and support these claims with relevant citations.

It is difficult for authors to respond to vague comments such as ‘This paper did not cite major relevant articles in the field’, so keep your comments as specific as possible. The provision of suggested resources can sometimes help authors to address your comments. These can include pertinent citations, databases and other resources. Good practice is the suggestion of open-access resources to ensure that authors may access them without financial barriers. Finally, you should not use a review as an opportunity to request that the authors cite your own work unless it is directly relevant.

Write for accessibility across cultures, languages and disciplines

A great benefit of participation in the peer-review system is the insights and perspectives that we gain from those outside our immediate collaborators and networks. However, communicating with those outside our discipline and culture can present challenges. Use simple language, avoiding idioms and jargon as much as possible. If you do use jargon that may be unfamiliar to the authors, explain it.

Maintain an upbeat, encouraging stance and provide actionable feedback

In graduate school, most of us are taught to ‘critique’ papers but relaying feedback to authors is very different from writing a critique. A peer-review report should be written to be clearly understood by the authors and should feel respectful and supportive, even if you recommend rejection of the paper.³³ Although your review may or may not be anonymous, depending on the journal, it is best practice to write as though your name was attached to the review.⁸ If an author feels supported, it may help them to feel less defensive and encourage them to respond more fully to your concerns.³⁴ For example, starting your review with an upbeat phrase such as ‘My comments are offered in the spirit of supporting the best version of this paper’, as in the first example in Box 1, can help you to set a positive tone.

Maintain this supportive tone throughout your review. Avoid words such as ‘sloppy’ or other words that may be interpreted as judgmental or harsh. In addition to being demoralizing, this type of comment is not specific enough to help authors to know what they need to do to revise their paper.³⁴ Instead, use constructive language with specific suggestions. For characterizing problems that need to be addressed, you can use neutral terms such as ‘unclear’ or ‘incomplete’, as necessary. Preferred language, however, includes phrases that focus on how to improve a paper such as: ‘To strengthen this paragraph, ….’

Additionally, the use of complete sentences rather than phrases may also help you to avoid unintentionally coming across as harsh. A comment such as ‘sample size?’ may feel abrupt and even unclear because of its brevity. Instead, write a complete sentence that describes an issue as well as your advice to address it, if relevant. For example, you might say:

The final sample size was reported as 631 in the abstract but was not reported in the results section of the manuscript. Be sure that the final sample size is included in the main paper and that it is consistent with the abstract.

Remember, you are more likely to see a revision that addresses your concerns if you explain them clearly.

Where possible, limit each comment to one issue to help authors to fully appreciate each point and avoid having an author miss one of your comments. Provide evidence-based feedback where possible, including citations as appropriate, and, if your comment is based on opinion, be transparent about it.³⁵ It can be helpful to number the comments. Be sure to reference the paragraph or line numbers to help the author to locate the area of concern.

Balance suggestions with positive feedback

Although most of your feedback will necessarily be suggestions, incorporate some positive feedback to share with the authors what they did well. Although the intention behind a generic comment such as ‘This is a good start’ or ‘Nice work’ is commendable, the key to effective positive feedback is to point out specific strengths. The example that is shown in Box 3 does a nice job of this. Note that a description of the strengths of a paper need not be long or elaborate.

Box 3.

Example of a review that cites specific strengths^a

Rapid economic development such as is occurring in China, Brazil, India and South Africa brings opportunities to reduce the rate of poverty and increase investment in health services. At the same time, it is often accompanied by a surge in alcohol (and tobacco consumption) as citizens gain more disposable income, social mores change and the multinational alcohol industry begins marketing and selling their products. From the point of view of alcohol problems, developing nations are thus an important focus in efforts to reduce the global burden of disease. This study is thus focused on a highly important issue.

An additional virtue of this study is that large sample community surveys are challenging in any event, but particularly so in rural, low- and middle-income regions of the world. The results presented are far more likely to give a reliable picture of the situation in rural India than are the more commonly conducted opportunistic and clinical small sample studies.

The report benefits as well from being sensitive to the possibilities that stigma has generated under-reporting and that unwillingness to seek treatment may stem in part from their being little treatment to seek. These acknowledgements in the discussion of the interpretational challenges here are consistent with the commendably modest tone of the paper in making claims.

^aThis example is reproduced verbatim from an open peer review of: Rathod et al., Epidemiologic features of alcohol use in rural India: a population-based cross-sectional study.³⁶

Copy editing and language use

Copy editing is not the responsibility of the reviewer. It is fine to point out a few typos to the authors as you come across them but do not spend your review time on the correction of spelling and grammar errors. Relatedly, for English-language manuscripts, do not presume the authors’ English-language status or assume that a paper that requires language editing has poor science.^37,38 In most cases, the simple statement ‘I recommend copy editing’ will suffice (as in the third example in Box 1), letting you focus on reviewing the scientific aspects of the paper. If there are places where language errors have made the text unclear, ask for clarification.

Suspicion of ethics violations

Instances of data fabrication, plagiarism and use of generative artificial intelligence (AI) are increasingly entering the public conversation. Although it is not the responsibility of the peer reviewer to detect such issues, you should be aware of and report any suspicions to the editor. Avoid uploading the paper that you are reviewing to online tools, as that would be a breach of confidentiality. More on the role of peer reviewers and suspicions of ethics violations can be found in COPE’s (Committee on Publication Ethics) open-access Ethical Guidelines for Peer Review.³³

Comments to the editor

The online interface through which you provide your reviewer report features a comment box for confidential comments to the editor that authors will not see. Although you will not always have comments for the editor, this box is a good place to let the editor know about certain types of issues. One such comment might note that there were aspects of the paper that you were not as comfortable to review. For example, if you did not conduct a rigorous review of the statistical approach, then consider sharing that with the editor. Additionally, comments regarding review decisions (e.g. ‘I’m on the fence about rejection vs major revision’) should be directed to the editor only—not to the authors. If you consulted anyone else in preparing your review (see more discussion below), that should also be reported to the editor confidentially.

Final steps before submitting your review

Be sure to proofread your review before submitting it. Typos may confuse authors or even change the meaning of your comment. Additionally, be prepared to thoughtfully respond to questions that the journal may ask of reviewers. Above, we discussed structured reviews but, even if the journal uses unstructured reviews, you may be asked to rate the paper in terms of originality, clarity, interest to the field or other metrics.

Reviewing a revised paper

If you are asked to review a revised paper, you will see the authors’ responses to all reviewer reports. Your main objective is to determine whether the authors adequately responded to your concerns. The authors may not reply as you wanted but their response may still be sufficient to address the concern. If some of the changes are not sufficient, then you may explain why they do not address your concerns and offer further suggestions on how to do this. Multiple rounds of responses to reviewers without meaningful change are not productive and, if major concerns are not addressed, consider recommending that the editor reject the paper. When reviewing a revision, it is appropriate (but not necessary) to comment on how well the authors responded to other reviewers’ comments, as it is part of the overall revision.

We each bring different strengths to a review

Writing a review can feel intimidating, particularly for less experienced reviewers. Some may feel that they lack knowledge or insights in a particular area or that they might be missing something in their assessment. In reality, no review is perfect. This is why the peer-review system typically includes two to five reviews for each manuscript. It is important to recognize that each of us brings different strengths to our reviews. Although some may be content experts, others bring expertise in a methodological approach. Your specific expertise may be captured in a positionality statement to the authors, if you included one, or in your comments to the editor (e.g. ‘I work in this area, so I feel confident about my comments in the introduction and discussion. However, I am not familiar with these specific methods, so am I not confident in my ability to review them rigorously’).

Take advantage of key opportunities for improving your peer-review skills. First, reading the other reviewers’ comments is an excellent way to build your skills and confidence. Second, less experienced researchers may collaborate with their mentor on reviews.^11,39,40 Such collaborations are an excellent way to learn content, conventions and professional norms.^39,40 Although this type of collaborative writing represents a valuable learning opportunity, a few professional considerations should be kept in mind. All contributors should agree to keep the manuscript confidential and avoid entering the manuscript text into online interfaces, including generative AI. Mentors should consider asking their mentees to participate in a confidentiality agreement that makes these expectations explicit and formalized. Invited reviewers should be transparent about additional contributors by noting them in the confidential comments to the editor when the peer review is submitted. In open reviews (those in which peer reviewer names are published), all contributors should be given credit. To encourage such practice, journal editors may articulate a policy that formally allows co-participation, such as: ‘If you consult with anyone on this review (e.g. trainees), please include their name in your comments to the editor.’ Finally, journals with open peer review provide publicly available examples from which to learn. Guidance on accessing these and other helpful resources is listed at the end of the Supplementary material (available as Supplementary data at IJE online).

By following the approach that we have outlined in this article, you will improve the peer-review process for both yourself and the authors, increasing the likelihood of the publication of high-quality, revised papers. By committing to a respectful and encouraging tone in your reviews, you will make the peer-review process a more pleasant experience for all involved and invite the authors to do the same when they review papers. In this way, you are maximizing the impact of your time and effort, and your service to science and the scientific community.

Ethics approval

Ethical approval was not necessary for this paper because we did not include data.

Supplementary data

Supplementary data are available at IJE online.

Author contributions

E.A. conceived of the paper and wrote the initial draft. A.F.B. created the examples in Boxes 1 and 2, composed key checklist text and critically revised the entire paper and Supplementary material.

Use of artificial intelligence (AI) tools

Artificial intelligence was not used in any capacity for this paper.

Funding

No specific funding was received for this work. E.A. was supported by the Center for International Reproductive Health Training at the University of Michigan (CIRHT-UM).

Conflict of interest

None declared.