A call for civility in peer review

“Criticism, like rain, should be gentle enough to nourish a [person]’s growth without destroying [their] roots.”¹

– Frank A. Clark

Peer reviewers play a crucial role in advancing science.¹ They can meaningfully enhance the rigor and impact of published research through their comments to authors,² while also donating countless hours³ to assisting editors in making publication decisions, identifying ethical issues, and contributing to the dissemination of scientific knowledge. However, concerns have grown about the quality and professionalism of peer review, particularly in light of an increasing number of submissions and journal outlets; reduced time and availability of reviewers; and the impact of artificial intelligence in scholarly writing, including reviewer reports.^2,4,5 In a study of 1491 sets of reviewer comments from two journals, 12% contained at least one unprofessional, personal comment about the author(s) and 41% contained an incomplete, inaccurate, or unsubstantiated critique (IIUC).⁶ Another study of 850 reviewer criticisms submitted to the tumblr site shitmyreviewerssay⁷ found that 24.5% were personal criticisms of the author(s).²

Although the majority of Sleep Health reviewer reports are positive and constructive, we have noticed an increasing number of reviewer reports with unprofessional comments and IIUCs in our roles as editor in chief (OMB), senior associate statistical editor (MLW), senior journal assistant (AMA), and associate editor (AAW), and more broadly as authors and reviewers in the field. Unfortunately, whereas most journals and/or publishers enforce their clearly posted author guidelines, with recent expansions and much-needed guidance on respectful, antiracist,^8-10 and inclusive language,¹¹ few outlets provide or enforce similar guidelines for reviewers. Further, graduate programs rarely include training on peer review and providing feedback more generally. Consistent with the Committee on Publishing Ethics (COPE), Sleep Health’s publisher Elsevier encourages reviewers to “…generally treat authors and their work as they would like to be treated themselves and to observe good reviewing etiquette” when providing feedback to the authors and editor(s).¹ COPE’s ethical guidelines for peer reviewers also stipulate that reviewers should avoid “unfair negative comments or unjustified criticisms” in their feedback, and, when providing feedback on language and style, ensure communication is respectful and sensitive to variation in authors’ style and linguistic experiences (e.g., writing in a language that is not their first).¹² COPE additionally recommends that journals communicate clear policies on unacceptable language and content in reviewer reports, including the extent to which these reviews can be modified by the editorial team.¹²

Sleep Health associate editors are trained to identify such reviewer reports, notify the overseeing senior associate editors and/or editor-in-chief, and edit the reviewer reports as necessary. An important next step in promoting professionalism in peer review is to formally communicate guidance for peer reviewers and authors on unacceptable language and content in reviewer reports. Drawing on recommendations from COPE and Elsevier, this editorial provides guidance and “best practices” to help prevent unprofessional comments and/or IIUCs.⁶ This guidance, along with examples of anonymized unacceptable review comments and recommendations for subsequent action(s) by authors and editorial team members, will appear on the journal’s website. Our intent is not to diminish the rigor of reviewer critiques, shield authors from feedback about conceptual and methodological flaws in their work, compromise associate editors’ autonomy in evaluating reviewer reports, or cause additional burden for editorial team members. Instead, we aim to support reviewers, editors, and authors in professional dialog and constructive criticism.

The language we use to communicate with one another can be helpful or detrimental to the peer review process and the journal’s peer review ecosystem—and particular care is needed for international journals, where language barriers only increase the chance of misunderstandings. It is also important to note that sparse, incomplete, and/or overly effusive reviewer reports are just as unhelpful as those with inappropriate content. Below we list “best practices” for peer reviewers, contrasted with inappropriate elements we have noticed in reviewer reports. These example reviewer comments have been modified to maintain confidentiality.

Evaluate the work (not the authors).

Reviewers are encouraged to focus on the work’s scientific merits, rather than the (assumed) merits of the authors. To this end, reviewers should avoid writing as if they are speaking to an author (e.g., “Your introduction has just been poorly written”).² Comments that are disparaging (e.g., “It seems that the authors were out of touch with current practice and had no reservations whatsoever about insulting clinicians”) or accusatory (e.g., “…inaccurate testing and scoring in your study is unethical and unlawful data collection”) must be tempered and, wherever reasonable, treated as a need for additional information and or identification of underlying assumptions.

Use respectful, inclusive, and antiracist language.

Reviewers should be aware of and adhere to the same standards as authors for use of respectful, antiracist, and inclusive language.^8-11 In cases of limited awareness (e.g., “You write that race and ethnicity were examined as socio-political constructs, how exactly did you do that?”), guidance around these standards can be provided. Expressions of bias, including microaggressions¹³ related to the authors’ perceived race and ethnicity, country of origin, gender, and other identity features, have no place in reviewer reports. Unfortunately, they can and do appear in reviews (e.g., in a comment to an author who self-identified as Black: “So do only Blacks need a positive racial identity? This is troubling and the authors need to consult with their Black colleagues to see how offensive this paper is.”). Bias can also manifest in more positive language and review content based on authors’ names¹⁴ and/or prestigious university affiliations, further perpetuating inequities in academia.

Stick to the facts, avoiding emotional arguments and negative assumptions about the authors.

Critiques should focus on the aspects of the manuscript that are factually correct or incorrect, and incorporate citations to support the critique when possible. In contrast, emotional arguments reflect how the science makes the reviewer feel, rather than the science itself (e.g., “It makes me cringe to think of this nonrigorous approach being used in clinical practice.”). Such comments can be inflammatory and detract from the intellectual rigor of an important critique. If the manuscript lacks some information, approach the review from the perspective that the author(s) may have already considered including the necessary information, but chose not to, due to limited word count or differing perspectives about scientific writing. Making negative assumptions about authors (e.g., “I’d encourage the authors to do some research about this topic as it is directly relevant to the validity of their conclusions.”) or their work (e.g., “You didn’t bother to correct for multiple comparisons, are you comfortable reporting results that are statistically insignificant under the pretense of exploration?”) does not promote scientific civility.

In the spirit of an open-minded assumption that everyone’s right, but nobody has all of the information, at least when everyone involved is well-intentioned, unstated assumptions will differ between reviewers and authors. As Carlo Rovelli eloquently argues in his biography of Anaximander, “Scientific knowledge is the process of continuously modifying and improving our conception of the world, selectively and constantly questioning the assumptions and beliefs on which it is based, searching for modifications that prove to be more effective.” (p. 112).¹⁵ Identifying, illuminating and working through the shared, not shared, or unidentified assumptions or theories within a critique or manuscript can often underlie transformative discoveries. “…science is a process that builds continuously upon existing theories—that is, upon existing cumulated knowledge—but continuously revises this knowledge, keeping the possibility open of questioning any aspect of it, including the general rules of thinking that appear to be most certain and beyond question” (p. 121).¹⁵

Include constructive feedback.

Lengthy criticism without suggestions for improvement can also herald a lack of humility and rigor in the review. Extended critical review text often lacks clarity about facts and assumptions, omits citations supporting the reviewer’s claims, and frequently requires that editors verify claims. Wherever possible, reviewers should provide suggestions (e.g., “More justification for this study’s contribution is needed, given extensive prior research on insomnia and psychological problems”), rather than posing questions without constructive feedback (e.g., “It is no surprise that insomnia is linked with psychological problems. So, what is the contribution of this study?”). Providing citations to support statements about perceived flaws is also more helpful than making generalizations without sufficient details or evidence (e.g., “This is an incorrect way of analyzing data and misreports the study outcomes.”).

“Don’t let the perfect be the enemy of the good.”

We should all work to make sure that only accurate, well-done science is published. However, it is important to balance this with what is possible within the context of one paper, written by one research team, within the constraints of one journal. Reviewer reports that encourage authors to acknowledge and expand on the limitations of their work are more helpful than reviews that expound upon perceived flaws and failures.

Finally, it is important for all members of the scientific community to recognize that, while science itself is fighting for respect in our society, the peer review process has become increasingly fraught. Authors are under ever-increasing pressure to publish. Reviewers face competing demands, with little incentive to engage in uncompensated peer review. Editors are increasingly challenged to balance demands in a way that satisfies authors and reviewers, while ensuring that only accurate, meaningful work is published. Publication timelines only get longer, as researchers’ available time dwindles. These stressors can be even more salient for early career researchers and those who are marginalized due to racial and ethnic minoritization and/or lower-resource contexts. While editorial teams can help reduce reviewer burden and minimize the potential for negative and/or inappropriate comments by making the reviewer process as straightforward as possible (e.g., limiting the number of questions reviewers must respond to) and ensuring that only high-quality papers are sent for review, direct reviewer guidance to enhance the rigor and quality of reviewer reports is also needed. Instead of adding to burnout and cynicism, let us all endeavor to provide constructive peer reviews that inspire authors and other reviewers to not only improve the current paper, but to continue looking for scientific truth. At the end of the day, we all share the same goal of contributing to knowledge.