The ethics of peer review in bioethics

Abstract

A good deal has been written on the ethics of peer review, especially in the scientific and medical literatures. In contrast, we are unaware of any articles on the ethics of peer review in bioethics. Recognising this gap, we evaluate the extant proposals regarding ethical standards for peer review in general and consider how they apply to bioethics. We argue that scholars have an obligation to perform peer review based on the extent to which they personally benefit from the peer review process. We also argue, contrary to existing proposals and guidelines, that it can be appropriate for peer reviewers to benefit in their own scholarship from the manuscripts they review. With respect to bioethics in particular, we endorse double-blind review and suggest several ways in which the peer review process might be improved.

Commentators have proposed a number of alternatives to the current peer review system.^1–5 One or more of these proposals might turn out to be an improvement on peer review and ultimately may be adopted. Recognising this possibility, peer review currently has, and is likely to continue to have, a crucial role in shaping the academic literature for some time. Acceptance of papers for publication typically requires favourable peer review, and the quality of the published literature depends on comments and recommendations from peer reviewers.

Given this central role, it is not surprising that a good deal has been written on the ethics of peer review. And there have been a number of attempts to develop guidelines regarding the ethics of peer review in science and medicine, as well as peer review in general.^6–10 In contrast, we are unaware of any articles on the ethics of peer review in bioethics. Recognising this gap, we evaluate the extant proposals regarding ethical standards for peer review in general and consider how they apply to bioethics.

We assume that peer review of empirical bioethics research largely should follow guidelines for empirical research in other fields, such as social and behavioural research. Thus, we focus on bioethics manuscripts that are conceptual or normative, rather than empirical. This analysis is intended to cover manuscripts that are submitted to bioethics journals as well as manuscripts in bioethics that are submitted to journals in other fields, such as medicine or science.

BACKGROUND

Current practices of voluntary peer review give substantial power to reviewers with little accountability other than self-assessment. Editors may choose to ignore poor quality reviews and/or decline to solicit future reviews from those who produce obviously poor or biased reviews. However, editors may not have the expertise to judge the quality of a given review. Also, critical feedback to reviewers has the potential to reduce even further the already limited supply of willing peer reviewers.

Financial compensation for peer review would be one way to try to promote greater accountability and improve quality. A survey across a range of disciplines found that a small minority of journals provide financial compensation for peer reviewers.¹¹ However, this is likely to remain a minority practice. With respect to bioethics in particular, we are not aware of any journals that offer compensation to peer reviewers and the relatively small budgets of bioethics journals likely make this unfeasible. This suggests that self-assessment will continue to be the primary mechanism of accountability for peer reviewers.

In this context, it is important to develop standards for what constitutes appropriate peer review in bioethics. Establishment of standards will provide guidance for those who aspire to do a good job, and may make it more difficult for others to produce substandard reviews. The strategy of the present manuscript will be to consider and critically evaluate proposed ethical standards for peer review in general. We then use this analysis to propose a framework for ethical peer review in bioethics.

JOURNAL POLICIES AFFECTING PEER REVIEW

Journals differ in their approach to peer review. Medical journals typically practice single-blind review in which reviewers know the identity of the authors, but not vice versa. Concealing the identities of reviewers is intended to allow them to provide honest feedback without concern that authors may attempt to retaliate against negative reviews. However, it is possible that revealing reviewer identities may encourage more thorough and careful reviews. Arguably, if individuals are required to sign their reviews, they will take greater care in conducting peer reviews. In addition, it has been argued that blinding the identities of reviewers is inconsistent with academic standards for transparency and reduces reviewers’ accountability.^{12, 13}

Consistent with these arguments, the British Medical Journal practices open review in which reviewers are required to sign their reviews.¹⁴ And at least one scientific journal offers authors the option of publishing reviewers’ comments along with their own responses.¹⁵ Including reviewers’ names as part of this process might increase how carefully they conduct their reviews.

On the other hand, this potential benefit may be outweighed by a loss of candour in making critical comments, thus decreasing the quality of peer reviews. This concern seems especially pressing in relatively small fields, including bioethics, where reviewers recognise that next time around it may be the present authors who review their submission. Open review also is likely to discourage junior scholars from undertaking critical peer reviews of the work of prominent scholars for fear that doing so may have a negative impact on their career prospects.

Empirical evaluations have failed to detect significant effects of masking versus revealing reviewers’ identities.^16–18 A systematic review of nine studies found inconsistent effects of masking reviewers’ identities.¹⁹ A study in psychiatry found that signed reviews took longer and were of slightly higher quality.²⁰ Reviewers who signed their reviews were also slightly more likely to recommend acceptance. These findings were similar to those found in other studies.²¹ These data suggest that revealing reviewer identities may slightly increase the quality of reviews, slightly decrease recommendations for rejection and moderately increase the time of reviews.²²

Taken together, these findings do not seem sufficient to determine which approach is best. Hence, this decision will have to be made based on theoretical considerations and evaluation of which concern is more important in the case of specific journals and disciplines: the lack of transparency and accountability that comes with blinding or the possibility that revealing reviewer identities may result in less candid reviews and possible retaliation against those who give negative reviews.²³ With respect to conceptual bioethics, the need for transparency seems less acute than it does in science and medicine. In particular, conceptual arguments should stand on their own. The question for the peer reviewer is whether the analysis is convincing given what is included in the manuscript. And given that bioethics is a relatively small field, concerns regarding lack of candour and the potential for retaliation seem especially important. We conclude that it makes sense to mask the identity of peer reviewers in conceptual bioethics.

The general argument for allowing reviewers to know the identities of authors is that, in medicine and science, it can be important to evaluate whether the authors are qualified to carry out the experiments described in the manuscript. This argument is less compelling for normative and conceptual bioethics. For this type of scholarship, the established expertise of the authors is of little importance. In addition, in disciplines that lack any clear standards on what constitutes a valuable contribution to the literature, there are good reasons to be concerned about bias when peer reviewers know the identity of authors. Peer reviewers who are friendly with the authors may be disposed, even unconsciously, toward favourable reviews.

Unblinded review likely promotes false positive and false negative errors. Distinguished authors may get the benefit of the doubt, while unknown authors may face an unreasonable burden of proof. Open peer review may also lead to gaming the system. Young scholars seeking to establish a professional reputation may seek prestigious collaborators for the sole purpose of enhancing their chances of publication. In the process, an established collaborator may get a free ride without meriting authorship. These concerns are echoed in a survey of authors and reviewers which found that 68% of respondents preferred concealment of authors’ names and 72% preferred concealment of reviewers’ names on the grounds that concealment facilitates fairness and honesty.²⁴ We conclude that double-blind review may be the most appropriate for conceptual research in bioethics, although one study that looked at the impact of masking authors’ identities did not find an impact on the quality of the peer reviews.²⁵

Peer review done right requires significant time and effort. An obvious way to reduce this burden, while maintaining the quality of peer reviews, is to reduce the number of requested reviews. In our 50 years of collective experience reviewing bio-ethics manuscripts, we have been asked to review many manuscripts that clearly did not merit publication. One way to address this concern would be for editors to send out for peer review a smaller percentage of the manuscripts they receive. Rejecting without review 50% or so of submissions will reduce the onus on peer reviewers and also allow authors to more quickly find a suitable place for their work. This approach can work if the journal has a sufficient number of editors who have the expertise to judge the quality of manuscripts in different areas. While this requires securing a larger number of editors, and placing a greater burden on them, being an editor provides significantly greater benefit to individuals than being a peer reviewer. Hence, it seems to make sense to place greater burdens on editors relative to peer reviewers.

Our experience is that it often is comparatively easy for competent scholars to determine their view of the quality of a given manuscript. Specifically, in many cases, a brief review of a manuscript will make clear to a peer reviewer that, in his or her view, it does not merit serious consideration. It is much more difficult to take the time to write out clear and substantive comments. One possibility, then, for journals with limited editorial expertise would be to create a new category of peer review. Two peer reviewers could be asked to read a paper quickly, perhaps within a week, to judge whether it merits detailed review. If the reviewers agree that the manuscript does not merit serious attention, the editors could notify the authors quickly, much in the way that is now done when a manuscript is not sent out for review. The reviewers then could be given more time to write comments for manuscripts that merit review.

IS THERE AN OBLIGATION TO PERFORM PEER REVIEWS?

Scholars rely on peer review both as contributors to and consumers of the literature. Because professional reputations are based in large part on publication in peer reviewed journals, successful scholars benefit from the process of peer review. Additionally, they benefit from constructive comments by peer reviewers which help them improve their work. Finally, as readers of the literature, they also benefit from improvements that are stimulated by thoughtful and comprehensive peer reviews.

Reliance on and benefit from peer review gives rise to an obligation to do one’s fair share in performing peer reviews. As Rawls observed: “We are not to gain from the cooperative labors of others without doing our fair share.”²⁶ Failure to do one’s fair share of peer reviews constitutes a form of free-riding. The obligation in question takes the form of an ‘imperfect’ duty in the sense that scholars have an obligation to do a fair share of peer reviews, but are free to choose when to discharge their obligation to perform peer review.²⁷

Assuming that reliance on and benefit from peer review give rise to the obligation to do one’s fair share, those who rely on and benefit more from peer review should do more peer reviews. Frequent readers of the literature benefit from the peer review process and should be willing to perform occasional reviews. In addition, those who submit a greater number of manuscripts which undergo peer review should do more reviews compared with those who submit fewer manuscripts. It is difficult to justify a precise formula for the number of peer reviews that active scholars should perform. In general, the system works when the number of willing peer reviewers equals the number of needed peer reviews. This suggests that scholars should accept at least enough requests to zero out the demand that they personally put on the system.

One proposal points out that the average submission requires review by an editor and two reviewers, and concludes that scholars should perform three peer reviews for every manuscript of theirs that undergoes peer review.²⁸ Given that editorial positions bring at least some reward of their own in terms of prestige and academic credit, it seems less important to compensate for the efforts of editors. In contrast, it does seem important to compensate for the efforts of volunteer peer reviewers. For sole authored submissions, it makes sense to do approximately two peer reviews in response. For multiple authored submissions, the obligation to do peer reviews decreases as the number of authors per paper increases. For example, a submission with two authors does not require any more peer reviewers than a manuscript with one author. In this case, authors should accept approximately one peer review assignment.

The obligation to perform peer review is not discharged simply by doing the requisite number of reviews per year. Rather, the obligation is to conduct reviews competently and expeditiously. Other than quality, perhaps the greatest complaint with peer review is the time it takes to receive peer reviews following manuscript submission. In bioethics in particular, 6 months or more is not unusual. Peer reviewers can help reduce this time by completing reviews in 2–3 weeks, and regarding 1 month as an outside limit in standard cases. During particularly busy periods, when these deadlines cannot be met, individuals should decline to review, and wait for a less busy time.

While recognising the importance of timely reviews, it is also important to give sufficient time to prepare each review. It is unrealistic to expect that one can understand and provide a thoughtful review based on a single quick reading. In many cases, the best approach may be to read a manuscript once, reflect on the content and then, a short time later, read the manuscript again and write comments.

WHEN TO ACCEPT A REQUEST TO REVIEW

When should a request for peer review be accepted? An obvious consideration is whether the individual has the requisite expertise to undertake a competent review of the manuscript. Requests should be accepted only if one has sufficient expertise regarding the topic in question. In addition, the fact that the duty is imperfect implies that individuals can decline particular requests for personal reasons, impending vacation, significant other work or even trivial reasons, such as insufficient interest in the topic.

More difficult to manage is the issue of potential bias. This problem can be acute because we are not always aware of our own biases, and may overestimate our ability to be objective in the face of factors that promote bias. As a rule, we suggest that individuals decline to review papers that they know or strongly suspect are written by close friends or colleagues. Similarly, it is not always possible to overcome or ignore personal feelings that are either especially positive or especially negative toward a particular author. In these cases, requests should also be declined.

Perhaps the most difficult challenge is deciding how to handle requests that are directly on a topic where one holds strong views; indeed in some cases, a scholar may be working on the very same topic at the time of a request for peer review. Familiarity and strong views on a topic can put one in a position to provide an especially useful review. On the other hand, this position raises the possibility that reviewers may object to a manuscript and recommend rejection simply because it conflicts with their own views.

Some argue that reviewers should decline requests when they are working on the topic in question. For example, the Committee on Publication Ethics (COPE) suggests that individuals should decline to review ‘a manuscript that is very similar to one they have in preparation or under consideration at another journal.’ While this would obviate a source of bias, it seems undesirable as a general rule. Precluding review from all those who are working on a given issue would prevent editors from obtaining input from those who are likely most conversant with the topic in question.

Individuals undertaking peer reviews should always disclose any potential conflicts of interest to the requesting editor and allow him or her to make the final determination. The desire to keep opposing views out of the literature should never be a reason for undertaking a peer review. Similarly, the mere fact that one disagrees with the position endorsed in a manuscript is never in itself a reason to recommend rejection, although the reasons and arguments that one has for one’s view may well be. One of the biggest challenges is to try to ensure that one’s own position on a topic does not influence one’s evaluation of the quality of the arguments that the author makes for an opposing view.

We suggest that scholars should decline peer review when they have previously reviewed the same manuscript for another journal and recommended rejection. This is in contrast to other guidelines. For example, COPE maintains that peer reviewers should ‘review afresh any manuscript they have previously reviewed for another journal as it may have changed between the two submissions and the journals’ criteria for evaluation and acceptance may be different.’ⁱ Even when the paper has been revised substantially, it is reasonable to let others approach it de novo without a preconceived idea about its merits. One reviewer should not be able to determine the ultimate fate of a manuscript. Since explaining that one already recommended rejection of the paper may inappropriately bias the editor against it, it might make sense in this case to simply decline the request without providing a reason.

This general policy admits of at least two exceptions. First, if a reviewer knows that a manuscript has some fatal flaw that may not be known or obvious to others, this would be important information to convey to the editors. Second, reviewers sometimes get requests to review manuscripts they previously rejected by journals that publish essentially all submissions and use peer review only to improve manuscript quality. In this case, it does not seem problematic to accept the second review given that one is not controlling whether the manuscript is published, and given that one’s prior exposure to the manuscript may put one in a good position to suggest ways to improve it.

STANDARDS FOR PEER REVIEW

Empirical evaluations of different methods to teach peer review, mostly with respect to reviewing for medical journals, have found only very modest improvements in quality.^{29, 30} For example, one study evaluated the impact of attendance at a training workshop or receipt of training materials.³¹ The study found only slight increases in quality over the short term. Another study that used a similar intervention found only very modest impact of training on the detection of errors by reviewers.³² Perhaps most importantly, and somewhat surprisingly, a study of more extensive and individual training by high-quality senior reviewers also did not improve the quality of mentees’ future reviews.³³

These studies do not find significant improvement as the result of training peer reviewers. This may be because peer review is simply too difficult a skill to impart to others. Another possible explanation is that the existing data are confounded by the fact that teaching peer review is difficult, as is measuring the quality of peer reviews.^{34, 35} Given these possible limitations with the existing literature, and given the possible importance of even small increases in quality, it seems to make sense to encourage training.³⁶ Established scholars thus should consider doing occasional peer reviews along with a fellow or intern. This can be an opportunity to help train the next generation in the important task of conducting effective peer reviews.³⁷

Peer reviewers should notify journal editors when they plan to invite a trainee to help review a manuscript so that the editors can keep track of precisely who has viewed the work. In general, including one more person in the process should not be a problem given that several reviewers and several editors typically review most manuscripts, and it is difficult to see how one more person reviewing the manuscript will be problematic. This of course assumes that the trainee is well trained in the ethics of reviewing manuscripts, including the importance of maintaining confidentiality. Individuals who do not have access to experienced peer reviewers may consult guidelines that have been developed on how to write an effective peer review.³⁸

It might seem to go without saying that ‘the golden rule’ applies to peer review and peer reviewers should ‘read their comments and consider what their reaction would be if they received the comments.’³⁹ However, the situation of power without external accountability makes it all too easy to undertake review of others’ scholarship without considering whether one would regard the comments as just, accurate and helpful if they were directed to his or her own work. Beyond reciprocity, peer reviewers should be guided by an understanding of what constitutes a competent peer review. What are the criteria of adequacy and accuracy that should govern peer review?

To answer this question it helps to reflect on the functions of peer review.⁴⁰ Peer review serves a gate keeping and a quality improvement function. Peer reviewers recommend that manuscripts are accepted as is (rarely), accepted pending minor revision, accepted pending substantial revision, rejected and reconsidered pending substantial revision, or rejected. As in many areas of assessment, peer reviewers can err by making false positive and false negative judgments. They can recommend acceptance of a manuscript that deserves to be rejected or rejection of a manuscript that merits acceptance. It is unclear whether one of these types of error is more problematic, all things considered, than the other.

If manuscripts that warrant rejection are accepted for publication, the ‘damage’ is done and unlikely to be undone, except in the rare cases of subsequent retraction owing to research misconduct. Publication of articles without merit wastes space within a journal and accords credit for publication that is undeserved. Readers may waste time reading articles that are not worth reading. Papers that make seriously misguided recommendations for practice or policy may have negative consequences. Rejecting papers that merit publication deprives the scholarly community of valuable contributions, and deserving scholars may fail to get due recognition. This may only be temporary in disciplines, including bioethics, that offer multiple publication targets.

As indicated above, it is often relatively easy to arrive at a general sense of the quality of a manuscript. For those that the reviewer clearly thinks do not merit publication, even with substantial revisions, it may suffice to state that the manuscript does not make a valuable contribution to the literature. Thoughtful comments may encourage authors to systematically rethink what they have written, with the potential for leading to an eventually publishable paper or at least to improving their performance on subsequent papers. Though less fateful than the gate keeping function, the quality improvement function is more demanding with respect to time and attention.

Peer reviewers should take care in the way they formulate their comments.⁴¹ While sharp criticism may be warranted, critical comments are likely to be taken less seriously by authors when they are accompanied by gratuitously irritating remarks. In addition to not rejecting a manuscript simply because one does not agree with the conclusion, peer reviewers should not recommend rejection in light of minor flaws and concerns that could easily be addressed in a revised version. When peer reviewers do not understand a particular argument, they might consider asking a colleague for input, although in this case, as in the case of training, peer reviewers should disclose to the editor the names of any colleagues with whom the manuscript has been discussed.

BENEFITS AND ABUSES OF PEER REVIEW

If peer review is accepted as a duty, it is typically regarded as an onerous one without reward. Yet, peer reviewers likely benefit to a certain extent from the opportunity to exercise skills of critical appraisal and reasoning. In addition, peer review offers the rewards of being helpful to others, with the added merit that it is typically done anonymously.

Most commentators argue that reviewers should never benefit professionally. This position is illustrated by the COPE guidelines on the ethics of peer review which maintain that peer reviewers should ‘not use information obtained during the peer-review process for their own or any other person’s or organisation’s advantage, or to disadvantage or discredit.’ This policy makes sense given the possibilities for abuse. In particular, when journals conceal the identities of reviewers, the reviewers may feel free to plagiarise material that they have reviewed. While the editors know the reviewers’ identities, the possibility that an editor may identify a case of plagiarism seems remote; hence, it may not serve as much of a deterrent.

Recognising the very real potential for abuse, the principle that peer reviewers should never benefit in terms of their own scholarship seems impossible to follow. Peer reviewers may derive benefit in ways they do not even recognise.⁴² Recognising this possibility, one might modify the principle to state that peer reviewers should not benefit intentionally. Even this seems unfeasible. Imagine that the arguments presented in the manuscript under review convince the reviewer that his or her own current line of argument or analysis will not be successful. It makes little sense to argue that the reviewer should not then knowingly modify his or her own research.

Reading how others approach an issue also may spark ideas on one’s own work in the area. Similarly, one might benefit by discovering references that one was not aware of. Finally, peer review provides the opportunity to learn about manuscripts that are likely to be published in the near future. This information may provide peer reviewers with the opportunity to be the first ones to publish responses to the manuscript. This occurs formally when editors invite peer reviewers to submit commentaries on manuscripts they have reviewed. It also can occur informally when peer reviewers begin work on a response after completing a peer review. Given that there are so few tangible rewards from peer review, it seems reasonable for reviewers to benefit in these ways, as long as they do not unfairly take advantage of the work of others. Indeed, making explicit the ways in which peer reviewers can benefit appropriately provides the basis for clarifying the ways in which peer reviewers should not benefit.

Policemen who enter a person’s home during the course of their work may benefit by taking some ideas on how to remodel their own home. This seems ethically unproblematic. They should not benefit by taking the owner’s possessions. Similarly, while reviewers may benefit by getting ideas to pursue in their own work, they should not take the ideas of the authors and claim them for their own. With respect to systematic reviews, gathering the references is much of the product. Reviewers thus should not inordinately take advantage in their own work of the list of references compiled in a manuscript under review.

Another potentially abusive practice is recommending that the author of a manuscript cite one or more articles of the peer reviewer. Certainly, there may be occasions where the peer reviewer’s published work is directly relevant to the research under review and the lack of attention to this work may constitute a limitation of the work. It is one thing to recommend that the author(s) look at an article written by the peer reviewer with respect to a particular comment in the review; it is another to state that this work should be cited. Because scholars are evaluated in terms of the number of citations that their work receives in the published literature, the power of peer review can be unfairly turned to the professional advantage of the reviewers if they stipulate that the authors include citations to their own work to benefit the reviewer.

Finally, and perhaps most obviously, abuses of peer review may occur because academic scholarship is competitive. Publication of someone else’s work may make one’s own work less attractive, less timely, less original and less interesting to other journals. Unfortunately, there are documented cases of peer reviewers recommending rejection to reduce competition for their own work, or intentionally taking a long time to complete a review in order to provide themselves more time to get their own work published first.⁴³ More generally, one’s standing in the field is judged in part in comparison with the standing of others. Thus, less success on the part of one’s competitors can redound to one’s own benefit. In addition to intentional instances, this can occur when one unintentionally downgrades the work of a competitor. At a minimum, reviewers should disclose to the editors the extent to which they are working on similar issues.

CONCLUSIONS

Peer review currently is critical to ensuring the vitality of the bioethics literature. However, peer review raises a number of important ethical issues that deserve careful attention. A framework for addressing these issues is needed. We have argued that such a framework should provide guidance on the responsibility to undertake peer reviews, how to conduct a peer review effectively and accurately, how to manage reviewers’ conflicts of interest, and the ways in which it is appropriate and inappropriate for individuals to benefit personally from the reviews they conduct.