Perceptions of Ethical Problems with Scientific Journal Peer Review: An Exploratory Study

Abstract

This article reports the results of an anonymous survey of researchers at a government research institution concerning their perceptions about ethical problems with journal peer review. Incompetent review was the most common ethical problem reported by the respondents, with 61.8% (SE = 3.3%) claiming to have experienced this at some point during peer review. Bias (50.5%, SE = 3.4%) was the next most common problem. About 22.7% (SE = 2.8%) of respondents said that a reviewer had required them to include unnecessary references to his/her publication(s), 17.7% (SE = 2.6%) said that comments from reviewers had included personal attacks, and 9.6% (SE = 2.0%) stated that reviewers had delayed publication to publish a paper on the same topic. Two of the most serious violations of peer review ethics, breach of confidentiality (6.8%, SE = 1.7%) and using ideas, data, or methods without permission (5%, SE = 1.5%) were perceived less often than the other problems. We recommend that other investigators follow up on our exploratory research with additional studies on the ethics of peer review.

Introduction

Critics of scientific journal peer review have claimed that ethical transgressions, such a breach of confidentiality, theft of ideas, personal attacks, and bias, frequently undermine the quality and integrity of the review process [1–4]. A number of different solutions to these problems have been proposed, including using open (or unmasked) peer review, providing additional education and training for reviewers, disclosing conflicts of interest, and developing codes of ethics for reviewers and editors [1–9]. Although most scientists agree that ethical problems can occur in journal peer review, evidence has been anecdotal, consisting of personal accounts published in news stories, letters, or commentaries [3, 4, 10, 11,12]. In this article, we report the results of an exploratory survey of scientists’ perceptions of ethical problems with journal peer review.

Materials and Methods

The aim of this study was to describe researchers’ perceptions about ethical problems with journal peer review. We conducted an anonymous survey of researchers, research staff, post-doctoral trainees and technicians, at the National Institute of Environmental Health Sciences (NIEHS), a branch of the National Institutes of Health (NIH), located at Research Triangle Park, NC. The survey was distributed at mandatory training sessions on Responsible Conduct of Research, during the fall of 2006. Those who chose to take the survey had the option of turning it in at the end of the training session or sending it by mail after the training session. The survey asked for demographic information, such as age and education, and included questions about researchers’ perceptions of the journal peer review process. The survey asked respondents to answer “yes,” “no,“ or “don’t know” to specific questions. Responses to questions were analyzed using standard logistic regression analysis in SAS version 9.1, to determine the contributions of age, gender, number of publications, and position. The NIH Office of Human Subjects Research categorized the survey as exempt research.

Results

About 283 out of the 556 people who received the survey completed it (50.9% response rate). Since the purpose of this study was describe the perceptions of NIEHS researchers with the journal peer review system, in our data analysis we omitted 25 respondents who had not published any papers, as well as contractors, graduate students, respondents in non-research positions, and those who did not identify their position. This reduced our sample to 220 respondents. This sample included 94 post-doctoral fellows, 55 staff scientists, 38 principal investigators, and 33 technicians. Of these 220, there were 119 men, 97 women, and 4 with gender unreported. The average age of respondents was 42 years (SD = 11 years) and the average number of publications by the respondents was 34.7 (SD = 54.7) Respondents came from 22 different disciplines in the biomedical sciences, such as, biochemistry, cell biology, developmental biology, epidemiology, genetics, neuroscience, pharmacology, physiology, reproductive biology, and toxicology.

We focused on responses to the following seven survey questions (see Appendix) in our analysis: (i) Comments from reviewers included personal attacks, (ii) A reviewer was incompetent (which we defined in our survey as “he/she did not carefully read the article, was not familiar with the subject matter, or made mistakes of fact or reasoning in his/her review“), (iii) A reviewer was biased (which we defined in our survey as “didn’t give an article a fair hearing, prejudged it”), (iv) A reviewer breeched the confidentiality of the article without your permission, (v) A reviewer used your ideas, data, or methods without your permission, (vi) A reviewer delayed the review so that he/she could publish an article on the same topic, (vii) A reviewer required you to include unnecessary references to his/her publication(s). In each case the responses were measured as “Yes”, “No” or “Do not know”. In the statistical analysis we combined the categories “No” and “Do not know“. We compared the four groups of researchers after adjusting for their age, gender and number of publications. We converted the number of publications into three categories so that each category had roughly an equal proportion of respondents. The three categories were: up to five publications (70 respondents), 6–20 publications (80 respondents), and more than 20 publications (70 respondents).

Table 1 summarizes the aggregate data from our survey. Incompetent review was the most common ethical problem perceived by the respondents, with 61.8% (SE = 3.3%) claiming that it happened at some point during peer review. Bias (50.5%, SE = 3.4%) was the next most common problem. About 22.7% (SE = 2.8%) of respondents said that a reviewer had required them to include unnecessary references to his/her publication(s), 17.7% (SE = 2.6%) said that comments from reviewers had included personal attacks, and 9.6% (SE = 2.0%) stated that reviewers had delayed publication to publish a paper on the same topic. Two of the most serious violations of peer review ethics, breach of confidentiality (6.8%, SE = 1.7%) and using ideas, data, or methods without permission (5%, SE = 1.5%) were perceived to be rarer than the other problems.

Table 1.

Journal peer review survey data

Have any of the following ever happened to you during the peer review process?
	% Yes	SE
A reviewer was incompetent	61.8	3.3
A reviewer was biased	50.5	3.4
A reviewer required you to include unnecessary references to his/her publication(s)	22.7	2.8
Comments from reviewers included personal attacks	17.7	2.6
A reviewer delayed the review so that he/she could publish an article on the same topic	9.6	2.0
A reviewer breeched confidentiality	6.8	1.7
A reviewer used your ideas, data, or methods without your permission	4.5	1.5

When asked about their perceptions concerning personal attacks, we found a significant age effect; the older the person, the more likely he/she would claim to have experienced at least one personal attack in his/her research career (P = 0.006). There were also differences between researchers in various positions (P = 0.0006), with post-doctoral fellows and principle investigators having greater negative perceptions of peer review (e.g. incompetent review, biased review) than staff scientists and technicians. We speculate that this difference could possibly be due to the fact that principle investigators and post-doctoral fellows are more likely to be either the first and senior authors, respectively, on a paper and hence are more likely to be aware of and sensitive to reviewers’ comments. On the question of perceptions of reviewer’s incompetence, we again observed a significant difference between researchers in various positions (P = 0.01), with post-doctoral fellows perceiving the highest level of incompetence compared to all other positions (P < 0.05). We did not find significant differences between the two genders on any of the seven questions considered in this analysis (P > 0.05). Further, with regards to questions (iv) through (vii), we found no significant association between the responses and age, position, or number of publications (P > 0.05). However, researchers with more publications were more likely to claim to have experienced incompetent review (P < 0.0001) or biased review (P = 0.04). We speculate that the perceptions of bias or incompetence increase with the number of publications because the person who has published more papers has had more opportunities to receive comments that he/she perceives to be biased or incompetent.

Discussion

One of the significant limitations of our study is that it is an opinion survey and does not provide objective data on ethical problems with journal peer review. There is no way to independently verify whether the problems respondents perceived with peer review actually occurred. Indeed, some of the perceived problems may simply reflect scientists’ frustration with having manuscripts rejected or critiqued. For example, answering “yes” to the question “a reviewer was biased” may result from the respondent’s own displeasure with the reviewer’s comments or recommendations. A person who answers “yes” to the question “a reviewer breeched confidentiality” may have little factual basis for this assertion, since it is often difficult to know whether confidentiality has been broken. While we recognize that this is a significant limitation of our research, we think that our study identifies some areas of concern among researchers and suggests some directions for additional investigation. Our research can stimulate discussion and debate, but clearly more work needs to be done.

We would like to point out, however, that documenting that scientists perceive that there are ethical problems with journal peer review can be an important finding in its own right, because a scientist may change his/her behavior in response to what he/she perceives to be a problem. A researcher who is concerned that his/her ideas will be stolen, for example, may not disclose all the information that is needed to repeat his/her experiments [4, 10]. A researcher who is concerned that a reviewer is incompetent or biased may choose to ignore the reviewer’s comments rather than address the concerns (which may in fact be valid), especially if they involve further time and effort in the laboratory. Additional studies can help determine whether scientists’ perceptions of ethical problems with journal peer review influence their behavior.

Another important limitation of our survey is that the sample may not represent all biomedical researchers, since the participants came from the same research institution. Thus, the researchers who work at different institutions or in different sectors of the science economy (i.e. academia or industry) may have different perceptions of journal peer review. Although we recognize this as an important limitation, we hypothesize that the researchers in the sample are not very different from researchers elsewhere, since they submit to the same journals used by other researchers, they have similar education and training, and many of them have worked at other institutions. Certainly, it would be useful for other investigators to follow-up on our research with a national (or international) survey. Our study represents exploratory work and suggests avenues for further inquiry.

A third limitation of our study is that it did not provide an estimation of the frequency of perceived ethical problems in peer review, since it did not ask respondents how often the problems they experienced had occurred. Thus, even though 61.8% of respondents had received incompetent reviews at some point in their careers, the actual rate of incompetence might be much lower. Again, we recognize this limitation but still maintain that the survey provides useful information, because it is important to know whether researchers have perceived problems with peer review, since these perceptions can have an important impact on their behavior and their attitudes toward peer review. Some ethical transgressions, such as breach of confidentiality or the use of ideas without permission, can have such a negative impact on peer review that it is important to document that scientists perceive that these problems occur. Additional research can focus on establishing an objective, quantifiable estimate of the frequency of these ethical problems in peer review.

As mentioned earlier, commentators have made various proposals for reforming the peer review procedures used by scientific journals. Our survey provides support for these reforms, since it demonstrates that biomedical researchers perceive that there are some problems with the integrity and quality of peer review. Since incompetence and bias were by far the most frequent problems respondents claimed to have experienced during peer review, journals, research institutions, and scientific societies should consider ways of dealing with these problems, such as providing additional education and training for reviewers on the scientific and ethical standards for peer review, requiring reviewers to disclose conflicts of interest, and paying more careful attention to the selection of reviewers. A more radical reform, such as open review, may be needed to counteract the perceptions of the most serious violations of peer review ethics, such as breach of confidentiality and use of data, methods, or ideas without permission [3, 13].

Appendix: Survey Questions used in this Study

1. Approximately how many articles have you published in peer-reviewed scientific or professional journals?

   Have any of the following ever happened to you during the peer review process:

2. The review period took longer than 6 months.

3. The review period took longer than a year.

4. Comments from reviewers included personal attacks.

5. A reviewer was incompetent (i.e. he/she did not carefully read the article, was not familiar with the subject matter, or made mistakes of fact or reasoning in his/her review).

6. A reviewer was biased (i.e. didn’t give an article a fair hearing, prejudged it).

7. A reviewer breeched the confidentiality of the article without your permission.

8. A reviewer used your ideas, data, or methods without your permission.

9. A reviewer delayed the review so that he/she could publish an article on the same topic.

10. A reviewer required you to include unnecessary references to his/her publication(s).