Cover the eyes of Lady Justice

Summary

An appeal for double-blind peer reviewing

Blind Lady Justice, who dates back to the ancient Greek goddess Themis, has become the symbol for fair and equal justice. No matter how she is painted, drawn or otherwise depicted, her eyes are always covered so that she cannot be influenced by the person being judged and thus does not fall prey to prejudice or corruption. Indeed, the idea of a judicial system that is 'blind' to anything other than facts is an important pillar of democratic societies that aim for fair and equal justice for all their citizens.

The scientific community is a society that should aim for the same high standards of justice to establish equal chances for everybody—at least in those aspects that concern the judgement of scientific contributions from individuals or research groups. In some ways, the publication system in science reflects the judiciary system in democracy, with the reviewers being the judges. Although they do not assess crimes against society, reviewers evaluate scientific achievements. Their verdicts are often important for the future lives and careers of the people being judged, so there is an evident need for a fair reviewing system. But the reality clearly does not match this demand, which puts an undue burden particularly on the younger and not yet established scientists who still need to build their reputation by publishing in high-impact journals. It is therefore necessary to take a

In some ways, the publication system in science reflects the judiciary system in democracy, with the reviewers being the judges

look at the shortcomings of the current system and to reconsider alternatives. In particular, I propose the establishment of a double-blind reviewing system, in which the author of a publication and its reviewer are not aware of each other's identity.

As discussed in detail in a recent commentary by Peter Lawrence (2003), there is tremendous pressure in many areas of research to publish in high-impact journals. This pressure is created by the fact that both positions and funding are very limited, and this in turn puts a heavy burden on many scientists, particularly younger ones. Because the selection criteria for scientists or projects are based mainly on a scientist's impact factor score, many postdoctoral fellows and younger group leaders need publications in high-impact journals to advance their scientific careers. This is not likely to change soon, as there is, at present, no other way to evaluate the quality of a person's work—at least in a semi-objective way.

Some of the limitations that cause this publication pressure are inherent in the way in which science is organized. Why, for instance, is it necessary to keep the majority of younger scientists in an uncertain financial situation by giving them only temporary contracts, whereas most other sectors offer permanent contracts? This does not imply that scientists should have a lifetime guarantee of employment, but providing contracts without a time limit would certainly reduce some of the pressure that leads to unwanted side-effects in our culture of scientific publishing. There are other restrictions that are certainly not that easy to overcome: cutting-edge research is becoming more and more expensive, and both public institutions and private companies that perform research are forced to restrict or focus their budgets to fulfil the obligations and demands of citizens and shareholders, respectively. Thus, even established scientists are competing for limited funding possibilities. This problem is further aggravated by the fact that a large number of graduate students and postdoctoral fellows compete for a disproportionately small number of available positions. All these factors contribute to the pressure that is placed on scientists. And at the end of the day, scientists are judged primarily on their publication records, because there is no other basis for comparison.

And at the end of the day, scientists are judged primarily on their publication records, because there is no other basis for comparison

Of course, scientific publishing is an important part of science—it spreads knowledge and expertise, making it available to a broad spectrum of people. It also prevents important findings and conclusions from being kept secret and motivates scientists to obtain as much information as possible from their research and to critically review their own data. In this respect, scientific publishing adds the important aspects of information transfer and quality control to research that may otherwise only have been performed for the mere satisfaction of an individual's curiosity.

But what is the reality of our scientific publication system, how is it related to the pressure that scientists have to bear and why does it seem to be unfair in some respects? One important point is that almost all reviewers are also authors and referees on committees that make decisions about funding or positions. When we compare this situation to the organization of democracy, with its separation of powers, this would mean that reviewers are the executive, legislative and judiciary authorities all in one—something that modern societies have separated to prevent conflicts of interest. But in our complex world of research, it would be practically impossible to separate these powers, mainly because it is absolutely necessary to do research to be able to judge it.

Another problem—and this is the crucial point—is that reviewers know the names of authors and their institutions, which is in stark contrast to the idea of Blind Lady Justice, who is not able to see the person she must judge. Thus, reviewers are even more prone to conflicts of interest, which arise from their joint function as authors and referees. The overall effect is that the politics of publication have often become more important than the quality of the research itself. Certainly, many reviewers try not to profit from this situation, but it is evident that it is difficult to judge the manuscript of a good friend—or a competitor—in an objective way when the reviewer knows the identity of the author. Sometimes this situation leads to mutual favours in reviewing or to the emergence of scientific lobbies that tend to keep other scientists, newcomers or even new ideas out of their elitist circle. It also leads to the establishment of 'popes' or 'aristocrats' in the scientific community who have a substantial influence in their field, with the power to define dogmas and to dictate to the community what is scientifically fashionable. It is evident that a reviewer cannot easily give a negative report on a manuscript from such an influential scientist, even if the data presented is neither good nor solid.

On the other end of the scientific spectrum, young or as yet unknown scientists have considerable difficulty in getting their manuscripts published in high-impact journals—even if the results are excellent—because the publication capacity of many journals is primarily occupied by the 'science aristocrats' and their friends. This is further aggravated by the importance of the reputation of the institution with which the authors are affiliated. The worst situation that you can have in research is to be a young, unknown scientist in an unknown institute. You might be a genius, but nobody would notice it. Many ambitious scientists, having realized this situation, try to get a position at a famous research institution and work with a famous guru with the aim of optimizing their publication score as fast as possible—sometimes publishing their results before their data are sufficiently solid or confirmed.

...reviewers are the executive, legislative and judiciary authorities all in one—something that modern societies have separated to prevent conflicts of interest

Obviously, this can lead to misinterpretations, oversimplifications and the propagation of immature or even false concepts and conclusions. It is a fact that the results of many projects are published in such a way as to optimize the sum of the impact factors rather than the efficiency and quality of the information transfer. As a result, a large number of rather low-quality manuscripts is published, a phenomenon that will eventually constrict the 'information channels' in science. This problem has become evident in many research fields of late, so that it is actually impossible to read and digest the vast bulk of literature that is published. At the same time, a significant part of the information is immature or even questionable.

What might be a solution to this problem, which is caused, at least in part, by the current publication and reviewing system in science? One possibility would be to make the reviewing process completely transparent, as suggested in a comment last year (Gura, 2002), which means that both authors and reviewers are aware of each other. Although this may bring more straightforwardness into the way in which results and publications are discussed and judged, it is questionable whether established scientists would still be willing to review others' work if they might face potential conflicts with the authors.

The other alternative—namely to set up a double-blind reviewing process, in which the reviewer does not know the identities of the authors—might be more interesting and promising. Analogous to Blind Lady Justice, the reviewer would not be influenced by the identity of the author—something that might have an

The worst situation that you can have in research is to be a young, unknown scientist in an unknown institute. You might be a genius, but nobody would notice it

enormous impact on our current scientific culture. As most journals are now installing online-submission systems, it would be an easy task for the editors to remove the authors' names and institutions. Of course, authors would not be allowed to give away their identity by referring to their own work, for example, by using phrases such as “as we have shown previously” followed by the citation of their publication. This requirement could be clearly stated in the instructions for authors and reviewers, and it should be made clear that any violation of this rule would lead to an immediate rejection of the manuscript.

What might be the effect of such a system? Undoubtedly, many big players in science could no longer rely merely on their good name and reputation, as their manuscripts would be judged according to the quality of research and the innovative ideas they contain. This might downgrade some science aristocrats, whose work is not—or is no longer—proportional to their reputation and influence. More importantly, however, young or as yet unknown scientists would have a much better chance of publishing their work in high-ranking journals. This would help to raise their impact factor score and certainly make it easier for them to get funding for future projects. In principle, such a system would mix up much of the 'social ranking' in science, but in a fair and honest way. Although elitist circles and lobbies will probably always exist, a double-blind reviewing system would make it more difficult for such lobbies to influence or dictate the direction of a whole research field. It would also prevent individual scientists or lobbies from handing down their influence to their students and followers independently of their real scientific skills, a phenomenon of science aristocracy that is frequently observed these days.

It is obvious that such a change in our publication culture will not occur overnight, because most of the important journals would have to switch to a double-blind reviewing system at the same time. This is unlikely to happen, simply because some of the high-ranking journals profit from the current situation. In addition, well-established scientists will not put pressure on journals to change their reviewing process.

However, one approach to starting this process and establishing this new reviewing system might be to generate a section in a journal that contains only articles that are evaluated by double-blind peer review. Thus, a journal's editors would be able to test whether this reviewing system is accepted by the scientific community—both by authors and readers—as indicated by submissions to that specific section and the number of citations of double-blind peer-reviewed articles. Indeed, publishing in such a section might become a new quality criterion for 'publishing by fair means'—even for well-established scientists.

If this is met with broad acceptance, the journal could then switch all of its manuscripts to the new system. This might even strengthen its position in the competitive field of scientific publishing, given that the journal would gain the reputation of its articles being unbiased and unaffected by science politics. This could kick-start a trend and may convince other editors to switch their journals to a double-blind reviewing system. If journal editors or well-established scientists on the editorial board resist such a change in their reviewing system, the scientific community should demand it.

Certainly, there will be no revolution in scientific publishing, but we should call for an evolution. It is time to establish new means of judging scientific contributions in a fair and equal way—something that is important for maintaining the health of the overall 'organism' of science.