Evidence on peer review—scientific quality control or smokescreen?

Peer review—the process by which experts advise editors on the value of scientific manuscripts submitted for publication—is traditionally surrounded by an almost religious mystique. Published papers are an important part of most assessment systems that decide how academic posts and research grants are distributed. Peer review confers legitimacy not only on scientific journals and the papers they publish but on the people who publish them. But if peer review is so central to the process by which scientific knowledge becomes canonised, it is ironic that science has little to say about whether it works.

Editors have described peer review as “indispensable for the progress of biomedical science.”¹ They argue that peer review helps them distinguish between good and bad papers and between good and bad research, that it improves the presentation of what is being published, and even that it educates editors and authors.² When they ask reviewers to comment on a paper’s scientific reliability, originality, relevance, appropriateness to the journal, and other matters, editors hope they are providing some kind of intellectual quality control, allowing the best science to be selected and improved. But is this belief more than just wishful thinking and self aggrandisement by editors and other beneficiaries of the peer review system? The question is all the more relevant because peer review is so time consuming, complex, expensive, and prone to abuse.³

Summary points

• Blinding reviewers to the author’s identity does not usefully improve the quality of reviews

• Passing reviewers’ comments to their co-reviewers has no effect on quality of review

• Reviewers aged under 40 and those trained in epidemiology or statistics wrote reviews of slightly better quality

• Appreciable bias and parochialism have been found in the peer review system

• Developing an instrument to measure manuscript quality is the greatest challenge

The evidence so far

In 1990, at the first international congress on biomedical peer review, some editors began to examine critically their own activities.⁴,⁵ The most recent insights into what, if anything, is achieved by peer review and how it might be improved were presented at the third such congress in Prague last autumn and were brought together in the July 15 issue of JAMA.⁶

Before 1990, most articles on biomedical peer review reported descriptive or observational studies. Many were heavy on opinion and speculation and light on evidence. Some speculated that blinding reviewers to authors’ identity (“blinding”), asking reviewers to sign their reviews (“signing”), or passing the comments of one reviewer to other reviewers (“unmasking”) might improve the quality of reviews by increasing objectivity and eliminating prejudice and bias. Others said that there might be special characteristics associated with high quality reviews, such as age, seniority, holding an academic post, or having published widely. There were numerous anecdotes of biases and of abuses of the peer review system.⁷,⁸

Blinding, signing, and unmasking

McNutt et al were the first to use a randomised controlled trial to examine the issues of blinding and signing.⁹ In their 1990 study of 127 consecutive manuscripts submitted to an American internal medicine journal, blinding increased the quality of reviews in a statistically significant way, though the improvement failed to reach their predefined threshold for administrative significance. The authors found no association between signing of reviews and review quality. Limitations of this trial were its small size, the specialist nature of the journal, the fact that reviewers were not randomly assigned to signing or not signing their reviews, the lack of a previously validated instrument for assessing review quality, and inability to exclude a “Hawthorn effect” (the possibility that reviewers’ behaviour changed merely as a result of being studied).

In the most rigorous investigation of the question so far, published in the 15 July issue of JAMA, Van Rooyen et al were unable to confirm the effect of blinding on the quality of review.¹⁰ They randomised 527 consecutive manuscripts submitted to the BMJ with regard to whether the reviewers were masked or unmasked to other reviewers, or unaware that a study was taking place. The latter group allowed a Hawthorn effect to be excluded. Two reviewers for each manuscript were randomised to receive either a blinded or an unblinded version. But when review quality was measured with a validated instrument, neither blinding nor masking was found to make an important editorial difference to the quality of the review. A smaller trial conducted across a range of American journals found a similar lack of effect.¹¹ A third study conducted partly at smaller, specialist journals showed that masking, even were it to improve review quality, is not always possible, and that reviewers guess the authors’ identity correctly in around 40% of cases.¹²

The most recent piece of research, published this week and from the BMJ (p 23), shows that making the reviewer’s identity known to authors had no effect on quality.¹³

Reviewer characteristics

In a separate analysis of data from the same trial, Black et al found that the characteristics of reviewers, such as demographic factors, specialty, seniority, or academic appointments, had little association with the quality of the reviews they produced, explaining only 8% of review quality.¹⁴ A logistic regression analysis found that training in epidemiology and statistics, and younger age, were the only characteristics significantly associated with higher quality ratings. Paradoxically, membership of an editorial board was associated with lower, not higher, review quality.

Bias

Other contributors to JAMA’s issue on peer review illustrate the worrying number of biases by which peer review is beset, including nationality bias,¹⁵ language bias,¹⁶ specialty bias,¹⁷ and perhaps even gender bias,¹⁸ as well as the recognised bias toward the publication of positive results.^19–21

Major challenges

For all the progress that has been made since 1990, some of the most important questions remain unanswered. The greatest challenge for peer review researchers is perhaps the quest for an instrument capable of measuring the most interesting and least accessible outcome of all—manuscript quality. Up to now, researchers have had little choice but to study the intermediate outcome of review quality; but to discover whether peer review is an effective intervention, we want to be able to trace its effects on the manuscript.

The other major challenge is obtaining funding for this new area of research, which falls outside the sphere of interest of almost all grant giving bodies. Much of the research so far has been conducted “on a shoestring,” using small, one-off grants and time borrowed from researchers’ other, paid commitments.

Where does this leave us?

Researchers have an interest in knowing about the fairness of the systems by which their research is judged. If the peer review process should turn out to be worthless or, worse still, hopelessly corrupt, researchers would be better off committing their findings to the internet. Meanwhile, it may be some small comfort to those who conduct research and submit papers to journals that editors, forced to grapple with the challenges of designing their own trials, are now receiving a salutary taste of their own medicine.

Editorial by Smith Papers p 23