Can we trust peer-reviewed science?

Discussion is heating up regarding honesty and effectiveness in our system of doing science and communicating it. Refereed journal publications are the main mechanism by which research output is evaluated and presented or not presented to the world and individuals or groups frequently assess how well the system is working. These assessments assume greater importance when the research environment and the methods of journal publication have undergone major changes. Two relevant changes are the massive increase in the number of researchers and the proliferation of for-profit, online only, open access (OA) publishers. As an example, 1 publisher in this category started in 2010 but lists 120 journals on its website. An academic librarian, Jeffrey Beall (1) has a website that lists “questionable, scholarly open-access publishers,” which are considered to be probably predatory; there are presently 471 journals on this list.

Peer review is central to the assessment of quality of research that is published in scientific journals. However, a recent article in The Economist(2) cites evidence that the system is not working well. The authors of this article believe that there are problems of poorly conducted studies, unsatisfactory analysis, and inadequate review. One piece of evidence cited refers to reports of failure of independent researchers to replicate studies published in refereed journals. They noted that researchers at an American pharmaceutical company were able to reproduce the published results in only 6 of 53 important basic science studies on cancer and that other researchers at a German pharmaceutical company could reproduce results in only about 25% of 67 major studies related to drug discovery. Concerns were also raised about weaknesses in statistical analyses of data, a common problem being inadequate statistical power. For example, a power of 0.8 is accepted to be a reasonable basis for testing hypotheses, but in one field the typical statistical power achieved by researchers was said to be 0.21, and in another, 0.35. Low statistical power increases the chances of a conclusion that there is no difference between the groups being examined based on the statistical results (P > 0.05), when in fact a real difference does exist and there was inadequate statistical power to detect it.

Inadequate peer review is blamed for much of the perceived state of science publication. The Economist(2) placed considerable emphasis on a sting operation carried out by science reporter John Bohannon. He created a fake paper with serious flaws, including lack of controls, and submitted it to 304 peer reviewed, open access journals, nearly half of which were on Beall’s list of disreputable science journals. The fact that 157 of the journals accepted the article was used as a basis for decrying the state of peer review in open access science journals. The Directory of Open Access Journals (DOAJ) (3) drew 3 conclusions from the sting operation: “there are predatory publishers, there are problems with peer review, some of the exposed journals are indexed in DOAJ.” In response, the DOAJ removed 114 journals from its directory.

Several of the points raised in The Economist are worth closer examination. First, let us consider the matter of irreproducibility of experiments. These failures must be due to either fraud or inability to precisely reproduce the studies (if a study is truly replicated then the results must be the same as in the original). It is difficult to believe that 75 to 89% of the studies investigated by the 2 drug companies were fraudulent. It is more likely that inability to obtain the published results of the study is most frequently due to difficulties in replication of experiments (4). It is amazing how some unrecognized aspect of the replication attempt that is not identical to that in the original can lead to a different result.

Next is the matter of poor statistical power. The Economist certainly has a point here — many published studies have low statistical power. Researchers cite limitations imposed by funding and by unavailability of cases as reasons for low statistical power. Nonetheless, this is an area that needs to be addressed by researchers, although it is worth noting that many studies do not require statistical analysis.

Then there is the matter of the weak peer review system. The overall poor results of peer review (apparently no peer review in some cases) are highlighted in the Bohannon sting operation. This is partly due to the journals that were selected but it has also been shown that non-open-access journals with high reputation and very high impact factor, such as Science, also occasionally publish and highlight terribly flawed studies. Reasons suggested for this state of affairs include: journals favoring what appears to be dramatic new findings, lack of interest in funding or publishing studies that replicate previous work, overworked academic reviewers, increasing specialization resulting in small pools of experts, reliance on the assessment of only 2 or 3 persons, the rise of journals that will publish almost anything for a fee. Interestingly, shoddy work often does get exposed after publication.

Sensationalism has resulted in exaggeration of the weaknesses of peer review, which appears to work well in most cases. However, there are ways in which our present peer review system can be improved. Some journals have put in place protocols designed to do just that. Others are replacing pre-publication review with post-publication review. I will discuss these developments in my next editorial.