Siler et al. (1) argue that “editors and peer reviewers generally—but not always—make good decisions regarding the identification and promotion of quality in scientific manuscripts.” Here, we provide a new perspective on the ongoing discussion regarding the quality of peer review and editing processes and discuss how the effectiveness of editors and peer reviewers can be measured. As well, we use a tool to analyze temporal trends in the effectiveness of the editing process and propose the use of a standardized corrigendum/erratum index as a surrogate of quality in scientific manuscripts.
Given that peer reviews and editing are fundamental for detecting mistakes and improving the quality of published scientific papers (2, 3), scientific journals such as Nature, Science, and PNAS, three of the most demanding of all journals publishing scientific research, could be used as good estimators and trend indicators of quality standards. These three high-quality journals are good examples of the rigor of published papers and of the effectiveness of peer reviewing and editing. We propose that an estimate of the annual number of items published by a journal divided by the number of corrections published could be used as a standardized index and as a surrogate of scientific and editorial quality, whose temporal trends could then be evaluated.
According to the Web of Science, in the period of 1993–2013, the total number of items published in Nature, Science, and PNAS was 59,849, 55,421, and 70,066, respectively. During the same period, the number of corrections published by these journals was, respectively, 1,353 (2.26%), 1,868 (3.37%), and 1,778 (2.54%). In all cases, the annual trend in the number of errata identified during this period has increased substantially (an average of 0.09%/y in Nature, 0.06%/y in Science, and 0.06%/y in PNAS). In other words, the number of mistakes in these journals, although still relatively small, is increasing progressively, independently of the number of items published.
The proposed index thus suggests that, albeit slowly, the quality of these journals (but not their contents) is decreasing. It is difficult to disentangle the factors that explain this trend, but it is likely that the pressure to publish reduces the time spent by authors in careful revisions of their manuscripts and that the review and editing process are both in part responsible. A more detailed analysis would allow us to identify the most common type of mistakes made by authors that are not identified by reviewers and editors. This information could thus enable us in the future to all but eradicate the most habitual mistakes and thus improve the scientific quality of published papers.
Footnotes
The authors declare no conflict of interest.
References
- 1.Siler K, Lee K, Bero L. Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci USA. 2015;112(2):360–365. doi: 10.1073/pnas.1418218112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Thomas SP. Conceptual debates and empirical evidence about the peer review process for scholarly journals. J Prof Nurs. 2011;27(3):168–173. doi: 10.1016/j.profnurs.2010.09.015. [DOI] [PubMed] [Google Scholar]
- 3.Park IU, Peacey MW, Munafò MR. Modelling the effects of subjective and objective decision making in scientific peer review. Nature. 2014;506(7486):93–96. doi: 10.1038/nature12786. [DOI] [PubMed] [Google Scholar]