Preprints versus peer-reviewed articles – Author's reply

Accustomed to seeing scientific discoveries presented in peer-reviewed articles, stakeholders have wondered how to weigh evidence reported in preprints. One way to measure the robustness of preprinted evidence is to compare data reported in a preprint with its published version. Accordingly, we matched preprints to keywords related to four epidemiological measurements: COVID-19 incidence, infection fatality rate, case fatality rate, and R₀.¹ Preprinted data were highly stable compared with their published versions. We concluded that our findings, alongside others’, support the use of evidence reported in preprints in decision making, as one component of the biomedical literature.

Kent Anderson conducts a follow-up analysis, poses an alternative interpretation, and raises a concern about data quality. The concern noted that while we reported 100 matched preprints, there are only 96 unique article identifiers. As we stated, we analysed article–keyword matches to ensure that data were central enough to be mentioned in the title or abstract. It is possible for articles to match more than one keyword, and this did occur at a low frequency (<5%). As we noted in the Methods, these multiple matches were analysed separately. This design choice facilitated comparisons of different types of epidemiological estimate, but entailed that the number of unique article identifiers is slightly lower than the number of article–keyword matches.

Anderson posits that some of the stability that we noted might be due to authors posting manuscripts after acceptance, potentially minimising differences between preprint and published versions by definition. Based on this comparison of preprint posting and article acceptance dates, up to around 20% of preprints could in principle be accepted versions. As we noted in our Discussion,¹ our study was not a measurement of the marginal effects of peer review on manuscripts, but rather a measurement of the expected change from preprint to publication version, because it is not knowable how much peer review (either for gatekeeping at journals or for improvement from informal peer feedback) has occurred before preprint posting. Nor is it knowable how much observed change was driven by reviewer feedback versus being author-initiated. This follow-up analysis reinforces our findings of high stability of data in preprints through to the published version, and proposes one small but notable potential contributor to that observation. Taken together, these results reinforce our interpretation that the use of evidence presented in preprints in decision making is warranted, as one valuable component of the biomedical literature.

I declare no competing interests.