Effect and Reach of Medical Articles Posted on Preprint Servers During the COVID-19 Pandemic

Abstract

This survey study examines the reach of studies about therapies for COVID-19 that were posted on preprint servers.

There are ongoing controversies about the evaluation and dissemination of medical and scientific research.^1,2,3 Preprint servers offer researchers the opportunity to post manuscripts before publication in peer-reviewed journals and regardless of whether they have been submitted to journals for review. The global coronavirus disease 2019 (COVID-19) pandemic has focused attention on the timely reporting and dissemination of research findings, and the respective roles of preprint servers and traditional peer-reviewed journals. We compared the effect and reach of studies about therapies for COVID-19 posted on the medRxiv preprint server, subsequent publications in medical journals of some of these studies, and journal articles that were not posted on either medRxiv or another preprint server.

Methods

We identified preprints and peer-reviewed publications of studies about clinical outcomes of proposed COVID-19 treatments, including chloroquine and hydroxychloroquine, lopinavir-ritonavir, angiotensin receptor blockers, angiotensin-converting enzyme inhibitors, convalescent plasma, and corticosteroids. The institutional review board at New York Medical College deemed the study as non–human participants research. We included articles in English posted at medrxiv.org or published between February 1 and May 10, 2020, as well as publications from these articles in medical journals between February 1 and July 10, 2020. We obtained data on attention and online engagement for each article as a preprint, and if published as a journal article, from February 1 to July 10, 2020. The data included the Altmetric attention score and the number of news outlets citing the article, individuals posting about the article on Twitter, page views, and scientific citations. Full details of the methods and search strategy are provided in the Supplement. To compare data between preprints and publications, we used bootstrapping to conduct nonparametric tests on the difference in medians. Statistical analyses were conducted using SPSS, version 26 (IBM), and statistical significance was set at P < .05.

Results

Between February 1 and May 10, 2020, we identified 45 publications in peer-reviewed journals that met the search criteria and had not been posted as preprints on medRxiv and 17 that had previously been posted; 1 preprint led to 2 publications. In addition, we identified 18 preprints on medRxiv that had not been published in a journal by the cutoff date of July 10, 2020. Table 1 shows the attention and online engagement metrics for the 34 preprints and the 62 publications. Although the publications had significantly more citations than the preprints (median [interquartile range] of 22 [4.3-52.5] vs 5.5 [1.3-20.3] citations; P = .01), there were no significant differences for the other metrics. There were also wide variations in the metrics between individual studies. The mean impact factor of the journals in which COVID-19–related articles were published was similar to that for articles that had been posted as preprints and those that had not (12.3 [95% CI, 6.2-18.4] vs 13.8 [95% CI, 7.6-20.0]; P = .20).

Table 1. Attention and Online Engagement Metrics for Preprints and Publications About COVID-19 Therapies^a.

Metric	Median (IQR) [range]		P value for difference
	MedRxiv preprints (n = 34)	Publications (n = 62)
Altmetric attention score	127.0 (18.3-1448.5) [1.0-10396	66.5 (14.3-601.8) [0-14 745]	.40
News outlets citing	1.5 (0-24) [0-411]	1.0 (0-16) [0-327]	.70
Individuals using Twitter	113.0 (28.3-1992.8) [4.0-15 131.0]	57.0 (11.5-477) [0-38 971]	.42
Citations	5.5 (1.3-20.3) [0.0-154.0]	22.0 (4.3-52.5) [0-883]	.01
Page views	4915.5 (1379-21 326.5) [427-41 7105]	20973.5 (3002.5-138 891) [812-1 330 608]	.19

Abbreviations: COVID-19, coronavirus disease 2019; IQR, interquartile range.

^aNew outlets citations were taken from tracked numbers on the Altmetric data page. Page views for preprints were taken from metrics provided on the medRxiv website; page views for publications were taken from metrics provided by individual journals. Data for preprints reflect metric scores from the preprint server, regardless of whether the article was subsequently published during the time the article was posted. All of the 17 articles posted as preprints before journal publication remained posted on medRxiv through July 10, 2020. One preprint article led to 2 journal articles.

Table 2 shows the attention and online engagement metrics of the 34 medRxiv preprints that were published and not published during the study period. Compared with the 18 preprint articles that were not published, the median Altmetric attention scores, number of news outlets citing, and the number of page views were significantly higher for the 16 preprint articles that were published, and there were trends toward greater numbers of individuals posting on Twitter and citations.

Table 2. Comparison of Attention and Online Engagement Metrics of MedXriv Preprints About COVID-19 Therapies That Were Published and Not Published During the Study Period^a.

Metric	Median (IQR) [range]		P value for difference
	Published (n = 16)	Not published (n = 18)
Altmetric attention score	407.0 (54-3669.8) [1-10 396]	32.0 (11-269.3) [4-7333]	.03
News outlets citing	5 (0-55.3) [0-411.0]	0 (0.0-5.3) [0-130]	.03
Individuals using Twitter	399.0 (43.3-4744.8) [4-11 108]	47.5.0 (15.0-389.8) [9.0-15 131.0]	.06
Citations	11.0 (3.8-28.5) [0-75]	3.5 (1.0-14.0) [0.0-154.0]	.08
Page views	9201.5 (1942-49 077.8) [724-417 105]	1826 (920.3-8950) [427-220 349]	.04

Abbreviations: COVID-19, coronavirus disease 2019; IQR, interquartile range.

^aFor data sources, see the footnote for Table 1. All data are for the preprint article and not for the subsequent publication. All 34 preprints remained posted on medRxiv through July 10, 2020, including the 16 preprints with subsequent publications during the study period. Metrics for the 16 publications are included in the data on publication in Table 1.

Discussion

In a small study comparing articles about therapies for COVID-19 posted on the medRxiv preprint server, subsequent publications in medical journals of some of these articles, and journal articles that were not posted on either medRxiv or another preprint server, we found widespread dissemination of reports that had not undergone traditional peer review. We also found that articles posted as preprints but that were not published during the study period received less attention that those that did across multiple metrics.

This study has limitations in addition to the relatively small number of reports considered and the short duration of the analysis. Attention and online engagement metrics are inexact and shed little light on how the traffic is being generated. We did not compare the metrics for the published and preprint versions of the same study, as the available follow-up time for the studies and versions varied widely. We also did not study the associations of any of the articles with medical practice, regardless of whether they were first posted on medRxiv or published in a medical journal.

Supplement.

eMethods.

eFigure. Search strategy and selection of papers for the analysis

eTable 1. References for COVID-19 preprints

eTable 2. References for COVID-19 publications