Abstract
This study examines publication timelines, completeness, and spin in the abstracts of all randomized clinical trials related to COVID-19 posted to medRxiv during the first 2 years of the pandemic and compared the latter 2 with their published counterparts.
Preprints are increasingly important in medical research communication.1 In rapidly evolving areas such as COVID-19, preprints, which by definition have not been peer reviewed, can influence practice and potentially cause harm.2,3 We examined publication timelines, completeness, and spin in the abstracts of all randomized clinical trials (RCTs) related to COVID-19 posted to medRxiv during the first 2 years of the pandemic and compared the latter 2 with their published counterparts.
Methods
We identified all COVID-19–related RCT preprints posted to medRxiv between January 1, 2020, and December 31, 2021. On June 1, 2022, and November 15, 2022, title and reverse-author searches were used to identify companion publications. Using the June 2022 data, assistants prepared identically formatted Word documents of all preprint (published and unpublished) and published abstracts with the headings removed to minimize journal-specific conventions that might allow inference of publication status.
Blinded abstracts were presented to 1 of 3 reviewers in random order. Reviewers scored each abstract on measures of completeness using a modified version of the Consolidated Standards of Reporting Trials criteria (Table). Next, a random sample of 75 of the 161 preprint abstracts and their companion published abstracts, and the 75 preprint abstracts without companion publications, were presented in random order to 1 of 2 reviewers, who scored whether spin was present using a modified version of the score developed by Boutron et al,4 including items such as “statistically significant secondary outcomes emphasized in a negative study” and “claims extend beyond target population of the trial.” These same abstracts were graded for global spin using a gestalt assessment from 0 (none) to 10 (high).
Table. Comparison of Characteristics, Completeness, and Spin Between Preprint Abstracts Posted to MedRxiv vs Companion Published Abstracts of COVID-19 Randomized Clinical Trials.
| No. (%)a | |||
|---|---|---|---|
| Preprint abstracts | Published abstracts (n = 161) | ||
| Not published as of June 1, 2022 (n = 75) | Published as of June 1, 2022 (n = 161) | ||
| Characteristics | |||
| No. of study participants, median (IQR) | 110 (46-353) | 210 (89-809) | NA |
| Trial intervention | |||
| Medication | 50 (66.7) | 106 (65.8) | 106 (65.8) |
| Vaccine | 10 (13.3) | 31 (19.3) | 31 (19.3) |
| Herbal or animal product | 7 (9.3) | 8 (5.0) | 8 (5.0) |
| Behavioral | 4 (5.3) | 10 (6.2) | 10 (6.2) |
| Not pharmacological | 4 (5.3) | 6 (3.7) | 6 (3.7) |
| Completeness b | |||
| Randomization indicated in the title | 50 (66.7) | 112 (69.6) | 116 (72.1) |
| Had general organization (abstract broken up into sections; eg, results and methods) | 66 (88.0) | 140 (87.0) | 127 (78.9) |
| Had methods section | |||
| Trial design described (eg, cluster randomized, parallel group, superiority or noninferiority)c | 19 (25.3) | 73 (45.3) | 80 (50.0) |
| Clear objective or hypothesis given | 39 (52.0) | 106 (65.8) | 116 (72.1) |
| Participants described | 67 (89.3) | 149 (92.5) | 150 (93.2) |
| Interventions described | 72 (96.0) | 154 (95.7) | 158 (98.1) |
| Primary outcomes defined | 38 (50.7) | 113 (70.2) | 122 (75.8) |
| Allocation to interventions describedc | 2 (2.7) | 9 (5.6) | 21 (13.0) |
| Blinding discussed | 52 (69.3) | 117 (72.7) | 121 (75.2) |
| Had results section | |||
| Indicated No. randomized | 29 (38.7) | 51 (31.7) | 62 (38.5) |
| Described recruitment (trial status)c | 15 (20.0) | 47 (29.2) | 59 (36.7) |
| Indicated No. analyzed in each group | 21 (28.0) | 61 (37.9) | 73 (45.3) |
| Provided results for primary outcome | 23 (30.7) | 86 (53.4) | 93 (57.8) |
| Indicated harmsc | 8 (10.7) | 27 (16.8) | 35 (21.7) |
| Had conclusion section | |||
| Provided a general interpretation of results | 73 (97.3) | 154 (95.7) | 155 (96.3) |
| Other | |||
| Trial registration listed | 75 (100.0) | 160 (99.4) | 159 (98.8) |
| Funding source listed | 73 (97.3) | 159 (98.8) | 157 (97.5) |
| Spin d | |||
| No. of abstracts with spin present | 75 | 75e | 75e |
| In title | 8 (10.7) | 8 (10.7) | 6 (8.0) |
| In results section | 16 (21.3) | 2 (2.7) | 3 (4.0) |
| In harms section | 7 (9.3) | 3 (4.0) | 2 (2.7) |
| In conclusion section | 49 (65.3) | 31 (41.3) | 26 (34.7) |
| Global extent score, median (IQR)f | 3 (1-6) | 1 (0-3) | 1 (0-2) |
Abbreviation: NA, not applicable
Unless otherwise specified.
Based on a modified version of the Consolidated Standards of Reporting Trials criteria; all preprints were blinded prior to review.
No credit given for partial fulfillment of selected criteria.
Based on modified criteria from Boutron et al4; all preprints blinded prior to review.
Randomly selected pairs from all included preprints of subsequently published articles.
Assessed by a single reviewer using a gestalt assessment from 0 (none) to 10 (high).
Comparative scoring was accomplished by placing blinded preprint and companion abstracts side by side in a randomly alternating sequence and having 1 of 3 reviewers score which had more favorable completeness and spin using a 5-point Likert-style scale (1 = greatly favors preprint abstract; 5 = greatly favors published abstract). Additional details appear in the eMethods in Supplement 1. The analysis was descriptive and performed using R version 4.2.1 (R Foundation).
Results
Of 236 RCT preprints, 182 (77.1%) had published counterparts as of November 15, 2022. Of the 182 published counterparts, 168 (92%) were published within 12 months of the medRxiv posting, with a median time to publication of 134 days (IQR, 86-222 days). Of the 54 unpublished preprints, 45 (83.3%) had been observed for at least 12 months (median, 552 days; IQR, 470-627 days) before censoring.
Of the 236 RCT preprints, 161 published counterparts (68.2%) were included in the analysis. The percentage of abstracts fulfilling each completeness criterion was lowest in the unpublished preprint abstracts across 12 of 17 categories. This difference was greatest for the inclusion of results for the primary outcome (present in 30.7% of unpublished preprint abstracts, 53.4% of published preprint abstracts, and 57.8% of published counterparts) (Table). For all 4 categories of spin, the percentage with spin was highest in unpublished preprint abstracts (eg, 65.3% with spin in the conclusion section vs 41.3% in published preprint abstracts and 34.7% in published abstracts). Unpublished preprint abstracts had a higher median global spin score of 3 (IQR, 1-6) compared with a score of 1 (IQR, 0-3) for published preprints and a score of 1 (IQR, 0-2) for their companion publications.
The blinded side-by-side comparison of preprint abstracts and their companion publications similarly favored the published version for both completeness and lack of spin (Figure). However, for all but 1 category, most pairs were judged to have no difference (range, 68.3%-91.3%).
Figure. Blinded Qualitative Comparison Between Preprints and Published Abstract Pairs (n = 161).
The bars represent the distribution of ratings on a Likert scale (1 = greatly favors preprint abstract; 5 = greatly favors published abstract) for the blinded comparison of the preprint and published abstract pairs for each item. When differences were deemed to exist, completeness was generally higher and spin was generally lower in the published versions.
aIndicates whether the investigation was described in sufficient detail so that a reader would not have further questions before being able to offer an opinion on the veracity of the result.
Discussion
One in 5 medRxiv preprint abstracts remained unpublished for at least 12 months after posting. These unpublished preprints were less complete and more highly spun than preprints that progressed to publication. Adoption of COVID-19 treatment protocols based on erroneous preprints2,3,5 suggests potential problems associated with less complete, more highly spun preprint abstracts.
Consistent with the literature,6 improvements in completeness and spin with publication were modest, suggesting that the peer-review process and journal editing were limited in improving reporting quality. However, the data demonstrate that a gap exists between preprint abstracts eventually published and those that remain unpublished in terms of completeness and spin, potentially indicating selection for desirable qualities in the journal review process.
Study limitations include the use of imperfect proxy measures for completeness and spin, the focus on a single topic (COVID-19) that may be unique, inclusion of a single trial design (RCT), consideration of abstracts only, and lack of assessment of the clinical implications of the differences in completeness and spin.
Section Editors: Jody W. Zylke, MD, Deputy Editor; Kristin Walter, MD, Senior Editor.
eMethods
eReferences
Data Sharing Statement
References
- 1.Fraser N, Brierley L, Dey G, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biol. 2021;19(4):e3000959. doi: 10.1371/journal.pbio.3000959 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Popp M, Reis S, Schießer S, et al. Ivermectin for preventing and treating COVID-19. Cochrane Database Syst Rev. 2022;6(6):CD015017. doi: 10.1002/14651858.CD015017.pub3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Axfors C, Schmitt AM, Janiaud P, et al. Mortality outcomes with hydroxychloroquine and chloroquine in COVID-19 from an international collaborative meta-analysis of randomized trials. Nat Commun. 2021;12(1):2349. doi: 10.1038/s41467-021-22446-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058-2064. doi: 10.1001/jama.2010.651 [DOI] [PubMed] [Google Scholar]
- 5.Oransky I. French hydroxychloroquine COVID-19 study withdrawn. Accessed November 21, 2022. https://retractionwatch.com/2020/05/21/french-hydroxychloroquine-covid-19-study-withdrawn
- 6.Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review: a systematic review. JAMA. 2002;287(21):2784-2786. doi: 10.1001/jama.287.21.2784 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eMethods
eReferences
Data Sharing Statement
