Comparison of COVID-19 Preprint and Peer-Reviewed Versions of Studies on Therapies for Critically Ill Patients

Abstract

Purpose

Significant increases in the volume of preprint articles due to the COVID-19 pandemic, we examined the reliability of preprint articles compared to their peer-reviewed publications.

Materials and Methods

Preprint articles evaluating experimental studies of select treatment options (anticoagulation, dexamethasone, hydroxychloroquine, remdesivir, and tocilizumab) for COVID-19 in the critically ill, available in a peer-reviewed publication were screened for inclusion within Altmetric (n = 2040). A total of 40 articles met inclusion criteria, with 21 being randomly selected for evaluation. The primary outcome of this evaluation was a change in a study's reported primary outcome or statistical significance between preprint and peer-reviewed articles. Secondary outcomes included changes in primary/secondary outcome effect size and change in study conclusion.

Results

One article (4.8%, 95% CI 0.12%-23.8%) had a change in the primary outcome. Seven articles (33.3%, 95% CI 14.6%-57.0%) had a change in the primary outcome's effect measure. Five studies (23.8%, 95% CI 8.2%-47.2%) had changes in statistical significance of at least one secondary outcome. Four studies (19.0%, 95% CI 5.4%-41.9%) had a change in study conclusion.

Conclusions

In preprint articles of COVID-19 treatments, the provided primary outcome is generally reliable, while interpretation of secondary outcomes should be made with caution, while awaiting completion of the peer-review process.

Introduction

In December 2019, a novel coronavirus infection, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was identified by the scientific community. In the months that followed, millions became sick, with thousands dying at a rate which initially outpaced the development of appropriate diagnostic testing and treatment. The World Health Organization (WHO) eventually declared coronavirus disease 19 (COVID-19) a pandemic and to date, nearly 6.65 million individuals have died as a direct result of COVID-19 infection. ¹

Initially, there were no evidence-based treatments available other than supportive care, nor any preventative strategies regarding COVID-19. Researchers were poised with rapidly disseminating information regarding disease pathophysiology, complications, and management due to its unknown nature and extreme worldwide impact. Therefore, data became increasingly available without the “norm” of peer review, such as press releases and the explosion in the use of preprint databases. ² This was critical for the potential improvement in clinical outcomes of patients hospitalized with COVID-19, especially patients with the highest severity, requiring admission to the intensive care unit (ICU).

Preprint articles are open-access versions (ie, publicly available) of articles available prior to being subjected to the peer-review process. During the peer-review process, articles are reviewed by identified members in the field of study, which is designed to enhance scientific validity and reliability of research findings before widespread public consumption. However, the peer-review process delays the availability of new research findings, with the average time from article submission to publication taking up to three months.^2–4 In the midst of a public health emergency like the COVID-19 pandemic, avoidance of such delays could have lifesaving implications. We thus sought to evaluate the reliability of findings reported in preprint articles compared to published peer-reviewed articles, focusing on COVID-19 pharmacotherapies. This insight is important, given the greater awareness and continued use of preprint databases, to determine if preprint versions of articles may be a reliable source of information for clinical decision-making.

Methods

We examined articles that were available in both a preprint and peer-reviewed publication, evaluating COVID-19 pharmacotherapy in pharmacotherapy in critically ill patients. Medications were defined as substances intended for the use of diagnosis, treatment, cure, mitigation or prevention of disease, or therapies that have been granted an Emergency Use Authorization (EUA) for COVID-19, identified from the National Institutes of Health (NIH) COVID-19 guidelines, regardless of level of recommendation.^5,6 Medications of interest were anticoagulation (therapeutic and prophylactic), dexamethasone, hydroxychloroquine, remdesivir, and tocilizumab. These medications were chosen due to their prominence in use as treatment for COVID-19. We restricted our analysis to critically ill patients given their high degree of morbidity and mortality, where the impact of novel pharmacotherapies could be the most substantial.

Articles were identified via the Altmetric data output database, which tracks the attention or citations a journal article or scholarly database from around the world generates from social media, traditional media, blogs, or online reference managers. This database was chosen to not restrict to a specific scholarly database (eg, MEDLINE) to allow for improved generalizability of our findings, given its ability to include multiple sources of relevant published articles, such as Mendeley, Web of Science, and social media. The Altmetric database includes data including information on the article (eg, title, journal of publication) and the Altmetric Attention Score, which is a qualitative data metric automatically derived to report the amount of attention or publicity an article receives using an automated algorithm. A search within the Altmetric database was searched using key terms focused on COVID-19 and medication of interest, from the date of the first United States laboratory-confirmed case of COVID-19 on January 20, 2020, through April 9, 2022.

Using Altmetric data, articles were identified which focused on the six NIH COVID-19 guideline medication therapies mentioned previously. ⁷ Articles of the prespecified medications were screened by two independent reviewers (CM, AP) for the following inclusion criteria: (1) observational study or randomized trial design; (2) inclusion of critically ill patients; (3) preprint version and peer-reviewed publication available; and (4) availability in English. Critically ill patients were defined as patients cared for within an ICU or required invasive mechanical ventilation (IMV), based on information provided within the study methods, demographics, or supplementary information. Preprint databases searched were bioRxiv, medRxiv, and Research Square. Any discrepancies were resolved by discussion or review by a third reviewer (AW). To minimize selection bias and to improve generalizability, an objective article inclusion/exclusion screening tool was created, and a random selection of up to five articles per therapy meeting inclusion criteria were further evaluated, regardless of the article's Altmetric Attention Score. If less than five articles for a therapy met inclusion criteria, then all articles for that therapy were evaluated.

A standardized data collection tool was used for data abstracted by two independent reviewers (CM, AP). Data included the therapy evaluated, number of days from preprint article availability to peer-reviewed publication, the peer-reviewed article's Altmetric Attention Score, changes in study title, changes in study methods, changes in statistical analysis, presence of new post hoc analysis (in published version), changes in data for the primary outcome (including statistical significance, as defined by the study methods), changes in data for secondary outcomes (including statistical significance, as defined by the study methods), changes in study graphics, changes in provided limitations, change in study conclusion, and percent article similarity between the preprint and peer-reviewed articles (defined using Turnitin® software). The use of Turnitin® was selected based on prior familiarity with the software, its use by journals in assessing article submissions, as well as studies evaluating the incidence of plagiarism in scientific journals.^8,9 Any noted discrepancies were resolved through discussion or through a third reviewer (AW).

The primary outcome of our study was the change in the evaluated study's primary outcome consisting of any difference in reported primary outcome (eg, change in study outcome endpoint of 30-60 days), effect size, or statistical significance. Secondary outcomes included changes in the study's primary outcome effect size/95% CI, changes in statistical significance within the study's nonprimary outcomes, and the change in study conclusions. The Clopper Pearson method for binomial proportions was used to calculate the estimated 95% confidence interval of the following: change in primary outcome (outcome/statistical significance, effect size), change in statistical significance of at least one secondary outcome, and change in study conclusion. All analyses were conducted in Stata, version 17.1 (College Station, TX).

Results

A total of 2040 Altmetric articles were screened, with 40 (1.96%) meeting inclusion criteria and 21 randomly selected for evaluation (Figure 1).^10–30 The most common exclusion was not being an observational study or randomized trial (n = 1272, 62.4%). There were no articles that met inclusion for prophylactic anticoagulation. Characteristics of each study are detailed in Table 1, with summary statistics by medication detailed in Table 2. The median Altmetric Attention Score of all included articles was 50 (IQR 4.5-1171) and the median time from preprint to publication was 112 days (IQR 63-135). The median percent similarity between the preprint and published versions was 57% (IQR 39.5-67.5).

Figure 1.

Screening and inclusion according to drug therapy.

Table 1.

Included Article Characteristics.

Article	Medication	Date of Preprint	Date of Publication	Time From Preprint to Publication, d	Preprint, n	Change in Publication, n	Altmetric Attention Score	Turnitin® Similarity
Patel et al 10	AC	8/26/2020	1/1/2021	128	1716	No	17	55
REMAP-CAP 11	AC	3/12/2021	8/26/2021	167	1089	Yes, n = 1103	757	26
Hoertel et al 12	D	6/22/2020	2/25/2021	248	6425	No	8852	14
Ko et al 13	D	10/27/2020	2/19/2021	115	12 217	No	8	64
Lee et al 14	D	2/5/2021	2/25/2021	20	262	No	50	67
Ranjbar et al 15	D	11/23/2020	3/29/2021	126	59	No	7	61
RECOVERY 16	D	2/1/2021	4/10/2021	68	86	No	238	70
Cadegiani et al 17	HCQ	11/4/2020	9/1/2021	301	585	No	2661	24
Gautret et al 18	HCQ	3/20/2020	7/1/2020	103	36	No	9990	57
Ip et al 19	HCQ	5/25/2020	8/13/2020	80	2512	No	27	68
Magagnoli et al 20 a	HCQ	4/23/2020	12/1/2020	222	368	Yes, n = 807	441	42
Mahevas et al 21 a	HCQ	4/14/2020	5/14/2020	30	181	Yes, n = 173	3532	37
Flisiak et al 22	RDV	11/3/2020	12/31/2020	58	333	No	18	49
Garibaldi et al 23	RDV	11/20/2020	3/24/2021	124	2299	No	160	30
Jo et al 24	RDV	9/27/2020	1/29/2021	124	N/A	No	2	50
Mehta et al 25	RDV	11/10/2020	2/26/2021	108	346	No	2	67
Fomina et al 26	T	6/12/2020	10/2/2020	112	89	No	1	79
Gokhale et al 27 a	T	10/14/2020	3/5/2021	142	269	No	1	65
RECOVERY 28	T	2/11/2021	5/1/2021	79	4116	No	1585	70
Rubio-Rivas et al 29	T	9/1/2020	10/6/2020	35	186	No	1	71
Somers et al 30	T	6/3/2020	7/11/2020	38	154	No	478	56

Abbreviations: AC, anticoagulation; D, dexamethasone; HCQ, hydroxychloroquine; R, remdesivir; T, tocilizumab.

^a Change in study period identified in methods from preprint to publication.

Table 2.

Summary of Articles by Medication Type.

	AC (n = 2)10,11	D (n = 5)12–16	HCQ (n = 5)17–21	RDV (n = 4)22–25	T (n = 5)26–30
Median days from preprint to publication, (IQR)	147.5 (137-158)	115 (68-126)	103 (80-222)	116 (87-124)	79 (38-112)
Median Altmetric Attention Score, (IQR)	387 (194-581)	50 (8-238)	2611 (441-3532)	10 (2-81)	1 (1-478)
Median Turnitin similarity, (%)	40.5 (33-47)	64 (61-67)	42 (37-57)	49.5 (40-58)	70 (65-71)

Abbreviations: AC, anticoagulation; D, dexamethasone; HCQ, hydroxychloroquine; IF, impact factor (from year of article publication); RDV, remdesivir; T, tocilizumab.

In comparing the preprint and published versions of the articles, there was one article (4.8%, 95% CI 0.12-23.8) that had a change in primary outcome or statistical significance, ²¹ with seven (33.3%, 95% CI 14.6%-57.0%) having a change in the primary outcome's effect measure.^{11,16,20,21,23,24,28} Five studies (23.8%, 95% CI 8.2%-47.2%) had a change in statistical significance for at least one non-primary outcome. Four studies (19.0%, 95% CI 5.4%-41.9%) had a change in study conclusion. Specific details on each of these changes are available in Table 3, while details on noted changes may be found in Table 4.

Table 3.

Comparison Between Preprint and Published Version of Included Articles.

	AC (n = 2)10,11	D (n = 5)12–16	HCQ (n = 5)17–21	RDV (n = 4)22–25	T (n = 5)26–30
Change in primary outcome, n (%)	0 (0)	0 (0)	1 (20)	0 (0)	0 (0)
Change in effect size of primary outcome, n (%)	1 (50)	1 (20)	2 (40)	2 (50)	1 (20)
Change in statistical significance of any secondary outcome, n (%)	0 (0)	2 (40)	1 (20)	1 (25)	1 (20)
Change in effect size of any secondary outcome, n (%)	1 (50)	2 (40)	2 (40)	2 (50)	3 (60)
Change in study title, n (%)	2 (100)	2 (40)	1 (20)	1 (25)	0 (0)
Change in study methodology, n (%)	0 (0)	0 (0)	1 (20)	0 (0)	0 (0)
Change in statistical analysis, n (%)	0 (0)	2 (40)	2 (40)	2 (50)	1 (20)
Presence of new post hoc analysis, n (%)	1 (50)	1 (20)	1 (20)	0 (0)	0 (0)
Change in study graphics, n (%)	1 (50)	4 (80)	2 (40)	1 (25)	0 (0)
Change in study limitations, n (%)	2 (100)	1 (20)	5 (100)	1 (25)	1 (20)
Change in study conclusions, n (%)	1 (50)	1 (20)	1 (20)	1 (25)	0 (0)

Abbreviations: AC, anticoagulation; D, dexamethasone; HCQ, hydroxychloroquine; IF, impact factor (from year of article publication); RDV, remdesivir; T, tocilizumab.

Table 4.

Changes in Study Primary or Secondary Outcomes, or Conclusion.

Change	Study	Preprint	Publication
Change in primary outcome; statistical	Mahevas et al 21	− Death/transfer to ICU in 7 days: RR 0.91 (0.47-1.8)	− Survival without transfer to ICU at day 21: HR 0.9 (0.4-2.1)
Change in primary outcome, effect size (95% CI)	REMAP-CAP 11 (Organ support-free days at 21 days)	0.87 (0.70-1.08)	0.83 (0.67-1.03)
	RECOVERY 16 (28-day mortality)	0.83 (0.74-0.92), P < .001	0.83 (0.75-0.93), P < .001
	Magagnoli et al 20 (Death)	2.61 (1.10-6.17)	1.83 (1.16-2.89)
	Mahevas et al 21 (See above in “Change in primary outcome, statistical”)	0.91 (0.47-1.80)	0.90 (0.40-2.10)
	Garibaldi et al 23 (Rate of clinical improvement)	1.55 (1.28-1.87)	1.47 (1.22-1.79)
	Jo et al 24 (# deaths averted w/ D) (# deaths averted w/ D in MV) (# deaths averted w/ RDV in non-MV; D in MV) (# deaths averted w/ RDV in non-MV)	1146 (572-1863) 638 (227-1140) 1111 (239-2027) 473 (29-1293)	689 (330-1118) 382 (140-679) 408 (229-1891) 26 (21-1497)
	RECOVERY 28 (28-day mortality)	0.86 (0.77-0.96), P = .007	0.85 (0.76-0.94), P = .0028
Change in secondary outcome, statistical	Lee et al 14 (Cox for HFNC/MV: Charlson)	1.335 (1.005-1.774), P = .046	1.032 (0.850-1.253), P = .750
	RECOVERY 16 (Subgroup: In no IMV @ baseline → IMV or death)	0.91 (0.82-1.00), P = .049	0.93 (0.85-1.01)
	Magagnoli et al 20	Changes in most secondary outcomes due to increase in sample size
	Garibaldi et al 23 (Cox for time to clinical improvement w/ RDV + D)	0.87 (0.53-1.44)	0.77 (0.62-0.97)
	RECOVERY 28 (Subgroup: Age 70-80 + 28-day mortality) (Subgroup: No vent support @ baseline + 28-day mortality)	0.84 (0.69-1.01) 0.84 (0.69-1.03)	0.82 (0.68-0.99) 0.81 (0.67-0.99)
Change in study conclusion	REMAP-CAP 11	Therapeutic heparin provides no benefit	Therapeutic heparin or LMWH provides no benefit
	Ranjbar et al 15	1-2 mg/kg IV methylprednisolone demonstrates better outcomes; superior immunosuppressive agent compared to dexamethasone	2 mg/kg IV methylprednisolone demonstrates better outcomes than dexamethasone 6 mg/day at days 5 and 10
	Magagnoli et al 20	No reduction in rate of mechanical ventilation; association of increased mortality in HCQ alone	No reduction in mortality or rate of mechanical ventilation
	Garibaldi et al 23	RDV + Dexa may reduce mortality	RDV + corticosteroid does not reduce time to death

Abbreviations: D, dexamethasone; D/C, discharge; IMV, invasive mechanical ventilation; IV, intravenous; MV, mechanical ventilation; RDV, remdesivir.

() Denotes the outcome that was evaluated within the study.

A total of six articles had a change in title (with only one indicating that it was changed from a preliminary report to a final report ¹² )^{10–12,15,21,23} Three articles had a change in sample size between the preprint and peer-reviewed versions.^11,20,21 Three articles had a change in study period identified from the methods from the preprint to publication.^20,21,27 There were six articles that had a change in statistical methods, which included changes in variables considered for multivariable analysis.^{12,15,20,21,23,24} The three articles containing a new post hoc analysis included interactions between therapies and outcomes or subgroup analyses.^11,12,20 Changes in study graphics (38.1%)^{11,12,14,15,16,20,21,23} and study limitations^{10,11,13,17–21,23,30} were relatively common. A total of four articles (19.0%) had no change in the study characteristics evaluated.^22,26,26,29

Discussion

Our study evaluated a total of 21 articles, available in preprint and peer-reviewed versions, focused on therapies for the treatment of COVID-19 in critically ill patients. We found that changes in the study's primary outcome was uncommon, and changes to the effect size of the primary outcome was more common. Changes in a study's statistical significance for secondary outcomes were more common than changes in the primary outcome and importantly, we found that approximately 20% of studies modified study conclusion, although most variation pertained to changes in wording and not amendments to study results. The results of our study suggest that the primary outcomes of a preprint article that is subsequently published do not significantly change compared to its published version and therefore, may be incorporated into clinical practice earlier than the “traditional” peer-review process.

These results are largely in line with prior studies evaluating the content of both COVID-19 and non-COVID-19 preprint and peer-reviewed literature.^31,32,33 These studies found that changes in text were minimal, with discordance in primary outcomes occurring in 20 to 30% of articles. Based on the time from preprint availability to publication, this may suggest that the peer review process may take at least 6 weeks, and in most cases at least 16 (although it is important to note that we did not have time of initial journal submission). However, this time frame may not be completely accurate given that some studies may have been finalizing their data collection/analysis. In a situation where the rapid dissemination of study results is critical, the need to have reliable information available without delay can be lifesaving. ³⁴ Our Turnitin® findings of approximately 60% similarity were interesting, with changes potentially due to changes in verbiage per peer-review or journal formatting. Given that the data evaluating similarity are focused on plagiarism, it is difficult to interpret this quantitative value.^8,9

Our findings suggest that changes between primary/secondary outcomes between the preprint and published articles are relatively uncommon, but they may still be clinically relevant. This is consistent with other published studies that have performed a similar analysis, although not specifically on articles regarding COVID-19.^31,33 It should be noted that though there is a difference in the effect size of the primary outcome, this was often minimal and did not affect statistical significance. Our data do suggest that secondary outcomes should be interpreted with caution and may not be reliable in the clinical decision-making process until peer-review has been completed, resulting in final publication. Changes in study conclusion were an important finding, given this may be what some clinicians focus on when reading an article. In the case of Garibaldi and colleagues, the article conclusion changed from remdesivir and dexamethasone may reduce mortality to remdesivir and corticosteroids do not reduce time to death. This suggests that for this study, the peer review process had an impact on the message clinicians would take away pertaining to the use of the above medications investigated. ²³ Given the lack of data on peer-review comments, it is unclear if changes in study methodology, statistical analysis, post hoc analyses, study limitations, and conclusion were due to intrinsic or extrinsic changes from peer-review. The type of journal (eg, impact factor), especially those with statistical consultation, may also have an impact on these changes.

In order to address the future use of preprint articles, potential recommendations to address discrepancies we identified include transparency within a preprint version of an article (possibly in the abstract and the conclusion) if additional data collection or analysis is pending, and articulation in the discussion indicating anticipated future steps prior to submission for peer-reviewed journal publication. Also, providing the peer-reviewers’ comments may help with transparency to address reasons for potential changes from the preprint version.

Our study has limitations, selection of articles available only in the Altmetric database, preprints available in three databases, and only underwent peer-reviewed publication. However, in our knowledge of COVID-19-specific articles from clinical practice, the three identified preprint databases were the most prominent in identifying potentially practice-changing articles. However, not all preprint articles are subsequently published and therefore, our results may only pertain to articles that subsequently are published, which is impossible to determine from the preprint version. ³¹ Additionally, our study only evaluated articles pertaining to the treatment of critically ill patients, which was due to its relevance to the potential severity of COVID-19. The studies included often included patients that were not critically ill and therefore, our findings may be generalizable to those populations. Furthermore, many articles available on preprint servers do not later become published, and thus could not be evaluated in this study. We aimed to minimize selection bias by randomly selecting articles regardless of their journal impact factor or Altmetric Attention Score, therefore improving the generalizability of our findings. Although we took these steps, we are not able to totally exclude potential bias. Our median Altmetric Attention Score was comparable to a previous study. ³¹ Given the discrepancies we noted, it is unclear if changes from the preprint to peer-reviewed publication were due to internal changes by the study team or the effect of the peer-review process; hence the value of published peer reviewers’ comments. Furthermore, this study did not look to categorize the types of changes made and its effects in clinical practice. Finally, since we focused on patients with COVID-19, the generalizability of our findings to other disease states is indeterminate. However, the limited studies evaluating preprints in other diseases states may indicate some potential generalizability.

Conclusion

As established therapies have become available for the treatment of COVID-19, there may be a less pressing need to rely on preprint literature for clinical decision-making. However, concerns about the effectiveness of certain therapies have been questioned as new COVID-19 variants are isolated. Additionally, given the explosion in the use of preprint databases, this may become an important method in rapid dissemination of literature in the future, outside of COVID-19 therapies. Based on our findings, one may reasonably justify clinical decision-making based upon a preprint study's primary outcome, specifically for the management of patients critically ill due to COVID-19. The results of our study, in conjunction with results of prior studies evaluating the impact of the peer-review process on an article outside of COVID-19-related illness, may allow for extrapolation of our results to lower acuity patients with COVID-19 and additional disease states, pending additional literature. Still, it is imperative to fully understand an article's strengths, limitations, and clinical relevance of a preprint article prior to incorporation into clinical practice.