Open science saves lives: lessons from the COVID-19 pandemic

Abstract

In the last decade Open Science principles have been successfully advocated for and are being slowly adopted in different research communities. In response to the COVID-19 pandemic many publishers and researchers have sped up their adoption of Open Science practices, sometimes embracing them fully and sometimes partially or in a sub-optimal manner. In this article, we express concerns about the violation of some of the Open Science principles and its potential impact on the quality of research output. We provide evidence of the misuses of these principles at different stages of the scientific process. We call for a wider adoption of Open Science practices in the hope that this work will encourage a broader endorsement of Open Science principles and serve as a reminder that science should always be a rigorous process, reliable and transparent, especially in the context of a pandemic where research findings are being translated into practice even more rapidly. We provide all data and scripts at https://osf.io/renxy/.

Introduction

The COVID-19 outbreak represents an urgent threat to global health. On October 15, 2020, the number of COVID-19 cases had exceeded 38 million and the death toll had exceeded 1,000,000 worldwide. Many important issues remain unresolved, including some crucial questions around both the diagnosis of patients with COVID-19 and optimal therapeutic strategies. Rapid scientific progress on these issues is needed to improve patient management, reduce mortality, and prevent new infections. The scientific community has responded accordingly, with the publication of over 80,000 preprints and peer-reviewed articles on COVID-19 or SARS-CoV-2 since announcement of the emergence of a new virus on 31 ^st December 2019 [1]. Many of these publications have contributed to the development of a body of knowledge that has since informed practice but a considerable number of these studies suffer methodological weaknesses, limiting the interpretability of their findings [2] or leading to false claims with a potentially dramatic impact on public health. While some of these studies have already been retracted [3, 4], others still contribute to the body of evidence and might be used by researchers and policy makers. In addition to the direct threat these publications pose to public health, these low-quality studies also exacerbate the waste of scientific resources [2] that is well-known to plague the scientific system [5]. Furthermore, many news outlets have recently amplified public exposure to low-quality research, sowing confusion among the public. In this paper we argue that many of the sub-optimal and non-transparent scientific practices witnessed during the pandemic, in conjunction with poor coordination across the global research community, have contributed to a dysfunctional scientific process for COVID-19 research. We support this view by providing results from an analysis of COVID-19 publishing data in recent months, including an analysis of reviewing times, conflicts of interests and misuse of non peer-reviewed material. We further argue that the widespread implementation of Open Science principles – known to increase the rigour, reliability and reproducibility of scientific results [6–10] – could help optimize research efficiency moving forward, and thus improve health outcomes and economic costs related to COVID-19.

Broadly speaking, Open Science aims to optimize scientific conduct and communication by exposing the scientific process, and results thereof, to the scientific community and broader public. This idea is implemented concretely through a number of core Open Science practices [8, 11, 12]: Open Access, Open Source, Open Data and Open Peer-Review. The best-known of those, Open Access, consists of making all scholarly communications freely available with full re-use rights. Open Access also encompasses early dissemination of manuscripts in the form of preprints (articles not yet published in scientific journals). The term “preprint” can refer to either pre-peer-review manuscript or peer-reviewed manuscript before formatting. Hereafter, we use the wording “preprint” to refer to pre-peer-review manuscript posted on archival websites. Even though preprints are not yet peer-reviewed and thus could contain mistakes which may have been identified through an independent review process, they contribute to a more transparent and open scholarly publication system, accelerating reviewing and communication within the scientific community [13]. Open Source and Open Data aim at ensuring that materials such as questionnaires, forms, procedures, collected data, metadata, and source code are shared to foster replication studies, increase data re-use, and facilitate the peer-reviewing process [14, 15]. Indeed, reviewers have the material at hand to verify the findings or detect any issues that could not be otherwise identified from the manuscript itself and to provide comprehensive peer-review reports. Then, following the Open Peer-Review principle, these peer-review reports should be publicly and transparently shared, along with the authors’ response. The scientific discussions between authors and reviewers are inherent to the process of creation of knowledge [16]. In addition, Open Peer-Review helps maintain high reviewing quality [17–19] and reduces the risk of concealed conflicts of interest. Therefore, the adoption of Open Science principles in the last decade has been particularly helpful in increasing the rigour, reliability and reproducibility of scientific results across research fields [6, 8–10].

There is evidence suggesting that the COVID-19 pandemic has served as a catalyst in the adoption of certain Open Science principles. For instance, major publishers such as Elsevier [20] and Springer Nature [21] have made newly written COVID-19 related articles freely accessible to all (Open Access). Furthermore, authors have shared their preprints more systematically than in previous pandemics [22] and reviews have been posted on external platforms (e.g., Pubpeer [23]). Specific initiatives, such as OpenSAFELY [24], have emerged to make data available to researchers while complying with the legislation regulating the use of medical data. Nevertheless, there have been many instances where these principles were ignored. One notorious example is the lack of transparency and sharing of the data provided by Surgisphere, which led to the retraction of the publication in The Lancet [25]. In other instances, some of the Open Science principles were adopted but misused. For example, news agencies have reported unreliable results based on the misuse of unreviewed preprints [2] and some open reviews took place on separate platforms (for example Pubpeer), and were thus not directly available to readers.

While we recognize that the faster embracing of Open Science during the pandemic is a step towards more accessible and transparent research, we also express concerns about the adoption of these practices for early and non-validated findings. Furthermore, embracing only some of these principles, while excluding others can have serious unintended consequences that may be as detrimental as not adopting open practices in some instances. The aim of the present paper is twofold. First, we identify the issues the scientific community has faced with regard to the publication process since the beginning of the pandemic. To do so we analyzed data collected on preprints and published COVID-19 research articles, as well as on retracted COVID-19 publications, in order to quantify issues related to reviewing time, conflicts of interest, and inappropriate coverage in the media. In light of this analysis, we then discuss how a wider adoption of Open Science principles could have potentially minimized these issues and mitigated their impact on the scientific community and broader public.

The structure of this article follows the stages of the publication process shown in Fig. 1. We first discuss issues arising at the data collection and interpretation stage (before the dissemination of the results). Then, we review the dysfunctions observed during the publication process (between the submission and the publication of research articles), before investigating the misuses of research outputs during science communication (after publication). We provide recommendations based on Open Science principles for each stage of the publication process, which we hope will contribute to better research practices in the future.

Fig. 1.

Outline of the publication process with its potential issues and our proposed solutions

Stage 1: data collection and interpretation

While previously deplored [5], waste of scientific effort has been particularly prominent during the COVID-19 pandemic [2] and has been more visible than ever before. In this section, we show that this waste has its roots in the early stages of the research process – at the data collection and interpretation stage – and discuss how study preregistration, registered reports, adherence to reporting guidelines and Open Source principles could help to minimize waste in research.

Identified flaws

Methodological and statistical issues

Conducting research during a pandemic is known to pose particular challenges [26] but scientists have raised concerns about methodological flaws in the design and analysis of various COVID-19 pharmacological studies [2, 27–29].

To better understand whether inappropriate study designs or statistical analyses contributed to the reasons behind the retraction of articles, we looked at the 29 COVID-19-related papers that had been identified on the COVID19 RetractionWatch dabatase as retracted or subject to expressions of concerns since January 2020 [3]. While the list is still updated by RetractionWatch, our original list of articles (both preprints and peer-reviewed) and the results of our analyses are available on the repository of this project: https://osf.io/renxy/. Of the 29 identified publications, 8 (27.6%) were retracted (or had an expression of concern from the editorial board) based on their data analysis or study design. More specifically, among these 8 publications, 2 (25.0%) papers were retracted, at the authors’ request, in order to conduct further data analyses and 6 (75.0%) were retracted because the methodology or the data analysis was wrong. Out of these 8 retracted papers, 3 were preprints and did not undergo peer-review, however 2 of them stand out from the peer-review data we could extract: one was peer-reviewed in less than 3 days and the second displayed editorial conflict of interest, thus highlighting the need for a more transparent review process.

Duplication of research

Another concern is the increased risk of research waste due to duplication. Many studies that aimed to assess the efficacy of hydroxychloroquine were conducted in parallel: 218 registered trials were ongoing or already completed as of 26 ^th April 2020 [30]. Many comparative effectiveness studies – randomised or not – were conducted without preregistration (e.g., Geleris et al. [31]), however, meaning that the broader research community only became aware of these studies at the time of the release of the results. This illustrates the general lack of cooperation between research teams, putting more patients at risk by exposing them to potentially harmful treatments in multiple underpowered studies, and also leading to a waste of research time and financial and human resources [2]. Given the additional workload for healthcare workers and clinical researchers these trials require, it may have contributed to the disruptions in the conduct of clinical trials during the pandemic [32]. Other studies have been pre-registered but conducted and reported with major deviations from the preregistration record without justification: for example outcome measures and their timing of assessment reported in the aforementioned study by Gautret et al. [33] were not those listed on the EU Clinical Trials Register

Ethical concerns

Ethical concerns have also arisen during the pandemic. While the research community needs to find ways to provide timely solutions to the COVID-19 crisis, it should not be at the detriment of good research and clinical practice. Among possible ethical risks, Xafis et al. [34] identified over-recruitment in trials, the conduct of human vaccine studies before the completion of animal studies, and the neglect of adverse effects in drugs studies. An example of the last is the little consideration given to the known cardiotoxicity of the combination of hydroxychloroquine and azithromycin early on in the pandemic [35]. Issues surrounding patients’ participation in clinical studies have also been observed: in her analysis of COVID-19 papers unsuitable for publication, Bramstedt identified issues surrounding informed consent as the second most common source of concerns [36]. In addition to the ethical problems this poses, it could also weaken the trust that patients and the broader community afford researchers, with detrimental consequences for public health in the long term.

Open science solutions

Here, we argue that the adoption of certain Open Science principles could have helped to detect or avoid the issues in data collection and interpretation described above (10). Two methods seem to be particularly relevant:

Study preregistration

First, study preregistration on dedicated platforms (e.g., ClinicalTrials.gov, OSF, or AsPredicted), with a thorough description of the study design, ethical approval, methods for data collection and data analysis, can help prevent some of the issues identified above (10). Indeed, study preregistration may reduce the amount of unnecessary duplication of research as researchers will be able to check whether specific studies are ongoing and design theirs to address complementary questions. Finally, study preregistrations can be used by Institutional Review Board for ethics approval and to fulfil the ethical obligation to transparently inform both the public about ongoing trials as well as the research community [37].

Another goal of preregistration is for readers and reviewers to make sure that a published study has been conducted and analysed as planned, thus limiting the risks of changes to the design, methods or outcomes in response to the data obtained other than the flexibility allowed by the protocol (in case of interim analyses of adaptive designs). Researchers should register studies prior to data collection. On the platform ClinicalTrials.gov, retrospective registrations or updates to the study protocol are flagged. Depending on the level of methodological details in the record, standardized study preregistration may help in limiting questionable research practices such as HARKing [38], p-hacking and p-fishing [7] and eventually lead to better subsequent reporting [39].

As COVID-19 was a new disease, there was no standardized diagnostic criteria or clinical outcomes. This led to a multiplication of different outcomes studies in the articles participating in the difficulty to replicate and compare results. Study preregistration could help researchers adopt the same criteria and outcome measurements and promote the use of validated international standardized criteria for variable and outcome measurements [40].

However, such preregistrations have two major limitations. First, they do not fully prevent duplication. While replication (defined as a deliberate effort to reproduce a study to validate the findings) is an important step of the research process, duplication (an inadvertent repetition of the research) contributes to research waste [2]. This waste has been noted among COVID-19 research [2], with a strikingly high amount of duplication despite study preregistration. Second, whereas preregistrations allow the detection of questionable research practices, they do not help prevent methodological issues before data collection since the preregistration is not itself peer-reviewed and the statistical analysis section of these records is often very brief. Therefore, standardized study preregistrations are necessary, since they encourage researcher to outline the study design and analysis strategy, but not sufficient to avoid the excessive waste of scientific resources.

Registered report

Peer-reviewed study protocols, also called registered reports [41, 42], can also have a major impact on the reduction of wasted resources. They essentially consists in articles with a two-stage peer-review, and provide details about the research question, hypotheses, methodology, statistical analyses and reporting strategy. Since protocols are peer-reviewed before the enrollment of participants and data-collection, potential omissions or mistakes in the proposed methodology can be corrected before any substantial resources are used, thereby limiting scientific waste [5, 43]. In these reports, researchers are also encouraged to provide details about the resources used, using for instance Research Resource Identifiers [44] when applicable, and specify the reporting guidelines that will be used (e.g., CONSORT [45], STROBE [46]). Registered reports can therefore contribute to higher quality research, with a reduced risk of bias and increased generalizability. One disadvantage of registered reports is that their reviewing takes time, while preregistrations are immediately available. However, some platforms for the submission of registered reports put in place measures to guarantee a timely review of COVID-19 protocols: stage 1 review of registered reports at Royal Society Open Science are performed within 7 days [47]. Furthermore, since they reduce the risk of publication bias, they also reduce the number of submissions needed to publish one’s results and should ultimately save both resources and time (e.g., [48]). Both pre-registration and registered reports contribute to a better visibility of ongoing research, and should be used at institution levels to coordinate research projects at an international level in a more efficient way, in order to optimize resources.

Beyond registration: open methodology and reforming the publication system

Preregistrations and registered reports are necessary but not sufficient to conduct reliable transparent research. Open Methodology [12] goes further. It consists in transparently sharing all the necessary details to allow replications of the research. In other words, Open Methodology relies on the authors to not withhold any details of their research project so that any outsider could exactly replicate it. While one might assume that preregistrations and registered reports are sufficiently detailed to allow replications, numerous past studies have shown that replicating research work is excruciating or impossible [49]. Journals should therefore support fully Open Methodology better by enforcing that all submitted articles must be fully reproducible before the manuscript is published. Beyond Open Methodology and registrations, researchers have suggested to reform scientific communication so that they would be less story-telling oriented and more focused on the methodology (see Octopus [50]). Every step of the research methodology (research-question formulation, hypothesis-making, data-collection plan, data analysis, interpretation...) is a smaller paper that builds onto the previous one and all of them are open for comments and reviews in order to make insights more robust and foster collaborations. Adopting this would, however, require a complete change of the scientific publishing system and further proof is needed to show its benefits.

Section conclusion

In clinical research, the adoption of study preregistration for clinical trials (clinical trial registrations) and observational studies, as well as registered reports, have contributed to better transparency and reliability of the findings. For researchers themselves, the availability of the report facilitates complete and transparent reporting in the final publication. Finally, study preregistration and registered reports enable the conduct of more exhaustive meta-research and systematic reviews, by reducing the risk of publication bias. We therefore argue that both study preregistration on dedicated platforms, already mandatory for some types of clinical trials, and registered reports should be used more systematically.

Stage 2: publication process

Publishing scientific evidence consists of several steps summarized in the second box in Fig. 1. In response to the COVID-19 global pandemic, an enormous number of research publications have been produced. Unfortunately, shortcuts have been taken in the publication process of some of these papers, jeopardizing the integrity of the editorial process and putting the rigour of scientific publications at risk. In this section we discuss three issues in the peer-reviewed publication process which have arisen during the COVID-19 pandemic: fast-track publication, conflicts of interest and lack of data sharing. We highlight the increasing retraction rate of COVID-19 preprints and peer-reviewed articles, and propose practical solutions to minimize these issues in the future.

Identified flaws

Expedite reviewing and conflicts of interests

The publication pipeline has been directly affected by COVID-19, in particular with respect to reviewing times of COVID-19 papers [51]. Although fast-tracking particular articles is not uncommon in the scientific publishing system [51], a number of journals have recently implemented specific policies to fast-track COVID-19-related research (e.g., PLOS, some Wiley journals [52], some Elsevier Journals [53], some SAGE journals [54], and PeerJ journals [55]). In addition a new overlay journal for fast and independent reviews of COVID-19 preprints has recently been launched [56]. While faster peer-reviewing does not necessarily equate with poorer review quality and faster peer-reviewed time are encouraged at a time of crisis, it remains unclear how thorough the peer-reviewing is and how potential conflicts of interest are handled. Palayew et al. [51] recently highlighted that COVID-19-related manuscripts between 1 January 2020 and 23 April 2020 had a median reviewing time of 6 days and that more than a thousand manuscripts were reviewed in less than 7 days.The authors also identified manuscripts for which it was unclear whether they had been reviewed at all. More recently, Homolak et al. compared median submission-to-publication times of COVID-19 publications with those of non-COVID papers during the same period and before the pandemic [57]. These times were reduced by a factor 10 to 15 for COVID-19 papers, consistently across research fields. It should be noted that the FASEB Journal, whilst acknowledging the degree of risks in doing so [52], allows editors to directly accept COVID-19-related submissions for Review, Perspectives, and Hypotheses, without peer-review, as per the journal’s fast-track policy [58]. In light of this, we have sought further information on the fast-tracking of peer-reviews with up-to-date information.

We searched for “COVID-19”(and related terms, see the full list in Appendix) on PubMed Central and found 12,682 published articles as of 1 ^st July 2020. Of these we could extract the reviewing time for 8,455 (66.7%) articles, as the difference between the date of submission and the date of acceptance. Of these 8,455 publications, 699 (8.3%) from 341 different peer-reviewed journals were reviewed and accepted for publication either on the day of submission (n=311) or the day after (n=388). We manually inspected these manuscripts to identify potential conflicts of interest. We focused only on editorial conflicts of interest, i.e., manuscripts for which at least one of the authors is an editor-in-chief, associate editor or a member of the editorial board of the journal in which the article was published. While publishing articles with editorial board members among the co-authors is not problematic in itself, and is often encouraged by publishers provided that a specific process is put in place [59], we were concerned about the lack of transparency in the peer-review process of these articles. We did not assess institutional or financial conflicts of interest. Among the 699 articles accepted within a day, an editorial conflict of interest was observed in 297 (42.5%) articles. In order to assess whether the frequency of these conflicts decreases over time, we also inspected the manuscript accepted in 16 days and 20 days, corresponding respectively to the median and mean time between submission and acceptance in our sample. To further investigate the distribution of these conflicts, we divided the articles into two groups: research articles, defined as publication with original findings and other types of publications, including editorials, letters and reviews (without a pre-defined methodology). The proportion and type of conflicts of interest per type of article and time between submission and acceptance are presented in Fig. 2.

Fig. 2.

Distribution of conflicts of interest according to the type of article for COVID-19 research articles with a submission-to-acceptance time of a day or less, 16 days and 20 days. COI: conflict of interest. Note: for fairness of comparison, we restricted our analysis to articles submitted before 11^th July 2020, since it was the last submission date at which an acceptance time of 20 days could be observed

As expected, conflicts of interest were most common for editorials letters and reviews, but were also surprisingly frequent for research articles (n=71,31.7% among articles accepted in a day or less). The prevalence of these conflicts was substantially heterogeneous across journals: the estimated intraclass correlation coefficient for the proportion of publications with any conflict of interest was 0.37, (95% confidence interval: [0.29;0.45]), which means that37% of the variability observed in the occurrence of conflict of interest can be explained by the journal in which the articles were published.The frequency of editorial COIs decreases with the increase in time to acceptance for both types of papers, although there are still common at 20 days (n=19,18.6% for research articles, n=24,33.8% for other types of manuscript). Unfortunately, it was not possible to perform the analysis for papers with longer time to acceptance, as it would have restricted the analysis to papers submitted in January and February 2020, reducing the sample size and the generalizability of the results. These findings raise concerns about the fairness and transparency of the peer-review process with such short acceptance times. For example, an opinion paper, reviewing the literature on Hydroxychloroquine and Azithromycin as an early treatment for COVID-19, written by a member of the editorial board of the journal, has been published in the American Journal of Epidemiology within 7 days of submission [60]. This article was followed 3 months later by an expression of concerns from other members of the editorial board, who identified major flaws in the review [61].

While the need for faster scientific dissemination during a pandemic is understandable, the possibility to publish without a rigorous and critical peer-review process is, in some circumstances, detrimental to the scientific community and the public at large. This is the case when these findings are used to inform medical practice or public health policies. For example, following concerns about the scientific validity of a study investigating the effectiveness of hydroxychloroquine, accepted for publication in less than a day after submission [33], post-publication reviews were commissioned. These reviews, published 4 months after the initial publication, [28, 62], pointed out major methodological and ethical flaws. Despite this, the paper was not retracted, on the grounds that it gives room for scientific debate [63]. This is problematic for several reasons. First, such papers with dangerous conclusions are still available to researchers, with no mention within the article of the existence of the post-publication reviews as a warning. Secondly, this type of study is likely to be included in any subsequent systematic review done on the topic, and, even though it may be flagged as a study with a high risk of bias, it might influence the results of the meta-analysis. Finally, the choice not to withdraw the publication, but instead encourage the submission of comments, will help increase the impact of the journal, despite the poor quality of the original publication.

Distrust of published results

In 2016, a survey identified a phenomenon that had later been called the ‘reproducibility crisis’ [49]. In this survey, more than half of the respondents reported that they had experienced trouble reproducing published results (including their own) at least once, raising doubts as to the quality of these published results. This issue has two main causes: the lack of data sharing and the lack of code sharing.

At the heart of biomedical research lies the need for high-quality data to answer relevant clinical questions.. While privacy issues raised by collecting medical data should be addressed, there is a need for data sharing between researchers to reproduce the results, enhance collaboration and obtain more timely results. Achieving both of these goals, however, remains a challenging task for the scientific community, especially during an emergency.

To date, four peer-reviewed articles related to COVID-19 have been retracted shortly after publication due to concerns about potential data fabrication or falsification (see our supplementary materials for the categorized list of retracted papers). Two retracted articles, in particular, attracted a great deal of attention: one in the New England Journal of Medicine [64] and one in The Lancet [25]. These articles reported the findings of an international study, based on data owned by the company Surgisphere. The data were not publicly shared but, even more disturbing, the data were not shared with all the co-authors. The initial publication of the study in The Lancet [25] indicated that hydroxychloroquine increased mortality for COVID-19 patients. These serious safety concerns led the WHO and the French National Institute of Health and Medical Research (INSERM) to interrupt the inclusions in the hydroxychloroquine arm of the Solidarity and DisCoVeRy trials ([65, 66]) while the clinical data of patients treated with hydroxychloroquine were reviewed. Our review of the retracted COVID-19 papers highlights that two additional papers using Surgisphere data have also been retracted. Surgisphere’s refusal to share data with the scientific community, including authors involved in the study, and even to a third trusted party, eventually led to the retraction of all articles. The most striking consequence of this affair, however, is that it may have made scientists, editors, readers, organizations and reviewers waste precious research time.

Open science solutions

Open review

We understand the pressure and the need to accelerate reviewing of submitted manuscripts, but journals and editors should carefully consider the trade-off between reviewing quality and reviewing time. Our findings relating to the fast-tracking of peer-reviews for COVID-19-related articles in “Expedite reviewing and conflicts of interests” section can, however, be perceived as particularly worrying. In a time of pandemic, medical management of patients and public health policies rely heavily on scientific findings. Fast-tracking of peer-review should therefore only be done when scientific rigour can be maintained as its loss might lead to disastrous consequences for public health as a whole. A greater transparency in the peer-review process is thus urgently needed. Sharing reviewers’ reports along with the authors’ response more systematically could contribute to this transparency. These scientific discussions are extremely valuable as they may help balancing the views expressed in the published article. In addition, reviews are usually found to be of higher quality when they are made publicly available [17–19]. The availability of the reviews may also help the scientific community and the different stakeholders to verify that the peer-review of a manuscript has been thorough and so help to increase public trust in scientific research. Several implementation of Open Reviews exists and the name is loosely used to define all of them [17]. In the present case, we argue that making reviewers’ reports available, should become the norm to help mitigate the issues previously highlighted in “Expedite reviewing and conflicts of interests” section, but as correctly argued in the literature (e.g., [17]) enforcing signed open peer-reviews might be problematic

Furthermore, the peer review process should evaluate all aspects of the publications, in particular the methodology, in order to identify inappropriate study designs or incorrect use of statistical methods (see “Identified flaws” section). In order to do so some journals have decided to resort to specialist statistical reviewers [67]. The British Medical Journal’s editorial board, for instance, includes a Chief Statistical Advisor and Statistics Editors [68]. The reviewing process is conducted by domain experts and statisticians in order to make sure that findings hold with respect to the conducted statistical analysis but also to ensure that the statistical analysis conducted by the authors is appropriate and correct. Statistical reviews help in the detection of mistakes in the data analysis and in detecting exaggerated claims in a manuscript before publication. A recent review of retracted research articles showed that among the 429 papers that were retracted from journals in which statistics are assessed, 81 (18.9%) had a specific statistical review, whereas the assessment of statistics was part of the reviewer or editor’s task in the remaining 348 (82.1%) [69]. Although the retraction rate was lower among articles involving a specialist statistical reviewer, the low rate or retraction prevents us from drawing any strong conclusions (5 per 10,000 vs 7 per 10,000).

In light of the lessons learned for reviewing during the current pandemic (“Expedite reviewing and conflicts of interests” section) we recommend the following to be adopted by journals and editors:

1. Authors should highlight, in their manuscript and its metadata, any affiliations with the editorial board of the journal to which they submit their paper. An example of such disclosure can be seen in Pardo et al. [70].

2. Journals should explicitly detail how the peer review was conducted: they should state how many referees were recruited and the duration of the complete process. This includes how long it took to find referees, how long each referee took to complete the review (time between submission and acceptance of the paper) and the number of iterations between the reviewers and the authors.

3. Journals should make the referees’ reports of accepted articles transparently available alongside the manuscript allowing authors, reviewers and the scientific community to benefit from the constructive comments in these Open Reviews. Journals should also consider the various Open Review implementations [17] and let referees themselves decide whether or not to sign their reviews.

4. When publications report quantitative findings, a systematic review of statistical methodology should be included.

Although the peer-review process implemented by journals contributes greatly to the quality of manuscripts, each submission is typically reviewed by a limited number of reviewers whose skills might be too specific to evaluate the different components of a manuscript. Crowdsourced peer-reviews (a.k.a portable peer reviews) reviews that are not ‘on request’ but carried out spontaneously, whether on preprints or post-publication are complementary to the journal’s reviewing. They are open because they are available to all and, very often, signed. The use of these reviews through dedicated platforms such as Pubpeer [23] or Zenodo [71], or even ASAPbio [72], Review Commons [73] and the COVID-19-pandemic-born OutbreakScience PREreview [74] that all allow solicitating reviews on preprints, has gained popularity during the pandemic. For instance, statisticians have published a detailed and comprehensive review of the preliminary report of the RECOVERY trial: the largest comparative effectiveness study on COVID-19 treatments to date [75]. Further examples are presented here [76]. It is notable that the reasons for the retraction of 5 COVID-19 papers were echoed on the Pubpeer link to those publications. Therefore, open reviews can contribute to an early detection of flaws in research articles. However, providing a thorough feedback on publications is time-consuming and reviewing activities are usually not highly valued by institutions. Therefore, changes in peer-review practices cannot be implemented without meaningful support and endorsement from research institutions. Many institutions have created repositories of accepted manuscripts to promote Open Research. These initiatives should go a step further by encouraging researchers to archive the reviewers’ reports as well.

We postulate that adopting Open Reviews and complete transparency in the reviewing process, in addition to all the already highlighted benefits identified in the literature [17–19, 77], would have helped in detecting potential mistakes in manuscripts or frauds and saved precious research time during the pandemic. Furthermore, it would make analyses of peer-review processes and their quality easier to conduct [78] and would accelerate the training of Early Career Researchers [78, 79] whose help might be critical during a public health crisis

Open data and open source

To aid a more thorough peer-review, enable data re-use, and facilitate reproducibility, researchers are now being encouraged to share their data on publicly accessible repositories [80, 81], along with the code used for the analysis. Even though sharing raw data may be challenging in medical research due to the need for compliance with data protection regulation, Open Data initiatives have been considered one of the main solutions to avoid a replication crisis [15, 82] and are often seen today as a critical part of the peer-review system [15].

Data sharing, including anonymized raw clinical data, is crucial during a pandemic and could accelerate the understanding of new diseases and the development of effective treatments [83]. Several researchers have thus, early-on, argued that journals and institutions alike should always ask authors of manuscripts to confirm that the raw data are available for inspection upon request (or to explain why they are not) [84, 85]. Considering that data fabrication and/or falsification has been observed in the scientific community and its frequency probably underestimated [86], some have suggested [15] that the policy of “sharing data upon request” is not enough. Therefore, recent years have seen an increase in policies, from journals and institutions alike, asking researchers to share their raw data by default, except when data could not be shared for privacy reasons [15, 82]. While data sharing is increasingly being adopted, it still does not appear to be a default practice [15]. A recent review of registered COVID19 trials showed that only 16% of 924 included trials indicated a willingness to share the trial data [87]. Maintaining a high-quality database is time- and money-consuming but sharing it allows other researchers to conduct research with the data and optimize the costs. The MIMIC database, a freely accessible critical care database, has led to thousands of articles using and citing this database, and increased clinical and scientific knowledge in this field [88].

To face the dangers of a reproducibility crisis [89], solutions have been proposed by the research community to promote and facilitate code sharing. An open code repository has been created [90] and many scientific journals are now asking researchers to publish the source code alongside their findings. The use of source code sharing platforms, such as GitHub or open source alternatives such as GitLab, is becoming common and has even been advised to improve open science behaviour [6, 91, 92]. The code should be published under a free license to encourage re-use and further developments, and when possible, open source software and programming languages should be preferred to maximize accessibility and reproducibility. The lack of code sharing is still, however, a major issue in the biomedical literature. In times of crisis, the need for open code is even more crucial when scientific evidence is at the root of major political decisions.

We therefore urge that data-sharing policies should be adapted to the following:

1. Research material for data generation and collection (e.g. questionnaires, simulation algorithms, etc.) should be made available on dedicated platforms, and, when appropriate, published under a CC-BY Creative Commons license.

2. Data should be shared by default: authors should not be able to submit a manuscript if they do not provide access to raw data and analysis scripts or a valid reason why they think it is not feasible. This is already the process used by some journals [93].

3. If authors are not able to share their raw data, journal editors should be able and should strive to demand that raw data be examined by a trusted third party (not the authors’ institution) to establish the existence of the raw data and validate the results of the analysis presented by the authors. Identification of the trusted third party can be left to the discretion of the ethics committee or the proprietary company but should not present any conflict of interest with the authors of the manuscript. The trusted third party should produce a public and signed report stating that the data are available and that the results presented in the manuscript can be confirmed.

4. To facilitate meta-analyses, abstracts of all manuscripts should contain links to preregistration numbers, data repositories and open source repositories. Additionally, publishers should consider directly including such information in paper’s XML to allow for even easier retrieval through text mining.

While we understand that changing data-sharing policies will take time, we hope that the scientific community, data-holding companies and governmental agencies protecting rights to privacy will learn the lessons from this pandemic and consider adapting their policies to the aforementioned points.

Stage 3: science communication

Once a manuscript is published, it is available for the rest of the scientific community to cite or conduct meta-studies on, but also to be communicated on by the media or to support policy makers. High quality independent journalism is crucial in our society to ensure a wide, accurate and balanced dissemination of knowledge. Journalists have been having an increasing role in the investigation, presentation, and contextualization of scientific results [94]. Although Open Science may facilitate their investigations, it also makes their work more complex due to the wealth of available information. In this section, we review some of the issues that the pandemic has highlighted with respect to media coverage of scientific results following the publication of preprints and peer-reviewed articles, and we propose alternatives for a more responsible communication of scientific findings.

A surge of preprints and their misuse

There are several benefits of preprints. They allow the communication of new findings to the research community in a more timely manner [13, 95], especially in the context of an emergency such as the current COVID-19 pandemic. Preprints also contribute to the reduction in wasted research by signaling ongoing research projects, avoiding duplication and potentially fostering collaborations. Preprints may also increase a researchers’ visibility and help in credit attribution and priority of DisCoVeRy [13, 96], reducing the risk of plagiarism. In addition, they are an opportunity for authors to get early feedback from a wider range of researchers and incorporate the suggestions received to enhance the quality of the publication [13, 96].

The COVID-19 pandemic has seen a surge in the numbers of preprints submitted by researchers [97]. While 807 preprints were deposited on medRxiv in the six-month period between 1^st July 2019 and 31^st December 2019, 6,771 preprints were submitted in the next six months (between 1^st January 2020 and 30^st June 2020), an increase of 739%. These figures are, respectively, 15,838 and 21,804 for bioRxiv (38% increase) and 87,942 and 112,197 for arXiv (28% increase). A recent study [98] gives more insights on the scale of this disruption. The use of preprints during outbreaks is certainly not new: a systematic review identified the publication of 174 and 75 preprints during the Ebola and Zika virus outbreaks, respectively [99]. Nevertheless, these figures are much smaller than the number of preprints submitted in the first 6 months of the COVID-19 pandemic.

Although the surge in preprints during the pandemic can be seen as an encouraging step towards Open Research, preprints – by their very nature – contain unreviewed findings. While scientific findings from a single study should always be interpreted with caution, the interpretation of results from preprints invites even further caution. Unfortunately, some COVID-19 preprints have been misused, notably regarding communication with the general public. Indeed, various news outlets and also scientists have used these non peer-reviewed articles as scientific evidence, increasing the impact of potentially inaccurate findings. While external peer-review does not guarantee the validity of the results and comes with its own limitations, it contributes to the improvement of the quality of research outputs. Thus, one of the benefits of preprints is to receive feedback from many other researchers with a broader range of expertise, which helps identify and correct potential flaws in the methodology, analysis or reporting, thus enhancing the quality of the article prior to submission. However, many preprints are never accepted for publication in peer-reviewed journals [100]. Looking at previous pandemic data in particular, a systematic review of preprints during the Ebola and Zika outbreaks [99] highlighted that only 48% of the Zika preprints and 60% of the Ebola preprints could be matched with peer-reviewed publications that later appeared. Although we cannot exclude the possibility that the authors never submitted their preprints to a peer-reviewed journal, a potential explanation for this phenomenon could be the presence of concerns expressed by the community on the preprints, highlighting poor methodology or other flaws that rendered the preprints unsuitable for publication.

In order to estimate how often preprints platforms were mentioned in the media, we queried the Factiva news database using the three major platforms hosting COVID-19-related preprints (arXiv, medRxiv and bioRxiv). On the 13^th of July 2020 the three platforms had been mentioned a total of 3,288 times in online news and in 313 blog posts since 1^st January 2020, the day after China alerted the WHO about a newly identified virus. Looking at English-only shares, we find 2,193 web news items reporting scientific findings from recent preprints and 121 articles addressing the surge of preprints during the pandemic and its challenges. These findings raise the concerns that the public may not have been correctly informed about the preliminary nature of these findings.

Next, to quantify the extent to which preprints themselves were shared in the media (news media and social media), we conducted a systematic search for preprints submitted to arXiv, medRxiv and bioRxiv between the 1^st January 2020 and the 30^th June 2020. We then used the altmetricAPI to determine the number of media shares as of the 8^th July 2020. For comparison purposes, we performed the same analysis on non-COVID preprints submitted to arXiv during the same time window. Finally, we performed the same analysis on retracted COVID-19 articles or preprints. The methodology is described in an appendix to this paper, and all the scripts for data extraction and analysis, along with collected data, are available on the OSFrepositoryoftheproject .

As can be seen in Fig. 3, arXiv preprints related to COVID-19 (n=1,462) were shared more often than preprints on other topics (n=80,786) submitted during the same period. A similar pattern has also been reported by Fraser et al. [101]. The difference was more pronounced for mentions in the news: whereas 1,066 (1.3%) of non-COVID-19 preprints were mentioned in the news, 156 (10.7%) of COVID-19 preprints were. The numbers for Twitter appear to be much larger than for any other source, probably because of bots that automatically tweet about new preprints, making the comparison on Twitter less relevant. However, the total number of citations (all sources combined) was larger for COVID-19 preprints, with a median of 6 ([Q1−Q3]=[2−15]) mentions versus 2 ([1−5]) for preprints on other research topics. The number of total citations in the media was even higher for preprints available on platforms dedicated to biomedical research. Out of 1,208 COVID-19 preprints found on bioRxiv, the median number of citations was 30 ([15−85]) and 444 (36.8%) of these preprints were mentioned at least once in the news. Similarly, out of 4,629 COVID-19 preprints submitted to medRxiv, the median number of citations was 12 ([6−36]) and 1,124 (24.3%) of these preprints were mentioned at least once in the news. These findings highlight the increasing trend in preprint sharing during the pandemic, raising concerns about the spread of potentially misleading and unverified data. However, Fraser et al. [101] found that COVID-19 preprints were more commented (on preprint platforms) than non COVID-19 preprints. While this suggests a higher scrutiny for COVID-19 papers, it also illustrate how preprints may encourage scientific debate.

Fig. 3.

Proportion of arXiv preprints shared in the media, broken down by research topic

Openly sharing results that have not yet been peer-reviewed can be very damaging if the media and the public take these findings at face value. An outstanding example is the current debate on the effectiveness of hydroxychloroquine as a treatment for COVID-19, which started after the publication of a methodologically flawed preprint [102]. This study quickly caught the attention of the public (1,458 shares in the media including 54 in the news as of 13^th July 2020), creating a high-demand for the treatment in the absence of valid scientific evidence [103]. This sudden interest contributed to slowing down research conducted on other promising therapeutic strategies. Moreover, the misuse of preprints may discourage researchers from sharing their own in the future, particularly in research areas in which the use of preprints is a quite recent phenomenon, as in medical research (medRxiv was launched in June 2019). Finally, our analysis of retracted papers showed that, among the 6 retracted preprints, the median total of shares was 723 [49−2488], emphasizing further the misuse of unverified and, ultimately, invalid findings.

Nevertheless, we would like to emphasize that while peer-review has a key role in maintaining quality, transparency and reproducibility of published articles, it is not sufficient to avoid the publication of flawed studies, and their use by the media and health authorities. Our analysis of retracted papers highlighted the role of some papers, later retracted, in informing public health policy: before the first expression of concern, the study by Mehra et al. [25] was mentioned in 4 policy documents, from 2 sources, including WHO. Besides, this study was the object of 1005 news articles before its retraction, among which 912 were not about the methodological concerns. Similarly, a study investigating the effectiveness of face masks [104] was used in 2 policy documents and was discussed in 132 news outlets.

A call for more reasonable communication

Many science journalists and news editors rely extensively on press releases from institutions [105]. Academic press releases have already been in the spotlight for their impact on the dissemination of exaggerated findings in the news (e.g., [106–108]) and can therefore directly exacerbate the spread of non-peer-reviewed findings if their communications are based on preprints. Concerns have already been raised about the role of preprints in the communication of science to the public and its potential dangers [109]. Nonetheless, the advantages of preprints for scientific communities are too important to completely give up on preprints [96] and preprints platforms have already implemented warning messages on manuscripts to explain that they have not been peer-reviewed.

An obvious solution to the issue would be to recommend that press releases from research institutions should be made only with respect to peer-reviewed studies, and should be written in collaboration with independent scientists. However, in some cases, it is necessary to report the findings of studies that have not yet been peer-reviewed, if they are expected to benefit the public at large. Should they do so, journalists and news editors must then be encouraged to search for potential conflicts of interest and check the availability of registered information and external peer-reviews to ensure the quality and trustworthiness of their article. Findings from preprints should be communicated with particular caution. Despite the advantages of making headlines simple or even exaggerated [110], journalists and news editors should make sure to accurately convey the inherent degree of uncertainty in scientific studies, which is often quantified and explicitly stated in the original articles. However, such measures are clearly not enough: with the increased productivity pressure on science journalists the fact-checking process needs to be sometimes less thorough [105]. While it seems sometimes difficult [105], journalists and scientists should work together, in particular for scientific results pertaining to public health or that could imply a change of behaviour of the public.

The misuse of preprints by some journalists emphasises the need for high quality journalism training. This issue is not new and has already been pointed out [111–113], for example during the Ebola crisis [114]. The misuses of research outputs are not limited to preprints but also includes poor quality studies in peer-reviewed and predatory journals. It is clear that a high quality dissemination of scientific information is essential to an appropriate public health response to a crisis such as COVID-19 [115]. The scientific community has addressed this issue by publishing some guidelines for a better dissemination of scientific news to the public [112] and also by fostering bridges between the scientific community and science journalists through exchanges and training [111]. As an example, the French association of science journalists, the Association des journalistes scientifiques de la presse d’information, has launched and funded an exchange program between researchers and journalists. In the UK, the Science Media Centre provides support to news reporters to help them to accurately interpret new findings from publications or press releases [116], by ensuring that journalists, scientists and statisticians work together. Training in science journalism in times of crisis is even more essential and has previously been attempted during the Ebola crisis [114]. During the COVID-19 crisis, the United Nations Educational, Scientific and Cultural Organisation (UNESCO) has organized a series of ‘webinars’ tackling the importance of journalists’ scientific literacy [117]. However, this process is still ongoing and – since many examples of problematic dissemination of scientific news are still seen – some researchers have proposed indicators to evaluate the scientific accuracy of news articles [118]. Some scientists are also grasping these issues by writings blogs directly addressing scientific integrity and exposing examples of bad science in the scientific literature (see e.g., the work of Elisabeth Bik [119]), directly motivating a more responsible science communication.

Discussion

In addition to previous concerns and investigations of the disruption that the pandemic has caused for research [2], we have found strong evidence of how COVID-19 has impacted science and scientists on several levels. All our findings and the solutions we argued for are summarized in Fig. 4. Firstly, we have highlighted the striking scientific waste due to issues in study designs or data analysis. Secondly, we have found that the fast-tracking of peer-reviews on COVID-19 manuscripts, which was needed to give vital treatment directives to health authorities as quickly as possible, led to potentially suspicious peer-reviewing times often combined with editorial conflicts of interest and a lack of transparency of the reviewing process. Thirdly, we highlighted that the lack of raw-data sharing or third-party reviewing has led to the retraction of four major papers and had a direct impact on the study design and conduct of international trials. Finally, we have found evidence of the misuse of preprints in news reports which seem to refer to non-peer-reviewed manuscripts as reliable sources. While we have focused on medical research in the present paper, these issues have also been identified in other research areas, where Open Science principles should also be implemented.

Fig. 4.

A summary of our findings and proposed solutions

The Open Science movement promotes more transparency and fairness in the access to scientific communication, the production of scientific knowledge and its communication and evaluation. Looking at the number of publishers removing their paywalls on COVID-19 related research, one might argue that the COVID-19 pandemic has been a catalyst in the adoption of Open Science principles. However, the aforementioned issues paint a more complicated story. The urgency of the situation has led to a partial Open Access policy but with a very opaque peer-review process coupled with a misuse of preprints and raw-data-sharing policies not being enforced. Full adoption of Open Science principles could, however, have saved precious research resources: open peer review would have helped in the detection of the editorial conflicts of interest and made it apparent whether manuscripts were thoroughly reviewed; adoption of registered reports would have strengthened study designs and data analysis plans; proper and monitored use of preprints would have helped the communication of early results between researchers; strengthening of the policies of raw-data sharing or reviewing could have prevented the Surgisphere scandal; and full Open Access might have accelerated the search for solutions to the pandemic both in medical and socio-economic contexts. In addition to this, statistics reviews could have helped to make studies and their results more robust and limit the impact of exaggeration or misinterpretation of results.

It remains, however, that these principles are not enough. The pandemic has highlighted other issues that Open Science cannot solve. For instance, the misuse of preprints by journalists probably stems from the fact that many journalists may not be trained to understand and navigate the complex academic publication system, and some journalist may be seeking sensationalist news headlines. The pandemic has also highlighted the already-existing science-literacy issue [120, 121]. Finally, we cannot exclude that some of the misuses and abuses that we have highlighted are a direct result of the current metric-centered evaluation of research and researchers which has already been shown to lead to questionable research practices in the past and has been the subject of criticism from scientists for decades [43, 122, 123]. Researchers have argued that the adoption of transparency should be coupled with the adoption of a more diverse set of metrics to evaluate researchers [124, 125] or a rejection of metrics altogether [126, 127] to truly limit questionable research practices. A wider adoption of these Open Science Principles cannot be achieved without the endorsement and support of institutions, publishers and funding bodies. International initiatives, such as the Declaration on Research Assessment (DORA), have been put in place to reform the process of research assessment and funding [128], promoting research quality over quantity of outputs. Senior academics have also been identified as key agents in the support of Open Research [129]. For Open Science principles to be clearly and widely adopted, all actors in the scientific community have a role to play: established researchers should encourage a transition to transparent research; institutions and funding agencies should diversify research evaluations; journals, editorial boards, and funding agencies should make all Open Science practices the de facto standard for submissions (especially Open Data and registered reports); publishers should strive to make all papers Open Access; and policy-makers and international review boards should consider opening sensible data to reviewers or trusted parties for external validation. We recognize that the adoption of Open Science principles is also dependent on the wider political context, and their implementation may be hindered by the governance in place, for instance in autocratic countries. More generally, there is a strong interplay between the direction of scientific research and political directions [130, 131]. Therefore, governments themselves should be an agent in the transition towards Open Science, and in turn, the adoption of Open Science principles by scientists could influence future political changes in the area of research and international collaborations [132].

We, as scientific researchers attached to transparency and fairness in the production, communication, use and evaluation of scientific knowledge, hope that this manuscript successfully argues and promotes a faster adoption of all Open Science principles.

Appendix

Factiva analysis

The Factiva analysis, to find occurences of preprints in the news, was done between January 1 ^st 2020 and July 13^th 2020 with the following query:

Altmetric analysis

To further analyse if COVID-19 preprints were used in the news more than preprints are regularly used we conducted an altmetric analysis of all COVID-19 preprints found on arxiv.org, medrxiv.org, and biorxiv.org. We first downloaded all the COVID19-related preprints from these three platforms from the 1st of January 2020 to the 30 ^th of June 2020, as well as all preprints from arxiv.org in the same period (to serve as a control group). Duplicates were removed. We then queried the altmetricAPI for each of these preprints using a Python script to process all entries, find their DOI and query Altmetric with the following command:

Analysis codes are available on the GitHub repository of the project: https://github.com/lonnibesancon/OpenSciencePandemic

PubMed central analysis

To extract the reviewing times, the metadata of 12,682 COVID-19 articles were downloaded on July 7, 2020 from PubMed Central using the query: “COVID-19”[abstract] OR “COVID-2019”[abstract] OR “severe acute respiratory syndrome coronavirus 2”[Supplementary Concept] OR “severe acute respiratory syndrome coronavirus 2”[abstract] OR “2019-nCoV”[abstract] OR “SARS-CoV-2”[abstract] OR “2019nCoV”[abstract] OR ((“Wuhan”[abstract] AND (“coronavirus”[MeSH Terms] OR “coronavirus”[abstract])) AND (2019/12[PDAT] OR 2020[PDAT])) The reviewing times were extracted from the data using a MATLAB script, available on the OSF repository.

Abbreviations

COI

Conflict of interest

COVID-19

Coronavirus disease 2019

OSF

Open Science Framework

SARS-CoV-2

Severe acute respiratory syndrome coronavirus 2

WHO

World Health Organization

Authors’ contributions

L.B. and C.L. conceived the study. L.B., C.L, M.D., H.J., and C.S. extracted the data and performed the analyses. L.B., N.P.-S., C.Se., P.M., C.Sm., M.D, and C.L. wrote the manuscript. All authors have read and approved the manuscript.

Funding

There was no dedicated funding for this research but CL is supported by the UK Medical Research Council (Skills Development Fellowship MR/T032448/1).

Availability of data and materials

The data and scripts are available on https://osf.io/renxy/.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.