COVID‐19 and the generation of novel scientific knowledge: Evidence‐based decisions and data sharing

Abstract

Rationale, aims and objectives

The COVID‐19 pandemic has impacted every facet of society, including medical research. This paper is the second part of a series of articles that explore the intricate relationship between the different challenges that have hindered biomedical research and the generation of novel scientific knowledge during the COVID‐19 pandemic. In the first part of this series, we demonstrated that, in the context of COVID‐19, the scientific community has been faced with numerous challenges with respect to (1) finding and prioritizing relevant research questions and (2) choosing study designs that are appropriate for a time of emergency.

Methods

During the early stages of the pandemic, research conducted on hydroxychloroquine (HCQ) sparked several heated debates with respect to the scientific methods used and the quality of knowledge generated. Research on HCQ is used as a case study in both papers. The authors explored biomedical databases, peer‐reviewed journals, pre‐print servers and media articles to identify relevant literature on HCQ and COVID‐19, and examined philosophical perspectives on medical research in the context of this pandemic and previous global health challenges.

Results

This second paper demonstrates that a lack of research prioritization and methodological rigour resulted in the generation of fleeting and inconsistent evidence that complicated the development of public health guidelines. The reporting of scientific findings to the scientific community and general public highlighted the difficulty of finding a balance between accuracy and speed.

Conclusions

The COVID‐19 pandemic presented challenges in terms of (3) evaluating evidence for the purpose of making evidence‐based decisions and (4) sharing scientific findings with the rest of the scientific community. This second paper demonstrates that the four challenges outlined in the first and second papers have often compounded each other and have contributed to slowing down the creation of novel scientific knowledge during the COVID‐19 pandemic.

1. INTRODUCTION

The COVID‐19 pandemic has impacted every facet of society and has created a multitude of challenges for national and supranational organizations, such as the WHO. COVID‐19 originates from a novel coronavirus and the scientific community has been faced with the daunting task of creating a novel model for this pandemic, or in other words, creating novel science. This series of papers explores the intricate relationship between the different challenges that have hindered biomedical research and the generation of scientific knowledge during the COVID‐19 pandemic. This second paper will also use research on hydroxychloroquine (HCQ) as a case study.

In the previous article, it was argued that, in the context of the COVID‐19 pandemic, the scientific community has been faced with challenges with respect to (1) finding and prioritizing relevant research questions and (2) choosing study designs that are appropriate for a time of emergency. First, a lack of research prioritization resulted in redundancy in research works and the dispersal of scarce resources (funding, hospital infrastructure, staff and patient base). The duplication of research works, combined with poor‐quality research, has greatly contributed to slowing down the creation of novel scientific knowledge. With respect to study designs, the previous paper has demonstrated that members of the scientific community took part in heated debates regarding the most appropriate design for an emergency. These oppositions, as well as the overall low methodological quality of studies on HCQ, suggest that methodological rigour and the notion of design complementarity have sometimes been abandoned.

This follow‐up paper will now examine the challenges presented by the COVID‐19 pandemic in terms of (3) evaluating evidence for the purpose of making evidence‐based decisions and (4) sharing scientific findings with the rest of the scientific community and the general public. This second paper will demonstrate how these challenges and those presented in the first paper have compounded each other to hinder biomedical research and the generation of novel scientific knowledge during the COVID‐19 pandemic.

2. MAKING EVIDENCE‐BASED DECISIONS

Questions about what kinds of evidence should be used, how evidence is to be evaluated and whether the answers to these questions change during a health emergency have often been discussed. This long‐lasting debate among decision‐makers, clinicians, researchers, and philosophers of science has been the essence of most discussions around the generation of novel scientific knowledge during the COVID‐19 pandemic. Two approaches for making decisions during emergencies are staples in the biomedical literature: the precautionary approach* or an evidence‐based approach. ¹ The precautionary approach is often used to justify the implementation of non‐pharmaceutical interventions. However, following the precautionary approach is often seen as harder to justify in the case of pharmaceutical interventions considering the perceived risks associated. ² Three factors, which this paper will examine consecutively, might explain why making evidence‐based decisions on investigational drugs has been difficult in the context of this pandemic.

2.1. The controversial nature of evidence

Thriving to understand the meaning of the term ‘evidence’ has been the essence of a long‐standing debate that has yet to find a definite answer. The growing influence of evidence‐based medicine (EBM) inspired evaluation frameworks to categorize studies as randomized controlled trial (RCT) or non‐RCT in a manner that is oblivious to the diversity of designs.† ³ The idea that RCTs are the ‘gold standard’ of evidence shapes most of the discussions regarding the efficacy of HCQ and has led to the neglect of relevant pieces of evidence. Most systematic reviews, including that of the WHO, ⁴ only take into consideration RCTs testing HCQ and automatically discard all other evidence. However, findings from non‐RCT studies have sometimes been more robust and generalizable. While RCTs are often considered the ideal design to determine causal inferences and reduce biases, they should not be considered flawless. ⁵ , ⁶ The three limitations that are significant in the context of COVID‐19 are:

1. Inability to draw robust causal inferences. Making causal inferences is essential in the context of the COVID‐19 pandemic since any proposed intervention must be accompanied by confidence that the intervention will change the outcome. Borgerson notes that ‘claims about the special ability of RCTs to isolate causes refer to probabilistic causes and downplay the possibility that mechanistic causes could be just as well established, just as epistemically strong, and just as useful in medical practice’. ^{5 (p.222)} ‡ In the context of COVID‐19, this is an issue since clinical trials were launched, and decisions were made, without thoroughly understanding the mechanisms of action and transmission of SARS‐CoV‐2. New information from lab‐based, mechanistic studies has sometimes undermined clinical trials.§

2. Randomization issues. Randomization is thought to eliminate confounding factors, thereby allowing researchers to isolate the intervention's effects. While randomization certainly has epistemic and scientific values, Worrall has argued persuasively that it only reduces the likelihood that confounding factors will affect the results but does not eliminate it.¶ ⁶ Even if randomization eliminated confounding factors, it would require a larger sample size than those used in the studies on HCQ. In the context of COVID‐19, exclusively focusing our attention on randomization has sometimes been an obstacle to recognizing the quality of findings generated by retrospective cohort studies.**

3. Lack of external validity. While a RCT is considered, in the evidence hierarchy, as the best design to ensure internal validity, it is not necessarily the best design to generalize results. Black argues that generalization to the whole population is more easily determined from an observational design (since it usually has broad inclusion criteria and preserves the context of care). ⁷ In the context of the COVID‐19 pandemic, the generalization of research findings has often been a challenge. For example, conclusions obtained by Gautret and colleagues ⁸ on the efficacy of HCQ could not be reproduced by Molina and colleagues. ⁹ Thus, their conclusions are not externally valid (either because they are not internally valid, the methods are not reproducible, or the population sampled in the second study was fundamentally different from that of the first study).

Given these limitations, proponents of EBM acknowledge that ‘there are always exceptions to the general rules’. ^{10 (p.165)} Nevertheless, it is arguable that non‐RCT designs and mechanistic studies should not be the exception but, instead, be considered complementary to RCTs. ⁷ Insights gained from research on HCQ and the theoretical limitations outlined above show that scientific rigour, although crucial, cannot be restricted to the use of RCTs. The scientific community should critically appraise the evidence available on a case‐by‐case basis, instead of relying on a set of predefined criteria.

2.2. Inconsistent and fleeting evidence

Upshur reminds us that ‘all evidence is capable of being overturned or modified in light of new findings’, ^{1 (p.109)} which further complicates making evidence‐based decisions. Evidence on HCQ has sometimes been uncontested for only a few days before being invalidated by new findings. Moreover, as Russell and colleagues ¹¹ suggest, the generation and evaluation of evidence cannot be completely judgement‐free. As such, basing decisions on a single study is problematic though, as we have seen, such decisions have been made frequent during the COVID‐19 pandemic.†† The threshold of evidence required to take action is ambiguous: there is always a tension between wanting to take immediate action and gathering more evidence. ¹ However, the biological, physiological and pharmacological complexity at work does not allow for rushed decisions regarding vaccine and drug approval.

2.3. Evaluating evidence: A time‐consuming process

Making decisions based on a body of knowledge requires that the evidence be first evaluated. However, conducting systematic reviews takes time (between 6 and 24 months). ¹² , ¹³ As such, rapid reviews were developed to evaluate evidence in less than 3 months and are commonly used during health emergencies. ¹⁴ Several methodological modifications are used to fast track the process, such as limiting the scope, the outcomes of interest and the number of databases reviewed, adding more reviewers or defining more restrictive search criteria. ¹⁴ Two studies show that very few differences exist between the conclusions reached by both review types. ¹⁵ , ¹⁶ Interestingly, however, there is no standard methodology to conduct rapid reviews – which is not an issue as long as the authors are transparent about their methods. ¹⁷ In the context of COVID‐19, one problem has precisely been the lack of transparency regarding the methods used in rapid and systematic reviews. Ruano and colleagues also claim that out of the 18 peer‐reviewed systematic reviews published on COVID‐19 up to 24 March 2020, 13 were considered of ‘critically low’ quality by AMSTAR 2. ^{18 (p.2)} This issue is compounded by a tendency to consider RCTs as the only source of evidence, thereby ignoring a valuable part of the knowledge base.

2.4. Non‐evidence‐based and rushed decisions

Given the limitations regarding the nature of evidence and the complexity of evaluating evidence in a timely manner, some might ask whether basing all decisions solely on evidence is meaningful. National public health leaders have often portrayed their recommendations and injunctions as evidence‐based but what is intended by this declaration is not entirely clear. Several organizations have modified their guideline development process. During an emergency, the WHO ¹⁴ is no longer bound to support decisions on systematic reviews and can rely exclusively on expert opinion,‡‡ which, interestingly, ranks at the bottom of the evidence hierarchy. The FDA has also developed ways to fast track the approval of therapeutics (fast‐track, breakthrough therapy, accelerated approval and priority review), ¹⁹ which have been used, and proved to be efficient, during emergencies. However, the threshold of evidence required to approve a drug under a ‘Fast‐Track’ approach remains unclear, especially given the fleeting and inconsistent nature of evidence.

While it might not be sustainable to maintain the traditional standards of evidence during a pandemic, disproportionately lowering these standards might also be problematic. Several rushed decisions based on a single study were made, ²⁰ such as the WHO's decision to halt HCQ treatment arms on 25 May 2020 and resume them on 3 June. ²¹ Other examples of rushed decisions include the FDA's Emergency Use Authorization for HCQ on 28 March ²² 2020, which was retracted on 15 June, ²³ and the addition of HCQ to the WHO's list of prioritized drugs (13 March). ²⁴ Chen and colleagues' ²⁵ study on 30 patients was the only completed study on HCQ that could be evaluated by peers before that date. Thus, it can be argued that this decision was primarily influenced by the growing international media coverage on HCQ. Retrospectively, and considering the number of trials that stopped enrolment in their HCQ arms, ²⁶ , ²⁷ these decisions seem to have been rushed. They also resulted in HCQ shortages for patients with conditions other than COVID‐19 and more frequent self‐medication incidents. ²⁸

Theoretically, the implementation of non‐pharmaceutical interventions (NPIs) might not need to be supported by as much evidence and can be safely implemented following the precautionary approach (given the low risks). Conversely, the approval of pharmaceutical interventions should be supported with much more evidence with respect to the associated perceived risks. However, what has happened since January is precisely the opposite: policymakers have sometimes waited for extensive evidence before implementing NPIs but have made rushed decisions regarding pharmaceutical interventions. This can be explained, in part, because, with respect to NPIs, adherence is more easily obtained if the population believes that the intervention is scientifically supported. On the other hand, the decision to allow the use of HCQ in the clinical setting and the FDA's emergency use authorization can be seen as a way to delegate decisions to clinicians' expert judgement. In that case, patients' adherence is facilitated by their trust in their family doctors, whom they see as authority figures.

The fleeting nature of evidence, as well as the complexity of evaluating a body of knowledge in a timely manner, has been an obstacle to the development of guidelines during the COVID‐19 pandemic. It has not always been possible to sustain prevailing standards of evidence. While evaluating methodological rigour is essential, other criteria should be taken into account, notably, whether the intervention can be easily and equitably administered, acceptable to patients and has a favourable cost–benefit profile. This is not something a RCT can always determine. ²⁹ These considerations support the idea that different kinds of studies might be more appropriate depending on whether the prioritized objective is to determine the intervention's effects, produce generalizable results, or draw robust causal inferences.§§ For evidence to be appropriately used in public health decision making, both the reliability (which is often assessed using tools such as GRADE) and the relevance of evidence must be evaluated. ³⁰ In the context of this pandemic, evidence that is relevant to the issue at hand may not exclusively originate from RCTs and can just as well be found in observational and mechanistic studies.

3. SHARING SCIENTIFIC FINDINGS

The fast reporting of accurate scientific knowledge also proved to be a challenge during the COVID‐19 pandemic. The sharing of scientific evidence contributes to the generation of new knowledge by allowing scientists to build on others' work and find new, relevant research questions. Policymakers also need to quickly and easily access these findings to readjust their decisions as new evidence is generated. During a health emergency, the need for rapid reporting of scientific knowledge must not come at the cost of compromising its accuracy. The reporting of inaccurate findings is detrimental both for future research efforts and the public's perception of the pandemic.

Issues regarding the rapid reporting of scientific findings have already been debated during past health emergencies. Concerns first arose in 2007 during the H1N5 outbreak in Indonesia when the country refused to share the virus' genome with the WHO.¶¶ ³¹ If shared, the Indonesian government feared that they would not derive any benefit from the development of future vaccines or therapeutics. This crisis incentivized the WHO to develop the Pandemic Influenza Preparedness (PIP) network to ensure that benefits derived from the sharing of genome data would be returned to local populations at a price they can afford. ^{31 (p.21)} The PIP network purports to facilitate the sharing of genome sequences and create a framework whereby the industry has to assist developing countries to have access to genome information. ³² Nevertheless, because the WHO has no international jurisdiction, it remains to be seen whether low and middle‐income countries will really have equitable access to the findings of COVID‐19 research. The question of transparency in the sharing of research findings was further debated during the Zika outbreak and resulted in the creation of Zika Open, a platform for the open sharing of papers related to the virus. ³³

On January 31, 2020, Wellcome, a foundation dedicated to addressing public health challenges, released a statement encouraging researchers, journals and funders to share COVID‐19 research findings as rapidly and openly as possible, in an attempt to keep the WHO informed of the latest advancements. ³⁴ This statement outlined five recommendations:

• All peer‐reviewed publications on COVID‐19 are made open access during the pandemic,

• Research findings are shared with the WHO upon journal submission,

• Research findings are made available on pre‐print servers with clear statements regarding the limitations of data,

• Researchers share interim and final research results, together with protocols and standards used to collect the data, as rapidly and widely as possible,

• Authors understand that data shared ahead of submission will not preclude their publication.

Scientific journals have explicitly stated that, in the context of COVID‐19, they will expedite all editorial steps. As such, articles have sometimes been published in less than 48 hours. ³⁵ Following these five principles and compared to past outbreaks, data sharing at the basic science level has been incredible since the beginning of the pandemic. On 11 January, the first full genome sequence of SARS‐CoV‐2 (obtained on 3 January) was shared on a discussion forum, virological.org. By 2 February, de Oliveira and colleagues had developed a software program to classify genomes of SARS‐CoV‐2. ³⁶ The development of reagents for diagnostic tests has been relatively fast (11 January), which is important progress compared to past health emergencies (e.g., SARS ³⁶ ). The use of pre‐prints has also been widely encouraged*** ³⁷ and has exponentially increased, allowing faster results reporting. While this is crucial for scientists, the growing use of pre‐prints has had negative consequences on the public's understanding of the pandemic. Indeed, information derived from papers posted on pre‐print servers was reported in the media, often without outlining the study's limitations, and has contributed to the spread of misleading information. ³⁸

The fourth recommendation outlined in the Wellcome statement regarding the sharing of clinical findings has been relatively poorly followed. While interim results of clinical trials have sometimes been shared, ²⁶ , ³⁹ most trials do not release interim results nor protocols. It seems that only the protocols of large, international, highly publicized trials were released (such as REMAP‐CAP and RECOVERY, but interestingly not SOLIDARITY). Sharing interim results also comes with its own set of issues: clinicians and researchers involved in the trial might subconsciously change their behaviour and alter the outcome as well as patient accrual, adherence and retention.††† ⁴⁰ , ⁴¹ , ⁴² The reporting of findings at the clinical level is also complicated by the need to accommodate different values and interests. During a health emergency, there is a strong incentive to publish quickly, given the number of knowledge gaps (i.e., the need for novel scientific knowledge) and lives at stake. Generating evidence to inform international and national decisions in a timely manner often comes back as a core ethical requirement during health emergencies. ⁴³ However, and in addition to the ‘publish‐or‐perish’ culture in academia, a health emergency provides strong incentives to publish articles that lapse into sensationalism, sometimes at the cost of quality. ⁴⁴ At the clinical level, there are also numerous potential financial and academic benefits associated with the commercialization or patenting of therapeutics and vaccines. Journals might also have strong incentives to expedite the publication process to be recognized as the first to publish a world‐changing paper. Therefore, the need for fast knowledge reporting is, sometimes, in conflict with reporting accurate information. While retraction of scientific papers has always happened, often without sparking public interest, several COVID‐related papers, that had been highly publicized in the media, have been retracted. As of 16 January 2021, 62 COVID‐19 related papers were retracted, and four are the subject of a statement of concern. ⁴⁵ Six of these retracted papers investigated the role of HCQ as a treatment for COVID‐19. However, the rapidity with which errors have been flagged by the scientific community and rectified by editors can be appraised. Compared to a paper published in The Lancet on a possible relationship between the MMR vaccine and autism, which was retracted 12 years after its publication, questionable papers related to COVID‐19 were retracted within days. ³⁵

The publication of findings during the COVID‐19 pandemic highlights the need to follow the five principles outlined by Smith, Upshur and Emanuel ⁴³ : ensuring scientific accuracy, social value (data must be released and (in)validated by the scientific community), protection of research participants, transparency and accountability on the part of journal editors. Contradicting evidence has been reported, almost in real‐time, by the media and has affected the public's understanding of the pandemic and trust in science. This has resulted in inappropriate behaviours, such as the panic buying of HCQ, leading to shortages for those who need it, ⁴⁶ and increased risks associated with self‐medication. ²⁸ , ⁵⁸ In a time where confusion, uncertainty and fear rule, and where mitigation strategies rely on people's adherence to science‐based guidelines, it is particularly important to communicate scientific findings, and their limitations, in a clear and transparent manner.

4. DISCUSSION

In the context of the COVID‐19 pandemic, research conducted on pharmaceutical treatments, and especially HCQ, has generated low‐quality evidence and inconclusive findings, which have had negative consequences on patients, the public and other ongoing research efforts. Part one and two of this series of papers have attempted to evaluate the factors that have interfered with the generation of novel scientific knowledge and have demonstrated that such challenges are to be found at each step of the research process. First, a lack of prioritization among research questions and therapeutics has, in part at least, been responsible for the duplication of research works and the dispersion of scarce resources. Study designs, aimed at minimizing biases and increasing objectivity, have, instead, been the subject of fruitless oppositions. During the pandemic, it seems that methodological rigour and the notion of design complementarity were somewhat abandoned. These two issues combined have resulted in the generation of fleeting and inconsistent evidence that has been an obstacle to the development of public health guidelines. Finally, the reporting of scientific findings has again highlighted the difficulty of finding a balance between accuracy and speed. Inter‐epidemic efforts have shaped and improved the COVID‐19 research response, especially in terms of expedited ethics approval and the sharing of basic science research. Interestingly, these achievements constitute the focus of our efforts since the last health emergencies, which should motivate researchers to address the remainder of challenges that are obstacles to the generation of novel scientific knowledge (such as the duplication of research works or the sharing of clinical data). The COVID‐19 pandemic will undoubtedly contribute to reshaping the way we think about research during health emergencies and encourage us to approach them in terms of alternate phases of preparation, response and learning instead of disconnected outbreak events.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Lucie Perillat: conceptualization, investigation, writing – original draft preparation, writing – review and editing. Brian S. Baigrie: conceptualization, writing – review and editing.

Perillat L, Baigrie BS. COVID‐19 and the generation of novel scientific knowledge: Evidence‐based decisions and data sharing. J Eval Clin Pract. 2021;27:708–715. 10.1111/jep.13548

DATA AVAILABILITY STATEMENT

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.