Will Evidence-based Medicine Survive the COVID-19 Pandemic?

While mankind is facing the worst global healthcare crisis of this century, our use of evidence-based medicine has suffered major setbacks. As recently discussed (1), social network, television shows, and other media platforms have been flooded by “experts,” who have voiced strong opinions on the treatment of patients with coronavirus disease (COVID-19). In this setting, hydroxychloroquine has been portrayed as a potential lifesaving drug in the current pandemic, mainly based on opinion or results of small clinical studies and uncontrolled experiments (2, 3).

One common argument for the use of an untested intervention during a pandemic is that “we must do something.” This may be dangerous and abandons the principles of evidence-based medicine. Indeed, physicians using an untested intervention at the bedside may suffer from attribution bias, which is a selective observation of favorable effects attributed to the intervention that lead to undue confidence in its effectiveness. The other available possibilities, such as harm resulting from the direct use of the intervention or a good recovery independent from it, can only be drawn with well-conducted clinical trials.

At the time of writing this editorial, the use of hydroxychloroquine for COVID-19 is a clear example of practice changing despite limited evidence to support its use. Hydroxychloroquine has failed to prove beneficial in trials when used as a potential treatment for previous viral diseases (4–6). Specifically, in COVID-19, the available evidence points toward a neutral effect of the drug (7, 8), with some studies suggesting harm (9). Nevertheless, to date, there are no well-powered randomized clinical trials testing hydroxychloroquine in this group of patients to inform safety and effectiveness.

In the face of the COVID-19 pandemic, it is understandable that physicians and patients are scared, overwhelmed, and want quick answers. There are concerns about the time to complete randomized clinical trials, and this is used as a justification to accept anecdotal and low levels of evidence. Indeed, the traditional approach to validate a new treatment is to complete lengthy phase I, II, and III studies. One major challenge during the pandemic is to design clinical trials that can mitigate these concerns and quickly identify effective or harmful interventions to improve patient outcomes. Careful consideration is required to optimize trial design to achieve this, and several novel trial designs are available and could be considered. These include adaptive sequential designs, response-adaptive randomization, historical and dynamic borrowing, multistage multiarm trials, shared controls, and strategies that aim to “pick the winner” to identify early which treatments are effective.

A good example of trial design that allows this flexibility is Bayesian design. Bayesian trials allow evidence about treatment to be continually updated with new information as it becomes available, maintaining trial integrity (10–12). Another benefit of the Bayesian approach is to estimate the probability that a treatment is effective rather than focus on whether it is effective or not according to P values. Frequentist classical trials rely on previous knowledge to calculate sample size and define features of the study according to known assumptions about the treatment, something not widely available during a pandemic. Also, classical designs are less flexible, and if the assumptions are not met, the study will end without providing useful evidence. In a pandemic with more than 75,000 new cases per day, the use of a Bayesian adaptive trial design can quickly incorporate existing evidence, drop interventions that have a higher probability of futility, redirect patients to be randomized to the most promising ones, and constantly include new and potential candidate interventions.

In this issue of AnnalsATS, Casey and colleagues (pp. 1144–1153) (13) describe the study protocol of a randomized, double-blind, placebo-controlled clinical trial (ORCHID trial) assessing the impact of hydroxychloroquine in hospitalized adults with confirmed severe acute respiratory syndrome coronavirus 2 (SARS-Cov-2) and symptoms of acute respiratory infection. The primary outcome of the study is the patient’s clinical status 14 days after randomization, assessed by a seven-category ordinal outcome scale. An important feature is that data can be collected from the electronic health record, decreasing the person-to-person contact, conserving personal protective equipment, and reducing the risk of infection. A major strength of the study is to employ a Bayesian framework, allowing multiple interim analyses, the possibility of incorporating new external evidence, and the early stopping of the trial according to predefined probabilities of benefit or futility.

In the last 10 years, Bayesian adaptive clinical trials have been increasingly used to hasten the overall trial process (10, 11). In general, these studies share common features, especially algorithms to greatly reduce the sample size needed to assess the intervention without lowering the statistical power of the study. As a result, interventions can progress more quickly through all the processes, which is urgently needed and of utmost importance during a pandemic. A quick approval of a potential intervention is expected by policymakers and consumers, as early use of an effective treatment may improve patient outcomes and prevent harm. However, the risk of approving an ineffective or unsafe intervention is not often considered. Indeed, withdrawing an approved therapy can be challenging, disruptive, and sometimes impractical.

The ORCHID trial investigators expect to finish the study with a sample size of 510 patients, which is realistic and acceptable as the United States of America currently has ∼20,000 new cases a day. In a recent simulation study (14), the optimal sample size for clinical trials decrease with the infectivity of the epidemic. This suggests that a Bayesian adaptive design allows the flexibility needed to adapt the study to specific parameters and stages of the epidemic.

One potential limitation of ORCHID trial, and several other trials testing therapies in COVID-19, is evaluation of the primary outcome in a short time frame. Though understandable because of the urgent need for answers, we have learned that these patients usually have a longer convalescent period. Fifteen days is relatively short and may not accurately capture the majority of patients who have died or recovered. A longer period of observation is somewhat problematic when there is very rapid randomization, such as during the pandemic, as it delays the trial results. This may interfere with the interim analyses, as many randomized patients will not have a measure of the outcome of interest, making it difficult for the data safety and monitoring committee to recommend stopping the trial if required.

In this time of uncertainty, when clinicians are desperately seeking effective therapies to fight COVID-19, the ORCHID trial provides several unique solutions to the constraints of traditional clinical trials. These include processes for rapid administrative and regulatory approvals, production of visually identical placebo pills, the use of interactive platforms for informed consent, and flexible Bayesian trial design. Studies like the ORCHID trial, and others in the field, are a whisper of rationality during irrational times and rekindle the hope that evidence-based medicine will survive the COVID-19 pandemic.