Evidence-based medicine in times of crisis

The COVID-19 pandemic represents the world's worst public health threat since the 1918 flu pandemic. Infected persons face the prospect of serious pneumonia, progression to critical illness, and possible death. In the face of this threat from a hitherto unrecognized disease caused by the novel virus, people are appropriately frightened and naturally seek medications that might prevent or cure the illness. In response to such sentiments, there is no shortage of suggestions—from public policy advice to essentially lock down societies, to the use of host of antibiotics, antiviral drugs, antimalaria drugs, and antibody-carrying plasma from people recovered from COVID-19.

We should be clear: making mistakes, sometimes with grievous consequences, is inevitable. The variability in public policy and clinical behavior in response to COVID-19 demonstrates those errors. From rapid (South Korea, Canada), to slow (England, United States), to minimal (Sweden) lockdowns; from minimal use of antiviral agents (Canada), to wider use (the United States, Sweden), to widespread use of multiple drugs (Poland): these varying decisions cannot all be correct! Because decision and inferential errors are inevitable [[1], [2], [3]], absolutely correct recommendations what to do in times of crisis are impossible. Here we will argue that adhering to the principles of evidence-based medicine (EBM) and GRADE gives us higher probability to get things right than making recommendations in any other way.

With momentous decisions before them, one would hope that governments, public health officials, and individual physicians utilize rational, evidence-based decision-making strategies. The response to COVID-19 pandemic has, however, been characterized by a paradox. On the one hand, we note unprecedented levels of scientific and political cooperation. On the other, governments, health institutions, and physicians have advocated unproven management strategies inconsistent with rational, evidence-based reasoning.

Particularly disturbing are statements by prominent politicians that have driven extremely influential public discourse, and decisions by the FDA to endorse emergency use of essentially untested interventions, giving the wrong message to the clinicians and the public. In this article, we outline how adhering to the principles of EBM [4], operationalized through the GRADE evidentiary system [5], can avoid such grievous errors.

The first EBM principle is that some health claims are more trustworthy than others [4]. Evidence comes on the continuum of quality—from evidence that leaves us completely uncertain to evidence that approximates near certainty. GRADE classifies evidence from high quality (top trustworthiness) to very low quality (the bottom) with categories of moderate and low in between.

In response to COVID-19, people are interested in recommendations regarding issues such as use of masks, or of drugs such as hydroxychloroquine. In general, implementing interventions based on high-quality evidence of substantial effectiveness will result in net benefit, whereas adopting interventions based on very low quality runs a high risk of net harm. Thus, prudence generally dictates not implementing interventions when only very-low-quality evidence exists.

Failing to heed this principle, based on in vitro data that the antimalarial hydroxychloroquine possesses antiviral activity against SARS–CoV-2, many physicians started using this drug for treatment and prophylaxis against COVID19. Possibly motivated by fear, and the resultant feeling that we must do something, this rush to judgment [6] is, for a number of reasons, very likely to do net harm. One is that optimism regarding important benefits is likely to result in disappointment: clinical success from interventions suggested by apparently promising results from preclinical studies is extraordinarily infrequent [7], as are successes following low-quality clinical evidence.

Moreover, in the pandemic setting, there are a host of adverse consequences of drug interventions that—whether or not the agents are beneficial—will accompany their early adoption. First, misleading public statements suggesting benefit will discourage people from enrolling in well-designed research studies, undermining the possibility of ever arriving at high-quality evidence. Second, drugs invariably come with adverse consequences, and toxic effects at doses extrapolated from treating people with chronic conditions but who are otherwise reasonably healthy may be considerably greater in those suffering from a severe acute infection. When drugs turn out to be useless, as emerging evidence regarding hydroxychloroquine suggests will be the case for this agent, their premature use will lead us to look back with dismay on all these problematic consequences [8].

A second key EBM principle incorporated in the GRADE system is that evidence is necessary but not sufficient for management recommendations and associated decision-making [4]. Depending on our values and preferences, each of us feels the consequences of our decisions differently. Value and preference considerations acknowledge that we all react differently to inevitable errors in scientific inference and decision-making [9]. Regardless if research is mature or emerging, absolute truth is unobtainable [[1], [2], [3]]; as a result, all our inferences and decisions are fraught with uncertainty. Thus, both individuals and society as whole have to decide on the basis of less-than-perfect evidence.

Errors in recommendation and action can occur in our assessment of the evidence, or in the process of moving from evidence to decisions. For instance, when a politician trumpets the virtues of a particular medication, or when the FDA promotes emergency use of a medication, they may be making one of two errors. First, they may be inferring that there is at least moderate-quality evidence supporting use of the intervention when the evidence is actually very low quality. Second, they may be operating under the belief that even if very-low-quality evidence, there is little to lose by its administration, when as we have pointed out, there is a lot to lose.

EBM and GRADE address both issues: providing an epistemological framework for judging quality of evidence, and evidence to decision frameworks that ensure consideration of all important health outcomes [10], our values and preferences, as well as considerations of equity and unintended consequences such as drug shortages.

The GRADE approach recognizes that value and preference considerations must consider the impact of potential errors, and the likelihood of those errors [[1], [2], [3],11,12]. How will we feel if we fail to give a drug such as hydroxychloroquine and it turns out to be beneficial? And how will we feel if it turns out to be useless, thousands of people endure serious adverse effects, drug resistance increases substantially, and people who need the drug for proven indications suffer because of its unavailability? Furthermore, in weighing these consequences, we must consider that when only very-low-quality evidence exists, it is far more likely a touted intervention will prove useless than providing important benefit.

Errors of falsely inferring benefit or lack of benefit are linked—as we avoid harm from endorsing interventions likely to cause net harm, we are missing benefits that may yet prove present. Public health officials and politicians need to attend to such considerations. The likely consequences of a mistake, and our values and preferences, play large in these decisions.

For example, until recently, there was little evidence for wearing a mask in open spaces, but given low level of harms associated with doing so (as long as the consequences do not include unavailability of masks in situations in which their use is more important), encouraging use may be reasonable. By contrast, locking down whole societies comes with grievous economic consequences. But how would we feel if lockdowns are in fact beneficial, and failure to implement results in tens, perhaps hundreds of thousands, of unnecessary deaths?

Although this will change in the near future, applying GRADE to most currently recommended treatments suggested for COVID19—from antibiotics to antimalarials [6] to antivirals, the quality of evidence for benefit is overwhelmingly only very low. Given the low likelihood of subsequently revealed important benefits for any single agent, the adverse effects, barriers to trial conduct, drug resistance, and unavailability for proven indications, their endorsement by authorities represents a grievous mistake.

Application of EBM and associated GRADE principles can reduce the likelihood of such errors. Failure to use these principles will result in two sets of errors: incorrect inferences regarding certainty of evidence and inappropriate application of values and preferences in balancing of uncertain desirable and undesirable consequences. Appropriate application of EBM and GRADE is never more important than in times of health crisis affecting millions of people.

CRediT authorship contribution statement

Benjamin Djulbegovic: Conceptualization, Writing - review & editing. Gordon Guyatt: Conceptualization, Writing - review & editing.