Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what

Abstract

Objectives

The public, policy makers, and science communities are subject to many false, uninformed, overly optimistic, premature, or simply ridiculous health claims. The coronavirus disease 2019 (COVID-19) pandemic and its context is a paramount example for such claims. In this article, we describe why expressing the certainty in evidence to support a decision is critical and why the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach matters now, perhaps more than ever and no matter what the specific topic is in relation to COVID-19. We finally also offer suggestions for how it can be used appropriately to support decision-making at global, national, and local level when emergency, urgent or rapid responses are needed.

Study Design and Setting

This is an invited commentary to address the objectives above building on examples from the recent COVID-19 pandemic. This includes an iterative discussion of examples and development of guidance.

Results

The GRADE approach is a transparent and structured method for assessing the certainty of evidence and when developing recommendations that requires little additional time. We describe why emergency, urgent, or rapid responses do not justify omitting this critical assessment of the evidence. In situations of emergencies and urgencies, such as the COVID-19 pandemic, GRADE can similarly be used to express and convey certainty in intervention effects, test accuracy, risk and prognostic factors, consequences of public health measures, and qualitative bodies of evidence.

Conclusions

Assessing and communicating the certainty of evidence during the COVID-19 pandemic is critical. Those offering evidence synthesis or making recommendations should use transparent ratings of the body of evidence supporting a claim regardless of time that is available or needed to provide this response.

Graphical Abstract

Highlights

• •GRADEing the certainty of the available evidence is more important than ever because of the unprecedented pressure for action and the large number of people affected by decisions.
• •The GRADE approach is a transparent and structured method for assessing the certainty of evidence and when developing recommendations that requires little additional time.
• •In situations of emergencies and urgencies, such as the COVID-19 pandemic, GRADE can similarly be used to express and convey certainty in intervention effects, test accuracy, risk and prognostic factors, consequences of public health measures, and qualitative bodies of evidence.
• •Requirements for emergency, urgency, rapid, and routine GRADE assessment may differ but should transition from one to another.

1. Background

The public, policy makers, and science communities are subject to many false, uninformed, overly optimistic, premature, or simply ridiculous health claims. The coronavirus disease 2019 (COVID-19) pandemic and its context is a paramount example for such claims: from media and politician's attention to biased interpretation of case series of patients on hydroxychloroquine to the injection of disinfectants and use of azithromycin [1]. Yet, for some interventions, such as personal protective equipment (PPE) and social distancing, there is an accumulating body of evidence in favor of their benefits [2]. To monitor misleading and appropriate claims and separate biased from unbiased research, understanding and expressing the certainty in effects of these and other clinical, public health, or health policy interventions is critical to weed out misleading or wrongful claims. It is more critical than in other situations because policy makers are under unprecedented pressure to react to claims and make decisions with varying degrees of certainty in the evidence and timing of their responses. And it is especially critical in this era because of the public access to information and expectation of a timely response.

The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach provides a transparent and structured approach to making judgments about the certainty of the evidence, and offers a transparent process to making recommendations and decisions [[3], [4], [5], [6]]. GRADE is the product of an open and inclusive community of people that has collaborated for 20 years (www.gradeworkinggroup.org) and is currently used by over 100 organizations globally, including the World Health Organization (WHO). The GRADE working group has used a carefully designed, rigorous, transparent, and inclusive process based on cumulative evidence about research methods, bias, and decision-making. GRADE offers solutions to the dilemma of expressing certainty in a body of evidence, which would have been very low certainty in efficacy of hydroxychloroquine because of imprecision and the nonrandomized designs used [7], and at least moderate certainty in physical distancing based on other indirect evidence from viral respiratory diseases research [2]. GRADE also offers a structured and transparent framework for making decisions, including recommendations, that should prove particularly helpful during the ongoing COVID-19 pandemic. In this article, after having made the point why we think expressing certainty in evidence to support a decision is critical above, we describe why GRADE matters now, perhaps more than ever and no matter what the specific topic is in relation to COVID-19. We finally also offer suggestions for how it can be used appropriately to support decision-making at global, national, and local level.

2. How to use GRADE in the COVID-19 era

Decision makers should, and many are (!), asking “what is the science?” and “how good is this test or this intervention?”. GRADE, although appropriately sophisticated in its full execution, can answer these questions and be relayed to decision makers by breaking its components down into straightforward questions about 1) the certainty of evidence and 2) the criteria for making decisions or recommendations. While ideally applied to rate the certainty of a body of evidence in a well-conducted and up-to-date evidence synthesis relevant to the question at hand (in terms of setting, population, intervention, comparator, and outcomes) with corresponding summary tables, such as evidence profiles; GRADE's application requires at least that “the evidence that was assessed and the methods that were used to identify and appraise that evidence should be clearly described” [4,[8], [9], [10]].

Those providing and using evidence should ask the questions in box 1 to understand if they can be certain about the effects of an intervention (including tests, public health strategies or other options being considered), regardless of the time available. Based on the answers to these questions, an expression of certainty in the body of evidence can be articulated (GRADE uses four levels of certainty: high, moderate, low, and very low) [4].

Box 1. GRADE questions to ask about the certainty in the body of evidence per outcome:

• -Are the study designs used appropriate?
• -Are there important limitations in the research design or execution of the research?
• -Are the results consistent across studies when the settings, populations, interventions, comparators, and outcomes are reasonably similar?
• -How directly do the results apply to the population (including setting), intervention, comparator, and outcomes (PICO) of interest?
• -Are the results precise enough or likely due to chance?
• -Is this all the research that has been conducted on the PICO question of interest?
• -Is there anything, in particular very large effects of an intervention, dose response gradients, or unfavorable scenarios still leading to convincing effects, that makes us more confident?

The questions proposed by GRADE to guide decision makers, including those formulating recommendations (and the recipients of them), are equally simple (Box 2 ), and can be tailored to the type of decision (clinical recommendations, public health recommendations/decisions, or health system recommendations/decisions) [11,12].

Box 2. GRADE questions when recommending for or against an intervention or strategy.

• -Are the expected health benefits greater than the harms or vice versa (this integrates considerations about the priority and severity of the problem, intervention effects, the values people place on the outcomes, as well as the certainty in the effects)?
• -What is the magnitude of the resource requirements (and associated cost) related to the intervention/strategy and is it cost-effective?
• -What is the impact of the intervention/strategy on equity, including societal implications and environmental impact?
• -Is the intervention/strategy acceptable to different stakeholders (this criterion includes ethical and other considerations)?
• -Is the intervention/strategy feasible (this criterion includes health system, social, legal, political, and other considerations)?

2.1. Emergency, urgent, rapid, and routine use of GRADE to assess the certainty of evidence

Early in the COVID-19 pandemic, it became evident that information is required with different levels of urgency. We previously described these different levels of time-based responses using GRADE (Figure 1 ), and categorized the use of GRADE to assess the certainty in 1) emergency responses or in an ultrashort time frame of hours; 2) urgent responses, allowing one to 2 weeks to respond; 3) rapid responses, in up to 3 months; and 4) routine responses, beyond 3 months [10].

Fig. 1.

The use of GRADE to assess the certainty in: (1) emergency responses or in an ultra-short time frame of hours; (2) urgent responses, allowing one to two weeks to respond; (3) rapid responses, in up to three months; and (4) routine responses, beyond three months.

An example of the first scenario, that is, providing GRADEd evidence within hours, is when it became apparent that a viral outbreak was the likely cause of what is now known as COVID-19. Under those circumstances, addressing the value of use of masks as PPE or distancing within hours became an emergency question to be answered. Indeed, indirect evidence from other viral outbreaks could have been used within hours. Credible systematic reviews of related conditions available at the time of the outbreak provide evidence about what measures to take. However, one would need to lower the original rating of certainty of the evidence to account for the indirectness of the population that was likely affected by an unknown virus [13]. This is because the answer to the question “How directly do the results of the research apply to the population and situation of my interest?” would have been that we know from other viral diseases that PPE has an effect but we cannot be as certain that it will be similar in this new outbreak. That an influenza virus or coronavirus were likely pathogens would have been supported by the type of clinical and public health presentation, and prior history of outbreaks caused by these viruses. Given the unknown exact virus, however, whether the effect of use of masks for COVID-19 is larger or smaller would have be unknown, as masks may be more or less effective depending on the exact mode of transmission of the virus as well as other factors. An informative narrative synthesis based on GRADE guidance for the state of the evidence about masks would have been “the use of masks may reduce the risk of transmission of respiratory virus (low certainty in the evidence due to indirectness based on seven case-control studies showing large effects of severe acute respiratory syndrome from coronavirus transmission) [13,14]. Similarly, one would have high certainty that a large distance from the source is effective, yet the certainty in how much distance to keep, for example, 1 m or 2 m or more, would be rated down for that same indirectness.

For the second scenario of using GRADE with urgency (one to 2 weeks), the COVID-19 pandemic has made it clear that systematic reviews can be conducted to respond to urgencies (defined as responding within 2 weeks) without “cutting corners”, for which rapid reviews have been criticized [15,16]. To illustrate this point, we present three standard urgent systematic reviews that we have conducted within 7 to 11 days each, two of them in parallel and one including many complex meta-analyses, including Bayesian approaches and a meta-regression. The three reviews addressed five questions on the use of masks, eye protection, and physical distancing; use of noninvasive ventilation; and handling of bodies of deceased individuals [2,[17], [18], [19]]. We produced five GRADE evidence profiles with a rating of the certainty after reviewing over 80,000 citations, and including over 70 studies. The review on the use of masks and physical distancing featured studies directly addressing COVID-19 and provides low to moderate certainty about large effects, and high certainty for any observed reduction in transmission by physical distancing although the exact effect is only of moderate certainty. Thus, for a situation of urgency and beyond, GRADE assessments based on systematic reviews are possible, albeit in the hands of a large experienced systematic review team. Contrary to what some believe, however, applying GRADE does not add significant time to the systematic review process. The time required to produce evidence profiles and add appropriate ratings of the certainty was approximately 1 hour. We believe the investment of time has paid off. In the earlier phases of the review, we saved time through streamlining the evidence assessments, structuring the questions and making decisions about what evidence to search for to address relevant outcomes as suggested by GRADE [9]. For example, to address the risk of COVID-19 transmission, we realized that studies addressing this risk during aerosolizing procedures such as noninvasive ventilation required a search for a different type of evidence, including mechanical and laboratory studies. While we conducted meta-analyses for many of our outcomes, something that it is often not possible, GRADE can be applied to narrative summaries of the evidence, and still provide guidance with informative statements about the findings [10,20].

For the third scenario of using GRADE within 3 months, we previously described the use of GRADE using a framework for developing rapid recommendations, that is, in up to 3 months [21]. In 2007, an expert committee at the WHO developed 23 separate recommendations, and a number of research recommendations for or against the pharmacological treatment of avian influenza [22]. Avian influenza was a serious public health threat of a pandemic at that time that, however, did not emerge, leaving more time to act compared with COVID-19 [22]. In addition to reviewing human studies, it included a review of mechanistic and animal research evidence. For COVID-19 addressing, the question about hydroxychloroquine, remdesivir, antibody testing, and other pharmacological interventions, rapid reviews (and subsequent monitoring) allow accumulation of evidence emerging during an outbreak for treatment, interventions, or strategies that may or may not be providing net benefit [23]. The accumulating evidence about mask use and physical distancing demands transitioning from urgent answers to rapid and routine answers, where new evidence is integrated, perhaps in living reviews, to provide best evidence for longer-term decisions [18,24].

Indeed, there will be answers to COVID-19-related questions that may need, perhaps after provisional urgent evidence assessments, routine monitoring using GRADE beyond 3 months. This may include addressing rehabilitation needs for patients suffering respiratory or neurological consequences or the impact of telemedicine on patient outcomes and health care utilization after COVID-19 infection. In a partnership with the Norwegian Institute of Public Health (NIPH) (https://www.fhi.no/en/qk/systematic-reviews-hta/map/) and authors’ institutions, we are making risk of bias assessments of individual studies and ratings of the certainty of a body of evidence for questions related to COVID-19 increasingly available which can be used globally by the evidence synthesis community, including Cochrane rapid COVID-19 reviews [25]. One of the goals is avoidance of duplication.

3. GRADE for COVID-19 and other recommendations

Using our rationale aforementioned, to optimally inform the public, it is inappropriate for politicians and organizations to not transparently convey the certainty narratively or provide ratings of it.

Policies have to be made regardless of the type of evidence that exists, even if there is low or very low certainty in the exact effect of an intervention or strategy, such as for the urgent answer about masks for the public. Those using GRADE are then often asked if policy makers do not shy away from making recommendations based on low or very low certainty evidence, and if they will be hesitant to communicate uncertainty of the evidence to the public. An adequate response is that evidence and studies are never perfect, and that there is always uncertainty but even in a new outbreak one can build on indirect evidence. An interpretation for policy makers of an effect labeled with low certainty would be: Although our confidence in the effect estimate is (still) low, a harm (transmission) reduction is likely assumed on the basis of the current best evidence. More research will likely increase our certainty in the effect. But, if all other factors necessary for a decision (Box 2) are considered (of which the effect is the most important but not the only one), it should be possible to reach a decision that is transparent. Decision makers are making complex decisions that balance the value placed on public health consequences, resources, equity, and feasibility. Using the structured GRADE framework, for example, for a recommendation against masks, it would be clear that it is not based on what we know about the effect, but on other factors such as equity, resource use, or feasibility.

To develop guidance, tools exist [26,27], and if used appropriately in the context of GRADE, we can inform decision makers appropriately and convey our certainty in any recommendation that evolves, on an emergency basis or over longer periods. When a structured and transparent approach is used, it facilitates the sharing of information and an understanding of the basis of decisions. To complement the Norwegian Institute of Public Health's evidence map of available COVID-19 evidence, we have partnered with NIPH and other groups, some represented by the authors of this article, to populate a recommendation catalog and map for COVID-19 that includes trustworthy recommendations and information to make decisions (https://covid19.evidenceprime.com/), both directly and contextualized to different settings, using the GRADE Adolopment methodology [28,29]. As with the evidence map, also here the goal is to avoid duplication, build on each other's work given the many guidance documents that are produced already, and make it available to platforms that link to recommendations such as the Guidelines International Network COVID-19 website and other initiatives [30,31].

4. Conclusion

Assessing and communicating the certainty of evidence during the COVID-19 pandemic is critical. Emergency, urgent, or rapid responses do not justify omitting this critical assessment of the evidence. Those offering evidence synthesis or making recommendations should use transparent ratings of the body of evidence supporting a claim. Although we focused on intervention effects, GRADE can be used to assess and communicate the certainty of evidence from models, prognostic studies, patient's values, tests, and preclinical animal research [[32], [33], [34], [35]]. There are good reasons to provide such certainty ratings and GRADE as the most disseminated and accepted approach globally has and can be used in the ongoing COVID-19 pandemic [2,18,19,36,37].

CRediT authorship contribution statement

Holger J. Schünemann: Conceptualization, Writing - original draft. Nancy Santesso: Writing - review & editing. Gunn E. Vist: Writing - review & editing. Carlos Cuello: Writing - review & editing. Tamara Lotfi: Writing - review & editing. Signe Flottorp: Writing - review & editing. Marina Davoli: Writing - review & editing. Reem Mustafa: Writing - review & editing. Joerg J. Meerpohl: Writing - review & editing. Pablo Alonso-Coello: Writing - review & editing. Elie A. Akl: Writing - review & editing.