Developing trustworthy recommendations as part of an urgent response (1–2 weeks): a GRADE concept paper

Abstract

Objectives

The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1–2 week) in the clinical, public health, and health systems fields.

Study Design and Setting

We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group.

Results

A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan.

Conclusion

An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.

What is new?

Key findings

• •We propose an approach for developing trustworthy recommendations as part of an urgent response (1–2 weeks).

What this adds to what was known?

• •The proposed alternatives are: adopting or adapting existing recommendations; using existent systematic reviews to develop new recommendations; and, relying on expert panel input to develop new recommendations.

What is the implication and what should change now?

• •It is critical to leverage collaborations, capabilities, and resources that allow for rapid assessment of adequate information to support recommendation development.

1. Background

The development of practice guideline recommendations requires a complex interplay of many participants and involves various tasks including setting priorities, scoping the remit of the guideline, identifying an expert panel group, and managing conflicts of interest [1,2]. The most time-consuming tasks are the identification, synthesis, assessment, and presentation of the evidence in ways that allow the guideline groups to formulate recommendations. Over the last 2 decades, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group has developed a systematic and transparent approach to creating evidence-informed practice guidelines that facilitate this. The method has been applied across the clinical, public health, and health systems fields [[3], [4], [5]].

Some situations raise the need for urgent recommendations to support the interventions of clinicians, public health practitioners, and policymakers, for example, to address the COVID-19 pandemic [6,7]. Under such circumstances, developing trustworthy recommendations in a sufficiently short timeframe is essential but can be challenging in rapidly changing contexts. Groups producing guidelines specifically need to balance the need for developing a timely response with the need to ensure the trustworthiness of their advice.

The World Health Organization (WHO) defined two types of guidelines developed in response to an emergency or urgent need: emergency (rapid response) guidelines (produced within hours to days) and rapid advice guidelines [8]. Thayer and Schunemann [6] defined four levels of urgency for developing recommendations: ultra-short emergency response (1–2 hours), urgent response (1–2 weeks), rapid response (1–3 months), and routine response (more than 3 months) (Box 1 ). Although more detailed advice exists for routine [1] and rapid responses, so far there is no formal guidance on how to apply GRADE in situations requiring urgent responses [6,9].

Box 1. Levels of urgency for developing recommendations.

Ultra-short emergency response: 1–2 h Urgent response: 1–2 wk Rapid response: up to 3 mo Routine response: more than 3 mo

The objective of this paper is to propose an approach for developing trustworthy recommendations as part of urgent responses (1–2 weeks) in the clinical, public health, and health systems fields.

2. Methods

The target audience for the proposed approach are guideline developers. We consider the timeframe from the perspective of the guideline developer, that is, the starting point for the approach is the receipt of a request for developing recommendations (if applicable), while the finishing point is the submission of guidance to the requesting organization. We use the WHO definition of guidelines as “systematically developed evidence-based statements which assist providers, recipients and other stakeholders to make informed decisions about appropriate health interventions” [10].

We developed this approach as members of the GRADE Rapid Guidelines project group. The GRADE Rapid Guidelines project group includes 31 members of the GRADE working group (gradeworkinggroup.org). These members collectively have expertise in preclinical, clinical, and epidemiological primary research, in modeling, in evidence synthesis (including systematic reviews and rapid reviews), and in trustworthy practice guideline development. This expertise spans the breadth fields of clinical, public health, and health policy. The development of the approach started in May 2016, and was based on reviewing selected relevant research (some of which was conducted by members of the GRADE Rapid Guidelines project group) [6,[11], [12], [13]], iterative discussions within the project group reflecting on the members’ experiences, informal consensus process, and feedback from the larger GRADE working group.

In terms of the review of selected relevant research, we first considered the findings of a 2018 systematic survey of methods manuals and published guidelines that were developed in a shortened timeframe [12]. The review found that rapid guidelines were usually commissioned to address emergencies, rapid increases in the incidence or severity of a condition emerging new evidence for a specific treatment. Overall, there was no consensus on the methods for developing rapid guidance or on the timeframe for development, ranging from 1 to 13 months, when reported. Second, we considered the findings of an interview of guideline developers about key aspects for rapid guideline development [11]. Third, we discussed published and unpublished examples of recommendations developed in response to rapid or urgent needs, with a focus on understanding how standard processes were modified [7,[13], [14], [15], [16]].

We debated, revised, and refined the concepts to be included in the approach based on iterative discussions during conference calls and a workshop by the project group (Rome, Italy, April 27, 2017). Members of the project group also drew on their experiences with developing recommendations in response to rapid or urgent needs [[17], [18], [19]]. Finally, the GRADE working group approved this paper using its expedited publication approval process, developed in light of the COVID-19 pandemic, and the need for development of trustworthy recommendations for its management internationally as part of urgent responses.

3. Results

The project group agreed to build the urgent response approach on the commonly accepted principles of standard guideline development (e.g., conflict of interest management) [1,2]. The group also agreed that the urgent response approach can also build on the GIN-McMaster guideline development checklist extension for rapid recommendations (see table here: https://bit.ly/rapidGDC) [11]. Ideally, the process should include the conduct of a rapid review, whenever feasible. However, and given the major challenges with feasibility, in the subsequent sections we did not consider a rapid review as an option.

The proposed approach assumes adherence to the criteria for applying or using GRADE [20]. These criteria include giving explicit consideration to the GRADE factors for determining the direction and strength of a recommendation and, ideally, using GRADE Evidence to Decision (EtD) frameworks and tables [21]. Under standard guideline development conditions, the EtD table is populated by evidence (typically from a systematic review) for a number of factors (the health effects of the interventions of interest, values and preferences, certainty of evidence, resource use, impact on equity, acceptability, and feasibility). The panelists review the evidence for each factor, then make judgments for those factors in relation to the interventions of interest, before deciding on the strength and direction for the recommendation [21].

In terms of sources of evidence, systematic reviews may not be feasible given the timeframe of 1–2 weeks for the complete recommendation development process. If, in spite of this tight timeframe, the guideline developers find it feasible to conduct a rapid review, the reviewers could refer to the Interim Guidance from the Cochrane Rapid Reviews Methods Group [22]. In addition, the guideline developers may consider a timely publication of the systematic review, including coordination with the coordinating organization/funder, reviewers, and journal [23]. The proposed approach offers the alternatives listed at the bottom of Fig. 1 , which are based on the use of existing guidelines and existent systematic reviews (discussed in more detail below).

Fig. 1.

Steps involved in urgent guideline development.

The following paragraphs discuss the steps of the urgent response approach and assume that existing and adequate guidelines and systematic reviews, but not new rapid reviews, are being considered. However, if a rapid review is performed it may be added at step 5 in this process. We provide illustrative examples of urgent situations related to environmental exposures and to the COVID-19 pandemic.

3.1. Assess the level of urgency

The purpose of this first step is to confirm that the response in developing the recommendation should be an “urgent” one, as opposed to “emergent” or “rapid” (see Box 1). Although there is no tool specifically developed for that purpose, one can use tools such as the WHO rapid risk assessment of acute public health events [24]. Risk assessment includes the assessment of the likelihood of the occurrence of a hazard causing an adverse event, as well as the likely magnitude of the consequences of the event over a specified period.

3.2. Assess feasibility

It is important for the guideline developers to assess the feasibility of meeting the “urgent” timeframe (i.e., 1–2 weeks). Aspects to be considered include the following: (1) the number and complexity of questions and comparisons; (2) the expected amount of work needed for collecting the literature; (3) the number of team members available to contribute to the different tasks; (4) sources of funding that can be promptly tapped to support the urgent response; (5) the availability of adequate coordination capacity; (6) the panelists who can be summoned to contribute to collecting the needed information, panel discussion, and peer review in a very intensive manner; and (7) the political and institutional support to expedite the administrative process—collaboratively with the relevant endorsing bodies (particularly the approval of the final product and if needed, of the dissemination and implementation strategies), if applicable.

3.3. Set up the organizational logistics

As the guideline developers start working with the requestors on specifying the questions (see next section), it is important to set up the organizational logistics for the project. Forming a steering group with executive power that would meet frequently (e.g., at least on a daily basis) to ensure the progress of the project and the compliance with the timeline is key. Similarly, the steering group should involve methodologists with prior experience in developing recommendations as part of urgent or rapid responses. Also, in the context of international collaboration, having members from different time zones can help ensure the ability to sustain project workflow.

Starting on day 1, the steering group should identify and quickly recruit members of the different groups (including methodologists, panelists, reviewers) and secure their solid commitment to the expected tasks with a clear timeline and schedule. However, it might be challenging to recruit stakeholders (particularly patients’ representatives) in the context of a rapid response (e.g., in the middle of a pandemic). Building on existing networks of stakeholders or having a roster of stakeholders willing to serve as panelists when needed can be helpful. The aim should be meaningful and equitable multi-stakeholder engagement [25].

Similarly, when applicable, the steering group needs to prospectively liaise closely with the organization’s oversight and quality committee to ensure compliance with quality requirements. The steering group should start as early as possible drafting the final report and fill in its different sections as the information becomes available, to expedite its submission and later publication. Other important logistics include a streamlined communication plan with the different groups involved, with the goal of minimizing email burden; and reliable and easy to use document sharing and online meeting platforms. Ideally, the organization should have in place a “preparedness plan” to mount an urgent response for developing recommendations (with the appropriate processes and tools).

3.4. Specify the question(s)

To make the process feasible, the guideline developers should work with the requestor to ensure the guideline’s scope is reasonably narrow. In addition, the guideline developers should ensure the questions are specific and well defined, for example, using the PICO (Population, Intervention, Comparator, Outcome) or PECO (Population, Exposure, Comparator, Outcome) frameworks for interventions and exposures respectively [26,27]. Even within the PICO or PECO frameworks, additional specificity (narrowing the scope) when defining the population, intervention, or exposure may facilitate the urgent development process. Annex 1 provides a brief description and examples of the elements of the PICO and PECO questions.

Prioritizing outcomes is an essential step of the guideline development process. However, it can be time consuming and not feasible under urgent conditions. Ideally, there would be a “ready to use” standardized set of outcomes for the condition under consideration (e.g., COVID-19 Core Outcomes initiative) [28]. When such a set is not available, the guideline developers could rely on previous prioritization efforts. For example, the Surviving Sepsis Campaign (SSC) guidelines on the management of critically ill adults with COVID-19, instead of performing a new prioritization of outcomes, used the outcome prioritization informed by the ongoing SSC guideline 2020 work and expert panel input [29]. The group could also rely on outcomes prioritized for a similar disease (e.g., influenza or severe acute respiratory syndrome [SARS]). In some urgent situations, the group might opt to prioritize only one critical outcome. For example, slowing the spread of the pandemic (“flattening the curve”) was initially considered as the sole critical outcome when considering public health interventions for the COVID-19 pandemic.

Urgent situations are typically associated with scarcity of data, particularly for clinical outcomes (e.g., clinical recovery). In such cases, the guideline developers might need to rely on surrogate outcomes (e.g., viral load) for informing recommendations. Indirect evidence can be used to inform other components of the PICO or PECO question. For example, the National Institute for Health and Care Excellence (NICE)’s “interim process and methods for developing rapid guidelines on COVID-19” calls for using information related to SARS, Middle East respiratory syndrome (MERS), and pandemic influenza [30].

It is important to determine a priori what indirect evidence would be considered and used, if needed. Similar key decisions in the design and conduct of the project (e.g., eligibility criteria) should ideally be done a priori. However, and given the limited time to fully consider the different aspects of the project, post hoc decisions are more likely in urgent compared with routine response when developing guidelines.

3.5. Collect the information needed

As mentioned earlier, the proposed approach considers using existing guidelines and systematic reviews and does not necessarily require the conduct of new rapid reviews. NICE has adopted such an approach in its “interim process and methods for developing rapid guidelines on COVID-19” [29]. They aimed to reuse existing guidance as much as possible, without conducting systematic literature searches. They first searched for published guidance, including WHO COVID-19 guidance, NICE guidance, and guidance from professional organizations, both inside and outside the UK [29]. If no guidance was identified, they searched for and prioritized the following types of publications, in the following order: systematic reviews, randomized controlled trials, and published expert opinions. Many organizations use a similar approach. For example, in 2008, the European Food Safety Authority relied on a previously published systematic review to develop its urgent guidance related to the presence of melamine in composite food products [31].

In addition to information about the health effects, the guideline developers may need to identify contextual information. In the example of mass evacuation in response to a nuclear incident, there is a need to collect information about the cost and feasibility of such a major undertaking, as well as the acceptability by relevant stakeholders (e.g., governmental agencies, citizens). This information may be available from administrative databases (e.g., for cost), and through consulting with community leaders (e.g., to assess the acceptability by stakeholders).

In terms of information sources, the published literature may not be the only optimal source in urgent and rapidly developing situations, like the COVID-19 pandemic. Information sources may include preprint servers, open research databases, and research data repositories. However, such sources may not include conventional processes of peer review and may require additional critical appraisal or caution in the interpretation of the accuracy or stability of the data. Specific to environmental exposure scenarios, it may be important to evaluate or model the evidence for health effects when little is known about the exposure (e.g., a chemical with uncertain effects).

3.6. Assess the adequacy of identified information

Once identified, the published literature (guidelines, systematic reviews, and if applicable, primary studies of any design) needs to be assessed for “adequacy” based on the three following criteria: relevance, credibility, and currency [32].

• •Relevance or directness, according to GRADE terminology [33], refers to the extent to which the population, exposure/intervention, and outcomes identified in the literature (whether for the original recommendation or for the systematic reviews) reflect the ones in the question being addressed by the recommendation. If any items are different enough that we would expect different relative effects, then the evidence is considered indirect.
• •Credibility reflects the quality of conduct of the guideline or of the systematic review being considered for use in the process of the urgent response. There are a number of tools that could be used for that purpose: the advancing guideline development, reporting and evaluation in health care II to assess the quality of conduct of guidelines [34]; the assessment of multiple systematic reviews-2 to assess the methodological quality of systematic reviews [35], and ROBIS to assess the risk of bias of systematic reviews [36].
• •Currency or recency: although the currency of a systematic review depends on its search date, the currency of a guideline depends similarly to a large extent on the search dates of the systematic reviews used to inform recommendations. In some cases, even if the search date is not current, a systematic review may still be up to date if no new evidence was published after the date of the search; however, a search would need to be conducted to verify this assumption. Rapidly updating a published systematic review may be efficient, if that review is credible and new evidence is known to have been published since the date of the original search [37].

The guideline developers do not need to conduct the above assessment for all identified relevant literature (guidelines, systematic reviews, and primary studies). For example, if they end up adopting an existing recommendation (addressed in step 6), although the recommendation under consideration needs to be assessed, this would not apply to all other identified literature (e.g., systematic reviews and primary studies).

The adequacy of identified literature is not an “all or none” assessment. For example, the guideline developer might identify three potentially relevant systematic reviews. Each one of the three may be deficient in relation to one of the three above criteria, but to different extents. We suggest considering the criteria in the following order: relevance, credibility, and currency. This means that the guideline developers would select a relevant but outdated systematic review over an updated, but less relevant review. The idea is that it is more efficient to “fix” the former review with a quick update than to use the latter less relevant review.

During the 2014 Elk River chemical spill, the most relevant information on the toxicology of the crude 4-methylcyclohexanemethanol (MCHM) was obtained directly from the manufacturer [38]. Within 6 days, the Scientific Review Panel for the National Library of Medicine’s Hazardous Substance Data Bank was able to expedite the evaluation of the toxicology, chemical composition, and chemical purpose to assess the credibility of the information [38]. This early evaluation was a critical step in the process of understanding the risks of exposure and the best course of action for cleanup. However, additional research was needed to better understand the short-term adverse health effects associated with MCHM, and no additional information was available at the time. To expand the knowledge base for MCHM and other chemicals associated with the spill with the most current data, the Centers for Disease Control and Prevention and the Agency for Toxic Substances Disease Registry enlisted the help of the National Toxicology Program to conduct a series of short duration studies [39]. The results obtained from these studies assisted the West Virginia Bureau of Public Health in responding to public health concerns related to the MCHM and other chemical exposures that occurred during the incident.

3.7. Develop the recommendations

Depending on the availability and adequacy of identified literature (guidelines or systematic reviews), the guideline developers may develop recommendations using one of the four potential approaches: (1) adopt existing recommendations; (2) adapt existing recommendations; (3) develop new recommendations using existing systematic and/or rapid reviews [15]; or (4) develop new recommendations using evidence provided by panelists (also referred to as expert input or expert evidence) [40].

3.7.1. Adopt vs. adapt existing recommendations

The literature search may find an original recommendation that is ‘adequate’ for the urgent response, i.e., is relevant, credible and current. At that point, the expert panel group should decide whether to adopt the original recommendation (i.e., use it as is) or adapt it. Practically, the expert panel group ‘adopts’ a recommendation when it does not change either the direction or the strength of that recommendation. Otherwise, the recommendation is ‘adapted’.

To decide on adopting vs. adapting an original recommendation, the expert panel group needs to assess whether the direction or the strength of recommendation may be affected by any difference in the following factors (ideally using a decision-making framework, e.g., EtD table) [21]:

• •Rating of the importance of outcomes: a change in the rating of the importance of outcomes might lead to a change in the judgment of the balance of benefits and harms. For example, an urgent response recommendation addressing the use of invasive ventilation in patients with COVID-19 might reflect a different prioritization of the importance of “patient survival” outcome relative to “transmission of infection to healthcare workers” outcome (in comparison to the original recommendation) [41].
• •Indirectness: what could have been judged as direct evidence in the context of the original recommendation might be judged as indirect evidence in the urgent response context (e.g., effectiveness of treatment with corticosteroids among patients with MERS-CoV vs. COVID-19) population. This might lead to rating down the certainty of evidence. One could use the GRADE framework to judge whether that situation warrants rating down the certainty of evidence for indirectness [33].
• •Rating of evidence: the expert panel group may rate the certainty of evidence differently than the original recommendation for reasons other than indirectness judgment (see above). Examples include different judgment of the risk of bias or imprecision, and consideration of new or emerging evidence.
• •Baseline risks: the absolute reduction of an outcome (e.g., mortality in laboratorians or other staff with a potential occupational risk for Ebola receiving a pre-exposure Ebola vaccination) might be higher or lower in the context of the urgent response (e.g., Ebola outbreak responders) compared to context of the original recommendation (e.g., due to a difference in baseline risk of mortality between the two contexts). This could lead to a change in balance of benefits and harms.
• •Perspective: the original recommendation might have been developed from an individual or clinical perspective, while the urgent response has a population or public health perspective. The new perspective may broaden the relevant stakeholders (e.g., beyond the patients, their caregivers, and healthcare providers), and increase the weight of EtD criteria such as cost effectiveness and impact on equity.
• •Contextual factors: the judgment on contextual factors (such as resource use, impact on equity, acceptability, and feasibility) might change when considering the context of the rapid response (in comparison to the original context).

For each of the above factors, the judgment (between the urgent guidance setting and the original one) needs to be sufficiently different for panel members to adapt the recommendation (i.e., modify its direction and/or strength). The recommendation statement wording may be edited to enhance the usability for the intended target group.

WHO developed interim guidance for the “clinical management of severe acute respiratory infection when COVID-19 infection is suspected” in consultation with the International Forum for Acute Care Trialists, International Severe Acute Respiratory and Emerging Infection Consortium, and SSC [14]. Guideline developers originally adapted this interim guidance from “Clinical management of severe acute respiratory infection when Middle East respiratory syndrome coronavirus (MERS-CoV) infection is suspected: interim guidance” [42]. In the first edition of COVID-19 guidance, developers adopted many document sections, research questions, and applicable guidance verbatim, adapting some directives to reflect underlying uncertainty about the microbiological profile of COVID-19 when informed from the indirect information from SARS (caused by SARS-CoV1) and MERS (caused by MERS-CoV) cases [14].

3.7.2. Develop new recommendation using existing adequate systematic review

One other reasonable scenario would be to find an adequate systematic review that would jump-start the development of de novo recommendation(s). The development would follow the standard GRADE evidence assessment and recommendation development process with some potential shortcuts to ensure the process is completed within the desired timeframe [1].

The Infectious Diseases Society of America developed rapid recommendations on the treatment and management of hospitalized patients with COVID-19, including a recommendation on corticosteroid treatment for hospitalized patients with acute COVID-19 [17]. At the time of the first iteration of the guideline, the review team did not identify any direct evidence to inform this recommendation; however, they identified a systematic review reporting on corticosteroid use among patients with SARS-CoV-1 or MERS-CoV [43]. The guideline panel determined this existing review to be direct enough to inform their recommendation.

3.7.3. Develop new recommendation using expert panel input

When no adequate recommendations are available for adopting or adapting, and no adequate systematic reviews are available to inform a de novo recommendation, using expert panel input as the sole source of evidence is a feasible alternative for an urgent response. It is important to clarify that panel input is still important when developing the recommendation based on existing recommendations or systematic reviews.

Fig. 2 shows the steps involved in developing a new recommendation using expert panel input. The panelists are asked to review the literature they are provided with and then fill out a “panelist EtD” table [21]. In that table, and in lieu of the systematically collected evidence, the panelists provide a description of their “expert evidence” consisting of their observations and experiences (equivalent to case reports and case series) [40]; these are expected to reflect “facts” (as opposed to opinions). In the next step, the steering group collates the input from all the panelists and populates the EtD that will be used as the basis for panel discussion and consensus building. Typically, the panel chair builds consensus with the panelists through discussion, and if needed through (iterative) voting. Alternatively, and for efficiency purposes, the chair can make suggestions and have panelists agree or disagree with them; however, caution is needed to ensure a broad range of perspectives is considered.

Fig. 2.

Steps involved in developing a new recommendation using panel input.

3.8. Consider an updating plan

Urgent situations are typically associated not only with scarcity of data but also with rapidly developing evidence base and contextual information. This raises the consideration of establishing an updating plan, ideally through a living process [8,44]. This is particularly relevant when emerging data can potentially lead to a change in the recommendation. The updating plan would need to define the frequency of reassessment of the recommendation (e.g., weekly, monthly), and to be adjusted to the speed of development of the urgent situation and of the emergence of the evidence.

In the case of an urgency related to an environmental exposure, there might be a need to continuously monitor the level of exposure within the population of interest for the purpose of triggering or updating the recommendations [44]. In the example of the nuclear incident, regular collection of environmental radioactive iodine levels from the field would be needed to reverse the recommendation for mass evacuation.

4. Discussion

Developing trustworthy guidelines in a relatively short timeframe are of utmost importance but can be extraordinarily challenging. In this paper, we propose an approach for developing trustworthy recommendations as part of an urgent response (1–2 weeks). The approach offers the alternatives of using existing guidelines to adopt or adapt recommendations; using existent systematic reviews to develop new recommendations; and, when the previous options are not possible, relying on expert panel input to develop new recommendations.

The alternatives of adopting, adapting, and using existent reviews to developing new recommendations are similar to the ones proposed by the GRADE-ADOLOPMENT methodology [45]. Although that methodology was designed to address guideline development primarily to save and share resources, the approach proposed in this paper is intended to address limited available time (and also potentially limited resources, recognizing that the absence of guideline recommendations may pose more of a threat than rapidly developed ones).

The alternative relying on expert panel input to develop new recommendations might be perceived as going against the principles of evidence-based medicine. However, Schunemann et al. [40] argued that expert opinion can be considered as “expert evidence.” They did caution however against confusing expert evidence with expert’s views and judgments. Similarly, Djulbegovic and Guyatt [46] refer to the “experience of individual clinicians” as a type of evidence. They also make the point that evidence requires interpretation and “a consensus process must determine that interpretation” in the context of guidelines.

Although we may not have captured all situations unique to creating urgent responses, we have prioritized providing a broad, structured, and pragmatic approach. The illustrative examples exemplify how a number of guideline organizations had to rapidly react to urgent situations related to environmental exposures or to the COVID-19 pandemic. Many of the examples highlight how flexibility was needed on the side of the developer in terms of maintaining trustworthy methodology.

Having recommendations for situations that might recur on an urgent basis may reduce the need for development of recommendations within urgent timeframes. For example, Public Health England has existing guidelines for handling nuclear incidents that are regularly updated, so that this material is available in advance of catastrophic events [47]. Nevertheless, there will continue to be new unexpected incidents that require decisions to be made for which no relevant recommendations are already available or could have been foreseen. In these instances, it is critical to leverage collaborations, capabilities, and resources that allow for rapid assessment of adequate information to support recommendation development. Moreover, it would be important for coordinated efforts by guideline developers to avoid duplication of efforts and sometimes inconsistent or even contradictory recommendations.

Supplementary data

Annex 1

Elements of the PICO and PECO questions

The use of the PICO vs. the PECO framework is dependent on the knowledge gap being addressed. For instance, when little is known about the outcomes arising from the exposure to a potential hazard, a PECO question is needed. Conversely, if the exposure is known to be a hazard, and information is needed about the effectiveness and safety of a mitigating intervention, a PICO question is needed [27].

• •Population: Includes those whose health outcomes might be affected (positively or negatively) by either the exposure in a PECO question (e.g., exposed to the released radioactivity during a nuclear incident), or the management options in a PICO question (e.g., mass evacuations or providing iodine following a nuclear incident). For the COVID-19 pandemic five populations of interest have been described: people known to be infected, people presumed to be infected, people exposed to those infected, people not known to be exposed or infected, and people who have recovered from infection [48].
• •Exposure/Intervention and comparator(s): These terms refer to either classifying the potential hazard (exposure) or to classifying the management options (interventions) under consideration. In some cases, there is a need to define the exposure level above which an action would be triggered and/or below which an action could be discontinued. It is important to note that interventions may consist of a comprehensive, coordinated response consisting of many components. Consequently, it is important to clarify the specific components, timing and duration of their delivery, and the individuals involved [27]. In the nuclear incident example, when considering the intervention of providing iodine vs. not providing iodine, the following needed to be specified: the dosage and number of doses of prophylactic iodine, and the timeframe within which iodine should be ingested.
• •Outcomes: These should ideally include all critical and important health outcomes that the alternative management options are expected to influence impact. In the above example, these could include the health and psychosocial effects of radioactive iodine exposure, and the benefits and adverse effects of prophylactic iodine therapy. Health-related outcomes should be prioritized based on patient importance. In some cases, policy makers might be interested in the impact on nonhealth outcomes, for example, socioeconomic impact of mass evacuation and road accidents caused by mass evacuation