COVID-19 therapies for inpatients: a review and quality assessment of clinical guidelines

Abstract

Owing to condensed development processes, expanding evidence and differences in healthcare system characteristics, many COVID-19 guidelines differ in their quality and treatment recommendations, which has consequences for clinical practice. This review aimed to identify COVID-19 treatment guidelines, assess their quality and summarise their recommendations. Guidelines were identified for five therapies most commonly used among inpatients with COVID-19 (remdesivir, dexamethasone, tocilizumab, baricitinib and casirivimab/imdevimab) from 11 countries. Guideline quality was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE-II) tool. Full details of recommendations and supporting evidence were analysed for high-quality guidelines, defined as those scoring ≥50% in Domain 3 (Rigour of Development) of AGREE-II. Overall, guidelines differed substantially in their quality and, even among high-quality guidelines using the same evidence, recommendations regarding specific therapeutics varied. Potential reasons for this heterogeneity, including the availability and consistency of clinical data, visibility of trial end-points and context-specific factors, are discussed.

Short abstract

This targeted review of #COVID19 treatment guidelines aimed to understand the heterogeneity in quality, recommendations and evidence bases for therapies commonly used among COVID-19 inpatients https://bit.ly/3T34G7Y

Introduction

The clinical research response to the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic has been unprecedented, rapidly yielding highly effective and safe vaccines, as well as data that guide the use of therapeutics across the spectrum of coronavirus disease 2019 (COVID-19) [1, 2]. Monoclonal antibodies (mAbs), direct-acting antiviral therapies, corticosteroids, interleukin-6 (IL-6) antagonists and Janus Kinase (JAK) inhibitors are commonly used to treat patients with COVID-19, following clinical trial results indicating their benefit in certain populations [3–7]. Yet, with the rapid generation of large amounts of data and sometimes conflicting clinical results, deciding on the best practice has been challenging.

To provide clinicians with evidence-based recommendations for managing patients with COVID-19 in their regions, many governmental or scientific organisations (both national and international) developed clinical treatment guidelines [8–10]. However, to deliver recommendations in a timely manner, guideline development groups often had to use rapid, condensed and less meticulous methods, reducing a process that often takes years to just a few weeks. Premature or draft datasets were sometimes utilised and, with an ever-expanding body of clinical data, evidence to support recommendations may have become outdated. Further, variable treatment availability and healthcare system characteristics could have led to differences in preferences or priorities. Such challenges have given rise to guidelines that are heterogeneous in their recommendations, quality and rigour; this can lead to confusion and uncertainty among clinicians, and disparate management practices around the globe [11, 12]. There is a need to identify high-quality COVID-19 treatment guidelines and provide clinicians with a synthesis of recommendations, especially those applicable to hospital settings.

Using a targeted literature review approach, this de novo review aims to: a) identify treatment guidelines for therapies most commonly used in patients hospitalised with COVID-19; b) systematically assess the quality of included guidelines; and c) compare the recommendations, evidence sources and considered outcomes of guidelines deemed to be of high quality.

Methods

Scope of review

COVID-19 treatment guidelines applicable to hospital settings were reviewed, with a focus on five key therapies used to treat COVID-19 in hospitalised patients: remdesivir, dexamethasone, tocilizumab, baricitinib and casirivimab/imdevimab. While some of these therapies may be used to treat non-hospitalised patients, this review focused only on their use in hospital settings.

This review focused on guidelines from governmental or scientific organisations within countries that had the resource capacity and the continued need (due to recent epidemiological disease burden) to develop and regularly update COVID-19 treatment guidelines. To this end, countries with the largest economies in their regions (within the top four highest 2020 total gross domestic product in the Americas, Asia, Europe, and the Middle East and Africa) [13] as well as the most severe, recent impact of COVID-19 (within the top 50 highest reported bi-weekly COVID-19 deaths globally, as of 1 January 2021) were identified [14]. Using these criteria, 11 countries were prioritised: Brazil, Canada, France, Germany, India, Italy, Japan, Mexico, Turkey, the UK and the USA. Guidelines from specific regional or international organisations with the resources and relevant influence to develop and update guidelines were also included. These comprised the World Health Organization (WHO), the European Respiratory Society (ERS) and the Surviving Sepsis Campaign (SSC). Initial searches to identify guidelines were conducted in August 2021; included guidelines were screened for updates on 4 January 2022.

Search strategy and selection criteria

Owing to rapid development and regular updating, the most recent COVID-19 treatment guidelines are often not published in journals or indexed in electronic databases. Therefore, targeted searches of continually updated websites, such as guideline repositories and libraries, were conducted (supplementary table S1).

Following an initial visual search, each source was queried using search terms tailored to the functionality, specificity and language of each source (supplementary table S2 and figure S1). Initial searches were conducted in August 2021.

Identified guidelines were screened against eligibility criteria (supplementary table S3 and figure S2) and prioritised for inclusion. Priority was given to guidelines that were applicable to the broadest possible patient population and to an entire country or region, assessed multiple therapies of interest, and directly assessed clinical data (rather than summarising information from other guidelines).

Quality assessments and data extraction

The quality of each included guideline was assessed by two reviewers using the Appraisal of Guidelines for Research and Evaluation II (AGREE-II) instrument [15]. Designed by an international team of guideline developers and researchers, AGREE-II has been used to rigorously assess the quality of a broad range of treatment guidelines, including those for infectious diseases [16–20]. The instrument includes 23 items across six domains: Domain 1: Scope and Purpose; Domain 2: Stakeholder Involvement; Domain 3: Rigour of Development; Domain 4: Clarity of Presentation; Domain 5: Applicability; and Domain 6: Editorial Independence (supplementary table S4) [15]. For each item, a score from 1 (strongly disagree) to 7 (strongly agree) was allocated. Any differences between reviewers of ≥2 points were resolved by discussion. A standardised domain score between 0% and 100% (low to high quality) was then calculated for each of the six domains.

For guidelines considered high-quality, the full details of recommendations were extracted by a single reviewer and verified by a second reviewer. High-quality guidelines were defined as those scoring ≥50% in Domain 3 (Rigour of Development) which determines whether recommendations are based on robust methodologies and reliable, up-to-date evidence. Similar thresholds of high quality have been used in other published AGREE-II assessments of guidelines [21–23]. Extracted details included the therapy, characteristics of the applicable population (e.g., supplemental oxygen requirement, disease severity, oxygen saturation), type of recommendation (for, against, clinical trials only, or insufficient evidence), and the evidence and outcomes used to support recommendations. Recommendations were categorised by the applicable population: hospitalised but do not require supplemental oxygen (“mild” disease); hospitalised and require low-flow supplemental oxygen (“moderate” disease); hospitalised and require noninvasive ventilation (NIV) or high-flow oxygen (“severe” disease); and hospitalised and require invasive mechanical ventilation (IMV) or extracorporeal membrane oxygenation (ECMO; “critical” disease).

Guideline updates

Included guidelines were screened on 4 January 2022 for updates since the initial searches in August 2021. Quality assessments were not repeated; however, data were re-extracted for updated guidelines. Where several versions of the same guideline had been published, only the most recent versions were considered.

Role of the funding source

Gilead Sciences Inc. provided funding to Costello Medical Inc. to conduct this study, under the direction of all authors. Five authors (AAE, LD, CYW, AL and RAB) are employees of Gilead Sciences Inc., and were involved in the study conception and design, the interpretation of data, in revising and approving the article and in the decision to submit the paper for publication. JKR and DAW were not compensated for their time in conducting this study.

Results

In total, 96 eligible treatment guidelines were identified; of these, 73 did not fulfil the prioritisation criteria, leaving 23 which were prioritised for quality assessment. Three were from international organisations and 20 were country-specific guidelines (supplementary table S5).

Guideline quality

There was considerable variability between the overall quality of guidelines, as well as the Domain scores within guidelines (supplementary table S5). The highest scores were generally achieved in Domain 4 (Clarity of Presentation), while the lowest scores were shown in Domains 5 and 6 (Applicability and Editorial Independence, respectively). A total of 13 guidelines achieved a score of ≥50% in Domain 3 (Rigour of Development), meeting the criteria for high-quality and full data extraction; of these, three were international guidelines and 10 were country-specific (from nine different countries). A broad geographical range was represented, including international guidelines and guidelines from Brazil, Canada, Germany, India, Italy, Japan, Mexico, the UK, and the USA [8–10, 24–41]. Supplementary table S6 provides a list of these 13 high-quality guidelines and a brief overview of their recommendations by therapy and applicable population.

High-quality guideline recommendations by therapeutic

Remdesivir

All 13 high-quality guidelines discussed remdesivir as monotherapy for use in patients hospitalised with COVID-19. Many also discussed the use of remdesivir within combination regimens. A summary of recommendations, supporting evidence and outcomes considered is presented in table 1.

TABLE 1.

Summary of evidence and outcomes reported by guidelines discussing remdesivir

Guidelines	RCTs						Outcomes considered
	ACTT-1 (NCT04280705) [5]	SIMPLE Severe (NCT04292899) [44]	SIMPLE Moderate (NCT04292730) [58]	WHO Solidarity (NCT04315948/ ISRCTN83971151) [42]	Wang 2020 (NCT04257656) [43]	Mahajan 2021 (NCT NR) [45]	Cost	Mortality	Clinical recovery	Discharge	Safety	Time to ICU admission	Progression to IMV	Other
Guidelines recommending for the use of remdesivir in at least one population described
SSC	✓		✓	✓	✓		✓	✓	✓		✓
COVID-19 Advisory Ontario	✓	✓	✓	✓	✓			✓	✓	✓		✓	✓	✓a
SITA/SIP	✓	✓	✓	✓	✓			✓	✓
J-SSCG	✓		✓	✓	✓		✓	✓	✓		✓		✓
Government of Mexico											✓
NICE	✓	✓	✓	✓	✓			✓	✓	✓	✓		✓	✓b
IDSA	✓			✓	✓			✓	✓		✓
NIH	✓		✓	✓	✓			✓	✓		✓		✓	✓c
Guidelines recommending against the use of remdesivir in any population described#
ERS	✓		✓	✓	✓		✓	✓			✓			✓d
WHO	✓		✓	✓	✓			✓		✓	✓			✓e
MoH Brazil	✓		✓	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓f
AWMF	✓		✓	✓	✓	✓	✓	✓			✓		✓	✓g
CIDS	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓		✓	✓h

“✓”: guideline explicitly mentions evidence or outcome. Empty cell: guideline does not explicitly mention evidence or outcome. ^#: includes guidelines which only provided recommendations against remdesivir or recommended against remdesivir for at least one disease severity category and reported insufficient evidence for any other categories. Organisations: AWMF: Association of the Scientific Medical Societies in Germany; CIDS: Clinical Infectious Disease Society; ERS: European Respiratory Society; IDSA: Infectious Disease Society of America; J-SSCG: Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock; MoH: Ministry of Health; NICE: National Institute for Health and Care Excellence; NIH: National Institutes of Health; SIP: Italian Society of Pulmonology; SITA: Italian Society of Anti-Infective Therapy; SSC: Surviving Sepsis Campaign; WHO: World Health Organization. Abbreviations: ICU: intensive care unit; IMV: invasive mechanical ventilation; NCT: National Clinical Trial; NR: not reported; RCT: randomised controlled trial. Other outcomes considered: ^a: ICU length of stay was considered; however, published data precluded the pooling of trials for ICU length. Need for oxygen support was considered. Clinical improvement outcomes were considered. ^b: septic shock within 28 or 30 days; acute respiratory failure or acute respiratory distress syndrome (ARDS) within 28 and 30 days. ^c: clinical improvement at Day 15, clinical status distribution on Day 11+Day 14. ^d: additional end-points which were searched for by the guideline development committee but were either not studied or data were not found in an extractable format were: deterioration in those not requiring ventilation at start of treatment; requirement for oxygen; hospital admission; ICU length of stay; need for noninvasive ventilation; hospital length of stay; severity of symptoms; improvement in oxygen saturations or arterial blood gases; relapse and duration of fever. ^e: viral clearance (7 days); acute kidney injury; delirium; time to clinical improvement; duration of ventilation. ^f: lack of benefit in patients using IMV. ^g: clinical worsening: new need for noninvasive mechanical ventilation or high-flow oxygen at day 28; clinical worsening: new need for oxygen by mask or nasal prongs at day 28; quality of life. ^h: progression to oxygen therapy; progression to noninvasive ventilation or high-flow oxygen. Sources: AWMF, October 2021 [24]; CIDS, June 2021 [26]; COVID-19 Advisory Ontario, May 2021 [34]; ERS, April 2021 [10]; Government of Mexico, August 2021 [35]; IDSA, November 2021 [36]; J-SSCG, September 2021 [37]; MoH Brazil, May 2021 [38]; NICE, January 2022 [8]; NIH, December 2021 [39]; SITA/SIP, May 2021 [40]; SSC, March 2021 [41]; WHO, January 2022 [9].

Of the guidelines that discussed the use of remdesivir in mild (N=9) [8–10, 24, 26, 36, 38, 39] and critical (n=12) [9, 10, 34, 36, 37, 39, 41] disease, most recommended against the use of remdesivir monotherapy (mild: n=6 [8, 9, 24, 26, 36, 38]; critical: n=10 [9, 10, 24, 26, 34, 36–39, 41]). Others suggested that there was insufficient evidence or that remdesivir (alone or in combination) should only be used in clinical trials [8, 10, 35, 39]. In moderate and severe disease, there was a mix of recommendations regarding remdesivir monotherapy, but remdesivir in combination with other therapies (particularly dexamethasone or baricitinib) was more frequently recommended. The WHO [9], Brazilian Ministry of Health (MoH) [38] and Indian Clinical Infectious Disease Society (CIDS) [26] guidelines did not recommend remdesivir in any disease severity. The ERS international guidelines [10] and the Association of the Scientific Medical Societies in Germany (AWMF) [24] guidelines recommended against remdesivir in some disease severities and reported insufficient evidence for others.

All guidelines, besides those from the Government of Mexico [35], reported considering evidence from the Adaptive COVID-19 Treatment Trial-1 (ACTT-1) [5], WHO Solidarity [42] and Wang et al. 2020 [43] trials. No guidelines recommending against remdesivir considered the SIMPLE Severe trial [44], but three which recommended for remdesivir considered results from this study: COVID-19 Advisory Ontario [34], Italian Society of Anti-Infective Therapy (SITA)/Italian Society of Pulmonology (SIP) [40], and the National Institute for Health and Care Excellence (NICE) [8]. Only two organisations considered results from Mahajan et al. 2021 [45], both of which recommended against the use of remdesivir in hospitalised patients [24, 26].

Mortality and safety outcomes were considered by most guidelines (mortality: n=12 [8–10, 24, 26, 34, 36–41]; safety: n=11 [8–10, 24, 26, 35–39, 41]), regardless of recommendation. However, the guidelines that recommended against remdesivir more frequently considered cost (n=4) [9, 10, 24, 38], and those which recommended for remdesivir more frequently considered clinical recovery (n=7) [8, 34, 36, 37, 39–41]. Regardless of recommendation, few guidelines considered time to discharge (n=5) [8, 9, 26, 34, 38, 39] and time to intensive care unit (ICU) admission (n=2) [34, 38].

Dexamethasone

All 13 high-quality guidelines discussed the use of dexamethasone in patients hospitalised with COVID-19 (table 2); of these, all recommended dexamethasone as monotherapy in severe and/or critical disease while most (n=10) [8–10, 24, 35–39] recommended against its use in mild disease. Many also recommended dexamethasone in combination with other therapies. For example, in the severe and critical populations, several guidelines recommended tocilizumab in combination with dexamethasone if patients had rapidly progressive disease (severe: n=8 [8, 10, 24, 27, 33, 35, 36, 40]; critical: n=7 [8, 10, 27, 33, 35, 36, 39]). A smaller number of guidelines also recommended remdesivir with dexamethasone (n=2) [8, 39] or baricitinib with dexamethasone (n=3) [27, 36, 39]. The National Institutes of Health (NIH) guidelines recommended the use of baricitinib or tocilizumab in combination with remdesivir and dexamethasone [39].

TABLE 2.

Summary of evidence and outcomes reported by guidelines discussing dexamethasone

Guidelines	RCTs					Outcomes considered
	DEXA-COVID19 (NCT04325061) [46]	RECOVERY (NCT04381936) [3]	CoDEX (NCT04327401) [47]	Jamaati 2021 (IRCT20151227025726N17) [59]	Ranjbar 2021 (IRCT20200204046369N1) [60]	Cost	Mortality	Clinical recovery	Discharge	Safety	Time to ICU admission	Progression to IMV	Other
Guidelines recommending for the use of dexamethasone in at least one population described
ERS	✓	✓	✓			✓	✓		✓	✓	✓		✓a
SSC	✓	✓	✓			✓	✓			✓			✓b
WHO	✓	✓	✓				✓		✓	✓		✓
MoH Brazil						✓	✓		✓	✓
COVID-19 Advisory Ontario		✓	✓				✓					✓
AWMF		✓			✓	✓	✓			✓		✓	✓c
CIDS		✓				✓	✓			✓		✓
SITA/SIP	✓	✓	✓				✓
J-SSCG	✓	✓	✓			✓	✓	✓
Government of Mexico	✓	✓	✓	✓	✓		✓			✓
NICE	✓	✓	✓	✓	✓		✓		✓	✓		✓	✓d
IDSA	✓	✓	✓				✓		✓	✓
NIH		✓	✓				✓		✓	✓	✓	✓	✓e

“✓”: guideline explicitly mentions evidence or outcome. Empty cell: guideline does not explicitly mention evidence or outcome. Organisations: AWMF: Association of the Scientific Medical Societies in Germany; CIDS: Clinical Infectious Disease Society; ERS: European Respiratory Society; IDSA: Infectious Disease Society of America; J-SSCG: Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock; MoH: Ministry of Health; NICE: National Institute for Health and Care Excellence; NIH: National Institutes of Health; SIP: Italian Society of Pulmonology; SITA: Italian Society of Anti-Infective Therapy; SSC: Surviving Sepsis Campaign; WHO: World Health Organization. Abbreviations: ARDS: acute respiratory distress syndrome; ICU: intensive care unit; IMV: invasive mechanical ventilation; NCT: National Clinical Trial; NR: not reported; RCT: randomised controlled trial. Other outcomes considered: ^a: additional end-points not included in the evidence table which were searched for but were either not studied or data were not found in an extractable format were: clinical resolution or cure (also includes the reverse, i.e. patients not cured); time to clinical improvement or resolution on an ordinal scale; requirement for oxygen; hospital admission; ordinal scale or clinical status at day 28; ICU length of stay; need for noninvasive ventilation; deterioration in those not requiring ventilation at start of treatment; severity of symptoms; improvement in oxygen saturations or arterial blood gases; relapse; viral clearance (negative SARS-CoV-2 test) and duration of fever. ^b: examined evidence from non-COVID-19 ARDS patients as well. ^c: ventilator-free days at 28 days; quality of life, including fatigue and neurological status long term; hospital-acquired infections during treatment per 28 days. ^d: outcomes assessed by end of treatment: gastrointestinal bleeding, bacterial co-infections, hyperglycaemia, neuromuscular weakness and neuropsychiatric effects. ^e: need for insulin; positive blood cultures at Day 7; sepsis by Day 28; score on 6-point WHO ordinal scale at Day 15; mean Sequential Organ Failure Assessment (SOFA) score; need for rescue strategies; need for oxygenation; organ-support free-days at Day 21; duration of fever; virus clearance time; proportion prescribed antibiotics and antifungal therapy. Sources: AWMF, October 2021 [24]; CIDS, July 2021 [27]; COVID-19 Advisory Ontario, August 2021 [33]; ERS, April 2021 [10]; Government of Mexico, August 2021 [35]; IDSA, October 2021 [36]; J-SSCG, September 2021 [37]; MoH Brazil, May 2021 [38]; NICE, January 2022 [8]; NIH, December 2021 [39]; SITA/SIP, May 2021 [40]; SSC, March 2021 [41]; WHO, January 2022 [9].

Except those from the Brazilian MoH [38], all guidelines considered evidence from the RECOVERY trial (n=12) [3]. Most also considered information from DEXA-COVID19 (n=8) [46] and CoDEX (n=10) [47]. In addition, several guidelines considered evidence from trials which investigated corticosteroids other than dexamethasone; e.g., REMAP-CAP (n=9) [48], CAPE COVID (n=9) [49], COVID STEROID (n=7) [50] and GLUCOCOVID (n=6) [51].

The NIH and ERS guidelines considered the broadest range of outcomes, including several “other” outcomes such as viral clearance and duration of fever [10, 39]. All guidelines recommending dexamethasone in severe and/or critical disease but against its use in mild disease (n=10) [8–10, 24, 27, 35–39] considered mortality outcomes, and most considered safety outcomes. Few guidelines considered clinical recovery (n=1) [37] and time to ICU admission (n=2) [10, 39].

Tocilizumab

Overall, 12 of the 13 high-quality guidelines discussed the use of tocilizumab in patients hospitalised with COVID-19 (table 3); the international SSC guidelines were the only not to discuss its use [41]. Three guidelines recommended against tocilizumab in mild disease [10, 24, 25]. Meanwhile, in severe disease, 10 guidelines recommended tocilizumab [8–10, 24, 25, 30, 35, 36, 39, 40]; eight of these explicitly recommended its use in combination (predominantly with dexamethasone) [8, 10, 24, 25, 30, 35, 36, 40]. The Brazilian MoH guidelines reported insufficient evidence to formulate a recommendation for the use of tocilizumab (as a monotherapy) in severe COVID-19 [38]. In moderate and critical COVID-19, guidelines generally recommended for the use of tocilizumab (n=5 [8, 10, 30, 35, 37] and n=8 [8–10, 25, 30, 35, 36, 39], respectively), predominantly in combination with dexamethasone. However, two guidelines in each disease severity category recommended against its use as a monotherapy (AWMF and Indian CIDS guidelines recommended against use in moderate COVID-19 [24, 25]; AWMF and Brazilian MoH guidelines recommended against its use in severe COVID-19 [24, 38]). Seven guidelines specifically recommend tocilizumab in patients who have progressive disease and systemic inflammation (as measured by increased levels of C-reactive protein) [8, 24, 25, 30, 35, 36, 39, 40].

TABLE 3.

Summary of evidence and outcomes reported by guidelines discussing tocilizumab

Guidelines	RCTs												Outcomes considered
	REMAP-CAP (NCT02735707) [63]	RECOVERY (NCT04381936) [4]	TOCIBRAS (NCT04403685) [52]	CORIMUNO-19 (NCT04331808) [53]	COVACTA(NCT04320615) [54]	EMPACTA (NCT04372186) [55]	Salvarani 2021 (NCT04346355) [64]	Stone 2020 (NCT04356937) [56]	Wang 2020 (ChiCTR2000029765) [67]	COVINTOC (CTRI/2020/05/025369) [65]	PreToVid (EU-CTR-2020-001375-32) [61]	Talaschian 2021 [66]	Cost	Mortality	Clinical recovery	Discharge	Safety	Time to ICU admission	Progression to IMV	Other
Guidelines recommending for the use of tocilizumab in at least one population described
ERS	✓	✓	✓	✓	✓	✓	✓	✓						✓			✓		✓	✓a
WHO														✓	✓	✓	✓		✓	✓b
COVID-19 Advisory Ontario	✓	✓	✓	✓	✓	✓	✓	✓	✓					✓		✓	✓		✓	✓c
AWMF	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓				✓	✓	✓	✓		✓	✓d
CIDS	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓			✓	✓	✓	✓	✓	✓	✓	✓e
SITA/SIP		✓	✓	✓	✓	✓	✓	✓		✓				✓	✓		✓	✓#	✓¶	✓f
J-SSCG	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓		✓			✓
Government of Mexico														✓			✓		✓
NICE	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓				✓	✓		✓	✓	✓	✓g
IDSA	✓	✓	✓	✓	✓	✓	✓	✓						✓			✓	✓	✓	✓h
NIH	✓	✓	✓	✓	✓	✓	✓	✓		✓				✓			✓		✓	✓i
Guidelines Recommending against the use of tocilizumab in any population described+
MoH Brazil	✓	✓	✓	✓	✓	✓	✓	✓						✓			✓	✓	✓

“✓”: guideline explicitly mentions evidence or outcome. Empty cell: guideline does not explicitly mention evidence or outcome. ^#: composite end-point of ICU admission, death or clinical worsening led to enrolment discontinuation for futility. ^¶: composite of death, mechanical ventilation and clinical worsening. ⁺: includes guidelines which only provided recommendations against tocilizumab or recommended against tocilizumab for at least one disease severity category and reported insufficient evidence for any other categories. Organisations: AWMF: Association of the Scientific Medical Societies in Germany; CIDS: Clinical Infectious Disease Society; ERS: European Respiratory Society; IDSA: Infectious Disease Society of America; J-SSCG: Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock; MoH: Ministry of Health; NICE: National Institute for Health and Care Excellence; NIH: National Institutes of Health; SIP: Italian Society of Pulmonology; SITA: Italian Society of Anti-Infective Therapy; SSC: Surviving Sepsis Campaign; WHO: World Health Organization. Abbreviations: ICU: intensive care unit; IMV: invasive mechanical ventilation; NCT: National Clinical Trial; NR: not reported; RCT: randomised controlled trial. Other outcomes considered: ^a: clinical worsening. ^b: duration of IMV. ^c: composite or mortality or invasive mechanical ventilation was considered; respiratory or cardiovascular organ support-free days was considered. ^d: need for new haemodialysis/haemofiltration.^e: ventilator-free days. ^f: disease progression; scoring higher than 5 on day 4 on a 10-point ordinal clinical scale. ^g: ordinal scale combining in-hospital mortality and days free of organ support by Day 21, days free of organ support in survivors by Day 21. ^h: clinical deterioration (follow-up: range 14 days to 30 days). ⁱ: progression to high-flow oxygen or noninvasive ventilation. Sources: AWMF, October 2021 [24]; CIDS, May 2021 [25]; COVID-19 Advisory Ontario, March 2021 [72]; ERS, April 2021 [10]; Government of Mexico, August 2021 [35]; IDSA, October 2021 [36]; J-SSCG, September 2021 [37]; MoH Brazil, May 2021 [38]; NICE, January 2022 [8]; NIH, December 2021 [39]; SITA/SIP, May 2021 [40]; WHO, January 2022 [9].

Regardless of recommendation, six different trials (RECOVERY [3], TOCIBRAS [52], CORIMUNO-19 [53], COVACTA [54], EMPACTA [55] and Stone et al. 2020 [56]) were considered by the large majority of guidelines (n=10) [8, 10, 24, 25, 30, 36–40]. The Government of Mexico did not consider any evidence, and the WHO guidelines only considered meta-analyses of trials [9, 35]. In terms of evidence considered, there were no distinct differences between guidelines which consistently recommended tocilizumab and those which recommended against its use in some populations.

All guidelines considered mortality and safety outcomes. The majority also considered progression to IMV (n=11) [8–10, 24, 25, 30, 36, 38–40], either as an individual outcome or as part of a composite outcome. Discharge, clinical recovery and time to ICU admission were each considered by approximately half of the guidelines. There were no clear differences in the outcomes that were considered between guidelines which consistently recommended tocilizumab and those which recommended against its use in some populations.

Baricitinib

Nine high-quality guidelines discussed the use of baricitinib among patients hospitalised with COVID-19 (table 4) [9, 24, 28, 32, 35–37, 39, 40]. Five guidelines discussed the use of baricitinib in mild disease, though recommendations were relatively inconsistent [24, 28, 32, 35, 39]. In moderate, severe and critical COVID-19, guidelines generally recommended baricitinib as a monotherapy (n=3 [24, 32, 37]; n=3 [9, 37, 39]; n=2 [9, 32], respectively). Some also recommended its use in combination with dexamethasone and/or remdesivir (n=2 [28, 39]; n=4 [28, 36, 39, 40]; n=2 [28, 39], respectively). The NIH guidelines discussed the use of baricitinib in combination with tocilizumab in severe and critical COVID-19 but recommended against this combination therapy except in clinical trial settings [39]. Across all disease severities, the Government of Mexico guidelines stated that baricitinib should be examined in the context of clinical trials only, preferably in combination with remdesivir [35].

TABLE 4.

Summary of evidence and outcomes reported by guidelines discussing baricitinib

Guidelines	RCTs		Outcomes considered
	ACTT-2 (NCT04401579) [6]	COV-BARRIER (NCT04421027) [57]	Cost	Mortality	Clinical recovery	Discharge	Safety	Time to ICU admission	Progression to IMV	Other
Guidelines recommending for the use of baricitinib in at least one population described
WHO	✓	✓		✓	✓	✓	✓		✓	✓a
COVID-19 Advisory Ontario	✓	✓		✓	✓		✓		✓	✓b
AWMF	✓	✓	✓	✓	✓	✓	✓		✓
CIDS	✓	✓	✓	✓	✓	✓	✓		✓	✓c
SITA/SIP	✓			✓	✓		✓			✓d
J-SSCG	✓	✓	✓	✓	✓		✓
IDSA	✓	✓		✓		✓	✓		✓	✓e
NIH	✓	✓		✓	✓		✓		✓
Guidelines recommending the use of baricitinib in clinical trials only
Government of Mexico	✓	✓		✓			✓

“✓”: guideline explicitly mentions evidence or outcome. Empty cell: guideline does not explicitly mention evidence or outcome. Organisations: AWMF: Association of the Scientific Medical Societies in Germany; CIDS: Clinical Infectious Disease Society; IDSA: Infectious Disease Society of America; J-SSCG: Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock; NIH: National Institutes of Health; SIP: Italian Society of Pulmonology; SITA: Italian Society of Anti-Infective Therapy; WHO: World Health Organization. Abbreviations: ICU: intensive care unit; IMV: invasive mechanical ventilation; NCT: National Clinical Trial; RCT: randomised controlled trial. Other outcomes considered: ^a: duration of IMV. ^b: progression to new noninvasive ventilation or high-flow oxygen. ^c: clinical status at day 15. ^d: need for noninvasive ventilation. ^e: disease progression (follow-up: 28 days), IMV-free days (follow-up; 60 days). Sources: AWMF, October 2021 [24]; CIDS, September 2021 [28]; COVID-19 Advisory Ontario, January 2022 [32]; Government of Mexico, August 2021 [35]; IDSA, October 2021 [36]; J-SSCG, September 2021 [37]; MoH Brazil, May 2021 [38]; NICE, January 2022 [8]; NIH, December 2021 [39]; SITA/SIP, May 2021 [40]; WHO, January 2022 [9].

Evidence from the ACTT-2 trial [6] was considered by all organisations, while COV-BARRIER [57] was considered by all guidelines except those from SITA/SIP [40].

All nine guidelines considered mortality and safety outcomes and seven (all except the Infectious Disease Society of America (IDSA) [36] and Government of Mexico [35] guidelines) considered clinical recovery. No guidelines considered time to ICU admission and only three considered cost. The Government of Mexico guidelines [35], which stated that baricitinib should be examined in clinical trials only, considered the fewest outcomes, while CIDS [28] and AWMF [24] considered the broadest range.

Casirivimab/imdevimab

In total, 10 high-quality guidelines discussed the use of casirivimab/imdevimab in patients hospitalised with COVID-19, though the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock (J-SSCG) did not make a clear recommendation (table 5) [8, 9, 24, 29, 31, 35, 37–40]. Overall, there was no clear consensus between guidelines regarding casirivimab/imdevimab recommendations. However, within guidelines, the same recommendation was often given across all disease severities for which casirivimab/imdevimab were discussed. AWMF [24] discussed casirivimab/imdevimab in mild and moderate disease, recommending its use in both; the Brazilian MoH guidelines [38] recommended against the use of casirivimab/imdevimab in all four severities; the NICE, COVID-19 Advisory Ontario and Indian CIDS guidelines [8, 29, 31] recommended its use in seronegative patients only; and the Government of Mexico and J-SSCG guidelines [35, 37] suggested that there was insufficient evidence to formulate a recommendation.

TABLE 5.

Summary of evidence and outcomes reported by guidelines discussing casirivimab/imdevimab

Guidelines	RCTs		Outcomes considered
	REGN-COV 2067 (NCT04425629) [7]	RECOVERY (NCT04381936) [62]	Cost	Mortality	Clinical recovery	Discharge	Safety	Time to ICU admission	Progression to IMV	Other
Guidelines recommending for the use of casirivimab/imdevimab in at least one population described
WHO	✓	✓	✓#	Y		✓	✓		✓	✓a
COVID-19 Advisory Ontario		✓		✓		✓	✓		✓¶	✓b
AWMF	✓	✓		✓		✓	✓		✓¶	✓c
CIDS (seronegative)		✓	✓	✓		✓			✓	✓d
NICE		✓		✓		✓	✓		✓	✓e
Guidelines recommending the use of casirivimab/imdevimab in clinical trials only
Government of Mexico				✓			✓			✓f
Guidelines recommending against the use of casirivimab/imdevimab in any population described+
MoH Brazil
SITA/SIP
NIH

“✓”: guideline explicitly mentions evidence or outcome. Empty cell: guideline does not explicitly mention evidence or outcome. ^#: qualitatively considered. ^¶: composite end-point of need for IMV or death. ⁺: includes guidelines which only provided recommendations against casirivimab/imdevimab or recommended against casirivimab/imdevimab for at least one disease severity category and reported insufficient evidence for any other categories. Organisations: AWMF: Association of the Scientific Medical Societies in Germany; CIDS: Clinical Infectious Disease Society; MoH: Ministry of Health; NICE: National Institute for Health and Care Excellence; NIH: National Institutes of Health; SIP: Italian Society of Pulmonology; SITA: Italian Society of Anti-Infective Therapy; WHO: World Health Organization. Abbreviations: ICU: intensive care unit; IMV: invasive mechanical ventilation; NCT: National Clinical Trial; RCT: randomised controlled trial. Other outcomes considered: ^a: admission to hospital; time to symptom improvement. ^b: time-weighted average daily change from baseline viral load in nasopharyngeal samples from day 1–7. ^c: need for dialysis; neurological function; viral clearance. ^d: progression to noninvasive ventilation; progression to organ replacement therapy. ^e: within 28 days of randomisation: noninvasive mechanical ventilation. ^f: reduction in hospitalisations. Sources: AWMF, October 2021 [24]; CIDS, August 2021 [29]; COVID-19 Advisory Ontario, November 2021 [31]; Government of Mexico, August 2021 [35]; IDSA, October 2021 [36]; J-SSCG, September 2021 [37]; MoH Brazil, May 2021 [38]; NICE, January 2022 [8]; NIH, December 2021 [39]; SITA/SIP, May 2021 [40]; WHO, January 2022 [9].

All guidelines which recommended casirivimab/imdevimab considered evidence from the RECOVERY trial [3] and two [9, 24] also considered REGN-COV 2067 [7]. Guidelines recommending against casirivimab/imdevimab did not consider either of these trials [38–40]. The NIH strictly followed guidance in the US Food and Drug Administration (FDA) Emergency Use Authorization (EUA) document that casirivimab/imdevimab should only be used in non-hospitalised patients.

Guidelines which recommended casirivimab/imdevimab more frequently considered mortality, discharge, safety outcomes and progression to IMV than those which recommended against its use. Guidelines which recommended for the use of casirivimab/imdevimab typically considered a broader range of clinical outcomes than those which recommended against its use.

Discussion

During the COVID-19 pandemic, guideline bodies around the globe have attempted to provide clinicians with recommendations for treating patients hospitalised with COVID-19. Assessments using the AGREE-II tool revealed considerable heterogeneity in the quality of the 23 guidelines included in this review. Even within guidelines, there was often substantial variation in quality across AGREE-II Domains, with few guidelines achieving consistently high scores. Overall, 13 guidelines passed the threshold for high quality, scoring ≥50% in Domain 3 (Rigour of Development).

For dexamethasone and tocilizumab, recommendations across the 13 high-quality guidelines were relatively consistent. Meanwhile, recommendations for remdesivir, baricitinib and casirivimab/imdevimab varied between guidelines. Recommendations relating to remdesivir were particularly varied in moderate and severe disease categories, while baricitinib recommendations were most varied in mild COVID-19. Across all disease severity categories, there was no agreement on recommendations for casirivimab/imdevimab.

There was a general trend between the number of guidelines discussing a particular therapeutic and the availability and extent of clinical trial data. Remdesivir [5, 42–45, 58], dexamethasone [3, 46, 47, 59, 60] and tocilizumab [52–56, 61–67] have been investigated in several clinical trials and were discussed by almost all guidelines, while baricitinib [6, 57] and casirivimab/imdevimab [7, 62] had been investigated in fewer trials and were discussed within fewer guidelines. For all therapies except remdesivir, there also appeared to be a trend between the availability of clinical trial data and the consensus between recommendations; a greater number of clinical trials was typically associated with increased consensus. It may be argued that when clinical data are not readily available, guideline development groups either make no recommendation or fill data vacuums with input from expert opinion, which can be highly subjective and inconsistent. When data become available, such recommendations may require changing or updating.

Even when clinical trial data are available, guideline development groups may not choose to use all data to inform decisions. For example, multiple trials [3, 46, 47, 59, 60] examined dexamethasone in COVID-19 patients, but most guidelines considered just a few studies (or often considered trials of other corticosteroids) and have not updated their dexamethasone recommendations since mid-2020. This is likely due to the publication of strong supportive evidence for the use of dexamethasone early in the pandemic [3], and later evidence corroborating earlier findings [47]. However, if trial results are less strong or consistent, considering different data (due to the timing of evidence publication, for example) could lead to differences in recommendations. For instance, many of the remdesivir trials came to contradictory conclusions: ACTT-1 [5] and SIMPLE Moderate [58] had broadly positive results; the WHO Solidarity trial [42] suggested negative outcomes; and Wang et al. 2020 [43] was insufficiently powered, stopping early with negative outcomes.

There were no clear trends between recommendations and the consideration of specific outcomes. However, where recommendations were more consistent, high-visibility end-points were considered. Guidelines relating to dexamethasone more frequently considered mortality, for example. Notably, for dexamethasone recommendations, few outcomes other than mortality were considered, likely because death is the most visible and important clinical end-point; when mortality data are available, other outcomes are less influential. As the number of approved therapies increased over the course of the pandemic, and more outcomes were included and achieved in trials, it is likely that these were increasingly favoured over less visible outcomes.

The timing of a study and its data cuts may also influence the specific outcomes to consider. Some ordinal scales, for instance, were developed over the course of trials, and mortality outcomes were sometimes only available as post hoc analyses. Consequently, the timing of guideline publication and/or guideline updates may have influenced which data cuts and outcomes were considered in guideline development. Further, the timing of the study may influence how outcomes were defined. For example, patients enrolled in the ACTT-1 trial [5] (which was conducted early in the pandemic) were sometimes required to remain hospitalised, despite not requiring supplemental oxygen or ongoing medical care, due to infection-control measures. As such, discharge-related outcomes required re-defining [5].

The inconsistency between recommendations, even when considering the same evidence, suggests that subjective factors, such as cost, accessibility, alignment with other guidelines and COVID-19 variants, also had an impact. Therefore, while the current review provides novel information to clinicians, they should be aware of and consider the totality of evidence prior to making treatment decisions. Additionally, clinicians should recognise when there is a possibility that non-clinical factors may play a role in influencing clinical practice decisions. For instance, dexamethasone is a relatively low-cost therapy, which may reinforce the preference to recommend its use. Meanwhile, remdesivir is more expensive, which, when combined with inconsistent data, could support decisions to recommend against its widespread use. Access may have contributed to the AWMF's [24] recommendation for casirivimab/imdevimab, given the purchase of a large stock of this therapy by Germany's Health Ministry [68]. In resource-limited settings, alignment with the WHO guidelines [9] may have an influential role; in their recommendations against remdesivir, the Brazilian MoH and Indian CIDS referenced the WHO guidelines, which also did not recommend this therapy. Finally, guideline groups may have shaped casirivimab/imdevimab recommendations on the most prevalent COVID-19 variant in circulation. This is an important consideration which was not investigated here but will likely play an increasing role in COVID-19 treatment guidelines, particularly for monoclonal antibodies. Finally, it would be important to look back at the actions and processes taken during the COVID-19 pandemic to see what can be learned for future situations.

Limitations

There are limitations to this analysis that should be considered when interpreting its findings. Foremost, the review focused only on five major therapies, although others (such as sarilumab and tofacitinib) have been recommended for use among hospitalised COVID-19 patients [8, 9, 36]. The review does not consider recommendations relevant to non-hospitalised patients, who arguably make up the greatest proportion of individuals infected with COVID-19 [69]. Further, while a broad geographical range was represented by the included guidelines, large geographical regions (e.g., the Middle East and Africa) were left unrepresented.

Though the review looked to identify the most up-to-date guidelines, the rapid development and regular updating of recommendations inevitably meant that more recent information may not have been captured in this review. For instance, treatment guidelines such as those developed by the European Society of Clinical Microbiology and Infectious Diseases [70] were not captured, as they were published after the initial guideline identification cut-off. Similarly, impactful data from recent clinical trials may have been missed. One example is the PINETREE trial (NCT04501952), which is referenced by multiple guidelines to support remdesivir recommendations in patients at high risk for progression to severe disease in hospital and ambulatory settings [71]. Other recent results from trials evaluating mAbs, oral antivirals and JAK inhibitors have also been considered, but these data were not incorporated into guidelines at the time of review.

The AGREE-II tool is just one method for assessing the quality of guidelines and has limitations in its approach. While the use of the AGREE-II Domain 3 to determine high quality is supported by other published guideline assessments, this definition disregards other characteristics which contribute to guideline quality, such as stakeholder engagement or clarity of presentation. Given the variability in performance across Domains, an alternative definition of high quality would have led to a different set of guidelines undergoing full data extraction. Additionally, since AGREE-II does not address in detail the methodology used for guideline development (e.g., analyses of bias and variance), analysis of the methodology could also result in a different set of included guidelines. AGREE-II also does not consider the quality of the evidence included in the guidelines, and thus the current review did not aim to make a judgement of the quality of the evidence included in any given guideline, but simply to assess the quality of the guideline in a systematic manner. Another potential limitation of AGREE-II is the lack of analysis of population/intervention/comparison/outcome (PICO) questions and formulations across guidelines; differences recommendations could arise due to differences in PICO questions. Further, though AGREE-II has been used to assess many types of guidelines, its applicability to those developed in health emergencies such as pandemics has not yet been verified. Finally, the findings of this review are themselves limited by the transparency of guidelines regarding their justifications for recommendations, and the rapid pace at which guidelines, clinical trial results and recommendations are being updated in line with evolving data and COVID-19 variants.

Conclusion

This review identified substantial heterogeneity in the quality of guidelines for the therapeutic treatment of patients hospitalised with COVID-19. Further, even among high-quality guidelines, it was found that recommendations regarding specific therapeutics varied, despite using the same clinical trials and specific outcomes. These findings suggest that unreported, subjective factors may also play a role, particularly where evidence is limited or conflicting, and call for guideline groups to justify their recommendations more transparently. In response to health emergencies, greater global collaboration to produce, synthesise and update evidence, along with country- or region-specific efforts to develop locally relevant recommendations, may provide clearer and more consistent guidance to clinicians.

Supplementary material

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary material 00236-2022.SUPPLEMENT ^{(745.9KB, pdf)}