Consistency of recommendations from clinical practice guidelines for the management of critically ill COVID-19 patients

Abstract

Background

A significant knowledge gap exists for the management of critically ill patients with coronavirus disease 2019 (COVID-19). This study aimed to systematically investigate the consistency of recommendations from the available clinical practice guidelines (CPGs) to those of the WHO on the management of critically ill COVID-19 patients.

Methods

We examined CPGs and UpToDate point-of-care resources on the management of critically ill COVID-19 patients that had been published as of 30 April 2020 and compared them against the CPG by the WHO. The main outcome was the rate of consistency among CPGs for the management of critically ill COVID-19 patients. Sensitivity analyses were conducted by excluding recommendation statements that were described as insufficient evidence and by excluding single CPGs one at a time.

Results

Thirteen reference recommendations derived from the CPG of the WHO were generated using discrete and unambiguous specifications of the population, intervention, and comparison states. Across CPGs, the rate of consistency in direction with the WHO is 7.7%. When insufficient evidence codings were excluded, the rate of consistency increased substantially to 61.5%. The results of a leave-one-out sensitivity analysis suggested that the UpToDate recommendation source could explain the inconsistency. Consistency in direction rates changed by an absolute 23.1% (from 1/13 (7.7%) to 4/13 (30.8%)) if UpToDate was removed.

Conclusions

We observed inconsistencies between some recommendations of the CPGs and those of the WHO. These inconsistencies should best be addressed by consensus among the relevant bodies to avoid confusion in clinical practice while awaiting clinical trials to inform us of the best practice.

Introduction

The unprecedented rapid spread of novel coronavirus disease 2019 (COVID-19) has created a significant knowledge gap for its management. The publication of an interim clinical practice guideline (CPG) on the clinical management of COVID-19 by the World Health Organization (WHO) is a much-needed initiative.¹ Major infectious disease/intensive care societies and national organisations have since followed suit and issued their own CPGs. Nevertheless, variation in recommendations with those of the WHO in each CPG may lead to ambiguity in clinical practice. We aim to investigate the consistency of recommendations in the available CPGs to those of the WHO on the management of critically ill COVID-19 patients.

Methods

This cross-sectional study adapted methods from Alper et al ² to evaluate consistency in CPGs that had been published as of 30 April 2020. We searched four sources that listed links to national and international CPGs: DynaMed Plus,³ UpToDate,⁴ BMJ Best Practice,⁵ and Turning Research into Practice,⁶ to identify CPGs on the management of critically ill COVID-19 patients. From our initial search, we selected CPGs that were currently active, publicly available, published in (or has been translated to) English, likely to be used as the primary source of guidance for the management of critically ill COVID-19 patients, and published by societies/organisations that have a membership of at least 1000 or more healthcare practitioners.

The recommendations concerning therapeutic management of critically ill patients with COVID-19 in the WHO CPG¹ was selected to be the reference standard for other CPGs to be compared against, since this was the first CPG issued on the management of critically ill COVID-19 patients, and it was intended to be the guide for a structured global response. There may be different interpretations among coders towards recommendations by the WHO in terms of patient population, intervention, and comparator, and thus to ensure consistency without ambiguity, we created a reference standard to compare recommendations against. A total of 13 reference recommendations derived from the CPG of the WHO¹ were generated using a combined population-intervention-comparison (PIC) concept to provide a consistent framework and scope for comparison to ease the comparison process without any ambiguity. The population was for whom the recommendations from the CPG of the WHO were intended, and intervention was defined as the approach adopted in the CPG of the WHO,¹ whereas comparator was defined as the approach in contrast to the CPG of the WHO.¹

To minimise bias, for each reference recommendation derived from the WHO CPG, a coder (KCS) independently compared and coded the recommendation from comparator CPGs in three steps. First, the coder assessed whether each recommendation was addressed in each comparator CPG and, if not, the recommendation was labelled ‘out of scope’ for that CPG and excluded from further analysis; second, each in scope recommendation was compared for consistency to the reference standard; and finally, the rate of consistency of all recommendations was assessed.

The consistency of the recommnedations in the comparator CPG to the reference standard derived from WHO CPG was coded as:

1. ‘for’ if the CPG recommended the intervention in favour of the comparison,

2. ‘against’ if the CPG recommended the comparison in favour of the intervention,

3. ‘insufficient’ if the CPG did not recommend ‘for’ or ‘against’ the intervention due to insufficient evidence from the CPG to recommend ‘for’ or ‘against’, but the PIC specification was within the scope to be addressed, or

4. ‘different’ if the assertion from the CPG could not be categorised as ‘for’, ‘against’, or ‘insufficient’.

A code reviewer (SSH) independently checked the coding of the coder. An investigator with clinical experience in critical care (STRZ) reviewed the codings from both the coder and the reviewer, and any discrepancy identified was resolved by consensus.

While assessing the rate of consistency, we did not include CPG coded as ‘out of scope’ or ‘different’ for a reference recommendation in any of the analyses for consistency, as these code for an absence of recommendation rather than the similarity of a recommendation. We assessed the rate of consistency only if two or more CPGs provided a coding of ‘for’, ‘against’, or ‘insufficient’. For assessments of the rate of consistency, we regarded the reference recommendation to be consistent with the WHO CPG if all comparator CPG codings were ‘for’, or if they were all coded as ‘for’ or ‘insufficient’ but ≥60% were ‘for’. In contrast, we regarded the reference recommendation inconsistent with the WHO CPG if all comparator CPG codings were all either ‘against’ or ‘insufficient’, or if they were all coded as ‘for’ or ‘insufficient’ but <60% were ‘for’.

We conducted a sensitivity analysis that excluded insufficient codings from the analysis. We also performed a leave-one-out sensitivity analysis to assess rates of consistency with each CPG excluded one at a time.

Results

The initial search yielded 300 CPGs (online supplementary figure S1). We excluded 293 of these based on our consensus (281 CPGs did not address management of critically ill COVID-19 patients, eight CPGs were published as review articles without endorsement from professional organisations, two CPGs addressed <3 reference recommendations, and two CPGs published in a language other than English and no English translation was available); the remaining seven CPGs came from the Australian and New Zealand Intensive Care Society (ANZ),⁷ Belgium (BEL),⁸ Canada (CAN),⁹ National Health Commission (NHC) of China,¹⁰ National Institutes of Health (NIH),¹¹ Surviving Sepsis Campaign (SSC),¹² a collaboration of Faculty of Intensive Care Medicine, Intensive Care Society, Association of Anaesthetists and Royal College of Anaesthetists of the UK (ICM),¹³ and UpToDate (UTD) point-of-care resources.¹⁴ Since many clinicians refer to electronic point-of-care resources, we included UpToDate, which is the most frequently used point-of-care clinical decision tool, and treated it functionally as an additional CPG. The reference recommendation derived from the CPG of the WHO¹ is presented in online supplementary table S1.

Considering all eight CPGs, we found consistency with the WHO CPG for only one of the 13 reference recommendations (7.7%). This consistency was for recommendation #6 (prescribing thromboprophylaxis where there are no contraindications), although this recommendation was labelled out of scope for five comparator CPGs as it was not covered in each guideline. In each reference recommendation, there was at least one CPG with ‘insufficient’ coding, except reference recommendations #6 (prescribing thromboprophylaxis) and #13 (not prescribing systemic corticosteroids). Recommendation #3 (empiric antibiotics) received only two ‘for’ and one ‘against’ codings and for two CPGs it was out of scope (table 1).

Table 1.

Summary of recommendations of the WHO and the different societies

	Reference recommendation	Population	WHO1	ANZ7	BEL8	CAN9	NHC10	NIH11	SSC12	ICM13	UTD14
#1	Initial oxygen therapy	Patients with SARI and respiratory distress, hypoxaemia or shock	Target SpO2 ≥94%	N/A	N/A	Target SpO2 ≥94%	N/A	N/A	N/A	N/A	N/A
#2	Maintenance oxygen therapy	Patients stabilised after initial oxygen therapy	Target SpO2 >90%	N/A	N/A	Target SpO2 90–96%	N/A	Target SpO2 92–96%	Target SpO2 92–96%	Target SpO2 92–96%	Target SpO2 90–96%
#3	Empiric antimicrobials	Patients with SARI and sepsis	Empiric antimicrobials within 1 hour of the initial assessment	Empiric antimicrobials within 1 hour of the initial assessment	N/A	Empiric antimicrobials within 1 hour of the initial assessment	Blind or inappropriate use of antimicrobials should be avoided	N/A	Empiric antimicrobials in mechanically ventilated patients with respiratory failure	N/A	N/A
#4	Fluid management	Patients with SARI	Conservative strategy	Conservative strategy	N/A	Conservative strategy	N/A	N/A	Conservative strategy	N/A	Conservative strategy
#5	Neuromuscular blockade	Patients with moderate-severe ARDS (PaO2/FiO2<150)	No routine neuromuscular blockade use by continuous infusion	N/A	N/A	No routine neuromuscular blockade use by continuous infusion	N/A	Continuous NMBA infusion to facilitate protective lung ventilation	No routine neuromuscular blockade use by continuous infusion	N/A	No routine neuromuscular blockade use by continuous infusion
#6	Thromboprophylaxis	Patients with SARI	Pharmacological prophylaxis if without contraindications	No pharmacological prophylaxis	N/A	Pharmacological prophylaxis if without contraindications	Pharmacological prophylaxis if without contraindications	N/A	N/A	N/A	Pharmacological prophylaxis if without contraindications
#7	Stress ulcer prevention	Patients with SARI	Administer histamine-2 receptor blockers or proton pump inhibitors in patients with risk factors for GI bleeding	N/A	N/A	Administer histamine-2 receptor blockers or proton pump inhibitors in patients with risk factors for GI bleeding	N/A	N/A	N/A	N/A	N/A
#8	Resuscitation for septic shock (1)	Patients with septic shock	Administration of crystalloid fluid	N/A	N/A	Administration of crystalloid fluid	N/A	Administration of crystalloid fluid	Administration of crystalloid fluid	N/A	N/A
#9	Resuscitation for septic shock (2)	Patients with septic shock	Use of hypotonic crystalloids, starches or gelatins is not allowed	N/A	N/A	Use of hypotonic crystalloids, starches or gelatins is not allowed	N/A	Use of hypotonic crystalloids, starches or gelatins is not allowed	Use of hypotonic crystalloids, starches or gelatins is not allowed	N/A	N/A
#10	Vasopressor for septic shock (1)	Patients in whom septic shock persists during or after fluid resuscitation	Norepinephrine is considered the first-line treatment	N/A	N/A	Norepinephrine is considered the first-line treatment	N/A	Norepinephrine is considered the first-line treatment	Norepinephrine is considered the first-line treatment	N/A	N/A
#11	Vasopressor for septic shock (2)	Patients who received vasopressor for septic shock	Target MAP ≥65 mmHg	N/A	N/A	Target MAP ≥60 mmHg	N/A	N/A	Target MAP 60–65 mmHg	N/A	N/A
#12	Inotrope for septic shock	Patients in whom poor perfusion and cardiac dysfunction persist despite achieving MAP target	An inotrope is considered	N/A	N/A	An inotrope is considered	N/A	An inotrope is considered	An inotrope is considered	N/A	N/A
#13	Systemic corticosteroids	Patients with SARI	No routine systemic corticosteroids use	No routine systemic corticosteroids use	No routine systemic corticosteroids use	No routine systemic corticosteroids use	Systemic corticosteroids in patients with progressive deterioration of oxygenation indicators, rapid progress in imaging and excessive activation of the body's inflammatory response	Low-dose corticosteroid therapy in patients with refractory shock	Systemic corticosteroids In mechanically ventilated patients with ARDS	No routine systemic corticosteroids use	Low-dose corticosteroid therapy for ICU patients who require oxygen supplementation and selected patients with refractory shock

ANZ, Australian and New Zealand Intensive Care Society; ARDS, acute respiratory distress syndrome; BEL, Belgium; CAN, Canada; FiO₂, percentage of inspired oxygen; GI, gastrointestinal; ICM, a collaboration of Faculty of Intensive Care Medicine, Intensive Care Society, Association of Anaesthetists and Royal College of Anaesthetists of the UK; ICU, intensive care unit; MAP, mean arterial pressure; N/A, no concrete recommendation available; NHC, National Health Commission of China; NIH, US National Institutes of Health; NMBA, neuromuscular blocking agent; PaO₂, partial pressure of arterial oxygen; SARI, severe acute respiratory infection; SpO₂, oxygen saturation; SSC, Surviving Sepsis Campaign; UTD, UpToDate (UTD) point-of-care resources; WHO, World Health Organization.

The recommendation with the highest rate of ‘insufficient’ codings was observed in reference recommendation #1 (prescribing initial oxygen therapy) where the target oxygen saturations were not specified in most guidelines (7/8; 87.5%) (online supplementary table S2). When we removed ‘insufficient’ ratings from consideration as part of the sensitivity analyses, rates of consistency in direction with the WHO (8/13; 61.5%) became higher. Also, there were two reference recommendations—#2 (prescribing maintenance oxygen therapy) and #11 (prescribing vasopressor to a target mean arterial pressure (MAP) ≥60 mmHg)—with the available CPGs that had exclusively ‘against’ codings when ‘insufficient’ codings were removed from consideration. The results of a leave-one-out sensitivity analysis suggested that the UpToDate recommendation source could explain the inconsistency. Consistency in direction rates changed by an absolute 23.1% (from 1/13 (7.7%) to 4/13 (30.8%)) if UpToDate was removed. If insufficient ratings are excluded in leave-one-out sensitivity analyses, the consistency in direction rates ranged from no change to change by an absolute of 15.3%.

Discussion

This cross-sectional study observed substantial inconsistency across CPGs for the management of critically ill COVID-19 patients, which could be explained by insufficient ratings, especially from UpToDate point-of-care resources. A high rate of insufficient ratings across each reference recommendation may be due to partial recommendations resulting from a lack of clinical studies addressing the specific area of recommendation in critically ill patients with COVID-19. For example, the WHO¹ recommended a target oxygen saturation (SpO₂) >94% on oxygen therapy during initial resuscitation for patients with severe acute respiratory infection (SARI) and respiratory distress, hypoxaemia or shock targets (reference recommendation #1), and though most CPGs (ANZ,⁷ BEL,⁸ NHC,¹⁰ NIH,¹¹ SSC,¹² ICM,¹³ UTD¹⁴) recommended the administration of oxygen therapy, the target SpO2 was not mentioned. Consequently, the inconsistency with the WHO CPG had resulted from only partial recommendations made in comparator CPGs on the use of oxygen therapy, and reflects the absence of evidence for optimal oxygenation requirements during resuscitation in critically ill COVID-19 patients. Indeed, the same can be applied to the high rate of out of scope ratings across each reference recommendation, where a lack of clinical studies focusing on COVID-19 patients addressing the specific area of recommendation may deter CPGs from giving their recommendations.

When independent societies/organisations have reviewed the same evidence with various factors including values and preferences being considered, and reach similar conclusions regarding a recommendation, the credibility of the recommendation may be higher. An analogy would be when findings from a research study are replicated by others, the credibility of the results is increased. However, when societies/organisations reach varying conclusions about a recommendation, the inconsistency can create confusion for clinical practice. Online supplementary table S3 presents some clinical implications for inconsistencies among comparator CPGs with the WHO CPG. The fact that the UpToDate CPG with one author was the biggest contributor to inconsistency scorings when compared with the WHO suggests a limitation with this resource in comparison with guidelines produced by consensus of national and international organisations. COVID-19 is exceptional in that, because there is such a significant knowledge gap, this emphasises the need for a collaborative approach to guide the production of CPGs based on evidence and expert consensus, and consequently requires a diverse number of specialist viewpoints to make reasonable recommendations for practice.

The most obvious example of inconsistencies would be the administration of corticosteroid therapy in severely ill patients with COVID-19. While the WHO¹ discouraged routine administration of corticosteroid therapy outside of the clinical trial, some CPGs recommended its administration in COVID-19 patients with refractory shock (SSC),¹² mechanically ventilated COVID-19 patients with acute respiratory distress syndrome (ARDS) (SSC),¹² and COVID-19 patients with progressive deterioration of oxygenation indicators, rapid deterioration in imaging, and excessive activation of the body's inflammatory response (NHC).¹⁰ In fact, UpToDate¹⁴ has changed from being against the routine administration of corticosteroid therapy to recommending low-dose dexamethasone for intensive care unit (ICU) patients with COVID-19 who require oxygen supplementation and for selected patients with shock who is refractory to fluid resuscitation, upon the announcement of the results of the RECOVERY trial.¹⁵ The RECOVERY trial¹⁵ randomised patients either to dexamethasone (6 mg once per day enterally or parenterally) along with usual care (n=2104) or usual care alone (n=4321). Compared with patients randomised to usual care alone, dexamethasone reduced the overall 28 day mortality rate by 17% (relative risk (RR) 0.83, 95% CI 0.74 to 0.92; p=0.0007). Subgroup analysis revealed that dexamethasone reduced deaths by about one-third in ventilated patients (RR 0.65, 95% CI 0.48 to 0.88; p=0.0003) and by one fifth in patients receiving oxygen only (RR 0.80, 95% CI 0.67 to 0.96; p=0.0021), but there was no mortality benefit among patients who did not require respiratory support (RR 1.22, 95% CI 0.86 to 1.75; p=0.14). Therefore, CPGs which recommend against routine use of systemic corticosteroids, including that from the WHO, should be updated accordingly based on the results of RECOVERY trial, since this is the first and only randomised trial of treatment intended for COVID-19 patients at the time of writing which has demonstrated mortality benefits. This reflects an advantage of the UpToDate point-of-care resources¹⁴ in that it allows rapid updates to be added to their recommendation without protracted peer review and consultation.

Another area worth attention due to disagreement with the WHO’s recommendation¹ was target SpO₂ during maintenance oxygen therapy, where some CPGs (CAN,⁹ NIH,¹¹ SSC,¹² ICM,¹³ UTD¹⁴) recommended not to exceed an SpO₂ of 96%, while the WHO¹ did not recommend the maximum target SpO₂. Those CPGs^{9 11–14} which recommended an upper limit of target SpO₂ of 96% cited the systematic review and meta-analysis¹⁶ which included 25 randomised controlled trials with over 16 000 patients; it reported that a liberal oxygen strategy is associated with increased risk of hospital mortality (RR 1.21, 95% CI 1.03 to 1.43) in acutely ill patients, with meta-regression also demonstrating a linear association between risk of death and higher SpO₂ targets. The upper limit of target SpO₂ of 96% was recommended since the baseline median SpO₂ was 96% in the liberal oxygen group across all trials included in the aforementioned systematic review and meta-analysis. Furthermore, the increased cost could be foreseen with liberal oxygen use in every patient requiring oxygen therapy during hospitalisation.

There was also disagreement in some CPGs with the WHO’s recommendation in the target MAP for patients who received a vasopressor for septic shock. Specifically, CAN and SSC allowed a lower target MAP up to 60 mmHg, instead of ≥65 mmHg recommended by the WHO. Two references have been cited by SSC to support their recommendation in which a 2017 individual patient-data meta-analysis¹⁷ of two randomised controlled trials, which randomised 894 adult patients with shock to either higher or lower MAP targets for vasopressor therapy, reported no significant difference in 28 day mortality, 90 day mortality, myocardial injury or limb ischaemia; however, the odds ratio (OR) for arrhythmias was increased among patients randomised to the higher MAP target (OR 2.50, 95% CI 1.35 to 4.77). Another more recent trial,¹⁸ not included in the above-mentioned meta-analysis, reported an absolute risk difference in mortality of 3% (RR 0.93, 95% CI 0.85 to 1.03) in favour of a MAP target of 60–65 mmHg compared with the higher MAP target.

In terms of neuromuscular blockade among patients with moderate-to-severe ARDS, there was only one CPG (NIH) which had opposing recommendations to those of the WHO. The NIH CPG allowed routine continuous infusion of neuromuscular blocking agents (NMBAs) to facilitate protective lung ventilation. This could be due to the mixed results reported thus far pertaining to this issue. A 2013 systematic review and meta-analysis,¹⁹ which included three randomised controlled trials with 431 patients, demonstrated a significant reduction in 90 day mortality with continuous NMBA infusion as compared with no NMBA infusion (RR 0.72, 95% CI 0.58 to 0.91). However, the 2019 ROSE trial¹⁹ reported no significant difference in 90 day mortality rate with and without routine continuous NMBA infusion (between-group difference, −0.3 percentage points; 95% CI −6.4 to 5.9; p=0.93). In addition, patients with routine continuous NMBA infusion were less physically active and had more adverse cardiovascular events than patients without routine continuous NMBA infusion. This is an area of controversy which may require consensus among professional societies to avoid confusion for clinicians, pending more randomised trials addressing this issue.

Conclusion

With observed inconsistencies in some recommendations of CPGs compared with those of the WHO, it is of the utmost importance that these inconsistencies are addressed by consensus among the relevant bodies, since there may be clinical implications for the care of critically ill COVID-19 patients. A consensus is important to avoid confusion in clinical practice where clinicians managing COVID-19 patients might hesitate in their clinical decision-making, potentially affecting the quality of care. Nevertheless, solving through consensus is only desired while awaiting clinical trials specific for COVID-19 patients. Results of clinical trials specific for COVID-19 patients should be adopted by CPGs to guide clinical practice if there is no ambiguity.