Guidelines and evidence-based recommendations in anaesthesia: where do we stand?

Abstract

Clinical practice guidelines are increasingly important to guide clinical care. However, they can vary widely in quality, and many recommendations are based on low-level evidence. The COVID-19 pandemic highlighted the need for new flexible formats for rigorously developed guidelines. Future guideline development should be standardised, graded, registered, and updated to ensure that they are ‘living’ works in progress.

The number of clinical studies published has accelerated to the point where the amount of new medical information doubled every 72 days in 2020.¹ As a result, anaesthetists are increasingly reliant on summarised evidence in guidance documents, with the most important being clinical practice guidelines. Clinical practice guidelines are designed to improve and standardise patient care based on systematic reviews of evidence and expert consensus. Adherence to guidelines is increasing tied to reimbursement for clinical services and determination of quality of care. Unsurprisingly, practice guidelines are often widely distributed, downloaded, read, discussed, and cited. However, guidelines are imperfect and may lack integrity and quality, and contain low levels of evidence, recommendations discordant with evidence, and redundancy. Recent experience during the COVID-19 pandemic has also highlighted the need for guidelines to be adaptable yet rigorous. To improve this aspect, clinical practice guidelines should be graded and registered, and efforts should be made to combine resources when feasible. Clinicians should understand that guidelines are only as good as the evidence behind them and the people who make them.

Surprisingly, standardised criteria for clinical practice guideline development do not currently exist, and methodology and approaches for development can vary widely. Issues regarding their variable quality have been reported for years and include insufficient levels of evidence for formal recommendations, lack of inclusion of key stakeholders, lack of independence leading to bias, and poor applicability.² Although this should have improved with the call to action by the WHO, the Institute of Medicine, and Guidelines International Network, recent studies have shown that clinical practice guidelines in anaesthesia are frequently limited by conflicts of interest, are based on low levels of evidence, and often the level of evidence and strength of corresponding recommendations are discordant. Recent discussions have highlighted other forms of guidelines as necessary to address the speed and volume of new information resulting from global health emergencies, such as the COVID-19 pandemic, that require rapid dissemination of urgent guidance, often despite lack of robust evidence. Table 1 describes current advantages and disadvantages of various guideline formats. Although the focus of this editorial is on academic-led guidelines, the issues raised may also be highly relevant to government and regulator-led guidelines, which are often less evidenced based and lack rigorous development processes. In the UK, the National Institute for Health and Care Excellence (NICE), a government organisation that publishes online recommendations for clinical practice, has built considerable influence. Its guidance is not based on standard guideline development and is published online only. In addition, there is limited opportunity to challenge or report concerns on controversial statements. Outside of peer-reviewed/academic guidelines, it is important to note that many governments or society-published online guidelines exist with these additional limitations.

Table 1.

Types of guidelines vs other guidance documents: definitions, advantages, and disadvantages. BJA, British Journal of Anaesthesia; EJA, European Journal of Anaesthesiology; NICE, National Institute for Health and Care Excellence.

Guideline type	Definition	Advantages	Disadvantages	Examples
Clinical practice guideline	Statements that include recommendations that are informed by systematic review of evidence and an assessment of benefits and harms of alternative care options	Comprehensive, Rigorous, Based on higher-level evidence	Rigid framework, Lengthy development process, May become outdated, No single standardised format for development	Ahmad and colleagues. Anaesthesia 2020: Difficult Airway Society guidelines for awake tracheal intubation (ATI) in adults26
Focused clinical practice guideline	Statements that adhere to the same methodological structure as clinical practice guidelines but cover a very specific topic and follow an expedited systematic review process	Efficiently produced, High readability, Frequently updated	May be considered less rigorous , Smaller area of concentration may lead to duplication, Lacks broad perspective on topic; end user may need to find multiple sources	Ahmed and colleagues. European guidelines on perioperative venous thromboembolism prophylaxis: patients with preexisting coagulation disorders and after severe perioperative bleeding.27 Afshari and colleagues. EJA 2018: European guidelines on perioperative venous thromboembolism prophylaxis. EJA 2018.28
Living clinical practice guideline	Statements that include recommendations based on continuous literature surveillance, rapid and constant updating of prioritised systematic reviews, and virtual consultations with expert panels	Flexible to new information, Balances speed and rigour, Easily disseminated, Rapid and constant updates	Potential confusion of end user as a result of to multiple updates, Limited use in fields where information moves at a slower pace	Agarwal and colleagues. BMJ 2020: A living WHO guideline on drugs for COVID-19.24
Rapid statement	Statements that include recommendations that are informed by expert opinion, which aim to provide a unified response to a health crisis or emergency	Adaptable, Practical, Easily disseminated	Based on low levels of evidence, Expedited process, Potential duplication of efforts: multiple author groups working on similar statements	Wei and colleagues. BJA 2021: Controversies in airway management of COVID-19 patients: updated information and international expert consensus recommendations.29 Updated living guidelines: BMJ30
Consensus statement	Statements based on expert consensus, where insufficient evidence exists for systematic review	Pragmatic, Analysis of best available evidence when systematic review not feasible, Provides clinicians with guidance when strong evidence not available, Expert led	Based on lower-level evidence, Methodology less rigorous than clinical practice guidelines , May become outdated as new evidence emerges	Healy and colleagues. Anesth Analg 2021: Expert consensus statement on the perioperative management of adult patients undergoing head and neck surgery and free tissue from the Society for Head and Neck Anesthesia.31
Guidance document
Standard operating procedure	A set of written instructions that describe the step-by-step process taken to properly perform a routine activity	Helps institutions to be consistent and in compliance with known standards of care	Institution specific, Not always evidenced based, Not published widely	Haslam and colleagues. Anaesthesia 2021: ‘Prep, stop, block’: refreshing ‘stop before you block’ with new national guidance’.32
Quality standards	Document that defines high-priority areas for quality improvement in a defined care or service area	Standardised and detailed development process, Clear and easily accessible, Derived from NICE guidelines, Consultation sought on topics	Not always evidenced based, No defined updating procedures, Medico-legal implications	NICE, 2021: Quality standard on venous thromboembolism in adults.33

What is wrong with guidelines?

Editorial independence

Editorial independence during guideline synthesis is a key quality metric.³ In the context of increasing conflicts of interest in medicine, many organisations, including the American College of Physicians, US Preventive Services Taskforce, and UK NICE, require anyone involved in clinical practice guideline development to disclose all financial/intellectual conflicts of interest in the preceding 3 yr according to their magnitude: high (active relationships with direct financial stakes in the guidelines), moderate (intellectual interest that may profit from the guidelines), and low (inactive conflicts of interests peripherally associated with the guidelines).⁴ In 2011, the National Academy of Medicine report Clinical Practice Guidelines We Can Trust recommended that all financial conflicts be disclosed, that only a small number of authors have conflicts, and that chairpersons should not have any significant financial and academic conflicts of interest.⁵ Given their high citation rates, participation of journal editors in guideline preparation creates another potential conflict of interest. Unfortunately, studies in the USA, Japan, and Australia have found that more than 70% of clinical practice guideline authors have financial conflicts with large discrepancies between publicly reported and self-disclosed conflicts of interest, and that chairpersons often have significant potential financial conflicts of interest.⁶ Finally, it is worth mentioning that the author list is not always reflective of relative contributions or influence, and valuable input from reviewers often goes unrecognised. These hidden contributors may also need to display transparency and disclose appropriate conflicts of interest. In addition, industry influence cannot be fully excluded based on statements of individual conflicts of interest. The influence of pharmaceuticals on guidelines produced by the American Pain Society preceding the opioid crisis is a case in point.

Conflicts of interests

Conflicts of interests are often grey areas. These statements are self-reported declarations, which remain undefined by journals and the wider scientific community. Some societies have attempted to define ‘significant’ interests in a landscape where it is increasingly difficult to be free from all conflicts. For example, clinicians or researchers especially knowledgeable on a topic may be more likely to be approached by companies for paid consultation. Usually only ‘moderate or high’ levels of conflicts are considered significant and undesirable in guideline development. However, it remains at the discretion of guideline committees and societies to appropriately screen and avoid contributory conflicts of interest.

It is impossible to ensure that authors do not intend to benefit from guideline development that would create future conflicts of interest. The guideline in question remains unaffected, although this may prohibit involvement in future guideline involvement. One way to avoid this may be anonymous guidelines published solely in the name of a society. This may have unintended consequences, however, allowing authors to hide their prior conflicts of interest and cannot be recommended. We suggest instead that for the process to be transparent, a third-party regulator or administrator should examine all potential conflicts of interest and grade the level of conflict before finalising involvement in the guideline development group.

Strength of underlying evidence

Another key quality indicator of clinical practice guidelines is the strength of the underlying evidence. Assessing the level of evidence is more complex than might be expected. Commonly used tools for grading evidence include the Grading of Recommendations Assessment, Development and Evaluation (GRADE)⁷ and the American College of Cardiology/American Heart Association (ACC/AHA) classification.⁸ These instruments categorise evidence to Levels A, B, and C, with Levels A and B based on prospective studies and Level C representing more subjective sources, such as clinical experience, case reports, and expert opinion.

Assessment of levels of evidence

Systematic reviews, meta-analyses, and RCTs are considered the highest level of evidence but can be of variable methodological quality. The number of systematic reviews and meta-analyses has increased nearly five times in the last decade compared with clinical trials, which have grown only of the 55% during the same time period.⁹ If best practices are not followed, many reviews may be biased and flawed, with questionable and often conflicting conclusions. Only 3% of meta-analyses produced represent high-quality work. In addition, meta-analyses of small trials are subject to publication bias and may overestimate treatment effects that are later disproved by large trials. If incorporated into guidelines, these reviews can affect the validity of recommendations. Inclusion of methodological experts as a gatekeeper is critical to maintaining guideline integrity.

Assessing RCT level of evidence can also be challenging. Using GRADE, studies can be upgraded or downgraded depending on the methodological rigour.⁷ Small RCTs and single-centre RCTs may overestimate treatment effects because of unintentional investigator bias, and RCTs using composite outcomes to decrease sample size may have clinically less relevant outcomes ‘drive’ the composite outcome. Superiority trials with high crossover rates or protocol deviations may bias the trial towards the null hypothesis. In contrast, large observational studies, representing a more ‘real-world’ clinical setting and providing temporal trends in disease prevalence or interventions, may have significant value and may be upgraded in quality of evidence. Therefore, it takes experts in trial methodology, digging deep into the details of individual trials, to grade levels of evidence accurately.

Concordance of levels of evidence and strength of recommendations

A recent systematic review evaluating the levels of evidence of North American and European perioperative care clinical practice guidelines in the last 10 yr found that a minority (16%) of recommendations were supported by Level A evidence, whereas 51% were supported by Level C evidence.¹⁰ More concerning, strong recommendations were often supported by a low level of evidence, a finding that did not improve during the study period. Tricoci and colleagues,⁸ in an analysis of 53 ACC/AHA clinical practice guidelines published between 1984 and 2008 and including 7196 recommendations, reported that only 11% of recommendations were supported by the highest level of evidence, whereas nearly half were based only on expert opinions or case studies. An updated analysis of 26 ACC/AHA guidelines published between 2008 and 2018 reported that only 8.5% of the recommendations were supported by Level A evidence, 50% by Level B evidence, and 42% by Level C evidence.¹¹ Similarly, of 25 European Society of Cardiology guidelines, 14% of recommendations were supported by Level A, 31% by Level B, and 55% by Level C evidence.

Another problematic issue with clinical practice guidelines is the correct translation of existing evidence into recommendations. Different tools have been introduced to evaluate this aspect, with the Appraisal of Guidelines, Research and Evaluation (AGREE) and the GRADE instruments being the most commonly utilised. The GRADE system defines how evidence should be translated into treatment recommendations and provides an explicit evaluation of the comprehensive criteria for downgrading and upgrading quality of evidence ratings.¹² The validity of GRADE is supported by its endorsement by organisations, such as the WHO, the American College of Physicians, the American Thoracic Society, and the Cochrane Collaboration.

Tools used to assess guidelines

GRADE instrument

Although the GRADE system has not been universally accepted, it has frequently been used to assess the quality of clinical practice guidelines in anaesthesia. In an analysis of 681 recommendations from 15 clinical critical care guidelines published from 2011 to 2017, Sims and colleagues¹³ found that amongst 215 Class 1 recommendations, 69 (32%) were discordantly paired with evidence other than Level A or B. Despite GRADE recommendations based on low-level evidence, 22 of 69 recommendations (32%) were based on expert consensus, and 47 (68%) of the guidelines were of low or very low quality.¹³

AGREE instrument

The AGREE instrument, updated to AGREE II in 2011, is arguably the gold standard for clinical practice guideline development, reporting, and evaluation.¹⁴ It has six domains, with Domain 3, Rigour of Development, regarded as the single most influential domain and most reflective of guideline quality. Several studies have used the AGREE II instrument to evaluate clinical practice guideline quality in anaesthesia. A study using AGREE II to access the quality of 96 guidelines in anaesthesia published between 2013 and 2018 found that 74% of them had low overall quality scores, and only 26% (25 out of 96) were of high quality.¹⁵ A recent review used the AGREE II instrument to analyse the quality of clinical practice guidelines from all anaesthesia subspecialties from 2016 to 2020 and reported an increased overall quality of guidelines, mainly driven by Domain 3. The absolute score of Domain 3, however, remained low with only 13.7% of clinical practice guidelines deemed to be of high quality. Furthermore, eight subsections of Domain 3 demonstrated a variability in scores with quality assessment of the underlying evidence base having the lowest scores.¹⁶ Future validated Domain 3 centric scoring systems may improve the accuracy of quality assessment and make AGREE II an even more reliable tool for clinical practice guideline evaluation.

A need for improved methodological approaches to anaesthesia guideline development is evident. Use of AGREE II together with the GRADE tool offers a potential solution. Mandating use of these tools together in guideline formation could lead to more-transparent high-quality documents. A recent review described high-quality guidelines as largely collaborative, with 78% having international involvement and 100% multi-institutional involvement.¹⁶ International and institutional cooperation should be further emphasised in anaesthesia guideline development and also reduction in duplication of efforts. The ADAPTE, A Guideline Adaptation and Implementation Planning Resource (CAN-IMPLEMENT), and Grading of Recommendations Assessment, Development and Evaluation – Adaptation, Adoption, De Novo Development (GRADE-ADOLOPMENT) documents may play a role, having been designed for the purpose of efficiently developing and adapting clinical recommendations.¹⁷ In addition, a standardised registry of clinical practice guidelines, similar to the International Prospective Register of Systematic Reviews (PROSPERO), should be implemented. Input from organisations, such as the WHO, Guidelines International Network, and Cochrane Collaboration, would be instrumental in supporting this process.

Role of journals

Clinical practice guidelines undergo a standard process of peer review before publication. However, journals do not comment on the quality of the guidelines, and publication in a specific journal does not always imply quality, particularly if the editorial team is involved in their development. Publication of an objective marker of guideline performance by the journal, such as AGREE II, along with the guideline itself should be considered to provide readers with a recognisable ‘quality mark’ reflecting the underlying integrity of the guideline. This does not imply that guidelines lacking this quality designation are inherently flawed, but that their recommendations should be viewed with some caution and with the understanding that there are grey areas in what is considered ‘best practice’. Similarly confusing for the reader are the various names and types of guidance documents (Table 1). Journals should define the type of guidance document that is being presented because titles can be misleading.

Alternatives to guidelines

Focused clinical practice guidelines

The current process of development and publication of clinical practice guidelines can take up to 2 yr. For COVID-19, the pandemic is just 2 yr old and hundreds of recommendations have been made. According to AGREE II, it is recommended that clinical practice guidelines are updated every 5 yr. For the COVID-19 pandemic, however, most guidelines have been based on low-level evidence and modified accordingly as higher-level evidence comes in, often reversing initial recommendations. In this context, updating clinical practice guidelines every 5 yr is inadequate. Conversely, for some recommendations, the evidence may not change significantly over time and updating them unnecessarily may waste resources.¹⁸ In anaesthesia, the European Journal of Anaesthesiology (EJA) recently introduced a new format to streamline guideline updates, ‘the focused clinical practice guidelines’.¹⁹

Focused clinical practice guidelines adhere to the same methodological structure as traditional guidelines, but they cover only a very specific topic and follow an expedited systematic review process. Therefore, they require less time from inception to publication, have a more efficient updating process, and have improved readability.²⁰ Whilst focused guidelines were initially considered of inferior quality, studies have found that there were no substantial differences in methodology, research strategy, and quality between traditional and focused clinical practice guidelines.²¹ However, it should be noted that these may be much less comprehensive than traditional guidelines, and the cost of efficiency may be duplication and confusion: many focused guidelines covering similar material may complicate the message to the clinician. In addition, there may be value to approaching guidelines from the broad ‘forest’ viewpoint than from the ‘trees’ perspective. More and more specific guidelines on smaller topics published in subspecialty journals are contributing to this trend, and it remains unclear what the optimal balance between efficient and comprehensive is and the best size and scope of a clinical practice guideline.

Rapid statements

It should be noted that consensus statements exist widely to provide guidance in situations where there is clinical equipoise yet there is not enough evidence on a topic for systematic review and subsequent guideline development. The professional community should regard these as suggestions for practice rather than as binding recommendations. The EJA provided its take on consensus documents as ‘rapid statements’, in response to the COVID-19 health crisis emergency. These statements are less rigorous than clinical practice guidelines, but they are adaptable to the urgent needs arising from health crises, such as COVID-19, by aiming to disseminate the best available evidence as quickly as possible.¹⁹ Rapid statements have been published prolifically during the COVID-19 pandemic, but many are of inferior quality and duplicative.²² Strict standards should still be used in these statements. There is a growing body of literature detailing the application of GRADE and guidance for conducting rapid systematic reviews in the context of a pandemic.²³

Using high standards in the development of clinical practice guidelines may in fact be even more crucial in times of crisis, and rigour and speed should both be emphasised.

Living guidelines

The WHO recently introduced the concept of ‘living clinical practice guidelines’.¹⁸ This approach combines continuous literature surveillance, rapid and constant updating of prioritised systematic reviews, and virtual consultations with expert panels to ensure that the latest evidence and updated recommendations can reach health workers worldwide as quickly as possible. For COVID-19, two clinical practice guidelines, one for treatment and one for prevention, have been developed adopting this strategy.^{24 , 25} This concept has yet to be used in anaesthesia clinical practice guidelines, and a pertinent question remains regarding how and when updates should be made. Details on how living clinical practice guidelines should be incorporated in anaesthesia and what clinical questions would best suit this format depend on the rate of new information and interest in specific topics. The rate and degree of change should be set by (i) the rate of new evidence available and whether this evidence should change practice, and (ii) balancing the risks and benefits of frequent changes confusing readers. For medico-legal purposes and the evolving nature of practice recommendations, clear date stamps will be necessary to understand when a practice was recommended during previous versions of the guideline.

Conclusions

Evidence, recommendations, and clinical practice guidelines exist in an imperfect world. However, there are clear steps to improve guidelines (Table 2 ). A more formal process must be performed to translate these suggestions into working recommendations; however, the hope is that these concepts will stimulate further debate. The integrity of guidelines must be taken seriously, and their development should be free from conflicts of interest. GRADE and AGREE II should be used to determine the quality of guidelines, assess evidence, and translate them accurately into recommendations. The ideal timing for updating guidelines is unknown. High-quality RCTs take years to complete, and clinical practice guidelines, rigorously performed, can also take several years to be finalised. Although it is surprising that so many recommendations are based on low-level evidence, this does not necessarily mean they are untrustworthy. Many clinical questions can be initially addressed with well-performed observational studies. Guidance for clinical practice is necessary despite the limitations of available evidence, but guidelines should be flexible to accommodate new information. When RCTs become available, their findings must be taken into account, perhaps even outside the timeline of traditional clinical practice guidelines. The COVID-19 pandemic has demonstrated the need for living guidelines that balance speed and rigour in the face of mass dissemination of information. These guidelines should still adhere to strict methodological standards that minimise duplication. Finally, the process of grading clinical practice guidelines should be standardised, and clinicians should be aware that all guidelines are ‘living’ works in progress that have limitations and may be updated as additional evidence accrues.

Table 2.

Recommendations to improve clinical practice guideline development in anaesthesia. AGREE, Appraisal of Guidelines, Research and Evaluation; GRADE, Grading of Recommendations Assessment, Development and Evaluation.

Suggestions to improve clinical practice guideline development in anaesthesia
Use of both AGREE II and GRADE during anaesthesia guideline development and assessment
Standardisation of AGREE II scoring with higher weighting assigned to Domain 3
International, multi-institutional cooperation in anaesthesia guideline formulation
Mandated registration of clinical practice guidelines
Publication of objective ‘quality mark’ for readers by journals
Recognition of reviewers and their conflicts of interest, and disclosure of society funding of guidelines
Third party to examine and screen significant conflicts of interest before guideline development
Use of focused clinical practice guidelines, rapid statements, and living guidelines appropriately for rapidly developing crises, such as COVID-19, and in areas of anaesthesia that require frequent updating

Authors' contributions

Conception/design: LQR.

Drafting/revising of paper: all authors.

Final approval: all authors.

Accountability for all aspects of the work: all authors.

Declarations of interest

LQR is a member of the associate editorial board of the British Journal of Anaesthesia.

Funding

National Institutes of Health; National Heart, Lung, and Blood Institute (K23 HL153836-01A1) to LQR.