A pragmatic approach to selecting a grading system for clinical practice recommendations in palliative care

Abstract

Background:

The limited palliative care evidence base is poorly amenable to existing grading schemes utilized in guidelines. Many recommendations are based on expert consensus or clinical practice standards, which are often considered ‘low-quality’ evidence. Reinforcing provider hesitancy in translating recommendations to practice has implications for patient care.

Aim:

To rationalize the selection of an appropriate grading system for rating evidence to support recommendations made in palliative care clinical practice guidelines.

Design:

Review of the methodology sections of international palliative care guidelines published in English identified five grading systems comparison: Grading of Recommendations, Assessment, Development and Evaluations (GRADE); the Scottish Intercollegiate Guidelines Network (SIGN); Infectious Diseases Society of America-European Society for Medical Oncology (IDSA-ESMO); Confidence in the Evidence from Reviews of Qualitative research (CERQual) and the National Service Framework for Long Term Conditions (NSF-LTC).

Results:

There is heterogeneity among grading systems used in published palliative care or terminal symptom management guidelines. GRADE has been increasingly adopted for its methodological rigour and inter-guideline consistency with other medical associations. CERQual has the potential to support recommendations informed by qualitative evidence, but its role in clinical guidelines is less defined. The IDSA-ESMO system has an intuitive typology with the ability to categorize tiers of lower-quality evidence.

Conclusions:

It is challenging to apply commonly used grading systems to the palliative care evidence base, which often lacks robust randomized controlled trials (RCTs). Adoption of IDSA-ESMO offers a feasible and practical alternative for lower-resourced guideline developers and palliative clinicians without a prerequisite for methodological expertise.

Key Statements

What is already known about the topic?

• Certain areas within the scope of palliative care lend themselves poorly to randomized controlled trials due to issues related to recruitment, attrition, bias and ethics.

• The available evidence informing recommendations on end-of-life issues often ranks low on the evidence hierarchy.

• The multiple different grading systems currently in use each have their own advantages and disadvantages, particularly in the context of the available evidence base.

What this paper adds?

• The grading system developed by the Infectious Diseases Society of America and adapted by the European Society of Medical Oncology is appropriate for supporting recommendations made within palliative care guidelines because its typology accounts for non-experimental methods common in the field.

• We provide a deliberation process for selecting a grading system in palliative care, including strengths and limitations of different systems.

Implications for practice, theory or policy

• Guideline development within palliative care would benefit from detailed documentation of the evidence appraisal process for consistency and transparency.

• To date, there is no single grading system that effectively integrates critical appraisal of qualitative and quantitative evidence sources alike. Systematically approaching recommendations rooted in psychosocial or spiritual evidence may help enhance clinician confidence in situations where the supporting qualitative data is limited.

• Guideline developers should strive for a balance between rigour and accessibility when grading their recommendations.

Introduction

Clinical practice guidelines provide recommendations with the goal of delivering consistent and informed patient care. They typically guide clinical decision-making in specific scenarios based on available evidence that has been systematically appraised ¹ and compared to alternative options.^1,2 Trustworthy guidelines assure methodological transparency, integrate multi-disciplinary stakeholder input and reconcile conflicting recommendations. ² The Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach ³ is widely used to ensure rigour and uniformity as part of an international movement towards enhanced quality standards for guideline developers. ²

GRADE is not optimal for the assessment of the evidence base in palliative care, which is often qualitative in nature or based on expert opinion and practice standards, rather than randomized controlled trials (RCTs) and other ‘high-quality’ experimental methods. While these are not scored favourably by GRADE,^{3 –5} they are pragmatic in providing clinical judgement and navigation in the absence of other evidence.^6,7 We acknowledge the hypothetical risk that such ‘low-quality’ evidence may seem synonymous with interventions that do not add value or are equivocal if not harmful. However, this term more commonly signifies a lack of available research. ⁷ The ability of guidelines to offer direction in the face of inadequate evidence is cited as a strength rather than a weakness, as it would be exceedingly difficult to make clinical decisions on the basis of high-quality evidence alone. ⁶ Controlled, experimental trials with high participant numbers may not be feasible given certain patient populations or clinical questions, a situation common in palliative care.

We report here the critical evaluation approach we adopted when addressing the need for a holistic, practical evidence appraisal system for use in updating the Palliative Symptom Management Guidelines published by our group at the BC Centre for Palliative Care (BCCPC). While our guidelines are targeted to clinicians, allied health professionals and caregivers for adults with advanced life-limiting illness, ⁸ the information contained in this paper is applicable to a broader audience of guideline developers and novices to clinical evidence appraisal alike.

Design

Websites of national and international palliative care organizations were searched for clinical guidelines published in English. The academic databases MEDLINE, Embase and CINAHL were searched for additional published palliative care guidelines. Methodology sections, bibliographies and supplemental appendices of the published guidelines were reviewed to identify critical appraisal processes and further guidelines for review. The search strategy outlined in Supplementary Figure 1 was used to expand the search and identify recent publications. We modified our search criteria to include methodological papers on grading systems. Briefly, the databases listed above were searched from inception to May 2023 using a combination of keywords related to palliative care, evidence-based practice and clinical guidelines. We excluded disease-specific guidelines not aimed at a generalist palliative care audience, development guides and guideline appraisal tools such as AGREE II. Our aim was to source publications on grading system methodologies, critiques and comparisons.

We identified five grading systems in use within North America, Australia and Europe. Four of these were selected based on their role in recent palliative care guidelines. The final system, NSF-LTC, was identified during database search and after exhausting the reference lists of methodological papers. It was selected for its theoretical applicability rather than current use.

Results: Summary of grading systems

Five grading systems were identified for review and comparison: GRADE; IDSA-ESMO; Confidence in the Evidence from Reviews of Qualitative research (CERQual); the Scottish Intercollegiate Guidelines Network (SIGN) and the National Service Framework for Long Term Conditions (NSF-LTC). The grading systems used by the identified organizations, and their geographic regions are detailed in Table 1. These were the only distinct grading systems identified during the search process. Other guidelines either did not explicitly name a grading system or relied on expert consensus. These grading systems were compared based on rigour, practicality, clarity of typology and breadth of use. This criteria addresses the systematization of evidence appraisal, feasibility of implementation and congruency with the current clinical landscape.

Table 1.

Systems for scoring or categorizing (grading) recommendations made in clinical guidelines ^a .

Organization	System(s) used	Region
Scottish Intercollegiate Network	SIGN (former), GRADE (current)	UK
Registered Nurses Association of Ontario	GRADE (modified), CERQual	Canada
European Society for Clinical Oncology	IDSA-ESMO	Europe
Alberta Health Services, Cancer Care Alberta	IDSA-ESMO	Canada
BC Centre for Palliative Care	GRADE (modified)	Canada
Ontario Palliative Care Network	Expert Consensus	Canada
The National Institute for Health and Care Excellence	GRADE	UK
National Consensus Project Clinical Practice Guidelines for Quality Palliative Care	GRADE	International
National Health and Medical Council of Australia	GRADE	Australia
National Comprehensive Cancer Network	Expert Consensus	US
European Association for Palliative Care	GRADE, Expert Consensus	Europe
American Society of Clinical Oncology	Expert Consensus	US

^aAs used in relevant palliative or end-of-life care guidelines only.

GRADE

Numerous organizations have adopted GRADE, including Cochrane, ⁹ the World Health Organization, ¹⁰ the Canadian Task Force on Preventative Health Care, ¹¹ the National Institute for Health and Care Excellence (UK), ¹² the National Consensus Project for Quality Palliative Care (US) ¹³ and the Registered Nurses Association of Ontario (Canada). ¹⁴

The separation of judgements on the strength of recommendation from the quality of evidence is a defining feature of GRADE.^3,4 This allows for nuance when high-quality studies are associated with burdensome therapy or low-quality studies with improved patient satisfaction. ⁴ The quality of evidence rating depends on the level of certainty (high, moderate, low or very low) in how close the true effect is to the effect estimate (Table 2).^4,5,15 To assess the quality of evidence supporting each recommendation, five domains are evaluated: study design, risk of bias, consistency, directness, precision and publication bias. Within guidelines, the gestalt of these studies serves as the basis for supporting practice recommendations, interwoven with clinical context.^{4,5,15 –17} In systematic reviews, GRADE provides structure for conveying judgements about how closely the collective body of evidence for a specific outcome approximates reality, and how influential future research is likely to be.^4,5,9,15 The strength of recommendation is dichotomous (Table 2) and is influenced by trade-offs between desirable and undesirable effects, patient values and preferences, evidence quality and equitable resource allocation.^4,18,19

Table 2.

Overview of GRADE Quality and Strength of Recommendation ratings.

Quality of evidence ratings	High	Future research is unlikely to change the effect estimate; benefits clearly outweigh the risks (or vice versa).
	Moderate Moderate	Future research may change the effect estimate; sufficient evidence for decision-making.
	Low Low	Future research is likely to change the effect estimate; some evidence for decision-making.
	Very Low Very Low	The effect estimate is uncertain; insufficient evidence for decision-making.
Strength of recommendation ratings	Strong	Most people in this situation would want the recommendation, and only a minority would not.
	Weak or Conditional	Most people in this situation would want the suggestion, but many would not.

RCTs are initially assigned a high level of certainty while non-randomized and observational studies are assigned a low level of certainty due to inherent bias,^5,9 which can be systematically evaluated using tools developed by the Cochrane Collaboration such as the Risk Of Bias In Non-randomized Studies – of Interventions (ROBINS-I).^9,20 ROBINS-I provides a way of comparing the non-randomized study against a ‘target trial’, which is a hypothetical RCT evaluating the same interventions in the same population, where feasibility and ethics can be ignored.^9,20

Observational studies may be upgraded if they have a large (i.e. two-fold) or very large (i.e. five-fold) effect size, a dose-response gradient, or residual confounding that inadvertently increases confidence in the estimated effect, whereby the actual effect size would be even greater had the overlooked confounder(s) been accounted for.^4,9,17,21 Critical consideration must be given towards the relative contributions of larger scale, higher quality studies. ⁴

Clinical applications

GRADE cannot be applied to qualitative studies or expert opinion, precluding its scope of use for much of the palliative care evidence base. Retrospective and case control studies represent most of the body of quantitative evidence on symptom management in end-of-life, which are rated as low-quality evidence under the GRADE system. In addition, much evidence from RCTs would be downgraded based on design limitations, such as low sample sizes,^{9,22 –24} or indirect evidence such as that extracted from systematic reviews on opioids for adjacent populations, such as patients with chronic or cancer pain.^{25 –29}

CERQual

The GRADE-CERQual approach (CERQual) was developed in collaboration with the GRADE Working Group to support the use of qualitative syntheses in guideline development.^30,31 The quality of evidence is appraised in a manner analogous to GRADE. Levels of confidence (high, moderate, low or very low) are assigned across four components (methodological limitations, coherence, adequacy of data and relevance) based on how well the review finding reasonably represents the phenomenon of interest.^{30 –32} Each review finding starts as ‘high confidence’ and is downgraded by one or more levels based on severity of concerns, for which there are no definitive criteria and much is subject to reviewer discretion.^30,33

Clinical applications

Although its application is less widespread in the literature compared to GRADE, CERQual offers a framework for clinical areas previously not amenable to traditional grading schemes by standardizing recommendations based on qualitative findings. ³¹

For example, the mixed-methods systematic review by Edwards et al. ³⁴ was able to state with a high degree of confidence that ‘programmes and services for people at the end-of-life require a comprehensive team approach incorporating symptom relief, psychological and psycho-social support and spiritual care’. This statement arose from the application of CERQual to qualitative studies on palliative care in people with severe mental illness. ³⁴ Being able to demonstrate thematic and contextual congruity conveys a certain degree of rigour. This adds weight to practice recommendations informed by qualitative evidence.

In a separate systematic review on ethical dilemmas faced by palliative care specialists, Schofield et al. ³⁵ held a high degree of confidence in the evidence that: ‘Participants reported ethical challenges related to the ethical principle of truth-telling. This focuses around whether it was appropriate to discuss both the terminal nature of a diagnosis and/or prognosis to both the patient and to family members’. The authors’ rationale was that individual studies had a high level of coherence with respect to this theme. ³⁵ CERQual provides the opportunity to address these ethical challenges within palliative care from an evidence-driven perspective, not just a theoretical one.

SIGN

SIGN is an expert group that independently releases guidelines with a historic grading system of their own. However, the organization began transitioning to GRADE in 2009 and moved away from their ABCD grading of recommendations in 2013.^6,36,37 The original SIGN approach of adding ‘+’ or ‘-’ to their level of evidence scores provided a spectrum for conveying strengths or weaknesses, although a significant limitation was their focus on study type rather than study quality.^38,39 The clinical applications of SIGN are limited, considering it has fallen out of use.^36,37,40

NSF-LTC

NSF-LTC developed a unique evidence typology to address the limitations of experimental quantitative research methods in long-term care based on the premise that research quality is more important than research design.^39,41 The intention was to address the challenges of making recommendations for diseases with complex trajectories,^39,41 which can be said about the natural history of many life-limiting illnesses under the purview of palliative care. On this basis, NSF-LTC was included in this discussion despite any evidence of its adoption by a recognized palliative care organization.

According to NSF-LTC, systematic assessment of the evidence involves upholding three principles: value in the opinions of experts, patients and families; belief that qualitative, quantitative and mixed-methods studies have equal validity with no implicit hierarchy; and emphasis on study design, integrity of its conclusions and relevance to the population of interest.^41,42

Evidence is divided into two categories: ‘E’ for expert evidence (including patients, caregivers and professionals), and ‘R’ for research-based evidence. Five criteria evaluating quality are scored for each study, and the overall study descriptor combines study design, quality and applicability. Research design is categorized into quantitative (P1), qualitative (P2) and mixed methods approaches (P3). For example, a high quality primary qualitative study that is directly applicable to the population of interest would be classified as ‘P2 High Direct’.^41,42 A recommendation using an ABC grading scheme is provided after holistic appraisal of the quality and relevance of the total body of evidence. Applicable to research evidence only, this includes quantitative, qualitative and mixed methods designs. The final assessment would be denoted by a combination of the type of evidence along with a rating of research evidence. For example, Grade A research evidence would be listed as ‘RA’, whereas expert evidence would be communicated as ‘E1’ (user expert, i.e. patient or caregiver) or ‘E2’ (professional expert, i.e. clinician) without a letter grade assigned. ⁴²

Clinical applications

NSF-LTC does not award higher points to studies based on their standing within the evidence hierarchy, which is conducive to the type of research available in palliative care. For example, two North American Delphi studies reached expert consensus and developed priority recommendations in palliative medicine and addictions, such as identifying opioid misuse regardless of prognosis and continuing buprenorphine in comorbid opioid use disorder and cancer pain.^43,44 Delphi studies use expert consensus to answer a research question and are often considered the lowest tier of evidence. However, each expert brings a unique combination of clinical expertise and theoretical knowledge to the dynamic. ⁴⁵ These studies strictly adhered to the Delphi process,^43,44 and their data is highly applicable to our population of interest. NSF-LTC would grade this favourably, building clinician confidence in safe opioid prescription among this population.

IDSA-ESMO

ESMO adapted the original scheme published in 2001 by the IDSA, modifying it by subdividing the levels of evidence from three to five.^46,47 IDSA-ESMO provides a letter grade for the strength of recommendation (Table 3) and a roman numeral for the quality and type of evidence supporting that recommendation (Table 4).^46,47 Certain ESMO guidelines explicitly note when clinical trials would be unethical or unfeasible to highlight recommendations supported by clinical expertise and practice standards, by adding an asterisk (*) to the respective grade of recommendation. ⁴⁸

Table 3.

IDSA-ESMO strength of recommendations.

Grade	Description
A	Strongly recommended; strong evidence for efficacy with a substantial clinical benefit.
B	Generally recommended; strong or moderate evidence for efficacy but with a limited clinical benefit.
C	Optional; insufficient evidence for efficacy or benefit does not outweigh risks or disadvantages.
D	Generally not recommended; moderate evidence against efficacy or for adverse outcomes.
E	Never recommended; strong evidence against efficacy or for adverse outcomes.

Adapted by Cancer Care Alberta ⁴³ from the Infectious Diseases Society of America and the European Society for Medical Oncology^46,47,49.

Table 4.

IDSA-ESMO levels of evidence.

Level	Description of evidence
I	Evidence from at least one large RCT of good methodological quality with low potential bias; meta-analyses of well-conducted RCTs without heterogeneity
II	Small RCTs; phase II RCTs; large RCTs with suspected bias, or meta-analyses of such trials or those with heterogeneity
III	Prospective cohort studies; post-hoc and ad-hoc analyses of RCTs
IV	Retrospective cohort studies; case-control studies; instrument validation studies
V	Studies without a control group; case reports; expert opinions; review articles or narrative reviews; Delphi studies; cross-sectional studies (interviews, focus groups, surveys)

Adapted by Cancer Care Alberta ⁴³ from the Infectious Diseases Society of America and the European Society for Medical Oncology^46,47,49.

In 2008, IDSA made the decision to adopt GRADE with the aim of conforming to their international collaborators to facilitate guideline availability.^6,50 However, the IDSA-ESMO system remains in use by ESMO.^46,47 While the September 2022 ESMO Standard Operating Procedures for guideline developers suggests that the phrasing of recommendations follow those set out by the GRADE Working Group, they still adhere to the letter and Roman numeral typology for level of evidence and strength of recommendation, respectively. ⁴⁶ Cancer Care Alberta also relies on the IDSA-ESMO system, with their adaptation outlined in their June 2022 Guideline Methodology Handbook. ⁴⁹

Clinical applications

Most opioid-specific guidelines for cancer pain management within the 2018 ESMO guidelines ranged from level II-III/grade B-C, although certain recommendations on immediate release opioids and transmucosal formulations for breakthrough cancer pain scored high (I, A). ⁵¹ There were several level II/grade A-C ratings for the recommendations on opioids in breathlessness. ⁴⁸

ESMO’s 2021 guidelines on the care of adults with cancer at end-of-life provide psychosocial and spiritual recommendations ranging from level I-III/grade A-B, partly informed by systematic reviews of cross-sectional and longitudinal studies. ⁵² The ESMO guideline on refractory symptoms and palliative sedation adds a preamble that all assertions are level V evidence because they are based on case series and expert opinion. ⁵³

Comparison of grading systems

Based on the above review, we critically appraised the identified grading systems by comparing the strengths and limitations of each. This was undertaken in the context of our development and review of interdisciplinary palliative care symptom management guidelines as an organization that does not have ample methodological support or extensive guideline development personnel (see Table 5). A summary of this comparison is shown in Table 6.

Table 5.

Comparison of grading systems.

System	Key features	Advantages a	Limitations a
GRADE	Judgements on both strength of recommendation (2 levels) and quality of evidence (4 levels).	Comprehensive. Two-judgement scales allows for nuance around aspects such as burden of therapy or types of outcomes. Widespread uptake. Systematically upgrades and downgrades evidence based on aspects of the study design and results. Requires reviewers to explicitly outline factors influencing their rating decisions.	Cannot be applied to qualitative studies or expert opinion. Retrospective and case-control studies are rated as low-quality evidence. Therefore, much of the evidence in palliative care would be rated as low or very low. Limited opportunity for nuance among lower tiers of evidence. Requires both methodological and content expertise to apply (i.e. resource intensive).
CERQual	System for appraising evidence in qualitative syntheses. Assigns 4 levels of confidence across 4 components based on how well the phenomenon of interest is represented.	Offers a framework for adding rigour to assessment of qualitative evidence. Evidence can be downgraded, allowing for nuance. Developed in collaboration with the GRADE Working Group, which confers legitimacy and standardization.	There are no definitive criteria for assessing levels of confidence among qualitative studies. Requires strong expertise in qualitative methodology. Downgrading of evidence is subject to reviewer discretion, hypothetical risk of low inter-rater reliability. Applicable to qualitative evidence only.
SIGN	4-level scale, with an added option to further indicate strengths or weaknesses (i.e. 1++ for high quality meta-analyses or systematic reviews, 4 for expert opinion). Grade of recommendation (A–D) is based on the highest scoring level of evidence.	Typology is simple and easy to interpret. Plus (+) and minus (–) symbols added to numerical scale allow for greater nuance in conveying judgement on strength of the level of evidence.	Considers study type on the evidence hierarchy rather than study quality. Has fallen out of use, with previous guidelines adopting GRADE in lieu which serves a similar purpose but with greater rigour.
NSF-LTC	Follows premise that research quality is more important than research design. Two evidence categories – expert evidence (patients, caregivers, professionals) and research evidence. Research evidence is scored based on study design, quality, and applicability.	Considers challenges of making recommendations for diseases with complex trajectories. No implicit hierarchy to different study designs. Plausible applicability to bodies of knowledge with a relative dearth of experimental evidence, but robust expert opinion (i.e. as seen in palliative care).	Does not differentiate between study quality and validity. Assesses quantitative and qualitative studies using the same rubric, suggesting inherent ambiguity in its criteria. Unique grading typology that is not widely adopted within other guidelines. This system was not identified in any of the palliative guidelines.
ISDA-ESMO	Judgements on strength of recommendation (5 levels, as a letter of the alphabet) and type of evidence (5 levels, as a roman numeral).	5 levels for strength of recommendation provides greater distinction around lower quality evidence. Typology is recognizable and intuitive. System is in use within published palliative care guidelines. Can be used for both quantitative and qualitative evidence without needing to resort to two different systems. Less resource-intensive from a methodological perspective compared to GRADE.	Unable to downgrade rating based on biases or methodological flaws within the studies included. Less methodologically rigorous compared to GRADE.

^aIn the context of palliative care and resource-limited organizations.

Table 6.

Comparison of grading systems based on rigour, resources, typology, breadth and type of evidence.

System	Rigour	Resources	Typology	Breadth	Evidence type
GRADE	+++	+++	Clear, intuitive	Widespread	Quantitative
CERQual	++	++	Unique, niche	Limited	Qualitative
SIGN	+	+	Clear, intuitive	Limited	Quantitative
NSF-LTC	+	+	Unique, niche	Very limited	Both
IDSA-ESMO	++	+	Clear, intuitive	Moderate	Both

Discussion

Here we have described our identification and characterization of five grading systems relevant to palliative care clinical practice guidelines. Based on this review, we critically appraised the systems by comparing the strengths and limitations of each in the context of interdisciplinary symptom management guidelines developed and reviewed by an organization with limited resources and timelines. Ultimately, we identified IDSA-ESMO as the most pragmatic system in grading practice recommendations made within palliative care guidelines.

Rationale for selecting IDSA-ESMO

Compared to GRADE, IDSA-ESMO does not have the same need for extensive academic methodological expertise. Delphi studies, surveys, interviews and expert opinion are included within its level of evidence scheme,^46,49 aligning with the palliative care context and the types of evidence included in our guidelines. Although IDSA-ESMO does not upgrade or downgrade evidence to the same extent as GRADE,^6,7,50,54 its level of evidence table considers methodological quality. ⁴⁶ Conveniently, it avoids having to rely on two distinct grading schemes. Its strengths of recommendation range from A to E,^46,49 providing greater distinction around ‘lower quality’ evidence and allowing for more nuance in considering clinical variables and patient preferences. Its typology is recognizable and intuitive, as desired to ensure accessibility among clinician audiences.

The ESMO guidelines on refractory symptom management and end-of-life care in cancer patients^{48,51 –53} demonstrate versatility and applicability of their system to various palliative care settings, serving as proof of concept and evidence of congruity. The simple addition of an asterisk (*) indicates recommendations that are justified based on expert consensus or standard practice but are not feasibly supported by clinical trials. ⁴⁸ This notation is both practically useful and assures transparency behind basing recommendations on non-experimental evidence.

Systems for quantitative evidence: GRADE and SIGN

Strengths of the GRADE methodology include its comprehensiveness, extensive adoption and systematic approach to downgrading or upgrading evidence. It enhances transparency by requiring reviewers to explicitly outline the factors influencing their evidence ratings.^3,5,7,9,17 It is held in high regard within academic medicine as evidenced by its adoption across various organizations, with an added advantage of familiarity among inter-disciplinary guideline users.

Despite concerns about deviating from the academic standard, we were hesitant to implement GRADE in the absence of sufficient organizational infrastructure. GRADE is resource-intensive, requiring both methodological and content expertise, training and dedicated staff. There are inherent financial and time considerations in assuring adequate technical understanding of this system.

In addition, GRADE has been criticized for having poor inter-rater agreement and lacking internal consistency, which may be attributed to discrepancies between ratings from clinicians and GRADE methodology experts.^3,7 This shortcoming may be compounded by inexperience or unfamiliarity.

The GRADE Working Group discourages modification of its methodology due to inconsistencies with best practice standards. ⁴ The investment required may outweigh the benefits, as palliative care evidence would likely have universally low levels of certainty or be ungradable. There is a lack of distinction between tiers of ‘lower quality’ evidence in GRADE. Even with tools such as ROBINS-I, most of the evidence in palliative care would be downgraded due to unavoidable bias and small sample sizes. There is a risk of reinforcing provider hesitancy and indecision. Implementing a fulsome GRADE approach may exceed the capacity of smaller organizations.

In contrast to GRADE’s complexity, the historic SIGN methodology was simpler in its reliance on the hierarchy of evidence heuristic. However, its ABCD scheme disadvantageously addressed the quality of evidence rather than the clinical importance of the recommendations.^38,39 This system would also have provided us with low quality of evidence scores, in addition to being less methodologically rigorous and largely obsolete.

Systems for qualitative or mixed-methods evidence: CERQual and NSF-LTC

CERQual shows promise in adding academic rigour to qualitative evidence, which is prominent in palliative care research and informs many important non-biomedical areas such as ethics, bereavement and spiritual distress. However, proper implementation of CERQual requires strong expertise in qualitative methodology. Its components are inherently vague, lacking strict criteria for upgrading or downgrading evidence.^30,32 We came across select systematic reviews ³⁴ but no palliative care guidelines that seamlessly integrated recommendations using a quantitative framework in conjunction with CERQual. Adding a secondary grading system may convolute our process and impair usability among the intended audiences due to the additional cognitive load of interpreting two different systems.

NSF-LTC was designed for use in long-term neurological conditions, which share parallels with other life-limiting illnesses under the purview of palliative care, such as the need to adapt interventions based on disease progression, non-linear trajectory and unquantifiable experiences of pain or other terminal symptoms. However, a major limitation is its relative obscurity; we did not identify any organizations who have adopted NSF-LTC in the palliative care context. It was included in our comparison for its theoretical utility, although we were deterred by a lack of widespread adoption to date. This system also lacks rigour in its ability to differentiate between study quality and validity. ³⁹ NSF-LTC assesses quantitative and qualitative studies using the same rubric, implying ambiguity in its criteria. One study found that three-quarters of assessors used NSF-LTC incorrectly. ³⁹ Finally, we were concerned that clinicians may find its unfamiliar grading typology difficult to decipher.

Conclusion

Acceptance of GRADE as the ‘gold standard’ in rating the evidence supporting practice recommendations is a compelling reason for large scale and well-resourced organizations to adopt it. However, the expertise and capacity requirements can be out of scope for smaller organizations. The lack of granularity for assessing non-RCT study designs does not lend itself well to the evidence base available in palliative care. By comparison, the implementation of IDSA-ESMO is more feasible for lower-resourced guideline development groups. Based on the grading systems we reviewed, the IDSA-ESMO approach was the most succinct and pragmatic. It has practical strengths despite lacking the methodological rigour of GRADE, and is appropriately suited to the palliative care evidence base.