More Thoughts than Answers: What Distinguishes Evidence-Based Clinical Practice Guidelines from Non-evidence-Based Clinical Practice Guidelines?

A widely accepted definition of clinical practice guidelines (CPGs) has come from the Institute of Medicine: CPGs are statements that are derived from a systematic review of evidence and an assessment of the benefits and costs of alternative care options.¹ These statements are recommendations intended to assist stakeholders to optimize patient care.¹ Djulbegovic and Guyatt² have suggested classifying all CPGs into two categories: evidence-based CPGs and non-evidence-based CPGs, so as to avoid using the term “consensus-based CPGs.” We agree with this approach as there is always some element of CPG developers’ judgment involved during recommendation development and therefore, all guideline recommendations are technically consensus-based to some extent. While most guidelines use some form of evidence to inform recommendations, do clear definitions exist to define what is meant by evidence-based versus non-evidence-based CPGs? For example, do evidence-based guidelines have to include a systematic review of the evidence? How do we classify guidelines that reviewed evidence, but not systematically? Some CPG agencies state that the literature has been reviewed, but do not provide the specifics around their literature search. Should we still consider these CPGs evidence-based?

Djulbegovic and Guyatt² suggested that an evidence-based CPG should include a systematic review of the literature. However, the definitions and understanding of what constitutes a systematic review in CPGs vary. A systematic search of the literature should be a key component of a systematic review, but what is the definition of a systematic search of the literature? Does a systematic review need to search multiple databases? If only one database was searched, should this still be considered a systematic review? If it is not a systematic review, then would this guideline be considered non-evidence-based? Alternatively, would this be considered a systematic review, but may be assessed as a lower quality systematic review? These are some of the common practical questions faced by CPG developers when trying to determine whether a review was systematic or not and, by extension, if a CPG can be considered evidence-based. This is particularly salient when CPG developers are searching exclusively for evidence-based CPGs to consider for endorsement or adaptation or searching for systematic reviews as a source of evidence.

A universally acknowledged standard of what should be included in a systematic review can be found in Cochrane’s Handbook describing how to conduct a Cochrane systematic review, from reporting tools for systematic reviews and CPGs, and from quality assessment tools for systematic reviews and CPGs. The Cochrane’s Handbook for Systematic Reviews of Interventions describes the criteria needed for Cochrane systematic reviews .³ For instance, they publish their protocols, search at least two databases (the Cochrane Library and MEDLINE), and provide their full search strategy. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement is a recommended reporting checklist,⁴ which requests that authors indicate if a protocol exists, provide the databases searched with the search dates, and list the full electronic search strategy for at least one database. Likewise, the Appraisal of Guidelines, Research and Evaluation (AGREE) Reporting Checklist⁵ outlines the reporting standards for a high-quality CPG and asks CPG developers to provide the source(s) consulted, the dates of the literature covered, the search terms used, and the full search strategy. The AMSTAR 2 tool,⁶ which was designed to assess the quality of a systematic review on an intervention topic, emphasizes that a systematic review should register their protocol, search at least two databases, provide keywords and/or a search strategy, and justify publication restrictions (e.g., language). The CPG quality assessment tool—AGREE II, which is a widely used tool to appraise the quality of existing CPGs, points out that in the domain of Rigour of Development, higher-quality CPGs include systematic reviews that have provided the search terms used, the source(s) consulted, and the dates of the literature covered.⁷

There is overlap in some content across these approaches and tools, but some content is inconsistent, such as whether it is important to register the protocol or report the full search strategy. However, these tools were not designed to differentiate a systematic review from a non-systematic review. There is no universally accepted minimum threshold of what constitutes a systematic review and we do not propose to present one. Instead, we suggest that CPG developers define a priori their own criteria for what they consider to be a systematic review for each CPG being developed because there may be different criteria needed for different purposes. For example, different databases may be required to be searched for different research questions, in which case, CPG developers may only consider systematic reviews that searched particular databases. Other options for criteria could be to consider CPGs with reviews that have provided the name of a least one database searched or with reviews that have registered their protocols. Furthermore, CPG developers may want to consider only high-quality systematic reviews, in which case they could set a threshold for quality using the AMSTAR 2 tool for interventions.⁶ CPG developers could also use the AGREE II tool to consider only high-quality CPGs based on a Rigour of Development score of 70% or above.⁷ Just as we set up inclusion criteria for primary studies, we suggest generating inclusion criteria for CPGs and systematic reviews. Ultimately, what distinguishes an evidence-based CPG from one that is not evidence-based is whether evidence is considered during recommendation development. Ideally, this evidence should be based on a systematic review. Since the definition for a systematic review can vary, the concept of an evidence-based CPG can also vary and should be defined upfront for transparency.

Compliance with Ethical Standards

All authors conduct clinical practice guidelines in the cancer area for Ontario Health (Cancer Care Ontario), Ontario, Canada.

Conflict of Interest

The authors declare that they do not have a conflict of interest.