How to critically appraise a clinical practice guideline

Abstract

Purpose:

Clinical practice guidelines play a critical role in guiding the evidence-based clinical practice of urology. We describe a systematic approach to critical appraisal of urology guidelines.

Materials and Methods:

Based on a focused clinical question derived from a clinical scenario, we identified a relevant clinical practice guideline that we critically appraised using the Users’ Guide to the Medical Literature framework as to whether the results are valid, what are the results, and can they be applied to the care of an individual patient.

Results:

A clinical practice guideline by the National Institute for Clinical Excellence on the use of sunitinib as the first line treatment for patients with metastatic renal cell carcinoma was identified. The guideline development process was found to be appropriately rigorous and included an explicit rating of the quality of evidence. The recommendations were clearly stated and appeared applicable to the specific patient in the clinical scenario.

Conclusions:

Clinical practice guidelines should be developed using rigorous evidence-based methodology. Urologists should have the skills and knowledge to critically appraise a guideline before applying it to the care of their patients.

INTRODUCTION

The evolution of evidence-based medicine has become well established in the medical community over the last two decades. Clinical practice guidelines play a critical role in guiding evidence-based practice and have been defined as systematically developed statements to guide patient and physician decision-making.[1] They are widely accepted in the urology community with multiple professional organizations such as the American Urological Association and the European Urological Association investing heavily in their development and dissemination.[2,3]

The recent explosion of clinical practice guidelines has led to the urgent need for urologists to be able to critically appraise these guidelines for their validity, impact, and applicability to the care of their individual patients. Clinical practice guidelines should be developed using rigorous methodology based on a systematic review of the best available evidence for specific clinical questions and provide a rating of the quality of evidence.[4] The process and considerations that are involved in formulating recommendations should be clearly identified, recommendations should be specific, rated according to their strength and be practically useful for the clinicians. One such methodologically rigorous and transparent approach has been developed by the GRADE Working Group (http://www.gradeworkinggroup.org/) that has been endorsed by over 55 professional organizations. To date, no unified system has been adopted by various urological organizations developing guidelines making it very important for urologists to understand what constitutes an evidence-based guideline, rather than one that is primarily consensus-based.[5] In this article, we describe a systematic approach to the critical appraisal of clinical practice guidelines that is based on the framework of the User's Guide to the Medical Literature.[6]

CLINICAL SCENARIO

A 62-year-old male patient presents to your office with painless macroscopic hematuria. Urological investigations reveal a 6 cm right upper pole renal mass. No other causes of hematuria are detected. Staging investigations reveal multiple small metastases in the lungs bilaterally. He is otherwise well with an ECOG status of 0. There is no other relevant past medical or surgical history. The patient has utilized the Internet at home to review various treatment options available for metastatic renal cell cancer and asks about the possibility of treatment with sunitinib. Being a general urologist 15 years out of training who takes care of patients with a broad spectrum of urological conditions you decide to seek out the current best evidence on the appropriateness of sunitinib in this setting. To guide your search, you formulate the following focused clinical question using the PICO format: In patients with metastatic kidney cancer (P patient), how does treatment with sunitinib (I intervention) compare to systematic immunotherapy (C comparison) with regards to overall survival (O outcome).

Finding the right guideline

You are familiar with the hierarchy of evidence-based resources which characterizes clinical practice guidelines as summaries that integrate the best evidence on a full range of treatment options.[7] Clinical practice guidelines are therefore one of the preferred resources of pre-appraised evidence for clinicians. An excellent resource to identify evidence-based clinical practice guidelines developed by different organizations is the National Guidelines Clearinghouse (NGC) funded and managed by the Agency for Healthcare Quality and Research.[4,8] This resource sets minimal methodological quality standards thereby excluding up front a large number of guidelines that cannot be considered evidence-based. For example, cancer guidelines by the National Cancer Center Network, although being widely used, are considered consensus and not evidence-based and therefore not included. An excellent feature of the NGC website is the option of displaying several guideline document side-by-side for comparison.

Using the search term “sunitinib” displays three guidelines of which one, “Sunitinib for the first-line treatment of advanced and/or metastatic renal cell carcinoma”, appears directly relevant.[9] This guideline is a publication of the UK's National Institute for Health and Clinical Excellence (NICE) which is funded by the UK's National Health Service (NHS). There is furthermore a link to freely download the full-text guideline document. You decide to review this document to ensure that it was rigorously developed, relevant, and applicable to your patient.

Are the guideline recommendations valid?

The critical appraisal of any study is essential to ensure the results are valid, or in other words an accurate representation of “the truth.” This applies to guidelines as much as it does to any other type of study. Similarly to the appraisal of an individual study such as a randomized controlled trial one needs to consider were all relevant outcomes considered, were the inclusion of appropriate studies complete (and were any unnecessary studies included), whether the development process was appropriate and whether potential conflict of interest were appropriately managed

Were all relevant outcomes considered?

Clinical decision-making should be based on outcomes that are important to patients. Such outcomes include overall and disease-specific survival, quality of life, and absence of complications secondary to the relevant disease as well as adverse events that are therapy related. However, very often, clinical research studies employ surrogate endpoints that are associated with patient-important outcomes yet distinct. Guideline developers and users should be aware of this distinction and clearly differentiate between the two types of outcomes. For example, tyrosine kinase inhibitors such as sunitinib may result in tumor responses but have not necessarily impact overall and disease-specific survival.

The review of the NICE guideline document finds that the developers explicitly considered and included information on overall, progression-free survival as well therapy-related adverse events. The latter included hypertension, fatigue, diarrhea, and hand-foot syndrome although rates and severities were not included. In addition, the guideline document included a formal economical analysis based on two different models, one independently performed by the NICE group, another provided by the drug manufacturer. Interestingly, the reported cost per quality adjusted life year (QUALY) by the NICE group was £104,715 as compared to £72,003 by the manufacturer. With the increasing importance of cost-effectiveness considerations in an era of burgeoning health care costs, the inclusion of formal cost-effectiveness analyses as provided by NICE guideline is a distinct strength of their guidance documents.

Was the inclusion of appropriate studies complete?

Clinical practice guidelines should be based on a systematic review of the current best evidence. The methodology for performing a systematic review is well established and includes predefined inclusion and exclusion criteria as well as a comprehensive literature search not limited by publication status or language.[10] For questions of therapy such as this, evidence should be derived from randomized controlled trials, which if well designed, executed, and reported has the potential to provide high quality evidence. Only in the absence of randomized controlled trials, may it be worthwhile to consider observational studies.

The search strategy described in the NICE sunitinib study clearly defined the clinical question of interest focusing on the effectiveness of sunitinib as first line treatment for locally advanced and/or metastatic renal cell cancer compared to other mainstay treatment options. For metastatic renal cell carcinoma they identified immunotherapy (interferon-alpha and/or interleukin-2) as the comparison. Observation was not considered as a comparator since systematic immunotherapy has established effectiveness and was therefore considered the standard of care prior to the advent of targeted therapy. The systematic review the guideline document is based upon has been independently published. The authors reported that systematic literature searches were performed in six electronic databases. Bibliographies of included studies were searched for further relevant studies and individual conference proceedings were searched using their online interfaces. Studies were selected according to the predefined criteria. All randomized clinical trials of sunitinib in combination with interferon for treating advanced metastatic renal cell carcinoma in accordance with the European licensed indication were included. Study selection, data extraction, validation, and quality assessment were performed by two reviewers with disagreements being settled by discussion.

The evidence search was grouped into four clinical settings for sunitinib use: First-line treatment for people suitable for immunotherapy, first-line treatment for people suitable for immunotherapy with at least three of six factors indicating poor prognosis, first-line treatment for people unsuitable for immunotherapy, and first-line treatment for people with poor prognosis unsuitable for immunotherapy. This was meant to provide users of the document more detailed guidance based on patient sub-group. The case scenario here fits into the first and largest group, those suitable for immunotherapy.

Who developed the guideline?

Clinical practice guideline development requires specialized knowledge both with regards to the content areas and evidence-based medicine/research methodology. If appropriate treatment of a given disease ideally requires input from different subspecialties such as urology, medical oncology, radiation oncology, palliative care, and nursing, these should ideally be represented on the guideline panel. In addition, inclusion of patient representatives usually adds a unique perspective and is increasingly recommended.

The NICE document identifies the guideline panel composition in its appendix. IT included a broad spectrum of representatives of various medical and surgical specialties, pharmacologists, researchers, public health officials, and “lay people.”

How was conflict of interest managed?

Undue influence of guideline panelists when making recommendation due to industry affiliation has recently become a major public concern.[11] As a result, professional organizations are increasingly tightening their requirements with regards to the allowable ties of guideline members and chairs. For example, the American Urological Association has recently decided that panel chairs need to relinquish all related industry ties prior to assuming the position until 1 year after guideline publication. This trend toward more stringent assurance of no potential conflict of interest is likely to continue. It has also been argued that potential conflict of interest management should go beyond financial conflict of interests and consider intellectual conflicts as reflected for example by a lifelong investment of one's career towards the development of a certain type of drug. Currently, it is a well-established standard that financial conflicts of interest of panel members are transparently reported and appropriately managed, for example by excusal of a conflicted panel member from a vote on a specific recommendation.

In case of the NICE guideline, this standard was met and guideline panelists self-identified any potential conflicts of interest and there were reported in the guideline document.

What were the methods of assessing the quality of evidence?

One of the central issues that determine the strength of a recommendation for or against a given therapeutic intervention is the quality of evidence for which a variety of rating systems exists. The most well known are the level of evidence rating system by the Centre for Evidence-Based Medicine in Oxford (http://www.cebm.net/). Well-recognized dimensions of the quality of evidence are the study design (i.e. randomized controlled trial) as well as study limitations such as allocation concealment, blinding, intention-to-treat analysis, and completeness of follow-up. GRADE further recognizes imprecision, indirectness, inconsistency, and outcome reporting bias as dimensions of the quality of evidence.[12] Irrespective of whatever system is used, the quality of evidence should be determined and graded.

In the case of the NICE guideline, the guideline authors only identified a single randomized controlled trial that compared sunitinib to interferon alpha.[13] In their evidence review, they commented on the fact that a number of methodological issues that might lower the quality of evidence were present. For example, a subset of patients crossed over to the sunitinib arm after disease progression. However, a discrete rating of the overall quality of evidence is missing and represents a shortcoming of this guideline document.

Has the guideline been subjected to peer review?

Peer review of a clinical practice guideline draft document is a frequently used method to seek input from a larger group of individuals from the target audience of prospective guideline users. This peer review can take the format of seeking input from a select group of content experts to the invitation of public commentary by posting of the document on the agency's website. Feedback from peer reviewers might apply to the scope of the guideline, the completeness of the evidence being considered and/or the practical value of the recommendations that are being made.

In case of the NICE sunitinib guideline, no external peer review process was reported. This issue is partially off-set by the fact that the guideline panel included a large number of individuals; nevertheless, this represents another shortcoming of this guidelines document.

Summary of the guideline validity

The NICE guideline on the use of sunitinib for the treatment of patients with metastatic renal cell carcinoma is based on a comprehensive and systematic review of the evidence, which identified a single, randomized controlled trial. The guideline had broad representation by different stakeholder representatives and potential conflicts of interests were appropriately managed. Main shortcomings of the guideline document are the failure to provide an explicit rating of the quality of evidence as well the absence of an external peer review process. Overall, the guideline raises no major concerns and can be considered valid.

What are the recommendations?

Busy urologists that consult clinical practice guidelines are the points of care that need clinically useful guidance that provides specific recommendations. In the most basic terms, guidance can be provided for or against a given intervention. These recommendations are the central deliverable and should be easily identifiable in the guideline document.

The four recommendations of the NICE guideline on sunitinib address four types of patients that differ by their suitability for immunotherapy and/or their prognostic risk. They are as follows:

1. Sunitinib is recommended as a first-line treatment option for people with advanced and/or metastatic renal cell carcinoma who are suitable for immunotherapy and have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1.

2. Sunitinib cannot be considered a clinically effective first-line treatment for people with poor prognosis, suitable for immunotherapy with advanced and/or metastatic RCC.

3. Sunitinib cannot be considered a clinically effective first-line treatment for those unsuitable for immunotherapy with advanced and/or metastatic RCC.

4. Sunitinib could not be considered a clinically effective first-line treatment for people with a poor prognosis who are unsuitable for immunotherapy.

In summary, these recommendations effectively limit the use of sunitinib to patients with a good performance status and a favorable prognostic risk that would also be eligible for standard immunotherapy. They are clearly stated and provide practical guidance for clinicians and patients.

How strong are the recommendations?

One of the most important advancements in clinical practice guidelines over the last decade has been the understanding that the quality of evidence and the strengths of recommendations are distinct from another and not necessarily correlated. The underlying reasons are that dimensions other than the quality of evidence impacts clinical recommendations. These include the balance of benefit to harm, the patients’ values and preferences (and their variability) as well as the associated costs to the healthcare system. Therefore, guideline panelists may judge that despite the presence of high quality evidence (i.e. systematic review of several well-designed randomized controlled trials) that a given intervention is effective, the risk of adverse events, or the costs are too high to justify a strong, unconditional recommendation for its use. Different professional organizations use different systems to rate the strength of a recommendation.[14] GRADE distinguishes between strong and conditional recommendations and clearly defines how physicians, patients, and policy-makers should interpret these recommendations.

The NICE guideline offers recommendations for the utilization of sunitinib but fails to offer grades of recommendation for its use. This must be considered a weakness of this particular guideline document.

Are the recommendations pragmatic?

Guideline developers should consider the feasibility of guideline implementations and its implications to the health care system. Should a guideline panel issue a strong recommendation for a treatment modality that is not readily available in part of the geographic reason for which is being created, parallel efforts may be necessary to make this treatment option accessible to all patients. Such considerations are highly relevant for guideline developers of the European Urological Association, for example, that serve many countries with different health care systems and varying socioeconomic status. Guideline developers should also consider the economic implications of guideline implementation.

The NICE guidelines primarily target patients living in the United Kingdom with advanced and/or metastatic renal cell carcinoma, which the guideline panel estimated to be a group of approximately 4000 patients. Within the national health care system, access to treatment is not an issue and therefore not explicitly discussed. The guideline developers further estimated the statistical life expectancy of these patients to be between 12 and 24 months and calculated the price of sunitinib treatment to lie between £72,000 per QALY and £105,000 per QALY gained, which they judged to be cost-effective in the end-of-life setting. In addition, the NICE guideline includes a specific section of implementation with tools for monitoring and cost-assessment.

Are the recommendations applicable to your patient?

Similar to assessing a therapy study, you need to ensure that the recommendations made in a guideline are applicable to your patient. The patient demographics are not described in the guideline but are assumed to be the general UK population. It is unlikely to be of concern in the area of RCC but you need to consider whether the general UK population is similar to your patient population. The NICE guideline broke down the results into four different patient groups that the investigators thought might need different recommendations for prior to performing a literature review.

Applying guideline recommendations to the care of your patient

In summary, the NICE guideline represents a methodologically rigorous and transparently reported guideline document that, despite some limitations, can be expected to provide valid and clinically relevant guidance that appears applicable to your patient. Having reviewed it systematically as outlined above, you feel comfortable in applying the recommendations to the care of your patient. Reviewing the guideline recommendation with the patient and his family applying a model of shared decision-making, the patient agrees to proceed with systematic therapy [Table 1].

Table 1.

GRADE's binary classification of strengths of recommendations[15]

CONCLUSION

Clinical practice guidelines are becoming increasingly important to guide the evidence-based practice of urology. It is important for urologist to recognize the defining features of what make a guideline “evidence-based” and distinguishes these from consensus-based guidelines and other non-systematic guidance documents. The outlined approach adapted from the Users’ Guide to the Medical Literature provide a useful approach to the critical appraisal of guideline which should be useful for all urologists seeking to practice evidence-based patient care.[15]