Critical appraisal of clinical practice guidelines for diagnosis and treatment of hepatocellular carcinoma

Abstract

Background and Aim:

Hepatocellular carcinoma (HCC) is the third most common reason for cancer-related death worldwide. Many countries either lack appropriate clinical practice guidelines for the diagnosis and treatment of HCC or the quality of their guidelines has never been evaluated. The main objective of our work was to identify published HCC guidelines and assess their quality with the Appraisal of Guidelines for Research and Evaluation instrument (AGREE) and their suitability regarding adaptation for future guidelines.

Methods:

We performed a systematic literature search on HCC clinical practice guidelines of MEDLINE, National Guidelines Clearinghouse and the Guidelines International Network. Methodological quality of selected guidelines was assessed by the AGREE instrument, Version 2001.

Results:

A total of 286 citations were screened and 32 relevant guidelines were identified. Overall, the guidelines performed well in the clarity and presentation domain with a mean score of 67%, followed by scope and purpose (55%) and rigor of development (50%). In contrast, poor scores were given for the remaining domains: stakeholder involvement (23%), applicability (28%) and editorial independence (31%). According to the AGREE instrument, four guidelines can be strongly recommended, 18 with provisos and alterations while the remaining cannot be recommended for adaptation due to poor methodological quality.

Conclusion:

Although existing HCC guidelines may accurately reflect agreed clinical practice, many guidelines lack proper methodological quality. Future guidelines should place more emphasis on these methodological shortcomings.

Introduction

Hepatocellular carcinoma (HCC) accounts for 80–90% of primary liver cancers and represents a major health burden, being the sixth most common cancer and the third most common cause of cancer-related death worldwide.¹ The general prognosis is still poor with overall survival rates of 3–5%. In a recently published study, the annual direct as well as indirect costs associated with HCC in the USA were estimated by using a Surveillance, Epidemiology and End Results (SEER)-Medicare database.² The study estimated that caring for a patient with HCC cost an average of $32 907 in 2005. With the calculated prevalence of approximately 14 000 patients with HCC in 2005, the total economic burden of HCC was estimated to be $454.5m. As more patients are being diagnosed at a younger age and with earlier stage tumors, the total cost of HCC will undoubtedly rise in the future.

Clinical practice guidelines (CPG) have been defined as systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.³ They are designed to evaluate and implement the ever-increasing amount of evidence and opinion on best current medical practice and to improve quality, appropriateness and cost-effectiveness of care.⁴ This implies that CPG need to follow a certain standard of methodology. However, recent studies revealed that the methodological quality of clinical practice guidelines is highly variable.⁵ Nonetheless, the recent increase in the production of clinical practice guidelines has been accompanied by a growing concern about variations in guideline recommendations and quality.⁶ Reasons for this variation have been suggested. For example, recommendations are often formed by consensus with research evidence being used to support this.⁷ Furthermore, the influence on the production of guidelines by public agencies and medical societies results in inconsistencies between guidelines dealing with the same topic.⁸ Ethical considerations, social influences and practical necessities vary between cultures and different health-care systems reinforcing differences between guidelines.⁹ The development of CPG also requires considerable time, expertise and resources. Therefore, the adaptation of already existing high-quality guidelines is not only cost-effective, but avoids duplication of effort.

Many countries either lack appropriate treatment guidelines for HCC or their quality has never been evaluated. The aim of this study was to search systematically for all existing HCC guidelines, and assess and compare their quality with the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument.¹⁰ This tool evaluates the process of practice guideline development and the quality of reporting. Consequently, already existing guidelines of sufficient quality can be used as a source for adaptation, especially in those countries with limited resources and experience on guideline development.

Methods

Data sources and selection process

Between October 2009 and January 2010 a systematic search was performed to identify evidence-based clinical practice guidelines for HCC. As computerized databases indexing guidelines, MEDLINE, the National Guidelines Clearinghouse (NGC) and the Guidelines International network (GIN) library were used by a search algorithm (Fig. 1). Furthermore, homepages of international medical societies and institutions were screened for current CPG publications. All full-text clinical practice guidelines published between 1999 and November 2009 on diagnosis and treatment of HCC were included in the study. Each guideline was checked for the following topics: prevention, screening and surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies and systemic therapies. Excluded were non-evidence-based expert consensus statements, secondary or multiple publications or adaptations of original practice guidelines, editorials, letters to the editor, case histories, hepatitis B and C guidelines and all guidelines not written in English or German.

Figure 1.

Searching and selecting guidelines flowchart.

Quality appraisal of guidelines

The AGREE instrument was selected as the appraisal tool. Twenty-three criteria of six domains, including scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability and editorial independence, were rated on a four-point Likert scale ranging from 1 (Strongly Disagree) to 4 (Strongly Agree) with two mid-points: 3 (Agree) and 2 (Disagree). Two experts independently conducted an evaluation of the chosen guidelines according to the instructions for using the AGREE instrument. Domain scores were calculated by summing up all the scores of each individual criterion in a domain and by standardizing the total as a percentage of the maximum possible score for this domain. Major discrepancies in the scores (> 1 point) assigned by the two reviewers were identified and resolved through discussion. A mean score was also calculated for each criterion, derived from the final scores assigned by the two reviewers. The final component of the AGREE instrument involves a recommendation regarding the use of a guideline in practice by taking each of the appraisal criteria into consideration. A guideline is “strongly recommended” if the guideline rates high (3 or 4) on the majority of items and most domain scores are above 60%. A guideline is “recommended,” if it rates high (3 or 4) or low (1 or 2) on a similar number of items with domain scores between 30% and 60%. A guideline is “not recommended” if it rates low (1 or 2) on the majority of items and most domain scores are below 30%, indicating that the guideline has a low overall quality and serious shortcomings.

Results

Searching for and selection of source CPG

Our literature search resulted in 286 citations identified through computerized database searches. An additional three citations had been identified through hand-searching in reference lists of papers, and website searches of CPG resources (Fig. 1). We identified 32 CPG for further evaluation and critical appraisal, which are listed in Table 1. For exclusion criteria see Figure 1. All 32 CPG covered either diagnosis and management of HCC (n = 11)^11–21 or only specific aspects, such as diagnosis or specific therapies (radiological intervention and others) (n = 21).^22–42 Diagnosis of HCC was addressed in three of 21 guidelines, with surgery and transplantation being covered by nine guidelines and locoregional therapies and radioembolization being covered in three and two guidelines, respectively. The role of systemic treatment was evaluated in three guidelines. All guidelines were published between May 2001 and October 2009. The former mainly originated from Europe (5/11), two CPG were published in the USA, three in Asian countries and one originated from Saudi Arabia. The specific guidelines were also predominantly produced in Europe (12/21), eight were from the USA and one was Japanese. Eighty-four percent of the guidelines (27/32) were disseminated national or regional, the rest of them (5/32) had an international scope; with the exception of the “Consensus statement from the Asian Oncology Summit 2009,”²¹ all guidelines were produced by a professional organization. Finally, 81% (26/32) of the guidelines were evidence-based. Quality scores of the CPG by the AGREE instrument are shown in Table 2 and are described below.

Table 1.

List of all 32 guidelines included in our analysis

Organization	Origin	Date	Type	Topics covered	Reference
FNCLCC	France	2001	EB	Diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	11
EASL	Europe	2001	EB	Prevention, screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	12
BSG	Great Britain	2003	EB	Screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	13
BASL	Belgium	2004	EB	Prevention, screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	14
AASLD	USA	2005	EB	Prevention, screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	15
SGA	Saudi Arabia	2006	EB	Prevention, screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	16
Japanese Ministry of Health, Labour and Welfare	Japan	2006	EB	Prevention, screening, surveillance, diagnosis, curative treatment, trans-arterial therapies, systemic therapies	17
Japan Society of Hepatology	Japan	2007	CB	Prevention, screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies	18
ESMO	Europe	2008	EB	Prevention, screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, systemic therapies including sorafenib	19
NCCN	USA	2009	EB	Screening, surveillance, diagnosis, clinical staging, curative treatment, trans-arterial therapies, radioembolization, systemic therapies including sorafenib	20
Asian Oncology Summit	ASIA	2009	EB	Prevention, screening, surveillance, diagnosis, curative treatment, trans-arterial therapies, systemic therapies	21
Haute Autorité de santé	France	2005	CB	Indications for liver transplantation	22
REBOC	USA	2007	CB	Radioembolization	23
EFSUMB	Europe	2008	EB	Contrast-enhanced ultrasound for diagnosis of liver tumors	24
SIR	USA	2009	EB	Transhepatic arterial chemoembolization	25
Department of Digestive Surgery, Nihon University School of Medicine, Tokyo	Japan	2008	EB	Surgical treatment	26
AFEF	France	2008	EB	Use of sorafenib	27
ACR	USA	2007	CB	Curative treatment, trans-arterial therapies, systemic therapies	28
ACR	USA	2006	CB	Diagnosis	29
ACR	USA	2008	CB	Radioembolization	30
CCO	Canada	2008	EB	Use of sorafenib	31
CCO	Canada	2006	EB	Surgical treatment	32
NICE	Great Britain	2007	EB	Microwave ablation	33
NICE	Great Britain	2003	EB	Radiofrequency ablation	34
NICE	Great Britain	2005	EB	Laparoscopic liver resection	35
NICE	Great Britain	2006	EB	Living-donor liver transplantation	36
NICE	Great Britain	2009	EB	Surgical treatment	37
NICE	Great Britain	2007	EB	Surgical treatment	38
CIRSE	Europe		EB	Radiofrequency ablation	39
SAGES	USA	2007	EB	Diagnostic laparoscopy	40
Royal COLLEGE of Surgeons in Ireland	Ireland	2005	EB	Diagnostic laparoscopy	41
SNLG	Italy	2009	EB	Diagnosis	42

ACR, American College of Radiology; AFEF, L’Association Française pour l’Etude du Foie; AASLD, American Association for the Study of the Liver; BASL, Belgian Association for the Study of the Liver; BSG, British Society of Gastroenterology; CB, consensus-based; CCO, Cancer Care Ontario CIRSE, Cardiovascular and Interventional Radiological Society of Europe; EASL, European Association for the Study of the Liver; EB, evidence-based; EFSUMB, European Federation of Societies for Ultrasound in Medicine and Biology; ESMO, European Society for Medical Oncology; FNCLCC, Fédération Nationale des Centres de Lutte Contre le Cancer; NCCN, The National Comprehensive Cancer Network ; NICE, National Institute for Health and Clinical Excellence; REBOC, The Radioembolization Brachytherapy Oncology Consortium; SAGES, Society of American Gastrointestinal and Endoscopic Surgeons; SGA, Saudi Gastroenterology Association; SNLG, Sistema Nazionale Linee Guida; SIR, Society of Interventional Radiology.

Table 2.

Assessment of HCC guidelines with AGREE instrument (Domain scores in %)

Reference	Organization	Origin	Scope & purpose	Stakeholder involvement	Rigor of development	Clarity & presentation	Applicability	Editorial independence	Overall recommendation
11	FNCLCC	France	28	0	19	50	0	0	Not recommended
12	EASL	Europe	11	8	16	46	5	33	Not recommended
13	BSG	Britain	61	8	62	75	16	0	Recommended
14	BASL	Belgium	55	16	29	66	33	0	Not recommended
15	AASLD	USA	61	12.5	45	87.5	61	33	Strongly recommended
16	SGA	Arabia	44	25	50	71	27	0	Recommended
17	Japanese Ministry of Health, Labour and Welfare	Japan	55	5	95	75	33	16	Recommended
18	Japan Society of Hepatology	Japan	66	25	26	75	33	16	Not recommended
19	ESMO	Europe	33	8	24	58	0	33	Not recommended
20	NCCN	USA	44	42	52	75	16	83	Recommended
21	Asian Oncology Summit	Asia	77	21	57	66	33	50	Recommended
22	Haute Autorité de santé	France	66	25	38	66	22	33	Recommended
23	REBOC	USA	88	33	50	58	11	50	Recommended
24	EFSUMB	Europe	44	25	29	50	11	17	Not recommended
25	SIR	USA	55	33	71	50	33	33	Recommended
26		Japan	55	25	48	58	22	33	Recommended
27	AFEF	France	88	25	10	42	22	0	Not recommended
28	ACR	USA	22	16	57	50	0	17	Not recommended
29	ACR	USA	22	16	57	50	0	17	Not recommended
30	ACR	USA	44	25	38	50	11	17	Recommended
31	CCO	Canada	100	50	77	83	55	50	Strongly recommended
32	CCO	Canada	100	33	62	83	44	66	Strongly recommended
34	NICE	Britain	55	50	52	92	33	50	Recommended
34	NICE	Britain	55	50	38	66	33	50	Recommended
35	NICE	Britain	55	50	48	83	33	50	Recommended
36	NICE	Britain	55	50	64	79	50	50	Recommended
37	NICE	Britain	55	50	64	79	50	50	Recommended
38	NICE	Britain	55	50	64	79	50	50	Recommended
39	CIRSE	Europe	55	33	71	50	33	33	Recommended
40	SAGES	USA	55	16	66	83	44	0	Recommended
41	Royal College of Surgeons in Ireland	Ireland	33	8	33	66	0	0	Not recommended
42	SNLG	Italy	77	45	81	83	83	55	Strongly recommended
		Mean	55.3	22.9	49.8	67	28	30.8
		Range	11–100	0–50	10–95	42–92	0–83	0–83

ACR, American College of Radiology; AFEF, L’Association Française pour l’Etude du Foie; AASLD, American Association for the Study of the Liver; BASL, Belgian Association for the Study of the Liver; BSG, British Society of Gastroenterology; CCO, Cancer Care Ontario CIRSE, Cardiovascular and Interventional Radiological Society of Europe; EASL, European Association for the Study of the Liver; EFSUMB, European Federation of Societies for Ultrasound in Medicine and Biology; ESMO, European Society for Medical Oncology; FNCLCC, Fédération Nationale des Centres de Lutte Contre le Cancer; NCCN, The National Comprehensive Cancer Network ; NICE, National Institute for Health and Clinical Excellence; REBOC, The Radioembolization Brachytherapy Oncology Consortium; SAGES, Society of American Gastrointestinal and Endoscopic Surgeons; SGA, Saudi Gastroenterology Association; SNLG, Sistema Nazionale Linee Guida; SIR, Society of Interventional Radiology.

Scope and purpose

This domain covers the overall aim of a guideline, the specific clinical questions/problems and the target patient population. Overall, the mean score was 55% (ranging from 11% to 100%) indicating that the criteria of scope and purpose were met by most of the guidelines. Of the 32 guidelines, only 11 scored more than 60% and four less than 40%. A closer look revealed that the target population in particular (criterion 3) was not specifically described in 22 guidelines, in contrast to criteria 1 and 2, where most of the guidelines performed well. The overall objective of the guideline was well defined in 19 guidelines (59%) and the clinical questions addressed in 27 guidelines (84%), respectively.

Stakeholder involvement

This domain evaluates the degree to which the guideline represents the views of its intended users. Included are questions regarding the composition of the guideline development group, whether patients were involved, whether the target users of the guideline were well defined, and whether the guideline was piloted among end-users. Overall, the mean score for this domain was 23% (ranging from 0% to 50%). Fifteen guidelines had been developed by a multi-disciplinary team (47%) and a total of 28 guidelines were produced by health organizations. Only seven guidelines addressed patients’ views and preferences during the development process. None of the guideline development groups described a process of piloting among target group members and just 13 guidelines provided some information about their target users (41%).

Rigor of development

This domain assesses if systematic methods and specific criteria were used for searching and selecting the evidence and for formulating recommendations, whether the recommendations and the supporting evidence were explicitly linked, whether health benefits, side-effects and risks have been considered, whether the guidelines were externally reviewed and whether a procedure for updating was provided. The mean score for this domain was 50% (ranging from 10% to 95%) with 21 guidelines scoring < 60%. Merely 12 guidelines reported details of the strategy used to search for evidence and 18 reported the criteria for selecting the evidence. Moreover, we observed that just over two thirds of the guidelines provided information about the methods used for formulating the recommendations. The recommendations were explicit in 19/32 of the guidelines regarding health benefits, side-effects and risks. In 14 guidelines there was an explicit link between supporting evidence and recommendations. A minority of 12 guidelines was reviewed before publication and most of the guidelines (31/32) did not provide any information about updating.

Clarity and presentation

This domain describes whether the recommendations were specific and unambiguous, whether the different management options were clearly presented, whether key recommendations were easily identifiable, and if the guidelines were supported with tools for application. Overall, the mean score for this domain was 67% (range, 42–92%), indicating that, on average, 67% of the criteria for clarity and presentation were met. Most guidelines performed well in this domain with only 12 guidelines scoring < 60% of which only two achieved less than 50%. The recommendations were specific in all of the guidelines and easily identifiable in 25 guidelines. Thirty of them described different disease management options, but only slightly more than one third of the guidelines (n = 11) included tools for application.

Applicability

This domain evaluates issues that are pertinent to guideline implementation, such as organizational barriers, cost implications, and monitoring criteria. The mean score in this domain was the lowest of all with a mean of 28% (range, 0–83%) and only two CPG scored > 60%. A total of 15 CPG discussed more or less potential organizational barriers and seven provided some kind of indicator for monitoring and audit purposes. Only two CPG discussed cost implications.

Editorial independence

This domain addresses conflict of interest, specifically whether the guideline was editorially independent from the funding body and whether potential conflicts of interest were reported for the members of the guideline development group. The score in this domain was also poor, with a mean score of 31% (range, 0–83%). Two guidelines scored > 60%. Potential conflicts of interests of CPG developers were stated in just two CPG and 13 guidelines (40%) were editorially independent from the funding body.

Overall recommendations

After completing the AGREE instrument, the authors made an overall recommendation for each guideline. The authors recommended for adaptation those guidelines that demonstrated acceptable quality, depending on their AGREE score. Four guidelines can be strongly recommended, the majority of the domains scoring above 60% showing a good overall quality of the guidelines. Eighteen can be recommended with provisos and alterations, the majority of the domains scoring between 30% and 60%. These guidelines revealed flaws in certain domains indicating their average quality. If in future CPG development more attention is paid to these shortcomings, the overall quality of them could be improved significantly. The remaining 10 CPG cannot be recommended (Table 2) due to their poor scoring in the majority of the domains.

Discussion

Over the last years, we have witnessed an increasing number of clinical practice guidelines in HCC produced by different bodies, not only with regard to diagnosis and treatment but also in several other clinical areas. This proliferation of guidelines has produced a need for standardized international criteria for their proper establishment. The AGREE instrument was developed in response to this need. It is a validated instrument for the evaluation of guidelines and for defining the steps in producing high-quality guidelines endorsed by the leading producers, raters, and compilers of international CPGs⁴³ and it is considered the best current tool for assessing guideline quality.⁴⁴ The AGREE instrument assesses both the quality of the reporting, and the quality of some aspects of the recommendations. Like other appraisal tools, the AGREE instrument does not differentiate between a guideline failing to meet a criterion due to poor methodology or lack of reporting, and which can result in distrust in and/or misuse of recommendations.⁴⁵ Furthermore, it has to be considered that the quality of the recommendations are beyond the scope of AGREE.

Overall, the assessed guidelines demonstrated considerable flaws. Almost all the guidelines scored poorly with respect to stakeholder involvement, applicability and editorial independence. Not a single guideline included the views of all relevant professional groups and/or addressed patients’ views and preferences. Furthermore, evidence of pilot testing was missing. Although few studies have assessed the impact of guideline development on patient outcome, it has been demonstrated that guidelines can improve clinical practice.⁴⁶

According to the results of studies relating to other clinical topics,⁴⁷ the assessed guidelines received the lowest scores on the applicability domain. These findings underline the fact that guideline producers should be more attentive both to potential barriers to guideline implementation as well as to monitoring criteria, which assess the guideline’s impact.

HCC guidelines also consistently failed to perform well in the domain of editorial independence. It should be borne in mind that potential conflicts of interest—for example, between a committee member and pharmaceutical industry—can have an impact on guideline drafting. Therefore, conflict of interests need to be clearly stated. A few studies recently underlined that authors of CPG were influenced by pharmaceutical industries and it is important to know to what extent these interactions might have influenced the recommendations.⁴⁸

One of the key factors regarding the adequacy of a set of guidelines pertains to the rigor of their development. Although most of the guidelines included references to published literature, many did not clearly describe the literature review methodology employed or the mechanism by which recommendations were formulated. This step is crucial in determining whether the recommendations are truly based on the best available evidence and also in understanding how the evidence is synthesized. For example, a recent study illustrated that less than one-third of cardiovascular risk management recommendations in national guidelines were based on high-quality evidence.⁴⁹ However, very often it is not clear how final recommendations have been arrived at⁵⁰ and these recommendations can vary as a result of local bias, differences in data interpretation or be a manifestation of available resources. Analyses of guidelines on other medical topics, such as the methodology of current psoriasis guidelines by Nast et al., have revealed similar results with a high score for scope, purpose and clarity, but low scores for stakeholder involvement, applicability and editorial independence.⁵ Since most guidelines on HCC scored low in these domains, special attention should be paid to these shortcomings in future guideline development.

As a consequence of our appraisal, we were able to strongly recommend four guidelines as source guidelines for future CPG. These included two guidelines with most domain scores above 60% developed by Cancer Care Ontario (CCO)³¹ and an Italian guideline by Sistema Nazionale Linee Guida (SNLG),⁴² indicating that these guidelines had a high overall quality. Two other guidelines, published by the American Association for the Study of the Liver (AASLD)¹⁵ and CCO³² scored only in three domains above 60%. However, these domains best reflect the adherence to the currently best available evidence and the transferability to local settings. Therefore, the authors considered, that they can be “strongly recommended.”

Eighteen other guidelines can be recommended with provisos and alterations. Most domains scored between 30% and 60% in these guidelines, while six guidelines could not be recommended because they rated low on the majority of items with a majority of domain scores below 30%, indicating a low overall quality and serious shortcomings. More than three domains had more than 30% in the other four guidelines,^14,18,19,41 but a closer look revealed that some of the domains were rated concisely about 30% and they especially failed to convince at the rigor of development and applicability domains. Therefore these guidelines cannot be recommended. However, it must be emphasized that the validity of overall assessment is limited due to the subjective nature of our appraisal, as there were no clear rules on how to weight the different domains in making a final recommendation.

Several high-ranking international guidelines have already been developed. Adaptation of already existing high-quality guidelines represents one possibility to save costs and avoid duplication of efforts when new guidelines need to be developed. However, until today there has been no validated process for adapting guidelines which have been produced in one cultural setting for use in another.

In conclusion, our analysis of current CPG for the diagnosis and treatment of HCC revealed several methodological flaws, although these may accurately reflect agreed clinical practice. Even if methodological standards for the development of GCP guidelines are published adherence to these remains unsatisfactory.