Clinical Practice Guidelines and Consensus Statements in Oncology – An Assessment of Their Methodological Quality

Abstract

Background

Consensus statements and clinical practice guidelines are widely available for enhancing the care of cancer patients. Despite subtle differences in their definition and purpose, these terms are often used interchangeably. We systematically assessed the methodological quality of consensus statements and clinical practice guidelines published in three commonly read, geographically diverse, cancer-specific journals. Methods Consensus statements and clinical practice guidelines published between January 2005 and September 2013 in Current Oncology, European Journal of Cancer and Journal of Clinical Oncology were evaluated. Each publication was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) rigour of development and editorial independence domains. For assessment of transparency of document development, 7 additional items were taken from the Institute of Medicine’s standards for practice guidelines and the Journal of Clinical Oncology guidelines for authors of guidance documents.

Methods

Consensus statements and clinical practice guidelines published between January 2005 and September 2013 in Current Oncology, European Journal of Cancer and Journal of Clinical Oncology were evaluated. Each publication was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) rigour of development and editorial independence domains. For assessment of transparency of document development, 7 additional items were taken from the Institute of Medicine's standards for practice guidelines and the Journal of Clinical Oncology guidelines for authors of guidance documents.

Findings

Thirty-four consensus statements and 67 clinical practice guidelines were evaluated. The rigour of development score for consensus statements over the three journals was 32% lower than that of clinical practice guidelines. The editorial independence score was 15% lower for consensus statements than clinical practice guidelines. One journal scored consistently lower than the others over both domains. No journals adhered to all the items related to the transparency of document development. One journal’s consensus statements endorsed a product made by the sponsoring pharmaceutical company in 64% of cases.

Conclusion

Guidance documents are an essential part of oncology care and should be subjected to a rigorous and validated development process. Consensus statements had lower methodological quality than clinical practice guidelines using AGREE II. At a minimum, journals should ensure that that all consensus statements and clinical practice guidelines adhere to AGREE II criteria. Journals should consider explicitly requiring guidelines to declare pharmaceutical company sponsorship and to identify the sponsor’s product to enhance transparency.

Introduction

Consensus statements and clinical practice guidelines are widely used in oncology to improve the quality of patient care [1], [2]. While both consensus statements and clinical practice guidelines are intended to provide guidance to clinicians, there are important differences between them. A clinical practice guideline (also called a medical guideline or clinical protocol) produces statements that are informed by a systematic review of the evidence and an assessment of the benefits and harms of alternative options [3]. A consensus statement is developed by an independent panel of experts, usually multidisciplinary, convened to review the research literature in an evidence-based manner for the purpose of advancing the understanding of an issue, procedure or method [4]. Both documents provide recommendations for optimizing patient care [3].

Although consensus statements address topics in which the evidence base is less extensive compared to clinical practice guidelines, their development should still be methodologically rigorous and transparent [4]. To assist with maintaining methodological rigor, organizations such as Appraisal of Guidelines for Research and Evaluation (AGREE) [5], Institute of Medicine (IOM) [3] and Guidelines International Network (GIN) [6] have developed criteria to ensure objective, scientifically valid, and consistent standards for the development and reporting of high quality guidance documents.

Given their widespread availability and importance for both clinical practice and funding decisions [7], we sought to evaluate the methodological quality of both consensus statements and clinical practice guidelines published in three commonly accessed oncology-specific journals through the domains of rigor of development and editorial independence Information around the transparency of document development was also collected to assess whether or not pharmaceutical company sponsored guidelines were more likely to endorse a product manufactured by the sponsoring company.

Methods

Three oncology specific journals were searched for consensus statements and clinical practice guidelines published from January 2005–September 2013. Current Oncology (CO), the European Journal of Cancer (EJC) and the Journal of Clinical Oncology (JCO) were chosen as they have editorial offices in different countries and for their perceived prominence in North America and Europe. January 2005 was chosen as the starting date for eligibility, as this was the date by which all three journals had accessible electronic archives. Each journal’s online search tool was used to search for the terms “consensus”, “consensus guideline”, “consensus statement”, “clinical practice guideline”, “practice guideline” or “medical guideline” in the title. Two reviewers (CJ, MC) reviewed each document retrieved to ensure they were consensus statements or practice guidelines, using the IOM criteria “statements that include recommendations intended to optimize patient care” [3].

As our primary focus related to evaluating the methodological quality, we opted to use Domain 3 of the AGREE II tool (Rigour of Development) and Domain 6 (Editorial Independence) to assess the documents. The rigour of development domain consists of 8 items, while the editorial independence domain consists of 2 items (items are shown in Table 1). AGREE II items are scored on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). Each domain score was calculated as per the AGREE II instructions included in the user’s manual [5]. Domain score = [score obtained – minimum possible score]/[maximum possible score – minimum possible score × 100], giving a percentage score between 0 and 100. As the Standards and Guidelines Evidence (SAGE) directory has used AGREE II to evaluate English language cancer guidelines released since 2003 [8], if a document had been included in the SAGE database, this appraisal was used and a primary assessment of our own was not performed. The SAGE assessment utilises two trained evaluators to assess each document, discrepancies of a certain magnitude are resolved by a third and if required, fourth evaluator [9].

Table 1. Items from AGREE II (Domains 3 and 6) and additional items collected to assess Transparency of Document Development.

Criteria collected	Source
Rigour of development
Systematic methods were used to search for evidence.
The criteria for selecting the evidence are clearly described
The strengths and limitations of the body of evidence are clearly described.
The methods for formulating the recommendations are clearly described.	Domain 3 of AGREE II [5]
The health benefits, side effects, and risks have been considered in formulating the recommendations.
There is an explicit link between the recommendations and the supporting evidence.
The guideline has been externally reviewed by experts prior to its publication.
A procedure for updating the guideline is provided.
Editorial independence
The views of the funding body have not influenced the content of the guideline.	Domain 6 of AGREE II [5]
Competing interests of guideline development group members have been recorded and addressed.
Additional items to assess transparency of document development
Was a systematic review performed? (yes – systematic review performed anddocumented, no – systematic review not performed or not documented)	IOM [3] JCO [10]
How was the guideline group established? (invited, not disclosed, other),	IOM [3]
Was the group privately funded? (yes, no, not disclosed)	JCO [10].
Was the group multidisciplinary? (yes, no, not disclosed)	JCO [10].
Consensus sponsor
For guidelines where a pharmaceutical product was evaluated was a specific productendorsed in the statement? (yes- name of product, no)
Name and manufacturer of product endorsed

IOM  =  Institute of medicine,

JCO  =  Journal of clinical oncology.

As we also wanted to assess issues surrounding the transparency of document development, and specific to whether or not pharmaceutical company sponsorship of the guideline development process was associated with product endorsement, each document was assess using an additional 7 items. These additional items were derived from the IOM standards for trustworthy clinical practice guidelines [3] and the JCO criteria for publishing consensus statements and clinical practice guidelines [10] (Table 1). These items included a statement on “Was a systematic review conducted?” Additional items related to transparency included, “How was the group established?”, “Was the group multidisciplinary?”, “Was the group privately funded?” and “What was the name of the funding body?” In order to assess any relationship between the sponsor of the group and recommendations, for pharmaceutical-related guidelines we also collected data on “Was a specific product endorsed in statement?”, and if so, “Who was the manufacturer of product?”.

Six reviewers appraised the documents, with each document appraised by two independent reviewers (see Acknowledgements). Discrepancy scores between reviewers for AGREE II were calculated using the concordance calculator for the SAGE database calculations [8]. We planned to resolve discrepancies in assessments as per SAGE, by third and if necessary fourth evaluators. For the additional items assessed, any discrepancies between the two reviewers were resolved by consensus.

Statistical analysis

For the two AGREE II domains of interests, we reported overall means with their 95% confidence intervals for each journal, stratified into separate categories of consensus statement and clinical practice guideline. We also stratified by year of document publication. We used the publication date of the IOM ‘Clinical practice guidelines we can trust’, March 2011 [3], as a time point in which to assess document quality over time. We compared overall differences between journals and between consensus statement or clinical practice guideline using analysis of variance (ANOVA). We also calculated the mean difference in scores between consensus statement and clinical practice guidelines with their corresponding 95% confidence intervals.

For the additional items collected addressing transparency of document development, we calculated the proportion of responses categorized as “Yes”, “No”, and “Not Reported”. We assessed for differences in the journals’ assessments using a chi-square test (or Fisher’s Exact test when dealing with small cell counts in summary contingency tables) at a significance level of 5% while stratifying analyses into categories of consensus statement and clinical practice guideline. Finally, we compared overall items responses according to their consensus statement or clinical practice guideline category.

Agreement between reviewers was assessed by a concordance calculator, determining the number of standard deviations between reviewers, over each domain. A ‘high’ discrepancy score occurred when greater than 2 standard deviations were present between reviewers, ‘medium’ if >1.5 but <2 standard deviations and ‘low’ if <1.5 standard deviations.

Results

Identified Literature

The search identified a total of 104 documents for review. Three were excluded as one was a physician survey, one was a review of guidelines, and one was a letter to the editor. Therefore, 34 consensus statements and 67 practice guidelines were retained for assessment. The numbers and types of documents for each journal were; CO-14 consensus statements, 24 clinical practice guidelines, EJC -9 consensus statements, 13 clinical practice guidelines and JCO-11 consensus statements, 30 clinical practice guidelines.

AGREE II Rigour of development scores

When assessed across all three journals (Figure 1, Table 2), the mean scores for consensus statements were 32% (95% CI 27–38%) and for clinical practice guidelines 64% (95% CI 59–69%). The mean difference between guidelines was 32% (p<0.0001), indicating that clinical practice guidelines were scored significantly higher than consensus statements in terms of rigour of development. Analyses stratified by journal showed that rigour of development scores were significantly lower for consensus statements than clinical practice guidelines for manuscripts published in CO (31% [95% CI 21–40%] consensus statements, 70% [95% CI 61–79%] clinical practice guidelines) and JCO (30% [95% CI 19–41%] consensus statements, 68% [95% CI 64–72%] clinical practice guidelines). There was no significant difference between manuscripts published in EJC (36% [95% CI 28–45%] consensus statements, 46% [95% CI 32–60%] clinical practice guidelines). When comparing each journal with the others, all three had similar scores for consensus statements; however EJC clinical practice guidelines scored lower than Current Oncology (EJC 46% [95% CI 32–60%], CO 70% [95% CI 61–79%]) and JCO (68% [95% CI 64–72%]). Discrepancy levels between the reviewers were low with the exception of one consensus statement published in the Journal of Clinical Oncology [11] which had a high discrepancy score.

Figure 1. Range and 95% confidence intervals for Rigour of development scores.

CO = Current Oncology. EJC = European Journal of Cancer. CS = Consensus statements. JCO = Journal of Clinical Oncology. CPG = Clinical practice guidelines.

Table 2. AGREE II: Rigour of development and Editorial Independence.

	CO	EJC	JCO	Overall	p-value, difference between means
AGREE II: Rigour of development (Domain 3)
Consensus Statement (n = 34)
Mean (95% Confidence Interval)	31 (21, 42)	36 (28, 45)	30 (19, 41)	32 (27, 38)	0.6400
Clinical Practice Guideline (n = 67)
Mean (95% Confidence Interval)	70 (61, 79)	46 (32, 60)	68 (64, 72)	64 (59, 69)	0.0006
Mean difference Consensus Statement vs Clinical Practice Guideline					32 (24, 40)
Overall p-value Consensus Statement vs Clinical Practice Guideline					<0.0001
AGREE II: Editorial Independence (Domain 6)
Consensus Statement (n = 34)
Mean (95% Confidence Interval)	50 (38, 62)	44 (34, 54)	63 (56, 70)	53 (47, 59)	0.0305
Clinical Practice Guideline (n = 67)
Mean (95% Confidence Interval)	75 (63, 86)	59 (52, 67)	66 (61, 70)	68 (63, 73)	0.0564
Mean difference Consensus Statement vs Clinical Practice Guideline					15 (7, 23)
Overall p-value Consensus Statement vs Clinical Practice Guideline					0.0003

AGREE II Editorial independence scores

When assessed across all three journals (Figure 2, Table 2), the mean score for consensus statements was 53% (95% CI 47–59%) and for clinical practice guidelines was 68% (95% CI 63–73%). The mean difference between consensus statement and clinical practice guideline scores was 15% (p  = 0.0003), indicating that clinical practice guidelines were scored significantly higher than consensus statements with respect to editorial independence. Editorial independence scores were significantly lower for consensus statements than clinical practice guidelines in documents published in CO (50% [95% CI 38–62%] consensus statements, 75% [95% CI 63–86%] clinical practice guidelines). This difference seen to a lesser extent in EJC (44% [95% CI 34–54%] consensus statements, 59% [95% CI 52–67%] clinical practice guidelines) and no difference was seen in JCO (63% [95% CI 56–70%] consensus statements, 66% [95% CI 61–70%] clinical practice guidelines). EJC (44% [95% CI 35–54%]) scored lower than JCO (63% [95% CI 56–70%]) on consensus statements, but similarly to CO. No journal appeared to perform better or worse than the other journals with regard to clinical practice guidelines. Discrepancy levels between the reviewers were low for all documents.

Figure 2. Range and 95% confidence intervals for Editorial independence scores.

CO = Current Oncology. EJC = European Journal of Cancer. CS = Consensus statements. JCO = Journal of Clinical Oncology. CPG = Clinical practice guidelines.

Additional transparency of document development item scores

Consensus statements infrequently referenced or conducted a systematic review on the topic of the guideline (6/34 = 18%), a step which was much more common with clinical practice guidelines (56/67 = 83%) (p = 0.018) (Table 3). The largest discrepancy was seen in JCO where 0/11 (0%) of consensus statements documented a systematic review compared to 28/30 (93%) of clinical practice guidelines. Neither consensus statements (50%) nor clinical practice guidelines (34%) consistently declared how their development group was established. Consensus statements were more likely than clinical practice guidelines to state that participants were “invited” (12/34 = 35% vs 14/67 = 21%; p = 0.01). Guideline groups were multidisciplinary in 21 out of 34 (62%) consensus statements and 50 out of 67 (75%) clinical practice guidelines groups. Group member roles were not declared in 35% (12/34) of the consensus statements nor in 25% (17/67) of clinical practice guidelines (p = 0.19).

Table 3. Additional items addressing Transparency of Document Development.

	CO n (%)	EJC n (%)	JCO n (%)	Overall n (%)	p-value
Systematic review performed
Consensus Statement yes (n = 34)	3 (21)	3 (33)	0	6 (18)	0.1350
Clinical Practice Guideline yes (n = 67)	21 (88)	7 (54)	28 (93)	56 (84)	0.0082
Overall Consensus Statement vs Clinical Practice Guideline difference					<0.0001
How groups were established
Consensus Statement (n = 34)
Invited	6 (43)	5 (56)	1 (9)	12 (35)	0.1440
Not reported	6 (43)	4 (44)	7 (64)	17 (50)
Other	2 (14)	0	3 (27)	5 (15)
Clinical Practice Guideline (n = 67)
Invited	4 (17)	5 (39)	5 (17)	14 (21)	0.0378
Not reported	7 (29)	7 (54)	9 (30)	23 (34)
Other	13 (54)	1 (8)	16 (53)	30 (45)
Overall Consensus Statement vs Clinical Practice Guideline difference					0.0106
Multidisciplinary
Consensus Statement (n = 34)
Yes	8 (57)	7 (78)	6 (55)	21 (62)	0.7182
No	1 (7)	0	0	1 (3)
Not reported	5 (36)	2 (22)	5 (46)	12 (35)
Clinical Practice Guideline (n = 67)
Yes	19 (79)	8 (62)	23 (77)	50 (75)	0.4716
No	0	0	0	0
Not reported	5 (21)	5 (39)	7 (23)	17 (25)
Overall Consensus Statement vs Clinical Practice Guideline difference					0.1857
Privately funded meeting
Consensus Statement (n = 34)
Yes	9 (64)	2 (22)	0	11 (32)	<0.0001
No	1 (7)	0	0	1 (3)
Not reported	4 (29)	7 (78)	11 (100)	22 (65)
Clinical Practice Guideline (n = 67)
Yes	1 (4)	5 (39)	0	6 (9)	<0.0001
No	15 (63)	0	16 (53)	31 (47)
Not reported	8 (33)	8 (62)	14 (47)	30 (45)
Overall Consensus Statement vs Clinical Practice Guideline difference					<0.0001
Consensus sponsors’ product endorsed
Consensus Statement (n = 34)
Yes	9 (64)	1 (11)	0	10 (24.4)	<0.0001
Clinical Practice Guideline (n = 67)
Yes	1 (4)	1 (8)	0	2 (3)	0.3012
Overall Consensus Statement vs Clinical Practice Guideline difference					<0.0001

While consensus statements were more likely to declare private funding (11/34 = 32%) than clinical practice guidelines (6/67 = 9%) (p<0.0001), many documents did not declare their source of funding (22/34 = 65% of consensus statements versus 31/67 = 46% of clinical practice guidelines). If a source of funding was declared, the funding body was recorded (Table 3).

With respect to whether or not a document endorsed a product made by the sponsoring company (Table 3), this occurred less frequently in clinical practice guidelines (2/67 = 3%) than in consensus statements (10/34 = 29%) (p<0.0001). In CO, consensus statements endorsed the product of the sponsoring company in 9/14 (64%) of cases. All of these documents declared financial support from the sponsoring company, but none explicitly declared the link between the sponsoring company and the product endorsed. Four percent of clinical practice guidelines published in CO endorsed the sponsor’s product. This trend was seen to a lesser extent in EJC with 11% of consensus statements endorsing sponsors products and 8% of clinical practice guidelines. No document published by JCO documented a relationship between pharmaceutical company funding and product endorsement in the guideline.

Have consensus statements and clinical practice guidelines improved over time?

When assessed chronologically, there is no association with document quality over time, using the date of publication of the IOM ‘Clinical practice guidelines we can trust’, March 2011 as a reference point (Tables 4,5 and 6). There may be a trend of declining pharmaceutical sponsorship of documents in recent years.

Table 4. Current Oncology Consensus Statements and Clinical Practice Guidelines.

Current Oncology
Consensus Statements
Paper	Year published			Pharma sponsored		AGREE Domain 3 (%)		AGREE Domain 6 (%)				Sponsors product endorsed
Recommendations of the Canadian Consensus Group onthe Management of Chronic Myeloid Leukemia [12]	2006			✓		50		50				✓
Updated recommendations from the Canadian Nationalconsensus meeting on HER2/neu testing in breast cancer[13]	2007			✓		7		17				✓
Colorectal Cancer Association of Canada ConsensusMeeting on Practice Guidelines - Raising the Standard ofCare in Canada for Early Stage Rectal Cancer [14]	2009			✓		22		64				✓
The role of the Epidermal Growth Factor Receptor TyrosineKinase Inhibitors as Therapy for Advanced, Metastatic andRecurrent Non-Small Cell Lung Cancer: A Canadian NationalConsensus Statement [15]	2009			✓		67		46				✓
Consensus recommendations for the use of anti-EGFRtherapies in metastatic colorectal cancer [16]	2010					6		54
Eastern Canadian Colorectal Cancer ConsensusConference: setting the limits of resectable disease [17]	2010			✓		32		21				✓
Consensus recommendations for the diagnosis andmanagement of well-differentiated gastroenterohepaticneuroendocrine tumours: a revised statement from aCanadian national expert group [18]	2010			✓		23		79				✓
Diagnosis and management of hepatocellular carcinoma:results of a consensus meeting of The Ottawa Hospital Cancer Centre [19]	2010			✓		33		25				✓
Consensus Statements pre IOM 2011 (n = 8), Mean (95% Confidence Interval)	30 (16, 44)			45 (29, 60)
Multidisciplinary Canadian consensus recommendations forthe management and treatment of hepatocellular carcinoma [20]	2011			✓		56		75			✓
Canadian Expert Group consensus recommendations: KRAStesting in colorectal cancer [21]	2011					19		58
Report from the 13th Annual Western CanadianGastrointestinal Cancer Consensus Conference [22]	2012					43		46
Consensus recommendations for cancer rehabilitation:research and education priorities [23]	2013					14		79
Endocrine therapy for postmenopausal women withhormone receptor–positive her2–negative advancedbreast cancer after progression or recurrence on nonsteroidal aromatase inhibitor therapy: a Canadian consensus statement [24]	2013			✓		40		63			✓
Eastern Canadian Colorectal Cancer ConsensusConference: standards of care for the treatment of patients with rectal, pancreatic, andgastrointestinal stromaltumours and pancreatic neuroendocrine tumours [25]	2013						29		33
Consensus Statements post IOM 2011 (n = 6), Mean (95% Confidence Interval)							34 (21, 46)		59 (45, 73)
Clinical Practice Guidelines
Bortezomib in Multiple Myeloma and Lymphoma: Asystematic review and clinical practice guideline [26]	2006				82				88
Guidelines for the diagnosis and management of carcinoidtumours, part 1: the gastrointestinal tract. A statement from a Canadian National Carcinoid Expert Group [27]	2006				20				13
The role of oxaliplatin combined with 5-fluorouracil andfolinic acid in the first- and second-line treatment of advanced colorectal cancer: a systematic review and clinical practice guideline [28]	2006				74				21
Canadian Recommendations for the Treatment ofGlioblastoma Multiforme [29]	2007			✓	19				17		✓
Melanoma Disease Site Group of Cancer Care Ontario’sProgram in Evidence-Based Care Mot. Temozolomide for the Treatment of MetastaticMelanoma: A PracticeGuideline [30]	2007				80				92
Ifosfamide-based combination chemotherapy in advancedsoft tissue sarcoma: a systematic review and clinical practice guideline [31]	2007				61				96
Management of Single Brain Metastases: A PracticeGuideline [32]	2007				79				92
Alemtuzumab in Chronic Lymphocytic Leukemia: ASystematic Review and Clinical Practice Guideline [33]	2007				81				100
Single-Agent Interleukin-2 in the Treatment of MetastaticMelanoma: A Clinical Practice Guideline [34]	2007				78				100
Biochemotherapy for the Treatment of MetastaticMalignant Melanoma: A Clinical Practice Guideline [35]	2008				43				75
Dose-intensive Chemotherapy with Growth Factor or Autologous Bone Marrow/Stem Cell Transplant Support inFirst-line Treatment of Advanced or MetastaticAdult Soft Tissue Sarcoma – A Clinical PracticeGuideline [36]	2008				89				100
Epidermal growth factor receptor targeted therapy in stageIII and IV head and neck cancer [37]	2010				85				79
Follow-up for women after treatment for cervical cancer[38]	2010				78				79
Clinical Practice Guidelines pre IOM 2011 (n = 13) Mean (95% Confidence Interval)					67 (54, 80)				73 (55, 91)
Canadian College of Medical Geneticists guidelines for theindications, analysis, and reporting of cancer specimens [39]		2011			18				46
Systemic therapy for advanced gastric cancer: a clinicalpractice guideline [40]		2011			84				83
Survivorship services for adult cancer populations: a pan-Canadian guideline [41]		2011			84				96
Invasive mediastinal staging of non-small-cell lung cancer: aclinical practice guideline [42]		2011			75				83
Management of a suspicious adnexal mass: a clinicalpractice guideline [43]		2012			80				63
Lenalidomide in multiple myeloma – a practice guideline[44]		2013			75				83
Chemotherapy (gemcitabine, docetaxel plus gemcitabine,doxorubicin, or trabectedin) in inoperable, locallyadvanced, recurrent, or metastatic uterineleiomyosarcoma: a clinical practice guideline [45]		2013			82				88
Role of endolaryngeal surgery (with or without laser)compared with radiotherapy in the managementof early (T1) glottic cancer: a clinical practiceguideline [46]		2013			74				79
Surgical margins and handling of soft-tissue sarcoma inextremities: a clinical practice guideline [47]		2013			77				100
Liver resection for colorectal cancer metastases [48]		2013			80				83
A pan-Canadian practice guideline and algorithm:screening, assessment, and supportive care ofadults with cancer-related fatigue [49]		2013			75				38
Clinical Practice Guidelines post IOM 2011 (n = 11) Mean (95% Confidence Interval)					73 (62, 84)				77 (65, 88)

Table 5. European Journal of Cancer Consensus Statements and Clinical Practice Guidelines.

European Journal of Cancer
Consensus Statements
Paper	Year published	Pharma sponsored	AGREE Domain 3	AGREE Domain 6	Sponsors product endorsed
EORTC consensus recommendations for the treatment ofmycosis fungoides/Sézary syndrome [50]	2006		43	50
Towards a pan-European consensus on the treatment ofpatients with colorectal liver metastases. [51]	2006		41	38
Consensus conference: Implementing treatmentrecommendations on yttrium-90 immunotherapy inclinical practice – Report of a European workshop [52]	2008	✓	27	42	✓
Diagnosis and treatment of melanoma: Europeanconsensus-based interdisciplinary guideline [53]	2010		39	38
Breast cancer in pregnancy: Recommendations of aninternational consensus meeting. [54]	2010		49	21
Consensus Statements pre IOM 2011 (n = 8), Mean (95% Confidence Interval)			40 (33, 47)	38 (29, 47)
Consensus on Lung Cancer, new clinical recommendationsand current status of biomarker assessment – First-line therapy. [55]	2011		15	38
Highlights of the EORTC St. Gallen International ExpertConsensus on the primary therapy of gastric,gastroesophageal and oesophageal cancer – Differentialtreatment strategies for subtypes of earlygastroesophageal cancer. [56]	2012	✓	28	63
Diagnosis and treatment of melanoma. Europeanconsensus-based interdisciplinary guideline – Update 2012 [57]	2012		35	63
German, Austrian and Swiss consensus conference on thediagnosis and local treatment of the axilla in breast cancer [58]	2013		50	46
Consensus Statements post IOM 2011 (n = 4), Mean (95% Confidence Interval)			32 (18, 46)	53 (40, 65)
Clinical Practice Guidelines
Guidelines for surgical treatment of hepatoblastoma in themodern era–recommendations from the Childhood Liver Tumour Strategy Group of the International Society ofPaediatric Oncology (SIOPEL) [59]	2005		21	46
Malignant ascites: systematic review and guideline for treatment. [60]	2006		63	46
EORTC guidelines for the use of granulocyte-colonystimulating factor to reduce the incidence of chemotherapy-induced febrile neutropenia in adultpatients with lymphomas and solid tumours. [61]	2006	✓	72	50
EORTC guidelines for the use of erythropoietic proteins inanaemic patients with cancer: 2006 update [62]	2007	✓	60	88
Tumour markers in colorectal cancer: European Group onTumour Markers (EGTM) guidelines for clinical use [63]	2007		38	64
Guidelines on the standards for the training of specialisedhealth professionals dealing with breast cancer [64]	2007	✓	0	46
Guidelines for the assessment of oral mucositis in adultchemotherapy, radiotherapy and haematopoietic stem cell transplant patients. European journal of cancer [65]	2008		69	58
Diagnosis and antimicrobial therapy of lung infiltrates infebrile neutropenic patients: Guidelines of the infectiousdiseases working party of the German Society of Haematology and Oncology. [66]	2009		18	58
The development of evidence-based guidelines on mouthcare for children, teenagers and young adults treated for cancer [67]	2010		76	63
2010 update of EORTC guidelines for the use ofgranulocyte-colony stimulating factor to reduce the incidence of chemotherapy-induced febrile neutropeniain adult patients with lymphoproliferative disorders and solid tumours [68]	2011	✓	63	50	✓
The development of evidence-based Europeanguidelines on the management of depression inpalliative cancer care [69]	2011		41	71
Clinical Practice Guidelines pre IOM 2011 (n = 11) Mean (95% Confidence Interval)			47 (32, 62)	58 (51, 66)
Paediatric intestinal cancer and polyposis due to bi-allelic PMS2 mutations: case series, review and follow-up guidelines. European journal of cancer[70]	2011		28	75
EASL-EORTC clinical practice guidelines:management of hepatocellular carcinoma [71]	2012		49	58
Clinical Practice Guidelines post IOM 2011 (n = 2) Mean (95% Confidence Interval)			39 (18, 59)	67 (50, 83)

Table 6. Journal of Clinical Oncology Consensus Statements and Clinical Practice Guidelines.

Journal of Clinical Oncology
Consensus Statements
Paper	Year published	Pharma sponsored	AGREE Domain 3	AGREE Domain 6	Sponsors product endorsed
Use of Positron Emission Tomography for ResponseAssessment of Lymphoma: Consensus of the ImagingSubcommittee of International Harmonization Project inLymphoma. [72]	2007		49	63
Definition, diagnosis, and management of intravascular large B-cell lymphoma: proposals and perspectives from aninternational consensus meeting. [73]	2007		9	67
Consensus Report of the National Cancer Institute ClinicalTrials Planning Meeting on Pancreas Cancer Treatment [74]	2009		7	67
Venous Thromboembolism Prophylaxis and Treatment inCancer: A Consensus Statement of Major Guidelines Panelsand Call to Action [11]	2009		52	71
Definition, Prognostic Factors, Treatment, and ResponseCriteria of Adult T-Cell Leukemia-Lymphoma: A Proposal Froman International Consensus Meeting. [75]	2009		38	67
International Myeloma Working Group Consensus StatementRegarding the Current Status of Allogeneic Stem-CellTransplantation for Multiple Myeloma [76]	2010		23	46
Renal Impairment in Patients With Multiple Myeloma: AConsensus Statement on Behalf of the International Myeloma Working Group [77]	2010		29	42
Hepatocellular Carcinoma: Consensus Recommendations ofthe National Cancer Institute Clinical Trials Planning Meeting [78]	2010		35	67
Future Directions in the Treatment of NeuroendocrineTumors: Consensus Report of the National Cancer Institute Neuroendocrine Tumor Clinical Trials Planning Meeting. [79]	2011		30	67
Consensus Statements pre IOM 2011 (n = 9), Mean (95% Confidence Interval)			30 (20,40)	62 (55,69)
Clinical End Points and Response Criteria in Mycosis Fungoidesand Sézary Syndrome: A Consensus Statement of the International Society for Cutaneous Lymphomas, the UnitedStates Cutaneous Lymphoma Consortium, and the Cutaneous Lymphoma Task Force of the EuropeanOrganisation for Research and Treatment of Cancer. [80]	2011		8	63
Platinum-Induced Ototoxicity in Children: A Consensus Review on Mechanisms, Predisposition, and Protection, Including a New International Society of Pediatric Oncology Boston Ototoxicity Scale. [81]	2012		45	79
Consensus Statements post IOM 2011 (n  = 29), Mean (95% Confidence Interval)			27 (0,63)	71 (55,87)
Clinical Practice Guidelines
American Society of Clinical Oncology GuidelineRecommendations for Sentinel Lymph Node Biopsy inEarly-Stage Breast Cancer. [82]	2005		75	73
Colorectal Cancer Surveillance: 2005 Update of anAmerican Society of Clinical Oncology Practice Guideline [83]	2005		65	79
American Society of Clinical Oncology/College of American Pathologists Guideline Recommendations for HumanEpidermal Growth Factor Receptor 2 Testing in BreastCancer. [84]	2006		82	65
2006 Update of Recommendations for the Use of WhiteBlood Cell Growth Factors: An Evidence-Based Clinical Practice Guideline. [85]	2006		57	77
American Society of Clinical Oncology Clinical PracticeGuideline for the Use of Larynx-Preservation Strategies in the Treatment of Laryngeal Cancer. [86]	2006		65	54
American Society of Clinical Oncology Guideline forAntiemetics in Oncology: Update 2006. [87]	2006		65	64
American Society of Clinical Oncology Guideline:Recommendations for Venous Thromboembolism Prophylaxis and Treatment in Patients With Cancer. [88]	2007		81	63
Cancer Care Ontario and American Society of ClinicalOncology Adjuvant Chemotherapy and Adjuvant RadiationTherapy for Stages I-IIIA Resectable Non–Small-Cell Lung Cancer Guideline [89].	2007		79	88
American Society of Clinical Oncology Endorsement of theCancer Care Ontario Practice Guideline on Nonhormonal Therapy for Men With Metastatic Hormone-Refractory (castration-resistant) Prostate Cancer. [90]	2007		69	92
American Society of Clinical Oncology 2007 ClinicalPractice Guideline Update on the Role of Bisphosphonatesin Multiple Myeloma. [91]	2007		60	75
Initial Hormonal Management of Androgen-SensitiveMetastatic, Recurrent, or Progressive Prostate Cancer:2007 Update of an American Society of Clinical Oncology Practice Guideline. [92]	2007		75	38
American Society of Clinical Oncology 2008 ClinicalPractice Guideline Update: Use of Chemotherapy andRadiation Therapy Protectants. [93]	2009		70	58
Use of 5-alpha-reductase inhibitors for prostate cancerchemoprevention: American Society of Clinical Oncology/American Urological Association 2008 ClinicalPractice Guideline. [94]	2009		76	71
American Society of Clinical Oncology Clinical Practice Guideline Update on Chemotherapy for Stage IV Non–Small-Cell Lung Cancer [95].	2009		71	71
American Society of Clinical Oncology Clinical PracticeGuideline Update on the Use of Pharmacologic Interventions Including Tamoxifen, Raloxifene, andAromatase Inhibition for Breast Cancer Risk Reduction [96].	2009		74	63
American Society of Clinical Oncology/American Society ofHematology Clinical Practice Guideline Update on the Use of Epoetin and Darbepoetin in Adult Patients With Cancer [97].	2010		57	54
American Society of Clinical Oncology Clinical PracticeGuideline: Update on Adjuvant Endocrine Therapy for Women With Hormone Receptor–Positive Breast Cancer[98].	2010		54	54
American Society of Clinical Oncology Clinical PracticeGuideline on Uses of Serum Tumor Markers in Adult Males With Germ Cell Tumors [99].	2010		58	63
American Society of Clinical Oncology/College of AmericanPathologists Guideline Recommendations forImmunohistochemical Testing of Estrogen andProgesterone Receptors in Breast Cancer [100].	2010		65	47
Clinical Practice Guidelines pre IOM 2011 (n = 19) Mean (95% Confidence Interval)			69 (64,72)	66 (60,72)
Antiemetics: American Society of Clinical Oncology ClinicalPractice Guideline Update [101].	2011		72	67
American Society of Clinical Oncology Endorsement of theCancer Care Ontario Practice Guideline on Adjuvant Ovarian Ablation in the Treatment of PremenopausalWomen With Early-Stage Invasive Breast Cancer [102].	2011		85	79
2011 Focused Update of 2009 American Society of ClinicaOncology Clinical Practice Guideline Update on Chemotherapy for Stage IV Non–Small-Cell Lung Cancer[103].	2011		43	58
American Society of Clinical Oncology Clinical PracticeGuideline Update on the Use of Chemotherapy Sensitivity and Resistance Assays [104].	2011		54	58
Sentinel Lymph Node Biopsy for Melanoma: AmericanSociety of Clinical Oncology and Society of SurgicalOncology Joint Clinical Practice Guideline [105].	2012		63	75
Appropriate Chemotherapy Dosing for Obese AdultPatients With Cancer: American Society of Clinical Oncology Clinical Practice Guideline [106].	2012		67	54
Use of Pharmacologic Interventions for Breast Cancer RiskReduction: American Society of Clinical Oncology ClinicalPractice Guideline [107].	2013		82	63
Fertility Preservation for Patients With Cancer: AmericanSociety of Clinical Oncology Clinical Practice GuidelineUpdate [108].	2013		65	71
Central Venous Catheter Care for the Patient With Cancer:American Society of Clinical Oncology Clinical PracticeGuideline [109].	2013		80	58
Breast Cancer Follow-Up and Management After PrimaryTreatment: American Society of Clinical Oncology Clinical Practice Guideline Update [110].	2013		48	71
Antimicrobial Prophylaxis and Outpatient Management ofFever and Neutropenia in Adults Treated for Malignancy: American Society of Clinical Oncology Clinical Practice Guideline [111].	2013		81	71
Clinical Practice Guidelines post IOM 2011 (n = 11) Mean (95% Confidence Interval)			67 (59, 76)	66 (61, 71)

Discussion

As the terms consensus statement and clinical practice guidelines are often used interchangeably and both are used to improve clinical care, their methodological rigour and transparency of development is essential. Here we report the results of a review of the methodological quality of consensus statements and clinical practice guidelines in a limited sample of the oncology literature. While others have published on quality assessment of clinical guidelines in oncology using either the AGREE or AGREE II tool [112]–[117], to our knowledge this is the first such comprehensive review of both consensus statement and practice guidelines in oncology.

As literature assessing the quality of consensus statements is limited [118], we used tools developed for clinical practice guidelines and collected additional information that would help assess the transparency of guideline development. AGREE II is a validated appraisal tool for assessing the methodological development quality and reporting of practice guidelines; it does not assess the actual content of clinical recommendations [5]. AGREE II assesses how well a guideline performs on each of the 6 domains (scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability and editorial independence). We felt the rigour of development (an assessment of the evidentiary base and methods used to formulate recommendations) and editorial independence (an assessment of bias and competing interests influencing recommendation formulation [5]) were the most appropriate for our evaluation.

For both the rigour of development and editorial independence domains, consensus statements scored consistently less well than did practice guidelines. In the only publication we found evaluating practice guidelines in comparison to consensus statements, although not specific to oncology [118], similar differences were seen, with consensus statements scoring significantly lower than clinical practice guidelines across 4 of the 6 AGREE II domains (stakeholder involvement, rigour of development, clarity, and presentation and applicability). We could show no improvement in document quality over time.

Performing a systematic review is an essential element of guideline development [119]. Both IOM [120] and JCO [10] state that “clinical practice guideline developers should use systematic reviews” and that “guidelines/recommendations should be driven by a high level of evidence” respectively. We felt it was necessary to specifically ask ‘was a systematic review performed?’ We asked this question even though AGREE II domain 3.1 assess if ‘systematic methods were used to search for evidence’ (scored on a continuum of whether a guideline reports what databases were searched, the search terms used, the search time periods and the inclusion of a full search strategy). In the current study systematic reviews were performed more frequently by clinical practice guidelines than consensus statements across all three journals. With respect to the processes by which a clinical practice guideline group was established and the role of individual members, this was inconsistently reported. There were however significant differences between these items in consensus statements and clinical practice guidelines.

Of particular interest was the role of the funding body for the development of the guidance document. While no information can be gleaned for whether this association is real or implied, several observations can be made. Overall, consensus statements and clinical practice guidelines published in the three journals studied either did not declare or were not explicit about the funding source for the document (funding source not declared in 65% consensus statements, 45% clinical practice guidelines). For documents with topics related to pharmaceutical products, when the document was sponsored by a pharmaceutical company, documents endorsed the sponsor’s product in both consensus statements (29%) and to a lesser degree in clinical practice guidelines (3%). However, in the CO journal, 64% of consensus statements published endorsed the sponsors product, whereas only 4% of clinical practice guidelines endorsed the sponsors product. Further, this association was not reported in the conflict of interest statement. This absence of reporting contravenes standards published by medical societies [121], [122] and could question the integrity and quality of published guidance documents [123], [124].

We acknowledge a number of study limitations. Although we feel that consensus statements should be subjected to the same rigorous criteria for their development as practice guidelines, the AGREE II tool has not been validated for evaluation of consensus statements [5], [118]. The additional items we included for assessment from the IOM guideline standards and JCO authorship guidance on consensus statements and clinical practice guidelines also have not been validated. Consensus statements and clinical practice guidelines analyzed here may not be representative of all oncology consensus statements and clinical practice guidelines released between January 2005 and September 2013, nor representative of all oncology journals. A brief PubMed search suggests over 900 oncology guidance documents were published in peer-reviewed journals over the same time period, translating to a sample of 11% of these documents. Finally, we chose only three journals from which to sample. Our rational for selecting them was that they commonly publish both consensus statements and clinical practice guidelines, are prominent journals in their locale of origin and are geographically diverse. We appreciate that these journals may not be representative of all oncology journals.

Conclusions

While consensus statements and clinical practice guidelines are developed with slightly different approaches and methods, both are used to inform clinical and policy decisions. As such both documents should be developed using equally rigorous and transparent methods and subjected to high quality standards. Here we have shown that consensus statements score lower than clinical practice guidelines for scores of rigour of development and editorial independence. Consensus statements are also less likely to include a systematic review of the literature and were more likely to be sponsored by a pharmaceutical company and to endorse a specific pharmaceutical product. Unfortunately transparency of document development was generally poor in both types of documents and there was infrequent documentation of sources of funding, how guideline groups were established and who comprised their guideline development groups.

Given the important role of guidance we feel that both consensus statements and clinical practice guidelines should be subject to the same rigorous and high quality development criteria. We suggest that journals encourage authors of guidance documents to use the AGREE II and IOM criteria when developing their documents and require journal reviewers to use these same criteria when undertaking their peer-review of these documents. While there are quality differences between each of the journals sampled in our study, this was most pronounced around the issues of private funding and product endorsement. Readers of guidance documents published within these journals should be made aware of the presence of private funding and sponsorship should be made transparent through their reporting so that readers can acknowledge such conflicts and potential bias.

Data Availability

The authors confirm that all data underlying the findings are fully available without restriction. All relevant data are within the paper.

Funding Statement

IG received funding from the Canadian Institutes of Health Research (CHIR) number; FRN:IOP 123548 (http://www.cihr-irsc.gc.ca/e/193.html). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.