Abstract
AGREE II is a widely used standard for assessing the methodological quality of practice guidelines. This article describes the development of the AGREE Reporting Checklist, which was designed to improve the quality of practice guideline reporting and aligns with AGREE II in its structure and content.
Summary points
AGREE II has become an internationally accepted standard for evaluation of the methodological quality of clinical practice guidelines
AGREE II items and criteria have been reformatted to create a reporting guide called the AGREE Reporting Checklist
The AGREE Reporting Checklist is intended to improve the comprehensiveness, completeness, and transparency of reporting in practice guidelines
The AGREE Reporting Checklist can be used by practice guideline developers, guideline users, funders, peer reviewers, and journal editors
The international Appraisal of Guidelines, Research and Evaluation (AGREE) research team developed a tool to assess the methodological quality of practice guidelines—the original was released in 2003 (AGREE),1 and the revised and updated version in 2009 (AGREE II).2 AGREE II has become a widely used standard for evaluating the methodological quality and transparency of practice guidelines internationally.3 It was also designed to inform development and reporting requirements for practice guidelines, but these functions have often been disregarded, in part because its formatting and presentation are geared more towards use as an evaluation tool. Similarly, other tools and methods designed to guide the development of high quality practice guidelines (for example, GRADE,4 IOM Standards,5 G-I-N Standards,6 and Guidelines 2.07) are not designed specifically to guide the reporting of practice guidelines and are not generally used as such.
The EQUATOR Network library (www.equator-network.org), a database of health research reporting guides, does not include any reporting checklists or guides specific to the reporting of practice guidelines. In 2003 Shiffman and colleagues released a proposed reporting framework for practice guidelines called the Conference on Guideline Standardization (COGS) checklist, based on existing resources and consensus of the COGS panel.8 This was a fundamental step forward and an asset to the guideline enterprise in providing a minimum reporting dataset for practice guidelines. However, the COGS checklist does not include key items included in the AGREE II evaluation tool that guideline users, guideline developers, and researchers have more recently deemed to be important quality aspects of practice guidelines, and its consensus process included participants from the United States, Canada, and United Kingdom only. For these reasons, the AGREE research team has developed a resource based on AGREE II, called the AGREE Reporting Checklist, specifically to facilitate reporting of practice guidelines. The AGREE Reporting Checklist is a contemporary resource based on a comprehensive review of the literature and consensus among a wider international team of practice guideline stakeholders. This article introduces the AGREE Reporting Checklist, describes its development, and outlines how it can be used.
AGREE Reporting Checklist development methods
To create the AGREE Reporting Checklist, we used the health research reporting development standards proposed by Moher and colleagues.9 Through the process of creating the original AGREE instrument and AGREE II and testing of the AGREE Reporting Checklist draft, all of the requirements in these standards were accomplished.9
The content for the checklist originated from the item generation and reduction stages used to create the original AGREE instrument. Here, a small working group of practice guideline experts reduced a list of 82 candidate items related to quality of practice guidelines, derived from existing appraisal instruments and relevant literature, to 34 items, and these were reduced further with feedback from international experts.1 10 Field testing and validity testing led to the development of the original 23 item AGREE instrument; each item had a definition and specific evaluation criteria to reflect the components of the concept.
In the development of AGREE II, modifications were made and an international sample of users and developers of practice guidelines rated all items as important determinants of the quality and usefulness of a guideline (study 111) and concluded that the instructions were appropriate and easy to apply (study 212). Moreover, the AGREE II domains were also found to be positive significant predictors of the overall quality of practice guidelines, and, with the exception of one, the domains were positive significant predictors of the endorsement of practice guidelines and users’ intention to use the recommendations.11
Across these studies, more than 200 international users and developers of practice guidelines participated. Together, these data show the importance and utility of each of the AGREE II items. For the concepts behind the AGREE II items to be evaluable, a practice guideline developer keen on creating a high quality guideline would need to consider how to embed the ideas in the development process and how to report the information in the guideline itself. Thus, using the content of AGREE II as its foundation, we created the first draft of the AGREE Reporting Checklist. We adapted the evaluation criteria for each item in AGREE II to create reporting criteria. We excluded redundant information and refined the language to match the style of a reporting guide.
To ensure the transferability and adaptability of the AGREE content from an evaluation style manual to a reporting checklist, 15 people with varied experience in developing practice guidelines evaluated the AGREE Reporting Checklist. On a five point scale, participants agreed that its structure was logical (mean=4.6), that its layout allowed for easy application (4.3), and that it facilitated the inclusion of information considered important in the reporting of practice guidelines (4.6). Thirteen respondents reported that they would use the AGREE Reporting Checklist, 14 reported that the appropriate amount of detail had been included in its items, 13 indicated that it would be useful for both new and experienced practice guideline developers, and most reported it would help to remind them about important details to include in their documents.
Scope
The AGREE Reporting Checklist incorporates the content of AGREE II to outline the reporting standards needed to achieve a high quality practice guideline. It comprises 23 items (each with specific reporting criteria) in six domains (see web appendix); the structure and design align with AGREE II. Authors of practice guidelines can use the AGREE Reporting Checklist prospectively during the drafting and final editing stage, to ensure that all necessary information is included, and retrospectively after the guideline is completed, as a quality assurance step. Organizations that develop practice guideline are encouraged to refer to the AGREE Reporting Checklist in their manuals and protocols. Practice guideline funders and research institutions may promote or mandate adherence to the AGREE Reporting Checklist as a means to optimize transparency of required guideline content and to reduce duplication. Finally, journal editors could use the AGREE Reporting Checklist to recommend reporting expectations of submitted manuscripts. The AGREE Reporting Checklist is sufficiently universal that it can be used by practice guideline stakeholders regardless of the more specific protocols or methods used to support guideline development (for example, GRADE,4 IOM Standards,5 G-I-N Standards,6 Guidelines 2.07).
Availability
The AGREE Reporting Checklist is available on the AGREE Enterprise website, a free and open access resource to support the practice guideline field (www.agreetrust.org), and it will be submitted to the EQUATOR Network website (www.equator-network.org/). The AGREE Enterprise website includes access to all AGREE related products, information about AGREE related research activities, the Guidelines Resource Centre, and the My AGREE PLUS interactive online guideline appraisal platform. EQUATOR is an international initiative that seeks to improve the value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guides.
Web Extra.
Extra material supplied by the author
Appendix
Contributors: MB, KK, and KS developed the AGREE Reporting Checklist. KK coordinated the external consultation process, analyzed the data, and wrote the first draft of the manuscript. MB and KS made revisions to the draft manuscript, and all authors, including members of the AGREE Next Steps Consortium, approved the final version. Members of the AGREE Next Steps Consortium include George Browman, Jako Burgers, Françoise Cluzeau, Dave Davis, Gene Feder, Beatrice Fervers, Ian Graham, Jeremy Grimshaw, Steven Hanna, Michelle Kho, Peter Littlejohns, Julie Makarski, and Louise Zitzelsberger. MB is the guarantor.
Funding: Development of the AGREE Reporting Checklist was supported by grants from the Canadian Institutes of Health Research (CIHR #125987).
Competing interests: All named authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: support from the Canadian Institutes of Health Research for this submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Research ethics approval for the development of the AGREE II, on which the AGREE Reporting Checklist was based, was provided by the Hamilton Health Sciences/McMaster University Research Ethics Board (#05-413). Research ethics approval was not obtained for the AGREE Reporting Checklist review survey because completion of the survey posed minimal risk to participants.
Provenance: Development of an instrument to appraise the quality of clinical practice guidelines was initiated in the 1990s because no suitable tool existed.10 Following a literature review and psychometric property testing, the original AGREE instrument was published.1 Further development was needed because the scale of the original AGREE instrument did not align with measurement standards, no construct validity testing had been done, and supporting documentation was limited. Additional validity testing, a reliability analysis, and user testing led to the publication of a revised instrument, AGREE II.2 3 11 12 The development of the AGREE Reporting Checklist built on the content of AGREE II to refine the structure to align with reporting goals.
References
- 1. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003;12:18-23. 10.1136/qhc.12.1.18. 12571340. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Brouwers MC, Kho ME, Browman GP, et al. AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010;182:E839-42. 10.1503/cmaj.090449. 20603348. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Makarski J, Brouwers MC. AGREE Enterprise. The AGREE Enterprise: a decade of advancing clinical practice guidelines. Implement Sci 2014;9:103. 10.1186/s13012-014-0103-2. 25123781. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64:383-94. 10.1016/j.jclinepi.2010.04.026. 21195583. [DOI] [PubMed] [Google Scholar]
- 5.Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, eds. Clinical practice guidelines we can trust. National Academies Press, 2011, available at http://iom.nationalacademies.org/reports/2011/clinical-practice-guidelines-we-can-trust.aspx. [PubMed]
- 6.Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P. Board of Trustees of the Guidelines International Network. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med 2012;156:525-31. 10.7326/0003-4819-156-7-201204030-00009. 22473437. [DOI] [PubMed] [Google Scholar]
- 7.Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ 2014;186:E123-42. 10.1503/cmaj.131237. 24344144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM. Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003;139:493-8. 10.7326/0003-4819-139-6-200309160-00013. 13679327. [DOI] [PubMed] [Google Scholar]
- 9.Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med 2010;7:e1000217. 10.1371/journal.pmed.1000217. 20169112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Cluzeau FA, Littlejohns P, Grimshaw JM, Feder G, Moran SE. Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 1999;11:21-8. 10.1093/intqhc/11.1.21. 10411286. [DOI] [PubMed] [Google Scholar]
- 11.Brouwers MC, Kho ME, Browman GP, et al. AGREE Next Steps Consortium. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010;182:1045-52. 10.1503/cmaj.091714. 20513780. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Brouwers MC, Kho ME, Browman GP, et al. AGREE Next Steps Consortium. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010;182:E472-8. 10.1503/cmaj.091716. 20513779. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix