Evaluating Guidelines

Abstract

Background

Guidelines of high methodological quality make an essential contribution to the quality assurance of medical knowledge. The detailed evaluation of guideline quality is a complex and time-consuming task. The answers to a few key questions generally suffice for an initial, rapid assessment of the quality and utility of a guideline.

Method

We selectively searched the pertinent literature for guideline-assessing instruments and analyzed selected ones with respect to their target group, purpose, orientation, and comprehensiveness. We identified key questions from brief instruments that can be used to assess guideline quality rapidly.

Results

A comparison of ten instruments revealed that most were designed to provide a highly detailed assessment of guideline quality. Four recently developed instruments enable a rough and rapid assessment. They focus, in essence, on four key questions: Was the evidence analyzed systematically? Does the evidence support the recommendations? Is the goal of the guideline formulated, and are the authors named? Is the organization of the guideline easy to follow, and are the recommendations clearly signposted?

Conclusion

Alongside the comprehensive instruments for assessing guidelines, such as DELBI and AGREE II, rapid-assessment instruments are a convenient tool for gaining a quick impression of the value of a guideline.

Guidelines, as defined by the Institute of Medicine (IOM), are aids to decision-making that have been systematically developed for the management of specific problems in medicine. Their purpose is to improve patient care (1, 2). The Association of Scientific Medical Societies in Germany (Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften, AWMF) classifies guidelines into three categories: S1 expert recommendations contain treatment recommendations developed by informal consensus of a group of experts, S2 guidelines are created by formal consensus-finding and/or a formal search for evidence, and S3 guidelines contain all of the elements of systematic development (3). Guidelines are now meeting with increasing acceptance among German physicians and are thus making a major contribution to the transfer of scientific knowledge into clinical practice (4, 5).

As the number of guidelines increases, so, too, does the amount of information that physicians must contend with. The length of the guidelines and the numerous, detailed recommendations found in them are not the only challenge. The broader the discipline, the more extensive the information that the relevant guidelines will contain, and that the individual specialist will have to master. Moreover, conflicting recommendations in the guidelines of overlapping disciplines commonly pose an additional barrier to guideline implementation (6).

Guidelines can only be successful if they are generated by proper methods and if they are of sufficiently high quality in matters of specialized expertise and content. According to the Appraisal of Guidelines for Research and Evaluation (AGREE) Collaboration, the key features of a good guideline are external validity, internal validity, and applicability in practice (7).

Internal validity means that any biases potentially affecting the recommendations were minimized in the guideline-developing process. This pertains, for example, to the mode of data acquisition and the way the data are presented (formulation of key questions and target variables; literature search and selection), the assessment of the evidence, and the use of the evidence to generate recommendations.

External validity means that the guideline actually leads to the intended improvement in patient care.

The applicability of guideline recommendations in practice is affected by further factors, including pilot testing, pre-implementation review, clarity rather than ambiguity in the formulation of recommendations, and the format of the guideline itself. Many manuals and instruments can be used for help in generating guidelines and assessing their quality (8– 13). Even though it was recently shown that the quality of guidelines in various countries has not appreciably improved over time (14, 15), active clinicians still need reliable guidelines and intelligible short versions (16).

There is international agreement on the elements influencing guideline quality (1, 12). The main ones are (17):

• the composition of the author group

• patient participation

• the identification and formulation of clinical questions

• systematic searching for, and assessment of, evidence

• the process of formulating guideline recommendations

• an external review of the guideline

• the guideline-updating process.

The goal of this article is to compare the main currently available instruments for assessing the quality of guidelines with respect to their comprehensiveness, orientation, and relevance to clinical practice. Rapid-assessment instruments such as the mini-checklist (MiChe) published by the authors (18) will be presented here to illustrate what practicing clinicians should look for when assessing guidelines, and what the key criteria are that should be used to tell good and bad guidelines apart without difficulty. Readers armed with this knowledge should be able to assess the utility and quality of a guideline rapidly.

Method

In January 2010, as an in initial step in the development of the rapid-assessment instrument MiChe, a systematic search was performed in guideline directories and bibliographic databases for instruments to be used in the creation and assessment of guidelines. Assessment criteria for MiChe were selected on the basis of those most commonly found in the retrieved instruments, as well as by questioning of guideline experts (18).

To identify current assessment instruments, relevant publications that appeared from January 2010 to February 2015 were selectively sought both in PubMed (with the “related citations” function, on the basis of publications retrieved by the initial search) and on the Internet pages of guideline-creating bodies and the AGREE Working Group. All of the retrieved references were examined by a reviewer (WB or TS). All articles that concerned new instruments for methodological assessment and that were written in either English or German were used for this review.

To characterize the assessment instruments, we extracted and compared information on their potential users, their purpose, and their orientation and extent. The main criteria to be considered in the rapid assessment of a guideline were determined by an analysis of rapid-assessment instruments, with a comparison of the key questions contained in each.

Results

Instruments for assessing guidelines

A literature search in January 2010 yielded 38 potentially relevant publications, 16 of which concerned instruments for assessing the methodological quality of guidelines (7, 19– 25, e1– e8). An update of the search in February 2015 yielded a further 12 potentially relevant publications; among these, four were selected for consideration in this review—an updated version of the already existing AGREE II assessment instrument (9, 26, 27) and three new instruments (28– 30) (Figure).

Figure.

Literature search and selection process

An assessment instrument was selected for further consideration when it was stated in the relevant publication that the instrument was intended for broad, public application, and that it could be used to assess guidelines systematically. Instruments that had been developed by individual research groups for their own use (e1, e2, e7) or that contained no assessment system or scale (e3– e6, e8) were excluded, as were two earlier versions of the assessment instruments AGREE II (9, 26, 27) and DELBI (19) (AGREE [7] and Helou et al. [21], respectively).

The main features of the ten remaining instruments and of the MiChe are shown in (Table 1 Most are intended to provide a highly detailed assessment of guideline quality, as evidenced by the multiplicity of assessment criteria and the use of multilevel assessment scales. These instruments are most suitable for use by guideline developers, decision-makers in health care policy, and organizations in the health sector. In recent years, however, a number of simple instruments have been independently developed for rapid assessment by physicians, with the purpose of providing a quick and informative overview. Aside from the MiChe (18), these include the iCAHE Guideline Quality Checklist (30), the Global Rating Scale (GRS) of the AGREE Collaboration (28), and the surgeons’217; checklist by Coroneos et al. (29).

Table 1. The instruments for guideline assessment that are included in the present study.

Instrument	Target group	Scope	Orientation/Application
Complex instruments
AGREE II 2010 (9, 26, 27)	Physicians, guideline developers, health-policy decision-makers, health-care organizations, teachers of medical students	23 items/6 domains; 7-level scale; ≥ 2 assessors	Detailed GL assessment (correctness & transparency of method) GL development
Cluzeau et al. 1999 (20)	Scientists, physicians, managers in the health-care sector	37 items; 3-level scale	Assessment of systematic GL creation comparison of multiple GL
DELBI 2008 (19)	Guideline developers, health-policy decision-makers, teachers of medical students, physicians	36 items/8 domains; 4-level scale; ≥ 2 assessors	Detailed GL assessment (correctness & transparency of method) GL development
IOM 1992 (23)	AHQR, guideline developers	46 items; 3-level scale	Standardized GL assessment (for teaching, nationwide GL implementation)
Liddle et al. 1996 (22)	Guideline developers	14 items/3 sections; 6-level scale	Checking of GL against the NHMRC requirements
Reed et al. 2002 (24)	APA members (psychiatrists, psychologists), guideline developers	33 items; 3-level scale	APA rules for the development and assessment of GL
Shaneyfelt 1999 (25)	Research institutions (specialists in research methods)	25 items; binary system; 2 assessors	Detailed GL assessment (correctness & transparency of method)
Rapid-assessment instruments
AGREE II-GRS 2012 (28)	Practicing physicians	4 items; 7-level scale	Initial assessment
Coroneos et al. 2014 (29)	Surgeons	8 items	Initial assessment – applicability
iCAHE 2014 (30)	Hospital-based physicians, decision-makers in health care policy	14 items; binary system	Initial assessment
MiChe 2014 (18)	Practicing physicians, hospital-based physicians, medical students	8 items; 3-level scale	Initial assessment

AGREE: Appraisal of Guidelines for Research and Evaluation; AHQR: Agency for Health Care Research and Quality; APA: American Psychological Association; DELBI: Deutsches Leitlinien-Bewertungs-Instrument; GRS: Global Rating Scale; iCAHE: International Centre for Allied Health Evidence; IOM: Institute of Medicine; GL: guideline(s); MiChe: Mini-Checklist; NHMRC: National Health and Medical Research Council

Key criteria in rapid-assessment instruments

Instruments for rapid assessment were developed because it was found that guideline assessment with the more comprehensive instruments, such as AGREE II, was often too time-consuming for clinical practice. The authors of the rapid-assessment instruments stated that they intended them to be used directly by physicians in hospitals or in private practice, where little time is available and staff face high demands. They are supposed to be complementary to the complex instruments, not a replacement for them.

For this purpose, the authors of these instruments sought to formulate a few, broad key questions to provide an overall picture of the methodological quality and applicability of guidelines, as follows:

• The AGREE GRS contains four assessment criteria (process of development, presentation, completeness of the report, clinical validity) subsuming a total of nine key questions.

• The iCAHE checklist contains 14 criteria covering six areas (availability, dates, underlying evidence, developers, purpose and target group, simplicity).

• The checklist of Coroneos et al. contains eight questions covering three areas (validity, development, and implementation in practice).

• The MiChe contains eight criteria that are not categorized under any broader headings.

The questions and criteria in each of the rapid-assessment instruments are presented for comparison in (Table 1, displayed according to their varying methodological properties (quality of guideline creation, quality of report, quality of presentation, quality of underlying evidence) and applicability. A number of criteria were found to be common to most of the instruments. From these, we derived four key questions that physicians should ask when they want to assess guidelines rapidly. The answers can be used to tell good guidelines apart from bad ones:

• Was the evidence analyzed systematically? A positive assessment means that a comprehensive, systematic literature search was carried out in electronic databases and other sources, that the report indicates the applied inclusion and exclusion criteria for studies, and that the method of study selection is intelligibly described.

• Is it clear what evidence the recommendations are based on? A positive assessment means that the connection between the evidence and the recommendations is clear, and evidence levels and recommendation grades are indicated.

• Is information provided about the developers of the guideline and its purpose? A positive assessment means that the members of the developer group are named, and that the relevant group of patients and the circumstances of their treatment are well presented.

• Is the organization of the guideline easy to follow, and are the recommendations clearly marked as such? A positive assessment means that guideline users can find what they are looking for quickly and easily, and that the recommendations are signposted to stand out visually.

A more detailed presentation of the eight key criteria of the MiChe is found in the eTable.

eTable. Explanation of the key criteria of the Mini-Checklist (MiChe)^*.

Feature to be assessed	Explanation
Intelligibility: The guideline is organized in such a way that it is generally easy to understand, and the key recommendations are easy to identify.	Guidelines are intended for use in everyday clinical practice and must therefore present the necessary information clearly and in such a way that it can be rapidly found. Their structure and organization should be based on those of the medical decision-making process (for example, clearly broken down into steps such as screening, prevention, diagnosis, treatment …) so that users can navigate through them with ease. Key recommendations on the main clinical questions should be made clearly recognizable as such by special signposting. The guideline should include a summary of the key recommendations.
Audience: The target users are clearly defined, as are the target situations in which the guideline is to be applied.	Guidelines vary in their audiences: they can be addressed to a broad group of medical and non-medical users, or else to a specific group of professionals or a specific target situation in health care. There is corresponding variation in the scope of applicability of the recommendations that they contain. To be useful, a guideline should therefore contain a clear statement and description of its target users and situations (e.g., physicians in particular specialties, inpatient vs. outpatient care, non-medical personnel).
Background, purpose, target group of patients: The background and purpose of the guideline and the patients for whom it is to be applied are clearly defined.	The guideline should contain a discussion of the background (prevalence, care situation, risk factors) of the medical problems that it addresses, its purpose, and its potential effects on the care situation (efficacy, cost-efficiency, practicality of implementation). Not only the target users, but also the target group of patients varies from one guideline to another, ranging from the entire population or a large part of it to a particular patient group or a single disease stage within a particular patient group. The implementation of the guideline can affect these patients’217; state of health and the quality of their care. The description of the group(s) of patients to which the guideline is to be applied should include clear information about their age, sex, type and severity of disease, and concomitant illnesses.
Conflicts of interest: The developers of the guideline are all identified by name. Their conflicts of interest are declared, and the financial independence of the guideline-issuing body is documented.	In order to cover all aspects of the care of the patients in the guideline’217;s target group, the group of persons developing it must be representative of the different medical specialties that participate in these patients’217; treatment. Moreover, the guideline should be developed independently of the authors’217; personal interests or preferences. Financial support from outside sources (e.g., the pharmaceutical industry or the government) for the creation of the guideline must not be allowed to influence the authors’217; work or the formulation of the recommendations. For the purpose of transparency, all members of the guideline development group should be identified by the specialty groups and associations to which they belong. All personal or financial conflicts of interest of the authors and any financial or other support for the guideline-issuing body should be explicitly declared.
Literature search: The search for evidence was performed systematically, and the criteria for the selection of evidence are described.	The recommendations in a guideline of high quality are based on the best available scientific evidence at the time of its creation, which is derived from high-quality clinical trials and systematic reviews. Important questions in medicine are often addressed in a great many publications; thus, a comprehensive and systematic search is needed to identify all of the available information on the topic, as far as this is possible. The search terms should reflect the specific questions that the guideline is intended to answer. The evidence to support each recommendation is selected on the basis of the area of application (patient group, type of treatment, .) and a variety of methodological factors (quality of clinical trials, language of publication, recency of publication,. ..). The guideline should contain a detailed description of the searching strategy for evidence, including information about the sources used (electronic literature databases, manual searches), the searching terms used, and the temporal limits of the search. The selection process should also be described transparently; this mainly applies to the method of selection and the criteria for inclusion and exclusion.
Recommendations and evidence: The recommendations are clear, and their derivation from the evidence is explicit.	Guidelines and the recommendations they contain are a directly usable aid to medical decision-making in patient care. Recommendations should, therefore, be formulated clearly and unambiguously, so that users will know what to do in each situation. The derivation of each recommendation from the underlying evidence should be made clear by specification of the recommendation grade and evidence class, and by the citation of appropriate references. If a clear and specific recommendation cannot be formulated on the basis of the available evidence, this should be stated.
Options for management: Multiple management options are presented with a discussion of their utility, side effects, and risks.	For many questions in medicine and health care, more than one recommendation or option for management or medical treatment is available, particularly when the question concerns multiple care situations (e.g., screening, prevention, diagnosis, treatment). These options should be clearly presented in the guideline, with full information about their potential benefits, side effects, and risks, so that guideline users will be able to weigh the options and choose the best one in each individual case.
Dates: The date of issuance of the guideline and its expiration date are clearly indicated.	Guidelines reflect the state of medical knowledge at the time of their creation. Medical knowledge is constantly changing through the pulication of new clinical research reports and must therefore be regularly checked for continuing validity. Guidelines may need to be updated at close intervals. Thus, the date of the last update of the guideline should be clearly indicated, as well as the projected date of the next update or the projected interval between serial updates.

^*These explanations of the eight key questions of the MiChe are the product of an analysis and summarized restatement of existing descriptions of comparable assessment criteria that are found in previously published instruments and in three systematic reviews of guideline-evaluating instruments (11, 36, 37).

Discussion

Acquiring key information is already a major challenge for practicing physicians and is only becoming more difficult over time. Physicians must be competent to deal with an enormous variety of conditions, and the time available is usually short: as a rule, patient contacts in the ambulatory setting are brief, and hospital stays are kept as short as possible. The time available for continuing medical education is limited as well. Physicians, therefore, often make decisions based on their own and their colleagues’217; personal experience, and on information sources other than clinical research reports (16, 31, 32). Despite this, physicians are now showing increasing interest in guidelines. The acceptance of guidelines seems to depend mainly on the availability of a short version and on the physician’217;s ability to assess the quality of guideline creation (16). To meet this growing demand, handy instruments are needed to help physicians evaluate guidelines rapidly on their own before incorporating them in their daily practice.

The literature search for the present article retrieved a number of guideline-assessing instruments that are intended for broad use. These were mainly instruments that provided a highly detailed evaluation of guideline quality and that were apparently most suitable for the use by guideline developers themselves and by specialists in guideline-development methods, in view of their comprehensiveness and their specified modes of application (e.g., by at least two raters). There were also three English-language rapid-assessment instruments designed for everyday use by physicians in clinical practice (28– 30), containing up to 14 questions or criteria and an assessment scale of 7 or fewer levels. MiChe (18), an instrument of this type in German, facilitates access for German-speakers (33) and encapsulates the evaluative process in eight key criteria, with a three-level assessment scale. Through the use of essential key questions, all four of the rapid-assessment instruments let physicians make a quick judgment of the methodological quality of a guideline, and often also of its practical applicability. A further instrument called AGREE-REX (Recommendation EXcellence)—a counterpart to the established AGREE-II instrument for guideline creation—is expected to become available in 2016; it is intended to enable users to make a competent assessment of the scientific and practical quality of guidelines (34).

Each of these rapid-assessment instruments offers a compact version of methodological guideline-development criteria that were previously worked out in detail. The compactly formulated criteria are very similar in all of the instruments; in their simplicity and their focus on essential features, they fit in well with the practicing physician’217;s need for efficiency.

The user must be able to judge not only the practical aspects—relevance to his or her own patients, naming of the key recommendations, readily intelligible organization of the guideline—but also the methodological quality of guideline creation. The mode of searching for evidence should be both systematic and clearly described, and there should be a clear documentation of the weight that was attached to each finding in the derivation of the recommendations. Further important and readily ascertainable criteria are a complete listing of guideline authors by name and a statement of their conflicts of interest.

By virtue of their focus on a small number of criteria, rapid-assessment instruments can be used not only by individual physicians, but also by groups of physicians (quality circles). Interested physicians can use them for a quick and easy assessment of the usefulness, for their own work in patient care, of guidelines that were developed through a long and methodologically demanding process. Clear and relevant criteria wih a simple, goal-directed rating system (“yes,” “partly,” and “no”) facilitate their use. Physicians, patients, and guideline developers all stand to benefit from the consistent implementation of guideline recommendations in practice (16, 35).

Limitations

The presentation of evaluative instruments in this article may be incomplete because of the way the instruments were searched for and selected. In the updated search, we looked for additional instruments only with the “related citations” function in a single database, and in the internet portals of a few selected guideline-sponsoring bodies. To be sure that no other important evaluative instruments were missed, we cross-checked the ones retrieved by our search against those cited in three different systematic reviews of the same topic (11, 36, 37): the three reviews did not mention any additional important evaluative instruments developed explicitly for broad, public use. Instruments retrieved by our search that had not been developed explicitly for broad, public use were intentionally excluded, as were the many instruments designed as aids to guideline development.

A specific limitation of the MiChe is that it has not yet been validated. A doctoral dissertation on the topic “The Validity and Reliability of the Mini-Checklist on the Basis of AGREE II” is currently in preparation at the Department of Internal Medicine, Goethe Universität Frankfurt.

The evaluative instruments described here can only be used to assess the methodological quality of a guideline. The instruments make use of information contained in the published guideline itself to test whether certain standards have been met in its development. Guidelines containing adequate information of this type—e.g., about how the data were acquired—will generally be judged to be of high (methodological) quality. In contrast, the instruments generally do not test whether the searching strategy and the databases used were, in fact, well chosen or appropriate to the topic at hand. Thus, although proper attention to methodological standards in the creation of guidelines can be assumed to promote internal validity, some guidelines of high methodological quality may still contain individual recommendations that are not internally valid. Multiple authors have concluded that the available evaluative instruments for guidelines cannot be used to judge their clinical content or the quality of the underlying evidence (7, 11, 38).

Overview

Rapid-assessment instruments serve a complementary function to the more complex instruments such as DELBI and AGREE II and are convenient tools with which interested physicians can quickly judge the value of a guideline for themselves. Without depending on any outside help, they can check whether the guideline meets certain essential quality criteria for applicability in patient care. Physicians are now increasingly using guidelines to acquire clinical knowledge; rapid-assessment instruments could contribute additionally to this positive trend and help bring about a sustained improvement in patient care.

Table 2. Comparative listing and grouping of key questions in various rapid-assessment instruments.

Aspects of guideline quality	AGREE-GRS 2012 (28)	iCAHE 2014 (30)	Coroneos et al. 2014 (29)	MiChe 2014 (18)
Presentation		Summary of recommendations
		Complete reference list
	Structured organization	Intelligible format		Intelligible format/Key recommendations
	recommendations easy to find
		Purpose and target group	Clinical question(s)	Target user group, purpose,
				Background, patient group
Guideline creation		Time point of literature search
	Systematic analysis of evidence	Search strategy indicated	Method of analyzing evidence	Systematic analysis of evidence
		Developers named	Developers named	Developers named/COI
	Appropriate developer group	Developers appropriate
Underlying evidence		LoE defined
		Assessment of the quality of evidence
	Recommendations consistent with literature	LoE and quality /recommendation	Link between evidence and recommendation	Recommendations clear and linked to evidence
Report	Transparency		Review process
		Date of issuance
		Date of planned update		Currency, expiration date
Applicability	Clinically relevant		Recommendations relevant
	Suitable for target group
		Full text freely available
			Recommendations can be implemented
			All endpoints taken into account
	Management options, benefits, risks, costs			Management options, benefits, risks

COI: conflict of interest; GoR: grade of recommendation; LoE: level of evidence

Key Messages.

• Physicians who treat patients in the ambulatory setting or in the hospital, or both, are increasingly consulting and implementing guidelines as part of their everyday practice.

• Most instruments for assessing guidelines are highly complex and are meant to characterize their methodological quality in great detail.

• They are mainly for use by guideline developers, health care decisors, and health sector organizations. Time constraints generally prevent their use by practicing physicians.

• Rapid-assessment instruments consisting of a few key questions let physicians quickly judge the quality of a guideline and the practical relevance of its recommendations.

• In the rapid assessment of guidelines, the main features to be judged are: systematic creation of the guideline, recommendations clearly linked to the evidence, naming of all guideline developers, a statement of goals, guideline organization for ease of use, and the clear signposting of recommendations.