Attributes of clinical guidelines that influence use of guidelines in general practice: observational study

Abstract

Objective: To determine which attributes of clinical practice guidelines influence the use of guidelines in decision making in clinical practice.

Design: Observational study relating the use of 47 different recommendations from 10 national clinical guidelines to 12 different attributes of clinical guidelines—for example, evidence based, controversial, concrete.

Setting: General practice in the Netherlands.

Subjects: 61 general practitioners who made 12 880 decisions in their contacts with patients.

Main outcome measures: Compliance of decisions with clinical guidelines according to the attribute of the guideline.

Results: Recommendations were followed in, on average, 61% (7915/12 880) of the decisions. Controversial recommendations were followed in 35% (886/2497) of decisions and non-controversial recommendations in 68% (7029/10 383) of decisions. Vague and non-specific recommendations were followed in 36% (826/2280) of decisions and clear recommendations in 67% (7089/10 600) of decisions. Recommendations that demanded a change in existing practice routines were followed in 44% (1278/2912) of decisions and those that did not in 67% (6637/9968) of decisions. Evidence based recommendations were used more than recommendations for practice that were not based on research evidence (71% (2745/3841) v 57% (5170/9039)).

Conclusions: People and organisations setting evidence based clinical practice guidelines should take into account some of the other important attributes of effective recommendations for clinical practice.

Key messages

• Specific attributes of clinical practice guidelines determine whether they are used in practice

• Evidence based recommendations are better followed in practice than recommendations not based on scientific evidence

• Precise definitions of recommended performance improve the use of guidelines

• Testing the feasibility and acceptance of clinical guidelines among the target group is important for effective implementation

• People setting evidence based guidelines need to understand the attributes of effective guidelines

Introduction

The increase in the number of clinical guidelines produced and published in different countries has stimulated discussion on their value. How good are they? Are they based on the best scientific evidence available? How effective are they in normal clinical practice? Interest has developed in many countries in the attributes of successful clinical guidelines.^1–5 The scientific validity and reliability of the guidelines receive most attention, less attention being paid to the features of guidelines that may determine their use in decision making in clinical practice. To date, research on these attributes has also been scarce.⁶ A good understanding of which attributes of guidelines influence their use in daily clinical practice is crucial for guideline development to be cost effective. Developing guidelines is laborious and usually expensive. Regrettably, too many guidelines do not remain in regular use, although the aim is to implement them in clinical practice. We therefore determined which attributes of guidelines or specific recommendations for appropriate care are related to their use in practice and in clinical decision making.

Subjects and methods

Determination of attributes of guidelines

We conducted a non-systematic study of scientific articles and documents from professional and scientific organisations in different countries on guideline development and attributes of guidelines to select attributes that might influence the use of guidelines in practice.^1–26 Attributes were concerned with the scientific validity of guidelines; their relevance and applicability in practice; the formulation and style of the recommendations; their compatibility with existing opinions and values; their complexity; their consequences for care providers, patients, doctors, and practice management; the risks of applying the recommendations; and the attention given to the guideline in the dissemination process. We formulated a set of 16 different attributes on the basis of these documents (box).

Attributes of guidelines

(1) The recommendation is based on scientific evidence—an explicit description of the scientific evidence for the recommendation is available; the research evidence is straightforward and not conflicting; the recommendation is based on the results of well designed clinical trials or meta-analyses

(2) The recommendation is based on clear and convincing arguments that are based on extensive clinical skills and experience

(3) The recommendation is concerned with a relevant aspect of care in daily practice

(4) The recommendation helps doctors to solve patients’ problems in daily care—it is concerned with difficult decisions or choices in daily care and it makes work easier

(5) The recommendation is one of the key features of the guideline—it is a central element in the guideline and represents the central aim

(6) The recommendation provides a concrete and precise description of desired performance—it gives detailed advice on which performance is appropriate in which situation and in what patient group and determines which factors or conditions should be taken into account

(7) The recommendation is vague and not specific

(8) The recommendation is complex—it is composed of many different elements and contains a complex decision tree or many different conditional factors influencing performance

(9) The recommendation is not compatible with existing norms and values in practice—it is controversial and provokes discussion

(10) The recommendation demands the acquisition of new competence (knowledge, skills)—it can be followed only when a doctor has specific knowledge and skills

(11) The recommendation has specific consequences for practice management—it requires adaptations in the organisation of care processes or demands extra resources, staff, equipment, etc

(12) The recommendation demands changing existing routines and habits and leaving what is seen as common practice in the target group

(13) The recommendation will provoke negative reactions in patients because it does not fit their common expectations—it may lead to a conflict of interest between patient and doctor

(14) The recommendation will provoke negative reactions among colleagues because it is not compatible with their views, position, or tasks

(15) The recommendation can be tried without any risks of possible damage for patients—experimenting with the proposed performance will not have negative effects on the health of patients

(16) The recommendation has been mentioned in the media and in implementation programmes

Attributes of guidelines in Dutch general practice

We selected 47 recommendations from 10 different national guidelines for general practice developed by the Dutch College of General Practitioners.²⁷,²⁸ These national guidelines are developed rigorously by combining the results of a systematic analysis of the scientific evidence with the results of consensus discussions of working parties of experienced general practitioners and specialists. These guidelines are published in the scientific journal for general practitioners Huisarts en Wetenschap [GP and Science] and disseminated through specially developed programmes of continuing medical education. Most general practitioners are knowledgeable about the guidelines 6-12 months after publication.²⁷ Each guideline consists of a set of 20-50 recommendations. In this study we selected a few recommendations on the management of acne vulgaris, acute sore throat, ankle distortion, acute otitis media, shoulder complaints, urinary tract infection, diagnostics in ophthalmology, type 2 diabetes, hypertension, and the prescription of oral contraceptives. Eight of the recommendations were concerned with management of chronic diseases and 39 with the management of acute problems in general practice. Twenty six focused on diagnostic performance, eight on therapeutic performance, six on patient education and advice, and seven on follow up or referral to specialists.

Four of us (JD, ST, CV, GR) independently determined whether the attributes given in the box were present, partly present, or not present in the 47 recommendations. Agreement between the four of us was moderate (kappa averaged 0.46 over 6×16×47=4512 paired assessments), so we decided that three of us had to agree on the score of a recommendation to determine the final score (this was the case in 87% of the assessments). The remaining scores were decided by consensus in a structured meeting. Attribute 2 in the box was excluded because we interpreted it in different ways.

Compliance with Dutch guidelines in relation to attributes

We related the attribute scores to use of the recommendations in clinical decisions using data on the 47 recommendations collected as part of a large clinical audit among 61 general practitioners in the Netherlands.²⁷ The general practitioners were selected on the basis of their interest in the project, and they were comparable to the national population of general practitioners in type and location of practice and their attitude towards the national guidelines but they were somewhat younger. Over three months they recorded their performance on specially designed forms after each consultation in which one of the 10 national guidelines was applicable. A test on the reliability of this self recording procedure was performed by comparing the scores of four general practitioners working in normal, busy practice settings with the scores of an independent trained observer in the surgery. These comparisons showed that the general practitioners completed the forms reliably (kappa averaged 0.76 for different decisions).

Analysis

The opinions of the assessors on the presence of attributes were dichotomised ((partly) present v not present) for each of the 47 recommendations. Attributes that did not discriminate—that is, were present or not present in less than four of the recommendations—were not included in further analyses. This was the case for attributes 3, 14, and 15 in the box. Thus a final set of 12 attributes was used in the analyses (see table 1). The presence of these attributes in the 47 recommendations was established. Compliance with these recommendations in decision making in clinical practice was determined by distinguishing between the recommendations with an attribute present and those without an attribute present. For example, in 3841 cases or decisions made by general practitioners it was possible to determine whether the recommendations of 15 evidence based guidelines had been followed, and in 9039 cases or decisions made by general practitioners it was possible to determine whether the recommendations of 32 non-evidence based guidelines had been followed (see table 1). Thus for each decision made by the 61 general practitioners performance in practice could be compared with a specific recommendation. We evaluated 12 880 different decisions of general practitioners in this way to determine compliance.

Table 1.

Compliance with 47 recommendations with different attributes of 12 880 decisions made by 61 general practitioners

	Attribute present			Attribute not present			Rate of compliance with recommendations (%) (No of decisions)		Strength of influence of attribute*
	No of recommendations	No of decisions		No of recommendations	No of decisions		When attribute present	When attribute not present
Recommendation (attribute No in box):
Is based on scientific evidence (1)	15	3841		32	9039		71 (2745)	57 (5170)	0.13
Helps to solve problems in clinical practice (4)	33	7893		14	4987		63 (4984)	59 (2921)	0.05
Is a key feature of the clinical guideline (5)	25	5957		22	6923		63 (3770)	60 (4145)	0.04
Is described concretely and precisely (6)	38	10384		9	2496		67 (6951)	39 (964)	0.23
Is vague and not specific (7)	7	2280		40	10600		36 (826)	67 (7089)	0.24
Is complex (8)	8	1798		39	11082		55 (997)	62 (6918)	0.05
Is controversial and not compatible with current values (9)	8	2497		39	10383		35 (886)	68 (7029)	0.26
Demands new knowledge and skills (10)	12	3970		35	8910		54 (2136)	65 (5779)	0.10
Has consequences for management (11)	10	3095		37	9785		50 (1554)	65 (6361)	0.13
Demands changing existing routines (12)	10	2912		37	9968		44 (1278)	67 (6637)	0.20
Will provoke negative reactions in patients (13)	7	1303		40	11577		47 (609)	63 (7306)	0.10
Has been mentioned in media (16)	4	671		43	12209		74 (494)	61 (7421)	0.06

^*Eta, which indicates the degree of association between a dependent (interval) variable and an independent (nominal) variable and in this table expresses the influence of an attribute on compliance with recommendations. Values range from 0 to 1, where 0=no influence and 1=strong influence. P⩽0.001 for all values in table.  

Differences in compliance rates for the different attributes were tested using analysis of variance. To evaluate which attributes had most influence a stepwise regression analysis was performed using the total compliance rate for 12 880 decisions as the dependent variable.

Results

The recommendations were followed in, on average, 61% (7915/12 880) of the decisions of general practitioners. All 12 selected attributes influenced the use of the recommendations in practice, but the degree of influence was different for each attribute (table 1). Whether a recommendation was controversial and incompatible with existing values in the target group and whether it was clearly defined were particularly important. Also its effect on daily work (attributes 10, 12, and 13; table 1) influenced use in clinical practice. Recommendations based on evidence were used more than those that were not.

The relative contribution of the different attributes to the total compliance rate for the 47 recommendations determined by stepwise regression analysis showed that the three attributes with the highest single correlation contributed most to the explanation of the variance (table 2). These were the recommendation is controversial and not compatible with current values; the recommendation is vague and not precisely defined; and the recommendation demands change of fixed routines. These attributes mainly had an independent effect on the compliance rate in practice. They explained 17% of the total variance.

Table 2.

Attributes determining compliance with recommendations in stepwise multiple regression analysis, with total score for compliance as dependent variable

Recommendation (attribute No in box)	R2	Beta	Sign (P value)
Is controversial and not compatible with current values (9)	0.07	0.09	0.000
Is vague and not specific (7)	0.10	0.21	0.000
Demands changing existing routines (12)	0.14	0.23	0.000
Will provoke negative reactions in patients (13)	0.15	0.12	0.000
Is based on scientific evidence (1)	0.16	−0.15	0.000
Has consequences for management (11)	0.16	0.09	0.000
Is complex (8)	0.16	−0.11	0.000
Demands new knowledge and skills (10)	0.17	0.07	0.000
Is key feature of clinical guideline (5)	0.17	0.10	0.000
Has been mentioned in media (16)	0.17	−0.02	0.02

Discussion

Scientists and policy makers need to understand the features of guidelines that relate to implementation of guidelines in decision making in daily practice.²⁹ Our study shows some of the important attributes of effective recommendations for clinical practice. Guidelines should be compatible with existing values among the target group and not be too controversial. They should not demand too much change to existing routines and be defined precisely, with specific advice on actions and decisions in different cases. They should be compatible with current values and routines. Indeed, some recommendations probably expressed what general practitioners were already prepared to do. The scientific basis of the recommendation was also important. Recommendations were more adhered to when an explicit description of the scientific evidence was available and the evidence was straightforward and not conflicting. The perceived consequences for doctors and practice management matter. A recommendation was used less when compliance affected the organisation of and staff in practices, when it demanded extra resources or acquisition of new knowledge and skills, or when it provoked negative reactions in patients.

The explained variance was not high (17%), but many other factors need to be taken into consideration to determine whether a guideline will be used. Setting guidelines is only one step in a comprehensive process of implementation and making patient care more effective.⁴,⁷ Further research on the implementation of guidelines needs to provide insight into why some guidelines are effective and others are not. Our findings are consistent with theoretical models on effective guidelines.²⁹ To our knowledge, this is the first time in one study that the influence of different attributes of guidelines has been shown using empirical data from clinical practice. Our findings show that developers of guidelines need to take these attributes into account. They show the importance not only of performing an analysis of the scientific literature but also of carrying out a test in clinical practice to evaluate the feasibility and effectiveness of guidelines in daily care. They show the importance of a precise definition of the recommended performance in clinical decisions. Professional writers might be helpful in analysing the drafts of guidelines for vague and ambivalent use of language.

We used only a limited number of recommendations. Our selection may have been too limited and the whole range of clinical decisions may have to be sampled. We did, however, include recommendations for acute and chronic diseases and on diagnosis, treatment, advice, and follow up. We included over 12 000 decisions in clinical practice. The assessment of the recommendations by a panel of general practitioners could be improved as the reliability of its assessments was moderate. On the other hand, four general practitioners were able to reach consensus after discussing each other’s arguments. Our results do not allow conclusions to be drawn about causal relations, for which prospective, controlled trials comparing the effects of different types of recommendations are needed. People who draw up guidelines or finance the development of guidelines may improve the effectiveness of their work by evaluating their products with the attributes we have described.