A review of clinical practice guidelines found that they were often based on evidence of uncertain relevance to primary care patients

Abstract

Objectives

Primary care patients typically have less severe illness than those in hospital and may be overtreated if clinical guideline evidence is inappropriately generalized. We aimed to assess whether guideline recommendations for primary care were based on relevant research.

Study Design and Setting

Literature review of all publications cited in support of National Institute for Health and Care Excellence (NICE) recommendations for primary care. The relevance to primary care of all 45 NICE clinical guidelines published in 2010 and 2011, and their recommendations, was assessed by an expert panel.

Results

Twenty-two of 45 NICE clinical guidelines published in 2010 and 2011 were relevant to primary care. These 22 guidelines contained 1,185 recommendations, of which 495 were relevant to primary care, and cited evidence from 1,573 research publications. Of these cited publications, 590 (38%, range by guideline 6–74%) were based on patients typical of primary care.

Conclusion

Nearly two-third (62%) of publications cited to support primary care recommendations were of uncertain relevance to patients in primary care. Guideline development groups should more clearly identify which recommendations are intended for primary care and uncertainties about the relevance of the supporting evidence to primary care patients, to avoid potential overtreatment.

1. Introduction

What is new?

• •The applicability of clinical practice guidelines to primary care has been questioned for individual conditions such as hypertension and depression, and concerns have been raised about guidelines promoting overtreatment of low-risk populations.
• •Until now, evidence from a systematic appraisal of the relevance to primary care of published guidelines has been lacking.
• •Nearly two-third of the research cited in support of National Institute for Health and Care Excellence guideline recommendations for primary care was of uncertain relevance to primary care patients, with little or no acknowledgment of this uncertainty.
• •Guideline development groups should more clearly identify which recommendations are intended for primary care and uncertainties about the relevance of the supporting evidence to primary care patients, to avoid potential overtreatment and adverse effects.

Clinical practice guidelines are an increasingly important driver of decisions about patient care. They have been defined as “recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” [1]. Guidelines have traditionally been developed to simply provide guidance for clinical decision making, but they are becoming embedded in the structure of UK primary care through their translation into indicators of quality of care in a national “pay for performance” financial incentive scheme (the Quality and Outcomes Framework) and through the development of quality standards to inform decisions on health care planning and commissioning [2]. This increasing use of guidelines to develop incentives and standards for primary care may lead to more patients at lower risk of adverse outcomes receiving treatment and exposure to potential adverse effects.

Groups developing guidelines about the care of primary care patients will use the current best evidence from primary care or lower risk populations where it exists. If high-quality primary care evidence is not found, the best evidence available may be from a secondary care or higher risk population. This entirely appropriate approach leads to problems when a guideline development group (GDG) assumes that the evidence from research conducted on a higher risk population can automatically be applied to a lower risk primary care population. If uncertainty about the evidence is not explicitly acknowledged, the integrity of the guideline is compromised and patient harm may result [3,4]. The benefits of treatment are usually lower in populations at lower risk of adverse outcomes, whereas the risk of harm from adverse treatment effects remains constant. Patients seen in primary care typically have less severe illness than those in hospital, and so evidence from trials conducted in secondary care may have limited relevance and result in harms outweighing benefits [5].

An example of taking evidence from a higher risk population and applying it to a lower risk population is the Quality and Outcomes Framework indicator and National Institute for Health and Care Excellence (NICE) heart failure guideline recommendation that all primary care patients with chronic heart failure (including low grade) should be offered β-blockers and ACE inhibitors. This indicator is supported by evidence generalized from higher risk populations (New York Heart Association grades III–IV), in which there is clear evidence of benefit, to lower risk populations, in which the evidence of benefit is more equivocal. The potential harm is the adverse effects of β-blockers experienced by some patients, and the substantial risk of acute kidney injury from ACE inhibitors, which may account for a tenth of the increase in hospital admissions because of an acute kidney injury [6]. It is therefore uncertain what the balance of harms and benefits might be in a typical primary care patient [7,8], and a general practitioner needs to know about this uncertainty when, for example, considering prescribing a β-blocker to a patient with a relative contraindication to a β-blocker therapy from a comorbid condition. This vital information about uncertainty and the balance of benefits and harms is hard to find in the Quality and Outcomes Framework guidance or NICE guideline, which presents a single approach rather than acknowledging that there are several acceptable alternatives for low-risk patients.

Another example where it is hard for the user of a clinical guideline to know about the balance of benefits and harms for a typical primary care patient is the Quality and Outcomes Framework incentive to prescribe aspirin or an alternative antiplatelet to all patients with peripheral arterial disease, most of whom do not have symptoms and are managed in primary care [9]. The evidence that antiplatelet therapy can reduce serious vascular events comes primarily from a large subgroup analysis of the Antithrombotic Trialists' Collaboration meta-analysis in high-risk patients and a similar review conducted by NICE [10,11]. However, the authors caution that their results may not be applicable to low-risk patients, and others have calculated that the number of potential reductions in coronary heart disease events exceeded the number of potential precipitated adverse bleeding events only for patients with a 1% or greater annual risk of coronary heart disease events [12]. A third example is chronic kidney disease (CKD), where there is evidence of benefit to high-risk populations but no evidence of benefit in people with early-stage CKD at a low risk of future disease [13]. Both primary care physicians and specialists have expressed concerns about potential harms from overtreatment resulting from expanding definitions of CKD in guidelines.

A small pilot study suggested that the evidence base for primary care guidelines might not be relevant to most primary care patients, with important implications for patient safety [14], and we wanted to systematically examine the evidence base for clinical guidelines used in primary care. We used guidelines from the NICE as it has been a leading provider of evidence-based clinical guidelines in the United Kingdom since 2002 [15]. NICE's highly respected methods compare well with the U.S. Institute of Medicine's standards for trustworthy guidelines [1,16–18] and with the international consensus that guidelines should be developed using an explicit and transparent process that minimizes distortions, biases, and conflicts of interest; should base recommendations on a systematic review of the existing evidence; should include experts and patient representatives on a multidisciplinary GDG; and should consider important patient subgroups and patient preferences [1,19,20]. The development of NICE clinical guidelines follows a well-established process [16]. When a topic has been chosen, a National Collaborating Centre (NCC) is commissioned to develop the guideline. The NCC prepares the scope which sets out what the guideline will and will not cover and recruits the GDG. Review questions are then developed to guide the evidence review and synthesis by the technical team, which the GDG uses to formulate recommendations. We aimed to measure the percentage of primary care relevant publications in the cited evidence base for NICE primary care recommendations, for all NICE guideline recommendations for primary care published over 2 years.

2. Methods

We reviewed the scope of all 45 clinical guidelines published by NICE in 2010 and 2011 and excluded any guideline explicitly aimed at non–primary care settings. Five academic family physicians on the research team classified recommendations in the remaining guidelines as specific, relevant, or not relevant to primary care, using our previously piloted methods [14]. Two reviewers independently rated each recommendation and then discussed any discrepancies by telephone and/or e-mail. If they could not reach consensus on a recommendation, it was sent to a third reviewer and then classified according to the majority view. Reviewers used the following definitions, along with examples: “primary care specific recommendations inform decisions that are almost always made by primary care providers such as general practitioners” and “primary care relevant recommendations inform decisions that could be made by health professionals in either primary care or another setting.” We included guidelines if more than 50% of the recommendations were judged relevant to primary care or if at least one recommendation was specific to primary care. We included all primary care–specific or -relevant recommendations within these selected guidelines.

We reviewed all publications cited as evidence for included recommendations and classified each publication according to whether the study population had been selected from primary care or community settings (ie, not from hospital outpatient or inpatient populations). We classified publications as “unclear” if we could not obtain this information from these or related publications. We considered systematic reviews as a single publication and included them if at least one study in the review had been conducted on a primary care or community population. Publications were excluded from the study and not considered further if they referred to other guidelines, nonsystematic reviews, or references that only appeared in the guideline appendices with no apparent link to the recommendations. The primary outcome was the percentage of included cited publications that were relevant to primary care. Secondary outcomes were guideline development center, primary care membership of GDG, recommendations for further primary care research, whether “Grading of Recommendations Assessment, Development and Evaluation” (GRADE) was used to assess the quality of evidence [21], and the clarity of links from the recommendation back to the supporting evidence base.

3. Results

Twenty-two of the 45 guidelines were assessed as relevant to primary care (Fig. 1). These 22 guidelines contained 1,185 recommendations, and the reviewers considered 777 of them to be relevant to primary care. The two independent reviewers initially rated 981 (82.8%) recommendations in the same category of specific, relevant, or not relevant to primary care, with discrepancies predominantly between the categories of primary care “specific” or “relevant.” After discussion between the two reviewers, full agreement was reached on 1,166 (98.4%) of recommendations. The remaining recommendations were classified after independent rating by a third reviewer.

Fig. 1.

Flow chart for selection of guidelines, recommendations, and research publications.

Of the 777 recommendations, 282 were excluded as they were based on GDG consensus in the absence of evidence or were standard clinical practice (such as a recommendation to take a history from the patient), leaving 495 (42%, range by guideline 9–100%) recommendations that were both evidence based and relevant to primary care (Table 1). The evidence for these 495 recommendations came from 1,573 research publications. The links in the guideline from recommendation back to supporting research sometimes appeared to have become blurred in the process of developing the guideline. The research was considered relevant to primary care in 590 (38%, range by guideline 6–74%) of these publications (Table 1). The research was not considered relevant to primary care in 916 publications. The relevance to primary care was unclear in 67 publications, in which the research population was not clearly described in the original articles.

Table 1.

Relevance to primary care of recommendations, and of publications cited as evidence, for all included clinical guidelines from the National Institute for Health and Care Excellence (NICE)

Clinical guideline and development center	Recommendations			Publications
	Total in guideline, N	PCR, N	PCR and evidence based, N (%)	Cited as evidence, N	PCR, N (%)
National Clinical Guideline Centre
CG95 Chest pain of recent onset	89	48	48 (54)	48	20 (42)
CG97 Lower urinary tract symptoms	82	48	22 (27)	92	13 (14)
CG100 Alcohol-use disorders	37	15	5 (14)	22	2 (9)
CG101 Chronic obstructive pulmonary diseasea	12	12	10 (83)	24	9 (38)
CG103 Delirium	27	27	24 (89)	129	8 (6)
CG108 Chronic heart failurea	25	22	17 (68)	48	5 (10)
CG109 Transient loss of consciousness	41	28	17 (41)	35	21 (60)
CG111 Nocturnal enuresis	99	93	39 (39)	103	35 (34)
CG126 Stable angina	50	32	19 (38)	35	9 (26)
CG127 Hypertensiona	37	37	24 (65)	147	66 (45)
Subtotal	499	362	225 (45)	683	188 (28)
National Collaborating Centre for Mental Health
CG113 Generalised anxiety disordera	49	46	35 (71)	124	58 (47)
CG120 Psychosis with coexisting substance misuse	78	45	20 (26)	27	14 (52)
CG123 Common mental health disorders	69	68	23 (33)	63	45 (71)
CG133 Self-harm (longer term management)	57	51	24 (42)	147	66 (45)
Subtotal	253	210	102 (40)	361	183 (51)
National Collaborating Centre for Women's and Children's Health
CG99 Constipation in children and young people	46	37	33 (72)	75	13 (17)
CG102 Bacterial meningitis	111	22	18 (16)	35	11 (31)
CG110 Pregnancy and complex social factors	53	53	53 (100)	137	101 (74)
CG128 Autism in children and young people	69	35	6 (9)	91	52 (57)
Subtotal	279	147	110 (75)	338	177 (52)
National Collaborating Center for Cancer
CG121 Lung cancera	91	11	11 (12)	12	1 (8)
CG122 Ovarian cancer	27	11	11 (41)	30	12 (40)
Subtotal	118	22	22 (100)	42	13 (31)
NICE Internal Short Clinical Guideline Technical Team
CG96 Neuropathic pain	17	17	17 (100)	95	12 (13)
CG116 Food allergy in children and young people	19	19	19 (100)	54	17 (31)
Subtotal	36	36	36 (100)	149	29 (19)
Total	1,185	777	495 (42)	1,573	590 (38)

Abbreviation: PCR, primary care relevant.

^aUpdate of existing guidelines with only new recommendations included.

There was a wide variation across guidelines in the percentage of both recommendations and supporting publications that were relevant to primary care. In some guidelines (eg, “pregnancy and complex social factors”) both the recommendations (100%) and the evidence base (74%) were judged to be relevant to primary care. Other guidelines (eg, “chronic heart failure” and “constipation in children and young people”) were relevant to primary care (68% and 72%, respectively) and yet were based on relatively little evidence from primary care (10% and 17%). Guidelines in mental health and women's and children's health had the highest percentages of supporting publications that were relevant to primary care (51% and 52%).

The number of primary care providers on GDGs ranged from 0 to 4 (12% of all GDG membership; Table 2). Family physicians (31 over all groups) outnumbered nurses [3]. The percentage of primary care providers on GDGs was not associated with the percentage of primary care relevant studies (correlation coefficient, 0.15). Further research in primary care was called for in 6 of the 22 full guidelines. Nine of the 22 included guidelines used elements of GRADE to assess the quality of included evidence [21–23]. These guidelines were published after NICE introduced GRADE in 2009 [23].

Table 2.

Guideline development group (GDG) membership and primary care research recommendations

Clinical guideline and development center	Primary care providers on GDG, N (% of GDG)a	Family physician	Academic family physician	Practice (office) nurse	Further primary care research recommended
National Clinical Guideline Centre
CG95 Chest pain of recent onset	1 (7)	1	0	0	No
CG97 Lower urinary tract symptoms	2 (17)	2	0	0	No
CG100 Alcohol-use disorders	1 (7)	1	0	0	No
CG101 Chronic obstructive pulmonary disease	3 (25)	2	0	1	Yes
CG103 Delirium	0 (0)	0	0	0	Yesb
CG108 Chronic heart failure	3 (30)	2	1c	0	Yesd
CG109 Transient loss of consciousness	2 (13)	2	0	0	No
CG111 Nocturnal enuresis	1 (8)	1	0	0	No
CG126 Stable angina	2 (15)	2	0	0	No
CG127 Hypertension	4 (40)	2	1	1	No
Subtotal	19 (15)	15	2	2	3/10
National Collaborating Centre for Mental Health
CG113 Generalised anxiety disorder	2 (18)	0	2	0	Yes
CG120 Psychosis with coexisting substance misuse	0 (0)	0	0	0	No
CG123 Common mental health disorders	4 (29)	1	2c	1	Noe
CG133 Self-harm (longer term management)	1 (7)	1	0	0	No
Subtotal	7 (13)	2	4	1	1/4
National Collaborating Centre for Women's and Children's Health
CG99 Constipation in children and young people	1 (8)	1	0	0	No
CG102 Bacterial meningitis	1 (7)	0	1	0	Yes
CG110 Pregnancy and complex social factors	1 (7)	1	0	0	No
CG128 Autism in children and young people	1 (8)	1	0	0	No
Subtotal	4 (7)	3	1	0	1/4
National Collaborating Centre for Cancer
CG121 Lung cancer	1 (5)	0	1	0	No
CG122 Ovarian cancer	1 (7)	0	1	0	No
Subtotal	2 (6)	0	2	0	0/2
NICE Internal Short Clinical Guidelines Technical Team
CG96 Neuropathic pain	1 (9)	0	1	0	No
CG116 Food allergy in children and young people	1 (9)	1	0	0	Yes
Subtotal	2 (9)	1	1	0	1/2
Total (22 guidelines)	34 (12)	21	10	3	6/22

^aGDG membership excludes technical team.

^bRecommended research in a long-term care setting but did not explicitly identify primary care.

^cAcademic family physician chaired the guideline development group.

^dRecommended research in the general population.

^eMentioned primary care in the justification to one research recommendation.

4. Discussion

Nearly two-third of the research cited in support of NICE guideline recommendations for primary care was of uncertain relevance to primary care patients, with little or no acknowledgment of this uncertainty in the published guideline. Only 38% of cited publications were based on patients typical of primary care. In many guidelines, the link from evidence to recommendation was not explicit. Where evidence was available, many studies did not report the setting or population used for the research.

The low percentage (38%) of studies relevant to primary care is likely to be due to the lack of suitable primary care research. This correctly requires GDGs to extrapolate from research that has been conducted on other, often higher risk, populations. The difficulty for the guideline user is not so much that the evidence is inevitably incomplete, but rather that it is not clear which recommendations are supported by primary care–based relevant evidence and which by evidence from other clinical settings or by consensus of the GDG. Judgments about the relevance of the evidence to different populations made using GRADE's “indirectness” domain [21] were useful but presented as a minor part of GRADE which did not come through clearly in final guidelines. Family physicians are experienced in managing clinical uncertainty and making decisions with incomplete information [24], and when deciding about treatment for a patient, they need to know the extent to which guideline evidence can be applied to that individual [25,26]. Information about uncertainty in the research, and potential applicability to a patient in primary care, is not currently easily accessible in guidelines in the way it needs to be to improve clinical decision making in primary care and avoid accusations of guideline bias [3,27].

Other authors have noted the difficulties in applying guidelines where there are important differences between trial participants and typical primary care patients in hypertension [5,28]. Only one group has looked specifically at whether guidelines are supported by evidence from primary care: a group in the Netherlands looked at 13 guidelines for depression and screened 804 publications, to find only two studies based in primary care [29]. They concluded that the guideline recommendations could not be considered evidence based. Our research is the first systematic appraisal of national guidelines, and our results show that uncertain applicability of cited research to patients in primary care is a problem shared to a greater or lesser degree by all the guidelines reviewed.

Strengths of the research include that it was based on a large sample of all recent NICE guidelines and used piloted methods [14]. The estimate of 38% of evidence being based on patients typical of primary care is likely to be an overestimate rather than an underestimate, as we only included guidelines with most recommendations relevant to primary care (in which the proportion of primary care evidence is likely to be greater than in other guidelines). The biggest challenge for the research team was to classify the guidelines, recommendations, and research studies in turn as either relevant to primary care or not. We used transparent and piloted methods, but the choice between “relevant” and “not relevant” was not always straightforward, and other groups might make different decisions in the gray areas. We responded to this uncertainty using a generous definition for classifying study populations as relevant to primary care, which again would tend to make the 38% an overestimate rather than an underestimate.

Future research is needed into the dynamics and approaches to reaching informal consensus used by GDGs and the influence of variable factors such as professional background and conflicts of interest [3]. Internationally, primary and community care is usually of lower technology than hospital care, and so contextualizing research within the primary care setting is relevant to both developed and developing countries. The extent to which research in high-technology settings can be considered relevant for guideline users in low-technology settings in low- and middle-income countries is under-researched.

Frameworks such as the Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) to assess the quality of practice guidelines [20] and the GRADE tool, which includes an “indirectness” domain to assess the applicability to different populations [22,23], have been developed to improve the integrity of clinical guidelines. NICE and other national guideline developers contribute to and follow this good practice, have primary care membership of GDGs, try to ensure that secondary care content experts do not dominate discussions regarding recommendations for primary care, and involve the public in the process [30,31], and these frameworks and methods should continue to be developed and applied. However, despite all the good practice, it is still hard for a primary care clinician to find the information they need to guide a decision about the potential benefits and harms of treatment. Guideline developers intend to present their recommendations with enough information for the user to track back to the strength of the underlying evidence, should they need to do so [1,16,19], but we found that “tracking back” to the evidence was usually very difficult. The relevance of published clinical practice guidelines to primary care patients is still too opaque.

5. Conclusion

An enhanced approach is needed to avoid the assumption of generalizability of research evidence from high-risk research populations to low-risk primary care patients and reduce potential overtreatment and avoidable harm. More explicit consideration of the primary care context is needed at all stages of development, to reduce dependence on the influencing skills of a primary care provider on the GDG. The relevance to primary care should be carefully considered at initial scoping. For relevant guidelines, the initial review questions for the evidence search should then be considered specifically for relevance to primary care, perhaps as a subgroup in the search, and negative findings reported if evidence is not found. Primary care relevance could be considered in terms of setting, severity of illness, or risk group depending on the guideline's intended audience. The guideline should be specific about where primary care research has or has not been used, including limitations or lack of evidence, and research recommendations where relevant primary care evidence is lacking should be clearly badged. Uncertainties in the evidence should always be clearly presented and not lost in the understandable desire to produce straightforward recommendations. A shift in focus during guideline development to outcomes (such as health gain) as opposed to simple processes (such as pharmaceutical prescription) has been proposed and should be encouraged [2].

An enhanced primary care perspective could come from either greater involvement from the outset of primary care professionals with relevant content expertise where feasible or from more explicit guidance to development groups combined with systematic checks at all stages of guideline development. Checks for compliance with development manuals and frameworks such as GRADE and AGREE II would help ensure, for example, that GRADE profiles properly address issues of indirectness [16,20,22]. At least one NICE committee has already started to consider research data that have been reanalyzed to show the results separately for low- and high-risk populations [32], with a consequent change in the decision made. Success will be when a clinical guideline user can quickly and accurately determine the broad relevance of each recommendation to their patient, including uncertainties, and can easily track back from the recommendation to the underlying evidence base, should they wish to do so.