Abstract
Background
Guidelines are a common and important tool in providing high-quality health care. The National Institute for Health and Clinical Excellence (NICE) guidelines are now being used to set standards for assessing the quality of care in UK general practice, and so the evidence behind them needs to be relevant to primary care.
Aim
To assess the extent to which guideline recommendations aimed at primary care are based on research conducted in a primary care setting.
Design of study
Purposeful selection of a sample of NICE guidelines for conditions commonly seen in general practice, with identification of the evidence underpinning recommendations that are relevant to primary care.
Method
Three recent NICE guidelines were selected: chronic obstructive pulmonary disease (COPD), hypertension, and respiratory tract infection in adults and children. Publications referenced as evidence for each individual primary care relevant recommendation were classified as to whether or not they were based in primary care relevant settings.
Results
In the three guidelines assessed, 160 studies were used to derive the 115 recommendations that were relevant to, or aimed at primary care. A wide variation was found in the proportion of studies that recruited patients from a setting relevant to primary care (range 26% to 80%).
Conclusion
In this sample of three NICE guidelines, a significant proportion of studies underlying the primary care relevant recommendations were derived from studies that were not conducted in that setting. In producing guidelines for a primary care audience, the guideline development groups should include explicit information about the setting of studies underpinning the recommendations.
Keywords: evidence-based medicine, practice guidelines, primary care
INTRODUCTION
The National Institute for Health and Clinical Excellence (NICE) is one of the foremost providers of evidence-based guidelines. NICE was inaugurated on 1 April 1999, with the aim of providing independent, authoritative, and reliable guidance on promoting good health and treating ill-health. Its guidelines are considered to be among the best available and are essential reading for those who are involved in the care of patients.
It has been well documented that the provision of clinical care varies greatly and is dependent not only on the illness but also on the patient, the setting, and the doctor.1–4 Guidelines have become one of the primary means of standardising care, and play a large role in the dissemination of new evidence and recommendation of best practice. However, interventions to implement clinical guidelines have, at most, demonstrated a modest improvement in the process of care, and further work is needed to consider factors that improve guideline dissemination and implementation.5,6
The current NHS drive to link quality to cost efficiency has provided another role for NICE guidelines. In this capacity, guidelines are being used to develop the clinical standards against which the performance of GPs will be judged and their performance-related pay calculated.7
The validity of a guideline depends on the data that are available and chosen for inclusion. In the development of NICE guidelines, the scope is based on an initial literature search and on consultation with stakeholders. These represent the NHS, healthcare professionals, patients, carers, and companies with a special interest in the guideline. The guideline review panel ensures their comments are taken on board. The guideline development group members are then selected, and a detailed literature search and evaluation of other guidelines performed.
How this fits in
Guidelines are a useful tool in providing universal and high-quality health care. It has been well documented that the research output from primary care is less than in other disciplines, but little is known of the effect of this on the evidence base for guidelines. This study has shown that, in a sample of three NICE guidelines, the evidence base for a significant number of recommendations neither originates from primary care nor is based in a primary care relevant setting. It suggests that guideline authors should include explicit information regarding the extent to which guideline recommendations for a primary care audience are derived from research performed in this setting.
NICE guidelines have been assembled from the best evidence available at the time of guideline preparation. Where the recommendations are targeted at a primary care audience, it is anticipated that the evidence will be derived from populations that are representative of the diverse group of patients encountered in primary care. If the evidence is derived from studies in more selected populations, for example, recruited through secondary care clinics, the recommendations may be less relevant to a primary care audience.
The primary aim of the study was to examine the extent to which guideline recommendations aimed at primary care are based on research conducted in a primary care setting.
In identifying the evidence that underpins the NICE recommendations, it is hypothesised that a greater proportion of more recent studies will be derived from primary care and there will be more UK-based primary care studies; hence, the secondary aims were to identify the percentage of studies conducted in primary care according to year of publication and assess the country of origin for studies based in primary care. Further, the primary care representativeness of guidelines was explored by counting the number of primary care professionals on the guideline development groups.
METHOD
Selection of NICE guidelines
The respiratory tract infection,8 hypertension,9 and chronic obstructive pulmonary disease (COPD)10 guidelines were selected for this study, based on two criteria: they refer to conditions commonly seen in general practice,11 and were the subject of guidelines published by NICE in the preceding 5 years. The respiratory tract infection guideline is aimed at a purely primary care audience, the hypertension guideline is aimed at all non-specialists but is particularly relevant to primary care practitioners, and the COPD guideline is aimed at both primary and secondary care (Table 1).
Table 1.
Selected guidelines.
| Guideline | Target audience | Publication date |
|---|---|---|
| Prescribing of antibiotics for self-limiting respiratory tract infections in adults and children in primary care (CG69)8 | Primary care | July 2008 |
| Hypertension: management of hypertension in adults in primary care (CG34)9 | Primary care | June 2006 |
| Chronic obstructive pulmonary disease — management of chronic obstructive pulmonary disease in adults in primary and secondary care (CG12)10 | Primary and secondary care | Feb 2004 |
Identifying relevant recommendations
Each guideline was independently analysed by two researchers. Recommendations were identified that were specific or relevant to primary care. The relevant recommendations comprise those drawn up for non-specialists. Given the target audience for the respiratory tract infection and hypertension guideline,8,9 this encompassed all recommendations. To identify primary care relevant recommendations in the COPD guideline,10 two researchers independently identified those recommendations that were relevant to primary care. Of the 188 recommendations, disagreement arose in eight (4.3%). This was resolved by consensus discussion between the two reviewers. Of these eight recommendations, two were subsequently included in the primary care relevant recommendations. Although a third reviewer was available to arbitrate where disagreement could not be resolved, consensus was achieved between the two reviewers. Of the 188 recommendations in the COPD guideline,10 97 were thus identified as being relevant to primary care.
Reviewing the original evidence
For all three guidelines, the studies supporting the evidence statements or recommendations were then independently reviewed by two researchers. The full-text article for each reference was assessed to establish the study setting and population recruited. The setting was classified as primary care if the study was entirely or partially conducted in primary care. The data were presented as the proportion of relevant studies at a guideline level; hence, where a study was cited for more than one recommendation it was counted only once. Similarly, where the same set of data was used for multiple companion publications, only the primary publication was counted. Again, where disagreement between the two reviewers arose, a consensus decision was made. Where disagreement remained, a third reviewer would arbitrate. To assess the attributes of primary care studies in the guidelines, the year of publication and the geographical locality of the study were identified. Details of the procedures for data collection and analysis are included as Appendix 1.
RESULTS
Primary outcome
Across the three guidelines, 115 recommendations were aimed at or relevant to primary care. These recommendations were derived from 160 studies. Given the differing target audiences of the three guidelines, there was a marked variation in the contribution of primary care research to the recommendations (Table 2). As expected, in the guideline aimed purely at primary care (respiratory tract infection8), 80% of the studies were primary care based. In the two guidelines aimed at mixed audiences (hypertension9 and COPD10), only 26% and 67% respectively of the studies used to derive the primary care relevant recommendation were based on research conducted in primary care relevant setting.
Table 2.
Setting of evidence used to derive primary care relevant guideline recommendations.
| Guideline | |||
|---|---|---|---|
| Respiratory tract infection | Hypertension | COPD | |
| Total number of recommendations | 7 | 11 | 188 |
| Number (%) of primary care relevant recommendations | 7 (100) | 11 (100) | 97 (52) |
| Total number of studies used to derive recommendations | 20 | 24 | 116 |
| Number (%) of studies with primary care relevant setting | 16 (80) | 16 (67) | 30 (26) |
| Number (%) of studies set in secondary care | 4 (20) | 5 (21) | 83 (72) |
Full consensus was reached by the two reviewers on all of the respiratory tract infection studies analysed. For the hypertension studies, the two reviewers were unable to agree on the setting in nine (27%) of the studies. In one study, this was resolved after review of the actual study protocol. In the remaining eight studies, the third reviewer, in discussion with the original reviewers, reached a consensus decision. Three of these studies were allocated to primary care and two to secondary care. It was not possible to allocate the remaining three studies – two because of lack of information and one because of difficulty in locating further study details.
After the initial independent review of the studies in the COPD guideline,10 there was disagreement between the reviewers on 23 of the 116 studies (20% of studies). After discussion between the two reviewers, consensus was reached on six studies: four of these were allocated to primary care and two to secondary care. In discussion with the third reviewer, consensus was reached on 14 studies: four were allocated to primary care and 10 to secondary care. Again it was not possible to allocate three studies, due to lack of information regarding the setting.
Secondary outcomes
Characteristics of studies included in NICE guideline recommendations
Just over half of all studies were published in the 5 years prior to the guidelines publication. In the decade prior to guideline publication, there has been little difference over time in the proportion of primary care relevant studies (Table 3). Studies set in the UK made a greater contribution to the recommendations in the three guidelines compared to other geographical areas for both primary- and secondary-care-based studies (Table 4).
Table 3.
Year of publication of research used to derive primary care relevant recommendations across all three guidelines.
| Research publication date (years preceding guideline publication date) | Number (%) of studies with a primary care relevant setting | Number (%) of studies with a secondary care setting | Total studies |
|---|---|---|---|
| <5 | 36 (43) | 48 (57) | 84 (100) |
| 5–10 | 14 (44) | 18 (56) | 32 (100) |
| ≥10 | 12 (32) | 26 (68) | 38 (100) |
Table 4.
Country of origin of research used to derive primary care relevant recommendations across all three guidelines.
| Country of origin | Number (%) of studies with primary care relevant setting | Number (%) of studies set in secondary care |
|---|---|---|
| Multinational | 10 (16) | 10 (11) |
| UK | 24 (39) | 26 (28) |
| US | 13(21) | 21 (23) |
| Europe | 10 (16) | 21 (23) |
| Other | 5 (8) | 10 (11) |
| Unknown | 0 (0) | 4 (4) |
| Total | 62 (100) | 92 (100) |
Members of the guideline development group
Five (12%) primary care professionals were listed out of the 41 members of the guideline development groups for all three guidelines. A GP and primary care tutor were part of the 14 members of the hypertension guideline development group. A professor of primary care and a lecturer in primary care were members of the nine-strong guideline development group for the respiratory tract infection guideline. One GP was present on the 18 members of the COPD guideline development group, and an additional consensus reference group was formed that included a professor of primary care and a GP.
DISCUSSION
Summary of main findings
In the three guidelines assessed, the research used to generate the recommendations aimed at or relevant to primary care did not always originate from a setting that is representative of primary care. A large inter-guideline variation exists, with the contribution of primary care research ranging from 26% to 80%. Guideline development is only as good as the pool of evidence available on which the recommendations can be based. The NICE respiratory tract infection guideline is exemplar of a NICE guideline informed by primary care relevant research.8 In contrast, in the current NICE COPD guideline, only 26% of the studies used to derive the primary care relevant recommendations were based in a relevant setting.10 As perhaps would be expected in a British guideline, the locality of studies with the greatest contribution to the evidence were UK based. This was true for studies undertaken in both primary and secondary care settings.
The difficulty in determining the setting of some studies was surprising, given the clear requirements in the CONSORT guidelines to record this. Many studies only vaguely referred to the setting; for example, in the COPD guideline there were 25 studies that listed the settings as only ‘clinical centres’ or ‘the setting was multicentred’.10
Which evidence is selected for use in a guideline will always be a matter of debate, but if the volume of evidence that is relevant to primary care is small then a significant bias may be an unavoidable occurrence. However, it is known that if clinicians do not have confidence in the evidence base for recommendations then they are unlikely to change their clinical behaviour.12 It is possible that even when there is limited primary care-based evidence, the involvement of primary care representatives in guideline development groups may support their implementation, and all three guidelines had GPs involved in their development.
Strengths and limitations of the study
To ensure the robustness of data extraction, the study data presented here have been collated after analysis of each guideline independently by two researchers. From this, a consensus view has been taken. Consensus varied between the three guidelines (percentage agreement; respiratory tract infection 100%, COPD 80%, and hypertension 66%), primarily due to difficulty identifying the setting of the studies. Even after arbitration by a third reviewer, the setting of six studies remained unclear. While an agreed procedure was developed for data collection and definitions, it should be acknowledged that there is still a potential for observer bias. Further, given that only three guidelines were reviewed and the results varied by guideline topic, it is not possible to comment on the generalisability of these results to other guidelines.
Comparison with existing literature
Primary care research has often been viewed as the poor relation to secondary care academia. It has been proposed that improvements need to be made to the output of primary care research and the number of indexed family medicine journals, and also there is a need for expansion of practice-based research networks.13,14 Askew et al noted that in Australia, GPs make up 38% of the medical workforce, yet in the years 2000 to 2007, only 3% of publications came from primary care.15 This compared poorly with hospital physicians who number only 15% of the workforce but are responsible for 72% of publications.
Concern has also arisen about the accessibility of primary care research. In this study the proportion of recommendations derived from primary care relevant studies has not improved with time. As early as 1993, a register for randomised controlled trials in primary care was proposed, after analysis of 5 years of publications found that only 23% of randomised controlled trials conducted in or directly relevant to primary care were published in primary care journals.16 More recently, concern has been expressed about the limited number of family medical journals listed in the journal citation reports and the effect this has on impact factors and availability.17 There is, however, some indication that with improved funding and the emergence of primary care research networks, the output and quality of primary care-based research is starting to improve.18–20 However, despite this, the anticipated trend that more recent primary care research would contribute to recommendations did not emerge from the findings of the present study.
Where relevant data do not exist, the guideline development group must continue to assess if research from other settings is generalisable to the setting of the target audience. This approach and its pitfalls have been well documented.21,22 High-quality randomised controlled trials use strict inclusion and exclusion criteria that may not reflect a typical primary care population. A primary care population encompasses extremes of age and severity of illness, with a lower probability of significant or severe disease than would be seen in secondary care. Additionally a large number of comorbidities are managed concurrently.23 Consequently, such ‘typical patients’ would fall outside the inclusion criteria of many studies.24,25
Implications for future research and clinical practice
This study found that there is a significant variation in the use of primary care evidence to derive the recommendations in three NICE guidelines. In view of this, these recommendations may not always be generalisable to primary care. Further research is needed to assess the use of primary care research evidence to underpin recommendations aimed at primary care, from NICE and other sources of authoritative guidelines.
Guidelines that suggest recommendations aimed at primary care should strive to use evidence that is from an applicable setting, and it is anticipated that this will usually be from a primary care setting. There is a need for more primary care research to increase the evidence base for those conditions commonly seen in primary care. However, research conducted in primary care does not guarantee that it is representative of a primary care population. Conversely, secondary care research may still be applicable to primary care populations. We suggest that guideline authors should include explicit information regarding the extent to which the recommendations in a guideline are derived from research performed in an appropriate setting. This clearly requires that the setting and population of a study should be included by authors and journal editors on publication.
Payment by results, through the Quality Outcome Framework (QOF), has now become a significant part of primary care funding, and NICE guidelines are now being used as a standard with which to judge clinical care in the QOF. However, the continued use of these standards in the future may become controversial unless it can be shown that the targets arise from evidence based on populations and settings that are typical of primary care.
Acknowledgments
The authors would like to express their gratitude to A Scott, who kindly assisted with data collection, and fellow members of cohort 2 of the International Primary Care Research Leadership Programme (‘Brisbane Initiative’), who inspired the development of this proposal.
Appendix 1 Protocol for data-collection and data analysis procedures
Data collection
Level 1: guideline information
Guideline audience.
Of the development panel members and consensus reference group members how many are primary care professionals?
Level 2
Where possible, guidelines will be scrutinised to match evidence to the recommendations.
Where there is no primary evidence to support a recommendation (for example, consensus opinion), then the recommendation should be recorded with — ‘no evidence available’.
Full text resources to be accessed to allow accurate determination of setting and population.
-
Data to be collected concerning the following headings (in italics). Where no answer is found record N/A for not available.
Recommendation — number and text.
Reference — where there is more than one reference for each recommendation, each reference should be listed under the specific recommendation.
Setting,
Year of publication of study
Country(s) of origin
Where a reference is used for more than one recommendation this should be included and multiple entries noted. However, in the final data analysis, duplication of the use of references or use of multiple references from the same study will be discarded.
All guidelines to be analysed twice by two independent reviewers.
Data analysis
Outcome measures
Primary:
-
Guideline development:
guideline audience
development panel members – percentage of primary care professionals.
For all primary care specific or relevant recommendations (cumulatively), calculate the percentage of studies based in primary care or with a primary care relevant setting. To be calculated separately for each guideline.
For all primary care specific or relevant recommendations (cumulatively), calculate the percentage of studies based in secondary care. To be calculated separately for each guideline.
Secondary:
Journal.
Year of publication.
Geographical area conducted in (multinational, UK, US, Europe, other, unknown.
Competing interests
Nadeem Qureshi and Nick Steele are members of the NICE Primary Care Quality and Outcomes Framework Indicator Advisory Committee.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
REFERENCES
- 1.Juncosa S, Jones RB, McGhee SM. Appropriateness of hospital referral for hypertension. BMJ. 1990;300(6725):646–648. doi: 10.1136/bmj.300.6725.646. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Steel N, Bachmann M, Maisey S, et al. Self reported receipt of care consistent with 32 quality indicators: national population survey of adults aged 50 or more in England. BMJ. 2008;337:957. doi: 10.1136/bmj.a957. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.McKinlay JB, Link CL, Freund KM, et al. Sources of variation in physician adherence with clinical guidelines: results from a factorial experiment. J Gen Intern Med. 2007;22(3):289–296. doi: 10.1007/s11606-006-0075-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mckinlay JB, Lin T, Freund K, Moskowitz M. The unexpected influence of physician attributes on clinical decisions: results of an experiment. J Health Soc Behav. 2002;43(1):92–106. [PubMed] [Google Scholar]
- 5.Grimshaw J, Russell I. Effect of clinical guidelines on medical practice; a systematic review of rigorous evaluations. Lancet. 1993;342(8883):1317–1322. doi: 10.1016/0140-6736(93)92244-n. [DOI] [PubMed] [Google Scholar]
- 6.Grimshaw JAM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):iii–iv. doi: 10.3310/hta8060. 1–72. [DOI] [PubMed] [Google Scholar]
- 7.National Institute for Health and Clinical Excellence. About the Quality and Outcomes Framework (QOF). http://www.nice.org.uk/aboutnice/qof/qof.jsp (accessed 23 Jun 2010)
- 8.National Institute for Health and Clinical Excellence. Prescribing of antibiotics for self-limiting respiratory tract infections in adults and children in primary care. CG69. London: National Institute for Health and Clinical Excellence; 2008. http://www.nice.org.uk/nicemedia/pdf/CG69FullGuideline.pdf (accessed 27 Jan 2011) [PubMed] [Google Scholar]
- 9.National Institute for Health and Clinical Excellence. Hypertension: management of hypertension in adults in primary care. CG34. London: National Institute for Health and Clinical Excellence; 2006. http://guidance.nice.org.uk/nicemedia/live/10986/30113/30113.pdf (accessed 27 Jan 2011) [Google Scholar]
- 10.National Institute for Health and Clinical Excellence. Chronic obstructive pulmonary disease — management of chronic obstructive pulmonary disease in adults in primary and secondary care. CG12. London: National Institute for Health and Clinical Excellence; 2004. http://www.nice.org.uk:80/nicemedia/pdf/CG012_niceguideline.pdf (accessed 27 Jan 2011) [Google Scholar]
- 11.ISD Scotland. General practice — practice team information (PTI). Top 10 conditions — GP and practice nurse. http://www.isdscotland.org/isd/3687.html (accessed 23 Jun 2010)
- 12.Michie S, Hendy J, Smith J, et al. Evidence into practice: a theory based study of achieving national health targets in primary care. J Eval Clin Pract. 2004;10(3):447–456. doi: 10.1111/j.1365-2753.2004.00520.x. [DOI] [PubMed] [Google Scholar]
- 13.Maeseneer JMD, van Driel ML, Green LA, et al. The need for research in primary care. Lancet. 2003;362(9392):1314–1319. doi: 10.1016/S0140-6736(03)14576-X. [DOI] [PubMed] [Google Scholar]
- 14.Van Weel C, Rosser WW. Improving health care globally: a critical review of the necessity of family medicine research and recommendations to build research capacity. Ann Fam Med. 2004;(2 Suppl 2):S5–16. doi: 10.1370/afm.194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Askew D, Schluter P, Gunn J. Research productivity in Australian general practice: what has changed since the 1990s? Med J Aust. 2008;189(2):103–104. doi: 10.5694/j.1326-5377.2008.tb01931.x. [DOI] [PubMed] [Google Scholar]
- 16.Silagy C. Developing a register of randomised controlled trials in primary care. BMJ. 1999;306(6882):897–900. doi: 10.1136/bmj.306.6882.897. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Van Driel M, Maier M, Maeseneer J. Measuring the impact of family medicine research: scientific citations or societal impact? Fam Pract. 2007;24(5):401–402. doi: 10.1093/fampra/cmm061. [DOI] [PubMed] [Google Scholar]
- 18.Society for Academic Primary Care. New century, new challenges. A report from the heads of departments of general practice and primary care in the medical schools of the United Kingdom. London: Royal College of General Practitioners; 2002. http://www.sapc.ac.uk/docs/Mackenzie2.pdf (accessed 23 Jun 2010) [Google Scholar]
- 19.No authors listed. Collaboration in primary care research. Lancet. 2009;373(9659):186. doi: 10.1016/S0140-6736(09)60050-7. [DOI] [PubMed] [Google Scholar]
- 20.Van Weel C, Smith H, Beasley J. Family practice research networks: experiences from 3 countries. J Fam Pract. 2000;49(10):938–943. [PubMed] [Google Scholar]
- 21.Rothwell P. External validity of randomised controlled trials: ‘To whom do the results of this trial apply?’. Lancet. 2005;365(9453):82–93. doi: 10.1016/S0140-6736(04)17670-8. [DOI] [PubMed] [Google Scholar]
- 22.McAlister F, Diepen S, Padwal R, et al. How evidence-based are the recommendations in evidence-based guidelines? PLoS Med. 2007;4(8):1325–1332. doi: 10.1371/journal.pmed.0040250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Beasley J, Hankey T, Erickson R, et al. How many problems do family physicians manage at each encounter? A WReN study. Ann Fam Med. 2004;2(5):405–410. doi: 10.1370/afm.94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Oswald N, Bateman H. Applying research evidence to individuals in primary care: a study using non-rheumatic atrial fibrillation. Fam Pract. 1999;16(4):414–419. doi: 10.1093/fampra/16.4.414. [DOI] [PubMed] [Google Scholar]
- 25.Fortin M, Dionne J, Pinbo G, et al. Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med. 2006;4(2):104–108. doi: 10.1370/afm.516. [DOI] [PMC free article] [PubMed] [Google Scholar]