Use of systematic reviews in clinical practice guidelines: case study of smoking cessation

Abstract

Objective

To examine the extent to which recommendations in the national guidelines for the cessation of smoking are based on evidence from systematic reviews of controlled trials.

Design

Retrospective analysis of recommendations for the national guidelines for the cessation of smoking.

Materials

National guidelines in clinical practice on smoking cessation published in English.

Main outcome measures

The type of evidence (systematic review of controlled trials, individual trials, other studies, expert opinion) used to support each recommendation. We also assessed whether a Cochrane systematic review was available and could have been used in formulating the recommendation.

Results

Four national smoking cessation guidelines (from Canada, New Zealand, the United Kingdom, and the United States) covering 105 recommendations were identified. An explicit evidence base for 100%, 89%, 68%, and 98% of recommendations, respectively, was detected, of which 60%, 56%, 59%, and 47% were based on systematic reviews of controlled studies. Cochrane systematic reviews could have been used to develop between 39% and 73% of recommendations but were actually used in 0% to 36% of recommendations. The UK guidelines had the highest proportion of recommendations based on Cochrane systematic reviews.

Conclusions

Use of systematic reviews in guidelines is a measure of the “payback” on investment in research synthesis. Systematic reviews commonly underpinned recommendations in guidelines on smoking cessation. The extent to which they were used varied by country and there was evidence of duplication of effort in some areas. Greater international collaboration in developing and maintaining an evidence base of systematic reviews can improve the efficiency of use of research resources.

Introduction

Good guidelines in clinical practice may help translate research findings into health gain.¹ Guidelines should be based on a valid summary of the available evidence, and it has been shown that Dutch general practitioners were more likely to adopt recommendations that were evidence based than those that were not.²

Systematic reviews are an important source of valid evidence.³ However, undertaking and maintaining systematic reviews of high quality is resource intensive. The Cochrane Collaboration represents an international attempt to organise this type of activity across most fields of health care.⁴

Use of Cochrane reviews in guidelines for clinical practice is a possible measure of the “payback” on investment in the collaboration, but we do not know the extent to which they are used for this purpose. Grant and colleagues evaluated payback on biomedical research based on papers cited in clinical guidelines.⁵ They found that only 3% of cited references were systematic reviews.⁶ However, their approach does not necessarily reflect the extent to which systematic reviews were used to provide the evidence base for formulating specific recommendations within a guidelines document.⁷

We used smoking cessation as a case study to investigate the extent to which developers of national guidelines make use of systematic reviews. We also aimed to determine the source of systematic reviews used and the proportion of recommendations that could have been developed using an existing Cochrane review. We chose smoking because a large number of systematic reviews on this subject have been available for some time.⁸

Methods

We searched the internet (using the free text terms “smoking cessation” and “guideline”) and the US National Guidelines Clearinghouse (www.guideline.gov) to identify national guidelines on smoking cessation for clinical practice fully published in English.

For each set of guidelines we identified the commissioning agency, the date that the guidelines were initially released, the date of the last search for evidence to be included in the guideline, the date of subsequent review of the guideline, the process for generating the guidelines, and individual recommendations. When there was more than one document associated with the guidelines, we focused on the recommendations contained in the document that targeted practitioners primarily.

We sought to identify the type of evidence used to support each recommendation, categorising evidence as systematic review, individual study(s), consensus of panel members, or not stated. We noted whether systematic reviews cited were Cochrane reviews, other published systematic reviews, or reviews undertaken as part of the development process for guidelines. When there was more than one document and the evidence supporting a recommendation was not identifiable in the primary document targeting practitioners, we examined other available documents.

For each recommendation we assessed whether a Cochrane review could have been used (at the time the guidelines were being prepared) to formulate the recommendation either directly or by extrapolation from the data in the review.

Results

We identified four national guidelines published in English without adaptation from an existing set of guidelines.^9–12 Table 1 shows the country and commissioning agency for the four guidelines, together with the dates of release, last search for included evidence, and scheduled review (where recorded). The guidelines for the United States, United Kingdom, and New Zealand were all developed within the past two years. An updated amendment to the UK guidelines was released in late December 2000 (after this study was completed) and was, therefore, not included.¹³ A review of the New Zealand guidelines was due in June 2001. The Canadian guidelines were developed in 1996 and reviewed during 1998, when the recommendations were still deemed relevant.

Table 1.

Description of guidelines included in case study of smoking cessation guidelines

Country	Agency with responsibility for developing and issuing the guidelines	Date guideline initially released	Date of last search for included evidence	Date for review
United States9	US Public Health Service	June 2000	1 January 1999*	Not stated
United Kingdom10	Health Education Authority (England)	December 1998	September 1998	Not stated†
New Zealand11	National Health Committee	August 1999	April 1999	June 2001
Canada12	Canadian Preventive Task Force	January 1994 (reviewed in 1998)	August 1993 (reviewed in 1998)‡	Not stated

^*The US guideline included articles identified for a previous version¹⁴ (which had been based on a search conducted for published articles between 1975 and 1994) as well as a more recent search for articles published between 1995 and 1999. It also contained some articles published after 1999 if they were known to the guideline panel and were deemed relevant. No final cut off date for the search was provided. 

^†The original guidelines did not state a date for review; an update was released in December 2000¹³ but was not available when the present study was completed. 

^‡Although the guidelines were reviewed in 1998 and were still considered current, there is no information about whether any additional searching was undertaken at that time. 

All four guidelines had clear documentation about the development process (table 2). In a number of cases this involved review and adaptation of existing material. Both the UK and New Zealand guidelines stated that they drew explicitly on Cochrane systematic reviews. All three non-US guidelines drew on reviews undertaken as part of an earlier US smoking cessation guideline.¹⁴ In addition, the guidelines for New Zealand explicitly drew on the UK guidelines. All assessed strength of evidence, although in the New Zealand guidelines this was provided in a separate document from the guideline document, which was targeted at practitioners.¹⁵

Table 2.

Development process for national guidelines in clinical practice

Guideline	Commissioning agency	Method of guideline development	Identification of recommendations	Categorisation of evidence base
United States	Public Health Service US Department of Health and Human Services	Process described in document Built on previous national guidelines released in 1996 Extensive literature review process of original and meta-analytical articles undertaken to update previous review Reviewed by expert panel	56 recommendations listed in bold print in main text Four additional recommendations identified from text	Recommendations categorised (A, B, or C) according to strength of evidence Ratings attached to each recommendation; references and results of meta-analyses (where undertaken) provided within related text
New Zealand	National Health Committee Ministry of Health	Process described in separate document Reviewed previous US and UK guidelines (including references) plus local New Zealand review No statement about additional literature searching Reviewed by expert panel	18 recommendations in primary document for practitioners highlighted in a bold box; additional details provided for each in another box No additional recommendations identified	No formal categorising of evidence supporting recommendations in primary document but evidence summary with references provided Separate summary of evidence and quality of evidence reported in document on literature review and background information
United Kingdom	Health Education Authority	Process described in document Based on expert panel review of systematic reviews conducted by Cochrane Tobacco Addiction Review Group and Agency for Health Care Policy and Research (as part of the former US guidelines) as well as the American Psychiatric Association guideline	14 numbered recommendations in main text targeting health professionals (separate recommendations targeting health commissioners not included here) No additional recommendations identified	Recommendations categorised (A, B, or C) according to strength of evidence Ratings attached to each recommendation; references provided within related text
Canada	Canadian Task Force on Preventive Health Care	Part of Canadian Task Force on Periodic Health Examination Process described in separate chapter within document Based on primary literature search with review by expert panel	Five recommendations presented in summary table No additional recommendations identified	Recommendations categorised (A, B, or C) according to strength of evidence Ratings attached to each recommendation; supporting references provided within an adjacent column of summary table as well as expanded in text

The number of clearly identifiable recommendations contained in the different guidelines varied from five (Canadian) to 60 (US) (table 3). Of these, we identified an explicit evidence base supporting the recommendations in most cases—that is, 68%, 89%, 98%, and 100%—for the UK, New Zealand, US, and Canadian guidelines, respectively.

Table 3.

Use of systematic reviews to underpin recommendations in clinical practice guidelines. Values are numbers (percentages)

	Total	Recommendations with explicit evidence base	Recommendations based on systematic reviews
United States	60	59 (98)	28 (47)
New Zealand	18	16 (89)	10 (56)
United Kingdom	22	15 (68)	13 (59)
Canada	5	5 (100)	3 (60)

A Cochrane systematic review (that is, one that was available at the time of the last search undertaken in assembling the guidelines) could have been used, either directly or indirectly, to formulate 39%-73% of recommendations (table 4). Despite the potential to use Cochrane systematic reviews in this way, the proportion of recommendations specifically supported by a Cochrane review was much lower (0%-36%). The two North American guidelines in particular made little use of Cochrane reviews, with the US guidelines undertaking 26 new meta-analyses as part of the guideline development process. Of these, approximately two thirds covered topics similar to those covered by Cochrane systematic reviews. The UK and Canadian guidelines also relied heavily on published systematic reviews, particularly those from the earlier version of the US guidelines.¹⁴

Table 4.

Use of Cochrane systematic reviews in clinical practice guidelines. Values are numbers (percentages) of recommendations

	Total	Cochrane systematic review could have been used	Cochrane systematic review was used	Paper publication of non-Cochrane systematic review was used	New systematic review was performed
United States	60	38 (63)	2 (3)	0	26 (43)
New Zealand*	18	7 (39)	2 (11)	8 (44)	0
United Kingdom	22	16 (73)	8 (36)	5 (23)	0
Canada	5	3 (60)	0	3 (60)	0

^*Figures presented here are based on the evidence base supporting recommendations identified in the document on primary guidelines. When the separate document on literature review and background information was reviewed, of 19 summary statements (that were not regarded as recommendations to practitioners), 13 (68%) included reference to a Cochrane systematic review. 

Discussion

This study of smoking cessation shows that guideline developers provide detailed information about the steps involved in the development process. They were careful to ensure that most recommendations had an explicit evidence base. However, systematic reviews were used to support only about half of the total number of recommendations (and they supported a slightly higher percentage of those recommendations that mentioned an explicit evidence base). Of these, the proportion of recommendations based on a Cochrane review compared with a non-Cochrane systematic review varied depending on the country of origin of the guideline.

Generally, the North American guidelines drew on Cochrane reviews to a lesser extent than UK guidelines did. With the New Zealand guidelines the proportion of recommendations based on evidence from Cochrane systematic reviews varied depending on which of the two documents was considered. The document on literature review and background information makes extensive use of Cochrane reviews.¹⁵ The reason that the practitioner document makes less use of Cochrane reviews probably reflects the fact that several of the recommendations related to an area where there are currently no Cochrane reviews—for example, the establishment of systems of care (including prompting) to encourage practitioners to ask their patients about their smoking habits and to advise in a systematic way.

The reason for the relative lack of Cochrane reviews in the two guidelines developed in North America is unclear. One explanation could be the availability of Cochrane reviews in relation to the timing of developing the different guidelines. The initial release of the Canadian guidelines preceded the availability of any Cochrane reviews, but the review of these guidelines in 1998 could have drawn on a number of Cochrane reviews. Similarly, the US Agency for Health Care Policy and Research's 1996 smoking cessation guidelines (www.ahcpr.gov) were prepared before any Cochrane reviews were available. The US 2000 guidelines specifically mention consideration of other meta-analyses (and reference the entire Cochrane Library) in the description of methods.⁹ However, the evidence base draws mainly on updates of meta-analyses previously undertaken for the 1996 guidelines. This suggests that there is scope for improving international cooperation to reduce duplication of effort (and hence cost) in research synthesis.

Although the present study is based on a small sample, the guidelines mentioned have been widely adopted and used around the world by other national organisations.¹⁶ The extent to which the findings generalise to other topics is an area for further research.

We do not suggest that all clinical practice guidelines must use Cochrane systematic reviews as their primary source of evidence, or that Cochrane reviews are necessarily better quality than other systematic reviews. However, smoking cessation is an area in which a sizeable body of evidence is organised and updated systematically by the Cochrane Collaboration. In contrast with Grant et al⁵,⁶ we found, in the case of smoking cessation, that the recommendations of UK guidelines made extensive use of Cochrane systematic reviews. This is one source of evidence of the “payback” for the investment of funding organisations in supporting the systematic review process.¹⁷

Finally, we recognise that not all guideline recommendations can be underpinned by systematic reviews. Guideline developers can and should make recommendations that are specific to local contexts. However, reliable, up to date summaries of the evidence available internationally can allow guideline developers to reduce the costs associated with local guideline development.