How confident should we be that smoking cessation treatments work?

Abstract

Aim

To determine a) the concordance among recent meta-analyses about which treatments for smoking cessation are vs are not efficacious, b) the similarity of odds ratios (ORs) across meta-analyses and c) among the validated treatments, the proportion of studies that found higher quit rates

Methods

Computerized literature search for meta-analyses in the last 5 yrs in PubMed and PsychInfo. Data were extracted from summary tables of overall effect of validated treatments

Results

The 14 meta-analyses agreed 100% on the presence/absence of efficacy of 17 proven treatments. The ORs differed by < 0.5 in 72/76 of the comparisons of meta-analyses. Among the 37 meta-analyses that reported individual study results, in 33 meta-analyses, > 85% of the individual studies reported numerical superiority for the active treatment.

Conclusions

The efficacy of treatments for smoking cessation are extremely reliable. This argues for inclusion of treatment as an essential feature of tobacco control and clinical practice and argues for reimbursement of smoking cessation treatments on par with other medical and behavioral disorders.

INTRODUCTION

As many as 70% of smokers do not believe smoking cessation treatments are effective [1]. In addition, many tobacco control advocates and health care funders appear to undervalue smoking cessation [2]. For example, a recent article on “Global regulatory strategies for tobacco control” in JAMA omitted smoking cessation products and services [3]. One possible reason for this undervaluing of treatment is that tobacco advocates and health care funders are unaware of how reliable the data are that smoking cessation treatments work. Recent releases of the two most-cited meta-analyses; i.e. those by the USPHS Treating Tobacco Use and Dependence Clinical Practice Guideline (www.surgeongeneral/tobacco) and the Cochrane Database of Systematic Reviews (www.cochrane.org), provide an opportunity to better illustrate this reliability.

METHODS

To examine the reliability of treatment effects for smoking cessation, I first located the individual meta-analyses tables in the 2008 USPHS Guideline [4] (Table 1). Next, I searched the appropriate Cochrane Reviews for the most similar meta-analyses [5-15]. For brevity, I compared only meta-analyses of generic use of treatments in the general population of smokers trying to quit; i.e., I did not include sub-analyses of different subsamples, treatment procedures, etc. Given the focus of this paper, I only examined meta-analysis of treatments that either the USPHS or Cochrane meta-analyses concluded were validated treatments. The investigators who conducted the USPHS and Cochrane meta-analyses (see acknowledgements).then reviewed these results for accuracy. This search resulted in 17 treatments to compare between the USPHS and Cochrane guidelines (the specific Cochrane meta-analyses and the specific comparisons can be obtained from the author). The median (and range) of number of studies/meta-analysis was 11 (2-93) among the USPHS meta-analyses and 14.5 (4-53) among the Cochrane meta-analyses. Only the Cochrane provided total sample sizes and the median was 6385 participants/meta-analysis (417-22,240). When all NRTs were grouped in the Cochrane the number of studies was 111 and the number of participants 43,040.

Table 1.

Odds ratios (ORs) (and proportion of comparisons with numerical superiority) for treatment in meta-analyses of recommended smoking cessation treatments

	USPHS 2008	Cochrane 2001-2007	Others 2003-2008
Psychosocial Tx
Group tx	1.3 (?/52) a	2.1 (15/17) b	2.1 (4/4) c
Individual tx	1.7 (?/67) a	1.5 (15/17) d	1.5 (13/15) c
MD advice	2.2 (?/11) a	1.9 (26/28) e	1.9 (?/?) f	? (11/13) g
Non-MD advice	1.7(?/39) a	1.3 (23/31) h	1.3 (19/26) i	1.8-1.9 (?/?) f	? (21/39) g
Phone tx	1.6 (?/11) a	1.4 (7/8) j	1.6 (16/22) k	1.4 (?/?) l
Rapid smoking	2.0(?/19) a	2.0 (12/12) m
Self-help	1.2(?/93) a	1.3 (12/17) n
Medications
Bupropion	2.0 (?/26) a	1.9 (31/31) o	1.7 & 2.9 (10/10) p	2.1 (16/16) q	1.6 (9/12) r
Clonidine	2.1 (?/3) a	1.9 (6/6) s
Nicotine gum	1.5 (?/15) a	1.6 (46/53) t	1.7 (21/22) q	1.8 (11/12) l	1.6 (?/?) r	1.8 (?/7) u
Nicotine inhaler	2.1 (?/6) a	2.1 (4/4) t	2.2 (4/4) q
Nicotine lozenge/microtab	2.0/2.8 (?/2) a	2.0 (5/5) t	2.1 (6/6) q
Nicotine nasal spray	2.3 (?/4) a	2.4 (4/4) t	2.4 (4/4) q
Nicotine patch	1.9 (?/32) a	1.8 (39/41) t	2.0 (30/30) q	1.6 (?/?) k	1.7 (?/33) u
NRTs (all)		1.4 (102/111) t	1.5 & 1.6 (47/51) p	2.0 (12/13) v	1.7 (61/70) r
Nortriptyline	1.8 (?/5) a	2.3 (6/6) o	2.4 (5/5) w
Varenicline	3.1 (?/5) a	2.9 (7/7) x	2.4 (13/13) q	2.4 (5/5) l

Tx = treatment, MD = Physician, NRT = Nicotine replacement therapy, ? = not reported

^aFiore et al, 2008 [4]

^bStead & Lancaster, 2005 [5]

^cBala &Lesniak, 2007 [17]

^dLancaster & Stead, 2005 [6]

^eStead etal, 2008 [7]

^fMojica et al, 2004 [27]

^gGorin& Heck, 2004 [26]

^hRice & Stead, 2008 [8]

ⁱRice, 2006 [23]

^jStead et al, 2006 [9]

^kPan, 2006 [22]

^lMyung et al, 2007 [21]

^mHajek& Stead, 2008 [10]

ⁿLancaster & Stead, 2005 [11]

^oHughes et al, 2007 [12]

^pBala et al, 2008 [18]

^qEissenberg et al, 2008 [19]

^rWu et al, 2006 [24]

^sGourlay et al, 2004 [13]

^tStead et al, 2008 [14]

^uFagerstrom, 2003 [16]

^vEtter & Stapleton, 2006 [20]

^wWagena et al, 2005 [25]

^xCahill et al, 2008 [15]

Next, I searched Pub Med (www.ncbi.nlm.gov/pubmed) and PsychInfo (www.apa.org/psychinfo) using the keywords “smok* OR tobacco OR nicotin* OR cigar*) crossed with (meta-analy* OR “systematic review”) and restricted the search to the last 5 yrs. This resulted in 46 meta-analyses. From these I excluded those that evaluated a) a treatment for which neither the USPHS nor Cochrane did a meta-analysis (e.g. cytisine), b) a specific subgroup of smokers (e.g. among pregnant smokers), c) a specific setting (e.g. over-the-counter), or d) specific procedures within a given treatment (e.g. pre-treatment with NRT). This resulted in 12 other meta-analyses of generic treatment effects, in the general population of smokers trying to quit [16-27].

To examine “statistical reliability” across meta-analyses, I compared whether the meta-analyses were concordant on whether treatment effects did or did not meet the meta-analytic statistical criterion and also compared the numerical value of the reported Odds Ratios (OR). The more recent Cochrane reviews report Risk Ratios rather than OR; however the Cochrane group recalculated these as ORs for this paper so that a more fair comparison could be made. To examine “experimental reliability,” I examined, within each Cochrane meta-analysis, what proportion of all the within-study comparisons showed a numerical superiority for the treatment. This information was not available for the USPHS meta-analysis..

RESULTS

Before discussing the outcomes, readers should be aware the USPHS, Cochrane and other meta-analyses differed in many ways; e.g., the exact comparison being made, study inclusion criteria, use of usual care/placebo/no-treatment control groups, length of follow-up, definition of abstinence, use of biochemical verification, publication year, amount of adjunctive treatment, statistical tests, setting, and use of unpublished studies. Prior papers have discussed how these could influence meta-analytic outcomes [28].

Despite these differences, the USPHS, Cochrane and other reviews agreed on statistical significance (yes/no) on 17 of 17 treatments and neither recommended a treatment that the other did not. (The reviews also had a high concordance on which treatments were not efficacious -these data are available from the author). In addition, the ORs were strikingly similar (Table 1). When I used the USPHS or Cochrane as the gold standard, and compared the USPHS vs Cochrane ORs plus compared the other 14 meta-analyses vs both the USPHS and Cochrane meta-analyses, this resulted in 76 comparisons of ORs. The OR from one meta-analysis differed by < 0.5 from the OR in the other meta-analysis in 72 of the 76 (95%) comparisons. This concordance is especially striking given that many meta-analyses of the same treatment differed dramatically in methods. For example, the USPHS meta-analysis of nicotine gum included 15 comparisons of gum vs a control but the Cochrane included 53 comparisons. Even so the OR for the USPHS was 1.5 and for the Cochrane was 1.6.

The experimental reliability of effects was also striking (Table 1). Among the 37 meta-analyses that reported individual study results, in 33 meta-analyses, > 85% of the studies found a numerical superiority for active treatment over placebo. For example, among the 111 comparisons of NRT in the Cochrane meta-analyses, 102 (92%) reported numerical superiority for NRT. Similarly, among the 31 comparisons of bupropion in the Cochrane review, 31 (100%) reported a numerical superiority

DISCUSSION

The concordance of OR across the meta-analyses (despite multiple methodological differences and very different sample sizes) plus the high rates of numerical superiority for treatments, suggest the efficacy of smoking cessation treatments is extremely reliable. Such high reliability of treatment effects is consistent with other indirect evidence; i.e., cessation treatments effects have been found to persist across funders, settings, samples, outcome definitions, and amount of adjunctive treatment [14,16,29]. For example, nicotine replacement therapies (NRTs) have been effective in those in an intensive counseling session and those in an over-the-counter setting [30], in smokers differing in motivation [31], and in smokers of different races [32]. Although the high rates of numerical superiority could be due to publication bias [33], almost all meta-analyses that tested for publication bias in smoking cessation meta-analyses did not find it [21,22,26,27,29]. The high proportion of comparisons that report numerical superiority for treatment of smoking appears to be greater than that for many other treatments of behavioral disorders; e.g. several marketed antidepressants and antipsychotics are effective in only half to two-thirds of the RCTs conducted [34]. On the other hand, the proportion of numerically successful trials of acamprosate and naltrexone treatments for alcohol problems is quite high [35].

This strong statistical and experimental reliability is as great as, if not greater than, that for tobacco control policies [2]. For example, the Cochrane meta-analyses for sales restrictions [36], media [37,38], prevention [39,40], community interventions [41,42] and public restrictions [43] have fewer numbers of studies, often use non-randomized design, and report smaller proportions of positive studies or smaller ORs or effect sizes.

The strong reliability of treatment effects is important evidence to argue that provision of smoking cessation treatment should be an essential part of both tobacco control efforts and clinical practice and [44]. It is also a strong argument that treatments for smoking cessation should be reimbursed on par with other behavioral and medication treatments.