How stable are stop smoking practitioner success rates over time?

Abstract

Stop smoking practitioners appear to differ in effectiveness, but the stability of their success rates over time is unknown. The purpose of this study was to assess the degree of stability of success rates of stop smoking practitioners over several years of practice. Using routinely collected practice data, the success rates of 197 practitioners active between April 2009 and April 2012 in the English stop smoking services were correlated across years before and after adjusting for client and intervention characteristics. Changes in client and intervention characteristics were assessed. Success rates for individual practitioners correlated highly in successive years (r = 0.64 to 0.68, p < 0.001, ICC = 0.56) and moderately over non-successive years (r = 0.39 to 0.51, p < 0.001). There was no evidence for increasing effectiveness over time. Practitioners’ effectiveness is moderately stable over time. Research is needed to establish what characterises the practice of the more successful practitioners.

INTRODUCTION

Support for smoking cessation is now available in more than 80 countries [1], yet little is known about the effectiveness of delivery of support in practice and what underlies effectiveness. England is one of the few countries with a national network of stop smoking services. These services can be accessed by any smoker attempting to stop and provide a combination of medication and behavioural support. In 2012, the services treated more than 700,000 smokers; however, the services differ widely in their effectiveness, with short-term success rates ranging from 5 to 61 % in 2011–12 [2]. A significant proportion of variation in success rates of clients is accounted for by the individual practitioner who supports the client in their quit attempt [3]. Practitioners’ success rates may improve with experience and may be affected by changes in their client base; for example, a change in the proportions of clients with higher socio-economic status, older age or male gender may benefit success rates [4, 5], as may new developments in practice, including wider use of more effective medication options such as combination nicotine replacement therapy (NRT) or varenicline [4–8], or more frequent use of group support [4, 5]. In cooperation with North51, the National Centre for Smoking Cessation and Training (NCSCT) has collated routinely collected data from stop smoking services across England, which provides a unique opportunity for detailed analyses of practitioners’ practice and outcomes. The aim of the present study was to use a sample of stop smoking practitioners to examine the degree of stability of practitioners’ success over several years of practice.

METHODS

Sample

Many stop smoking services in England use an online database (QuitManager, North51, Nottingham, UK) to record information on their clients, the support provided and the short-term outcomes. We used anonymised data on 511,988 quit attempts between April 2009 and April 2012 that included information on which practitioner had supported the quit attempt. For each calendar year, the data included between 3,061 (in 2009) and 5,867 practitioners (in 2010). The present analyses were restricted to practitioners who had been active throughout the period of data collection. This was defined as having supported at least 25 quit attempts in each full year and correspondingly fewer in the incomplete years. Of the 561 practitioners who had supported any quit attempts in all four periods of time, 197 were classified as active throughout and included in the analyses. Outcome data from the database were used by services to report back to the Department of Health, so completion of data was crucial. Other fields, including information on the practitioner delivering the intervention, are less reliably completed; however, the included quit attempts appear to be representative of all quit attempts supported by the English stop smoking services in the years 2009 to 2012. The gender split, the proportion of clients in routine and manual occupation and the proportion of interventions set in specialist settings and delivered as one-to-one support in the sample are very similar to the national figures [2, 9, 10]. The only apparent difference is that the proportion of those paying prescription charges was lower in the present sample than in the national figures (around 30 % versus 46 to 52 %), suggesting more deprived clients.

Measures

All measures were averaged for each practitioner for the year in which the quit attempt took place. Carbon monoxide (CO)-validated 4-week quit rates were the main outcome measure and were measured according to the Russell Standard (Clinical) [11]. The success of a quit attempt is determined 4 weeks after a quit date, when the client reports no smoking for the 2 weeks prior to the follow-up, and a carbon monoxide level of less than 10 ppm is measured in expired air. To check that results were not due to the quality of CO data recording, self-reported 4-week quit rates, whether or not confirmed by a CO measurement, were included as secondary outcome measure. A self-reported quit is determined by the client reporting no smoking for the 2 weeks prior to the 4-week follow-up, regardless of whether this could be biochemically validated.

Some demographics of the clients seen by the practitioners were included. These were the proportion of men, the proportion of clients not exempt from paying NHS prescription charges (exemption can be used as a rough proxy for economic deprivation), the proportion in routine and manual occupations and clients’ average age. A measure of dependence, the Heaviness of Smoking Index (HSI [12]), was only available for clients of 30 practitioners.

Intervention characteristics included the proportion of interventions set in specialist clinics, the proportion that used one of the two most effective medication options (varenicline or combination NRT) and the proportion that used group support. Client demographics and intervention characteristics included were selected and coded in such a way that an increase would generally favour an increase in success rates. Exceptions were the HSI, as higher dependence generally makes successful quitting more difficult, and the proportion in routine and manual occupations. Any change in the proportion of clients in routine and manual occupations would require further investigation to determine the direction of the change, i.e. which other occupational grades changed at the same time.

The average number of quit attempts per month supported by each practitioner was calculated by dividing the total number of quit attempts in each year by the number of months included for that year.

Analyses

Mean number of quit attempts per month, CO-validated and self-reported success rates, demographics and intervention characteristics were compared across the 4 years using repeated measures means comparisons (general linear model) with Huynh-Feldt or Greenhouse-Geisser corrections as applicable, followed with pairwise Sidak-adjusted comparisons. For success rates, correlations across years were calculated. Effect size eta squared was calculated, which represents the proportion of variance attributable to an effect. Intra-class correlation coefficients (ICC, single measures) were also calculated for success rates to assess stability over time. To assess stability of success rates while adjusting for any other predictors, multiple regressions were modelled with change in success rates from 2009 to 2012 as outcome and demographics and intervention characteristics that changed significantly and with at least a medium effect as predictors. The significance level was set at p < 0.05 for all analyses.

RESULTS

On average, the included practitioners supported 11.4 (SD = 18.7) quit attempts per month in 2009, 12.4 (20.6) per month in 2010, 12.7 (20.6) per month in 2011 and 13.2 (21.0) per month in 2012, which are equivalent to around 150 quit attempts per year and indicate a linear increase across time (F(2.07, 405.3) = 3.18, p = 0.041, eta squared 0.016).

Success rates ranged widely between practitioners, but average self-reported and CO-validated success rates were stable across all 4 years, and consequently, the distance between them remained constant, with self-reported success rates about 14 percentage points higher than CO-validated success rates (Table 1). Most of the difference (>95 %) between the two was due to a lack of CO measurements, with only a small proportion of self-reported quits disconfirmed.

Table 1.

Success rates, client demographics and intervention characteristics over time for 197 practitioners

	2009 (Apr–Dec)	2010	2011	2012 (Jan–Apr)	F	p valuea	Effect size (eta squaredb)
Success rates % (SD), range
Mean quit rates as self-reported by clients	52.3 (15.30), 10.3 to 89.3	51.9 (14.46), 9.1 to 93.3	51.9 (14.35), 10.8 to 88.6	52.1 (16.53), 4.9 to 95.7	F(2.6, 500.8) = 0.07	0.97	<0.001
Mean CO-validated quit rates	37.8 (17.74), 0 to 82.4	38.6 (16.71), 0 to 90.0	38.2 (15.50, 0.1 to 87.3	36.9 (18.32), 0 to 91.3	F(2.5, 479.2) = 0.78	0.48	0.004
Difference CO-validated − self-reported quit rate	14.5 (12.05), 0 to 73.9	13.4 (10.91), 0 to 87.5	13.7 (11.09), 0 to 79.2	15.2 (14.85), 0 to 90.9	F(2.2, 434.5) = 1.79	0.16	0.009
Client demographics
Male, % (SD)	48.0 (11.05)	46.6 (10.23)	47.0 (9.67)	46.4 (12.56)	F(2.6, 512.1) = 1.80	0.15	0.009
Occupation, % (SD)
-Routine and manual	20.9 (12.42)	20.9 (10.73)	25.2 (12.37)	25.3 (14.81)	F(2.4, 89.2) = 17.03	<0.001	0.080
-Unable to code	17.3 (22.07)	15.0 (16.16)	7.4 (11.00)	6.1 (10.67)	F(2.0, 391.7) = 36.91	<0.001	0.158
Pays prescription charges, % (SD)	29.8 (17.10)	30.0 (15.54)	32.3 (14.11)	30.7 (15.83)	F(2.5, 483.0) = 2.90	0.034	0.015
Age at quit date, years (SD)	42.6 (4.25)	43.2 (4.00)	43.4 (3.58)	43.6 (4.32)	F(2.7, 522.3) = 7.05	<0.001	0.035
Intervention characteristics, % (SD)
Specialist settings	27.3 (41.79)	29.4 (42.73)	31.6 (44.45)	32.8 (45.51)	F(1.5, 287.7) = 10.52	<0.001	0.051
Group support	7.9 (22.57)	9.7 (24.46)	9.1 (24.87)	10.1 (26.74)	F(1.9, 373.1) = 2.94	0.06	0.015
One-to-one support	83.1 (33.07)	80.8 (33.74)	80.4 (34.41)	79.9 (36.30)	F(1.8, 185.9) = 3.53	0.04	0.018
Varenicline/combination NRT	46.6 (26.80)	53.8 (26.64)	63.3 (23.39)	62.3 (26.50)	F(2.0, 381.9) = 45.48	<0.001	0.188

SD standard deviation

^aGreenhouse-Geisser or Huynh-Feldt corrected, depending on the value of epsilon

^bEta squared is the proportion of variance attributed to the effect, interpreted as effect size: small = 0.01, medium = 0.06, large = 0.14 [22]

Significant associations were found when success rates for practitioners were correlated across all years. For consecutive periods, CO-validated success rates correlated with Pearson’s r between 0.64 and 0.68 (all p < 0.001, Fig. 1). Self-reported success rates correlated between r = 0.56 and r = 0.63 (all p < 0.001). Effects decreased with increasing distance in time but remained significant with moderate to large effect sizes over non-consecutive periods (CO-validated success rates: r = 0.39 to r = 0.51; self-reported success rates: r = 0.33 to r = 0.52, all p < 0.001). Intra-class correlation coefficients (single measures) across the 4 years were ICC = 0.56 for CO-validated success rates and ICC = 0.50 for self-reported success rates.

Fig 1.

Scatterplots for biochemically validated quit rates for 197 practitioners, change year on year (a–c) and change from first to last period of analysis (d)

As shown in Table 1, there were some changes in client demographics. In particular, the proportion of clients with routine and manual jobs increased significantly from 2010 to 2011 (p < 0.001) and remained at the higher level in 2012. Further investigation indicated that this may be an artefact of changes in data recording as the proportion of clients recorded as ‘unable to code occupation’ decreased significantly in an opposing pattern. A difference in the proportion of clients who were not exempt from paying prescription charges across all the years was indicated in the omnibus result, but none of the pairwise comparisons reached significance. Client age increased slightly across the years; in particular, clients in 2009 were younger than in later years (all p < 0.05). There were no significant differences in HSI scores across time in the small subsample with information available (p = 0.73; M (SD): 2009: 3.5 (0.71), 2010: 3.4 (0.47), 2011: 3.4 (0.49), 2012: 3.4 (0.47)).

Characteristics of the intervention provided also differed across the years (Table 1). A major change was observed in the proportion of quit attempts using the two most effective medication options, with significant increases from 2009 to 2010 and 2010 to 2011 (both p < 0.001). The proportion of quit attempts set in specialist clinics increased linearly (2009 to 2010 p = 0.05, 2010 to 2011 p = 0.03, 2011 to 2012 p = 0.29). The proportion of group support increased overall, but differences failed to reach significance. Because of the small proportion and large standard deviation of group support, the proportion of support delivered as one-to-one support was also tested. It decreased linearly over the years, with a significant reduction from 2009 to 2010 (p = 0.034).

In linear regressions, the change in CO-validated quit rates from 2009 to 2012 was not associated with change in medication (p = 0.21) or change in routine and manual occupations (p = 0.66). Similarly, change in medication (p = 0.32) and occupational grade (p = 0.93) did not predict change in self-reported success rates.

DISCUSSION

A sample of stop smoking practitioners achieved stable success rates in the quit attempts they supported across several years. Success rates in temporally adjacent years were associated with each other with large effects, and moderate associations between success rates were found for periods separated by more than 1 year. As is to be expected over a period of several years, there were some changes in client demographics and intervention characteristics, particularly an increase in the use of the most effective medication options. These were not associated with changes in clients’ self-reported success rates or the more reliable biochemically validated success rates. Other changes to demographics or intervention characteristics had no detectable association with practitioners’ success rates.

The practitioners included in this study were active for at least 3 years, and it is likely that most of them had been working in their role previous to the period of time assessed here; unfortunately, no further information on experience was available. There was no evidence of improvement in their effectiveness in supporting quit attempts which is in line with previous findings that length in job did not mediate an association with success rates for stop smoking practitioners [13] and some research showing a lack of improvement of outcomes for clients of psychotherapists or counsellors with increased counsellor experience [14]. The increase in the use of more effective medication options indicates that they incorporated new evidence into their practice to some extent, but these changes were not enough to positively affect overall outcomes. The level of success of this sample of practitioners was slightly above the average level of success in all English stop smoking services, which achieved overall self-reported 4-week quit rates between 48.7 and 49.1 % and CO-validated success rates between 33.4 and 35.3 % in the corresponding years [2, 9, 10]. Practitioners in this sample may have reached an asymptote in their improvement; analyses of practitioners new to the job may find evidence of improvement over time.

The included quit attempts were generally representative of all quit attempts supported by the English stop smoking services in the years 2009 to 2012[2, 12, 13], suggesting findings can be generalised beyond this sample of practitioners and quit attempts.

Generally, there is a lack of information on what makes individual practitioners more effective in the support they provide. The inclusion of certain behaviour change techniques (BCTs) in treatment manuals has been shown to be associated with success rates on a service level [15, 16]. Future research should examine the extent to which the delivery of BCTs and the quality of their delivery by individual practitioners is associated with their clients’ chances to succeed. Methods to assess the delivery of BCTs in practice are being developed [17]. Importantly, future research needs to address the extent to which training enables practitioners to improve their practice in a way that is reflected in outcomes. The number of days of training and the observation of an experienced colleague have been found to mediate the difference in success rates between groups of practitioners [13].

While the availability of CO-validated success rates is a strength, the present study had some limitations. There may have been changes throughout the period of data collection that potentially affected success rates beyond the demographics and intervention variables included in this study. A second limitation is that the different periods did not all include the same months of the year, so that if success rates varied across the year, averaged success rates for the incomplete years would be skewed. However, evidence suggests that those attempting to quit in January/February or September/October have slightly higher quit rates [18]. One of these slightly more successful periods is included in each of the truncated years, so the effect is expected to be negligible. Also, we could not include the amount of contact each practitioner had with clients. Although treatment guidelines recommend weekly meetings over a period of 6 weeks [19], the extent to which these recommendations are followed may vary across practitioners and time. The current results indicate some implications for practice and policy. Policy makers and managers of stop smoking services should establish systems for routinely assessing success rates of practitioners and use this information to improve the quality of the service delivered. To help practitioners further improve the service they deliver, opportunities for continuing professional development should be assessed and enhanced where necessary. Surveys have for example indicated that about a quarter of practitioners is not at all aware of the success rates of their clients [20], which suggests very little feedback is given, thus making the implementation of evidence-based practice more difficult [21].

CONCLUSIONS

Practitioners’ effectiveness is moderately stable over time. Research is needed to establish what characterises the practice of more successful practitioners and how the practice of less well-performing practitioners can be improved. Policy makers and managers of stop smoking support services should put in place systems for routinely assessing success rates of practitioners and use this information to improve the quality of service delivery.