Model‐based economic evaluations in smoking cessation and their transferability to new contexts: a systematic review

Abstract

Aims

To identify different types of models used in economic evaluations of smoking cessation, analyse the quality of the included models examining their attributes and ascertain their transferability to a new context.

Methods

A systematic review of the literature on the economic evaluation of smoking cessation interventions published between 1996 and April 2015, identified via Medline, EMBASE, National Health Service (NHS) Economic Evaluation Database (NHS EED), Health Technology Assessment (HTA). The checklist‐based quality of the included studies and transferability scores was based on the European Network of Health Economic Evaluation Databases (EURONHEED) criteria. Studies that were not in smoking cessation, not original research, not a model‐based economic evaluation, that did not consider adult population and not from a high‐income country were excluded.

Findings

Among the 64 economic evaluations included in the review, the state‐transition Markov model was the most frequently used method (n = 30/64), with quality adjusted life years (QALY) being the most frequently used outcome measure in a life‐time horizon. A small number of the included studies (13 of 64) were eligible for EURONHEED transferability checklist. The overall transferability scores ranged from 0.50 to 0.97, with an average score of 0.75. The average score per section was 0.69 (range = 0.35–0.92). The relative transferability of the studies could not be established due to a limitation present in the EURONHEED method.

Conclusion

All existing economic evaluations in smoking cessation lack in one or more key study attributes necessary to be fully transferable to a new context.

Introduction

The core strategies in reducing smoking prevalence are to prevent people from starting smoking, to reduce the number of smokers and to decrease the chances of relapse. This can be achieved by implementing population‐based tobacco control policies (e.g. legislations and mass media campaigns) and smoking cessation programmes (e.g. drug or behavioural therapies) targeted at current smokers. However, due to the increasing number of interventions now available, decision‐makers face difficulties in deciding which intervention to implement. Given scarce resources, relative costs and benefits of those interventions are one of the key decision‐making criteria, thus making the importance of economic evaluations rise in recent years 1, 2.

Economic evaluations combine the outcomes of interventions with their costs, in order to determine which intervention provides the best value for money 3. Such evaluations, for example, have shown that treatment with varenicline 4, 5 or behavioural support by mobile phone 6 can be cost‐effective. Model‐based economic evaluations are especially appropriate to extrapolate the benefits beyond clinical trials and when a single primary source of data is not sufficient 7. In addition, a model‐based economic evaluation has the ability to adapt itself to a new context, making the process of executing economic evaluations less time‐consuming and thus less costly 8, 9. Unfortunately, such evaluations often originate in affluent societies. The number of lives that can be saved from the use of such evidence elsewhere (e.g. countries in Central and Eastern Europe) is potentially enormous. Sadly, those countries often have too limited research resources to study cost‐effectiveness of such interventions in their own context, highlighting the importance of transferability assessments 9, 10.

The notion of transferability of evidence from one context to others varies widely in the literature. ‘Transferability’, ‘generalizability’ and ‘external validity’ are the concepts used to assess the ability of a study to be relevant to the decision maker's context to the extent the findings could actually be used 11, 12, 13, 14, 15. However, a distinction also exists between what is feasible/applicable and what is generalizable/transferable. Applicability refers to ‘how can I replicate the intervention in my own decision context?’ (the process question) and generalizability refers to ‘whether the effectiveness will be similar to that in the original context?’ (the outcome question) 12, 13, 15, 16. Therefore, these two underlying questions seem to have defined transferability in the literature.

Transferability assessments to date have focused mainly on the way in which a model is constructed and populated, as modelling provides a well‐defined structure helping us to recognize the limitations and their implications for generalizability of the results 7, 17, 18, 19. There has not been a systematic enquiry in to the transferability of economic evaluations in smoking cessation, although a few systematic reviews in this area exist 20, 21. The review by Kirsch et al. 21, for instance, limits itself to a narrow definition of study population and to a specific type of economic model. In this paper, we therefore set out to: (i) identify different types of models used in economic evaluations of smoking cessation; (ii) analyse the quality of the included models examining their attributes; and (iii) ascertain their transferability to a new context.

Methods

Search strategy and implementation

A systematic search was conducted to identify all relevant models used for economic evaluation in smoking cessation on the following databases: National Health Service (NHS) Economic Evaluation Database (NHS EED), Health Technology Assessment (HTA), Medline and EMBASE. They were searched for publications in English language between 1996 and April 2015. The search strategy was based on related published systematic reviews 20, 22, 23, 24, leading to the final search terms ‘smoking’, ‘nicotine’ and ‘tobacco’ in NHS EED and HTA. Medline and EMBASE required additional terms related to model‐based economic evaluation, which were based on Wilczynski et al. 25 and McKinlay 26 to acquire high sensitivity as well as high specificity 27. Supporting information, Table S1 shows an overview of the search strategies used by databases. All results were exported to EndNote (Thomson Reuters) version X7, where duplications were removed automatically and remaining duplicates checked manually.

Exclusion criteria and screening

Title and abstract screening for the first 50 papers was performed independently by two reviewers (M.H. and M.B.) based on the following exclusion criteria: (1) topic not in smoking cessation (as the focus was on the interventions to reduce tobacco use), (2) no original research (to avoid inclusion of review of evidence or opinion pieces), (3) no model‐based economic evaluation (to avoid inclusion of other designs, e.g. trial‐based evaluations), (4) no adult general population (to focus on adults, rather than children), (5) no high‐income country (to reduce study heterogeneity by including comparable, industrialized countries based on their income levels) and (6) not available in the English language (practicality reasons mainly to address resource constraints). No differences in exclusion/inclusion were observed between both reviewers; only minor discrepancies were recorded in the reason of exclusion. The inter‐rater reliability (IRR) gave a Cohen's kappa of 0.912, meaning almost perfect agreement 28. Remaining discrepancies were discussed, leading to full agreement. Screening of the remaining papers was then completed by one researcher (M.B.). Full text screening was performed independently by two reviewers (M.B. and K.L.C. or M.H.). There were only minor discrepancies between the reviewers, which led to full agreement after discussion. Supporting information, Tables S2 and S3 show an extended list of exclusion criteria for full‐text screening.

Data extraction

Data on the following items were extracted using an Excel template adapted from published studies 20, 29, 30 and included: study attributes (type of evaluation, interventions, comparator and country); model (type, transition or health states, time horizon and perspective); effectiveness (outcome and discount rate, primary measure of effectiveness and utility valuations); costs (perspective, categories, resource, index year and discount rate); uncertainty (type and outcome of sensitivity analysis); and results and major limitations.

As data from some included studies were already extracted by the University of York's Centre for Reviews and Dissemination (CRD) (n = 39 of 64), only one researcher (M.B.) extracted data independently on those studies and compared with the CRD extraction. The CRD database contains clear and structured summaries of the economic analyses by experts, and therefore it was deemed sufficient to compare the results of data extraction to these summaries. For the remaining studies that were not included in the CRD database, the data were extracted independently by two reviewers (M.B. and one of the following: M.H., K.L.C., R.D.K. and P.K). Any disagreement between the reviewers was resolved by consensus with a third reviewer.

Quality appraisal

In order to appraise the quality, 10% of the included studies were first assessed independently by M.B. and M.H., using a quality checklist and corresponding classification from the National Institute for Health and Care Excellence (NICE) Methodology Guide with the aim to filter out quality‐poor studies 31. The quality checklist was based on three major criteria: (1) the study was conducted from a relevant perspective (i.e. at least payer or health‐care perspective; (2) the study was a cost–utility or cost–benefit analysis with cost/quality adjusted life years (QALY) or benefit–cost ratio reported; and (3) limitations, either stated in the original study or identified by the reviewers during data extraction stage. Once the overall assessment using these criteria was completed, the studies were assigned to one of the following three classifications: (i) a study with minor limitations (ML); (ii) a study with potentially serious limitations (PSL); or (iii) a study with very serious limitations (VSL). As full agreement on quality classification was reached in the 10% of the included studies, M.B. then completed the quality appraisal of the remaining studies.

Transferability assessment

The studies appraised as the one with minor limitations (ML) were considered to be of sufficient quality to be included for transferability assessment applying the EURONHEED checklist 9. Two independent researchers (M.B. and one of the following: M.H., K.L.C., R.D.K. and P.K.) applied the checklist. The EURONHEED checklist was developed originally by Boulenger et al. 9 and described and updated further with guidelines by Nixon et al. 32. It consists of 42 questions, 26 of which relate to overall methodological quality and internal validity, and 16 questions relate to transferability. An overview of all questions is provided in Supporting information, Table S4. Every question can be answered by ‘yes/partially/no or not applicable (NA)’, assigning a score of 1, 0.5 and 0, respectively. While each item in the checklist is treated equally (but implicitly giving more weight to 16 of the 42 items), the assigned score to each question thus additionally provides another weight to reflect the extent to which each item was reported in the study being assessed 32. The combination of the questions generates an overall summary score 9, 10. We calculated two summary scores: the total summary score including all 42 items and the transferability score including the 16 items. The summary scores were calculated using the following formula; $\frac{1}{n-x}{\sum }_i\mathit{Si}\times 100$ , in which n is the number of questions, x is the number of questions for which the response was NA and S is the score of each question 9. The summary scores reflect how thoroughly key methodological items are reported as the quality of reporting is paramount for generalizability/transferability 32. In addition to this, we calculated the scored percentage of the total score possible per section. This showed us what sections within model‐based economic evaluations were of sufficient quality and which needed further improvement. For example, a score of 0.75 means that 75% of this section is of sufficient quality.

Results

Search outcomes

The systematic literature search yielded 1925 references. After removing duplicates, 1500 studies were included for title and abstract screening which led to a total of 101 studies selected for full text screening. On applying the exclusion criteria, 64 studies were judged to be eligible for data extraction. Thirteen of the 64 studies were included for transferability assessment. An overview of the process is provided in Fig. 1.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) flow diagram, based on National Health Service Economic Evaluation Database (NHS EED) and Health Technology Assessment Database (HTA). [Colour figure can be viewed at wileyonlinelibrary.com]

Overview of studies

An overview of the identified models is shown in Table 1. Most studies originated from Europe (n = 30 of 64) and the United States (n = 24 of 64), followed by Australia (n = four of 64) and Asia (n = two of 64). Three of 64 studies were multi‐continental.

Table 1.

Overview of studies by population, intervention, comparators and outcome.

Author, year	Country	Population	Intervention	Comparator	Outcome
Ahmad, 2005a	CA, USA	General Californian population	Raising legal smoking age from 18 to 21	Legal smoking age 18, 19, 20	QALY
Ahmad, 2005b	USA	General American population	Raising legal smoking age from 18 to 21	No intervention	LY gained and QALY
Annemans, 2015	Belgium	18+ Belgian smokers	Varenicline in retreatment	No treatment, and retreatment with bupropion or NRT	QALY
Annemans, 2009	Belgium	18+ Belgian smokers	Varenicline	Pharmacotherapies, brief counselling and unaided cessation	LY gained and QALY
Athanasakis, 2012	Greece	18+ Greek smokers	Varenicline	Bupropion, NRT and unaided cessation	QALY
Bae, 2009	South Korea	General Korean population	Varenicline	NRT, bupropion and no drugs	QALY
Bauld, 2011	Scotland	Not reported	One‐to‐one counselling or group‐based support programme	No intervention	QALY
Bertram, 2007	Australia	Australian smokers aged 20–79	NRT or bupropion	No intervention	DALY
Bolin, 2006	Sweden	Swedish smokers aged 35+	Bupropion tablets with four nurse visits for motivational support	NRT	QALY
Bolin, 2008	Sweden	Swedish smokers aged 18+	Varenicline	Bupropion	QALY
Bolin, 2009a	Sweden	Swedish adult population	12‐week varenicline treatment expanded with 12 weeks of maintenance with varenicline	12 weeks of varenicline +12 weeks of placebo	QALY
Bolin, 2009b	Belgium, France, Sweden	Not reported	Varenicline	NRT	QALY
Boyd, 2009	UK	Glasgow smoking population	‘Starting fresh’ and ‘Smoking concerns’	Self‐quit	QALY
Brown, 2014	England	16+ without having quit successfully in the last month	Stoptober	Usual situation for all other months	LY gained and QALY
Cantor, 2015	USA, Texas	Physicians and pharmacists from 16 communities in Texas Participants: 18+	The health‐care team approach to smoking cessation: ETOEP	Usual practice	QALY
Chevreul, 2014	France	Insured current French smokers aged 15–75 years	Full coverage of the medical management of smoking cessation	Current situation	ICER per LY gained
Cornuz, 2006	Canada, France, Spain, Switzerland, UK, USA	Smokers smoking > 20 cigarettes per day	Four NRTs (gum, patch, spray, inhaler) and bupropion, given as adjunct to cessation counselling	Not Reported	LY gained
Cornuz, 2003	A European country (some data used from Switzerland)	Smokers smoking > 20 cigarettes per day	Four NRTs (gum, patch, spray, inhaler) and bupropion, given as adjunct to cessation counselling	GP counselling during routine visit	Incremental cost per LY gained
Croghan, 1997	USA, Rochester	Smokers aged 18+	Non‐physician smoking cessation counselling	No intervention	LY gained
Dino, 2008	USA	Adolescents aged 17–25 years	American Lung Association's Not On Tobacco national teen smoking cessation programme	Brief intervention	Discounted LY
Feenstra, 2005	The Netherlands	Dynamic population	Face‐to‐face smoking cessation interventions	Current situation	LY gained and QALY
Fiscella, 1996	USA	Not reported	Nicotine patches as an adjunct to physician‐based counselling	Physician‐based counselling	QALY
Guerriero, 2013	UK	Smokers aged 16+	Text‐based support in adjunct to current practice	Current situation	LY gained and QALY
Halpern, 2007a	USA	Not reported	Varenicline	Nicotine patch, bupropion, and no pharmacotherapy	ROI, IRR, B–C‐ratio
Halpern, 2007b	USA	Reflection of US population	Work‐place smoking cessation coverage	No coverage	IRR, ROI
Heitjan, 2008	USA	American whites	Nicotine patch, bupropion, varenicline and tailored therapy based on genetic testing	No intervention	Residual LY
Hill, 2006	USA	Not reported	NRT (gum, patch, inhaler, nasal spray), Zyban or combinations	No intervention	ICER
Hojgaard, 2011	Denmark	General Danish population	Smoking cessation programme and a smoking ban	Current situation	LY gained
Hoogendoorn, 2008	The Netherlands	General Dutch population	Varenicline	No intervention, bupropion, nortriptyline or NRT	Number of quitters, LY gained, and QALY
Howard, 2008	USA	US adult 18+ population	Varenicline	Bupropion, NRT, and unaided quitting	QALY
Hurley, 2008	Australia	General Australian population	Australian National Tobacco Campaign	Current situation	LY gained and QALY
Igarashi, 2009	Japan	Japanese smokers aged 20+ smoking >20 cigarettes per day	Varenicline combined with counselling	Counselling	QALY
Jackson, 2007	USA	Not reported	Varenicline	Bupropion	Net benefit
Knight, 2010	USA	General American population making single quit attempt	Varenicline 12 + 12 weeks	Bupropion, NRT and unaided cessation	QALY
Lai, 2007	Estonia	Estonian smokers aged 15–59	Increase of tax, clean indoor air law enforcement, and NRT	No intervention (do‐nothing counterfactual)	DALY
Lal, 2014	Australia	Smokers aged 35–100	Telephone counselling	Self‐help	DALY
Levy, 2006	USA	Employees aged 18–64	Four coverage scenarios	No coverage	Changes in medical expenditures
Levy, 2002	USA	Hypothetical cohort of smokers	Coverage of costs of different combinations of treatment, and brief interventions by care providers	No intervention	Quit rates
Linden, 2010	Finland	Finnish adult smokers making a first quit attempt	Varenicline	Prescribed medicine, bupropion or unaided cessation	LY gained and QALY
McGhan, 1996	Not reported	Not reported	Self‐care, behavioural therapy, group withdrawal clinic or nicotine patch	Not reported	Net benefit
Nielsen, 2000	USA	Smokers enrolled on a smoking cessation programme	Nicotine patch, bupropion, or combination	Placebo	Net benefit
Nohlert, 2013	Sweden	General Swedish population	Low and high intensity smoking cessation program	No intervention	QALY
O'Donnell, 2011	USA	Dynamic population	Cold turkey, behavioural therapy, medication therapy or combinations	No intervention	LY gained
Olsen, 2006	Denmark	General Danish population	Group courses, individual courses or quick interventions	No intervention	LY gained
Ong, 2005	USA, Minnesota	Minnesota population of smokers	Free NRT	State‐wide campaign of smoke‐free work‐places	QALY
Over, 2014	The Netherlands	Dutch smokers aged 25–80	Tax increase or reimbursement	Current situation	QALY
Pinget, 2007	Switzerland	Swiss smokers	Physician training in smoking cessation counselling	Physician training in dyslipidaemia management	LY gained
Ranson, 2002	139 countries	Current smokers in 1995	Tobacco control policies (price increases, NRT, non‐price interventions)	No tobacco control policy	DALY saved
Shearer, 2006	Australia	General Australian population	Brief advice, telephone counselling, NRT or bupropion	No intervention, brief advice, counselling or pharmacotherapies	ICER
Simpson, 2013	USA	New York State aged 18+	New York Tobacco Control Programme	No intervention	Smoking costs avoided
Song, 2002	UK	Hypothetical cohort of smokers	Advice plus NRT, advice plus bupropion or advice plus NRT and bupropion	Advice or counselling only	LY gained
Stapleton, 1999	UK	Smokers in general	Transdermal nicotine patches with GP counselling	GP counselling	LY gained
Stapleton, 2012	Data used from USA and UK	Smokers in general	Cytisine, Agency for Health Care Policy and Research Guideline for smoking cessation, NICE appraisal of NRT, or effect size given as an odds ratio or relative rate	Placebo	LY gained
Taylor, 2011	UK	Hypothetical cohort of smokers who recently initiated quit attempts	NRT, bupropion or varenicline	No drug therapy	QALY
Tran, 2002	USA, Virginia	Smokers aged 21–70 who tried (at least once) to quit smoking	Cold turkey, nicotine patch, nicotine gum or bupropion	Self‐quit	QALY
Van Baal, 2007	The Netherlands	Dynamic population	Tobacco tax increase	Current situation	LY gained and QALY
Van Genugten, 2003	The Netherlands	Dutch population	Policy measures (‘Don't start’, ‘quit’, ‘tax’)	Future smoking prevalence is based on trend extrapolation	DALY
Vemer, 2010a	The Netherlands, Belgium, Germany, Sweden, France, and UK	Smokers aged 18+ in the Netherlands, Belgium, Germany, Sweden, France and the UK	NRT, bupropion or varenicline	Unaided quit attempt	QALY
Vemer, 2010b	The Netherlands	Dutch smokers aged 18+	Smoking cessation support	Current situation	QALY
Von Wartburg, 2014	Canada, France, Spain, Switzerland, UK, USA	Cohort representative of Canadian demographics, smokers who seriously consider quitting within the next 30 days	Standard 12 weeks of varenicline, or 12 + 12 weeks of varenicline	Bupropion, NRT, or unaided cessation	QALY
Warner, 1996	USA	Hypothetical cohort of blue‐collar workers	Work‐site smoking‐cessation programme	No intervention	LY gained, medical expenditures saved
Welton, 2008	UK	Not reported	Genetic testing of DRD2 Taq1ANRT, bupropion, their combination, or standard care	Brief advice or individual counselling	Incremental net benefit
Xenakis, 2009	USA	Not reported	Varenicline with counselling	Counselling + bupropion or placebo	Incremental costs
Xu, 2014	USA	US adult 18+ population	Anti‐smoking campaign	No campaign	LY gained and QALY

NRT = nicotine replacement therapy; QALY = quality adjusted life years; DALY = disability adjusted life years; NICE = National Institute for Health and Care Excellence; GP general practitioner; ICER = incremental cost‐effectiveness ratio; LY = life years; IRR = inter‐rater reliability; ROI = return on investment; B–C = benefit–cost.

The populations in the analyses were described mainly as the general adult population of smokers. In three studies the populations were described further as smoking 20 cigarettes per day or more 33, 34, 35, making or considering a single or first quit attempt 36, 37, 38, 39 or had recently tried to quit smoking 40, 41. In five studies the population was described only as a dynamic and/or hypothetical cohort 42, 43, 44, 45, 46 and in nine studies the population was not reported at all 47, 48, 49, 50, 51, 52, 53, 54, 55.

A significant part of the intervention was smoking cessation programmes, either pharmacotherapy 4, 5, 36, 37, 38, 40, 41, 48, 50, 51, 53, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, behavioural therapy 6, 42, 47, 66, 67, 68, 69 or a combination of these 33, 34, 35, 43, 45, 46, 49, 52, 54, 70, 71, 72, 73, 74, 75. Several studies evaluated wider tobacco control interventions 39, 44, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, whereas five studies included both smoking cessation programmes and tobacco control interventions 89, 90, 91, 92, 93.

In a number of studies, the authors selected ‘no intervention’ or ‘current situation’ as comparator. All other studies described the comparators in more detail (Table 1).

The main measure of outcome used is the QALY. In total, 23 of 64 studies reported QALY as their main outcome 5, 35, 38, 40, 41, 47, 48, 49, 56, 58, 59, 61, 62, 63, 65, 69, 70, 76, 78, 81, 86, 88, 94, followed by life years (LY) gained (n = nine of 64) 33, 43, 46, 66, 67, 68, 73, 74, 89 or a combination of these (n = 12 of 64) 4, 6, 35, 36, 37, 39, 42, 44, 57, 77, 80, 83. Five of 64 studies reported disability adjusted life years (DALY) as their main outcome 60, 82, 90, 91, 92, and only four of 64 (incremental) net benefit 52, 53, 55, 71. There were two of 64 studies reporting only the intermediate outcomes of the intervention 85, 93 (Table 1).

Overview of economic models

Table 2 shows the main model attributes used in the included studies. Thirty of 64 studies used a Markov model, 12 of which used a specific type called the benefits of smoking cessation on outcomes (BENESCO) model 4, 5, 36, 37, 48, 56, 57, 58, 59, 61, 62, 65. Decision‐tree models 41, 43, 52, 55, 63, 71, 75, 83, 93, discrete‐event simulations (DES) 45, 54, the chronic disease model (RIVM‐CDM) 44, 81, 88, the tobacco policy model (TPM) 76, 77, the quit benefits model (QBM) 80, the World Health Organization (WHO) model 90, the global health outcomes model (GHO model) 70 and the abstinent‐contingent treatment model (ABT model) 73 were also used. Twelve of 64 studies did not report explicitly the model used, reporting only decision analysis modelling or simulation modelling 39, 50, 51, 66, 69, 72, 74, 78, 86 or limiting the description to only dynamic or static modelling 42, 82, 92.

Table 2.

Characteristics showed per model and summary of most reported characteristics.

Type of model	Study	Characteristics
		Transition/health statesa	Time‐horizon	Perspective	Discounting		Analysis
					Effects	Costs	Primary measure of effectiveness	Sensitivity analysisb
Markov (n = 30)	Annemans, 2015	4	Life‐time	Health‐care payer	1.5 and 3%	1.5 and 3%	Abstinence rates	USA and PSA
	Annemans, 2009	4 + 6	Life‐time	Health‐care payer	1.5%	3%	Continuous abstinence rates	USA and PSA
	Athanasakis, 2012	5	Life‐time	Societal	3%	3%	Continuous abstinence rates	PSA
	Bae, 2009	NR	Life‐time	NR	5%	5%	Quit rates	USA and PSA
	Bertram, 2007	3	Life‐time	Health‐care system	3%	3%	Quit rates	PSA
	Bolin, 2008	NR	20 and 50 years	Health‐care and societal	3%	3%	Probability of cessation	DSA and PSA
	Bolin, 2009a	NR	50 years	NR	3%	3%	Smoking prevalence and quit rates	USA, MSA, and PSA
	Bolin, 2009b	SC intervention +4	Life‐time	Health‐care system	3.5%	3.5%	Continuous abstinence rates	PSA, MSA, and DSA
	Chevreul, 2014	3	Life‐time	Social Health Insurance	3%	3%	Quit rates	PSA
	Cornuz, 2006	NR	Life‐time	NR	NR	3%	Odds ratio for quitting	USA
	Cornuz, 2003	NR	NR	Third‐party payer	3%	3%	Odds ratio for quitting	NR
	Dino, 2008	Current smoker, quit, reduce, stay smoker	Life‐time	School	3%	3%	Quit rates	MSA and ECA
	Fiscella, 1996	NR	NR	Health‐care payer	3%	3%	Cessation rates	USA and PSA
	Guerriero, 2013	3 + MI, CHD, stroke, lung cancer, COPD	Life‐time	Health service (UK NHS)	3.5%	3.5%	Relative risk of quitting, relapse rates	DSA and PSA
	Heitjan, 2008	NR	NR	NR	NR	3%	Initiation rates and successful quit attempts	USA and ECA
	Hojgaard, 2011	2	10 years and life‐time	Societal	3.5%	3.5%	Quit and relapse rates	ECA
	Hoogendoorn, 2008	4 + 6	Life‐time	Health‐care payer	1.5%	4%	Abstinence rates	USA and PSA
	Howard, 2008	4 + 6	Life‐time	Health‐care system	3%	3%	Continuous abstinence rates	USA and PSA
	Igarashi, 2009	Success‐alive, failure‐alive, sick‐smoke, sick‐non‐smoke, death	Until age 90	Health‐care payer	3%	3%	Abstinence rates	USA, MSA, and PSA
	Knight, 2010	NR	Life‐time	NR	3%	3%	Quit rates	USA and PSA
	Lal, 2014	3 + Mortality due to: cancer, COPD, CHD, stroke, other diseases	Life‐time	Health sector	3%	5%	Quit rates	PSA
	Levy, 2006	NR	20 years	Employer	NR	5%	Probability of smoking cessation	DSA
	Linden, 2010	4 + 6	Life‐time	Societal	5%	5%	Continuous abstinence rates	USA, MSA, and PSA
	Olsen, 2006	3	Life‐time	Payer	3.5%	3.5%	Abstinence rates	USA and PSA
	Pinget, 2007	NR	1 year	Third‐party payer	NR	3%	Point abstinence at 1 year	USA
	Simpson, 2013	Quit or continue smoking	20 years	NR	3%	3%	Rates for media awareness and quitline and (NYTCP) NRT utilization rates	NR
	Taylor, 2011	Recent quitter, smoker (lung CA, CHD, MI, stroke, COPD), former smoker (lung CA, CHD, MI, stroke, COPD), dead	Life‐time	Health service (UK NHS)	3.5%	3.5%	Abstinence rates	USA
	Vemer, 2010a	4	Life‐time	Health‐care system	0–5.0%	3.0–5.0%	Change in incremental net monetary benefits	NR
	Von Wartburg, 2014	Exclusive health states as a function of their demographics and smoking status.	Life‐time	Health‐care system and societal	NR	5%	Quit rates	USA and PSA
	Welton, 2008	NR	Life‐time	Health service (UK NHS)	Not discounted	Not required	Abstinence rates	MSA and PSA
Most reported		NR (n = 11), 4 (n = 3) and combined with 6 (n = 4)	Life‐time (n = 21)	Health‐care system/payer (n = 17)	3% (n = 12)	3% (n = 16)	Quit/abstinence rates (n = 24)	USA with PSA (n = 9)
Decision‐tree model (n = 9)	Boyd, 2009	NR	4 or 52 weeks	Health service (UK NHS)	NR	NR	Quit rates	USA and MSA
	Levy, 2002	Quit attempt or no quit attempt, quit or fail	1 year	Health‐care payer	NR	Not required	Predicted quit rates	USA and MSA
	McGhan, 1996	NR	NR	Employer	NR	NR	Quit rates	NR
	Nielsen, 2000	NR	NR	Employer	NR	3%	Quit rates	USA
	Song, 2002	NR	NR	Health service (UK NHS)	NR	Not required	Quit rates	ECA
	Tran, 2002	NR	1 year	Payer	3%	Not required	Continuous abstinence rates	USA
	Halpern, 2007b	Quit attempt or no quit attempt, quit or fail, resume	2, 5, 10 or 20 years	NR	NR	3%	Quit rates	NR
	Jackson, 2007	Quit or continue smoking	1 year	Employer	NR	Not required	Continuous abstinence rates	NR
	Xu, 2014	Current smoker, quit attempt or continue smoking	NR	Funding agency	3%	3%	Quit rates	USA
Most reported		Quit attempt or no quit attempt, (quit or fail) (n = 4)	Short‐term (n = 5)	Health‐care system/payer (n = 4)	3% (n = 2)	3% (n = 3)	Quit/abstinence rates (n = 9)	USA (n = 3) or in combination with MSA (n = 2)
Remaining models reported (n = 25)
Markov & Monte Carlo	Bauld, 2011	Ex‐smoker, smoker, death and smoking‐related death	1 year or life‐time	Health service (UK NHS)	3.5%	NR	Continuous abstinence rates	DSA
DES	Warner, 1996	NR	50 years	Societal and employer	NR	3%, 3.5%, 4%	Quit rates	USA and ECA
	Xenakis, 2009	NR	1 year	Health‐care payer	NR	Not required	Continuous abstinence rates	USA
CDM	Over, 2014	1 + age, gender, SES	75 years	Health‐care system	NR	1.5% and 4%	Quit rates	USA and MSA
	Van Baal, 2007	1 + 14‐smoking related chronic diseases	100 years	Health‐care system	1.5%	4%	Price elasticity of tobacco consumption	USA
	Vemer, 2010b	NR	20 years and life‐time	Health‐care system	1.5%	4%	Additional number of successful quitters	NR
TPM	Ahmad, 2005a	1	50 years	Societal	3%	3%	Initiation rates	NR
	Ahmad, 2005b	1	50 years	Societal	3%	3%	Initiation rates	USA
QBM	Hurley, 2008	NR	Life‐time	NR	3%	3%	Reduction in smoking prevalence	DSA, MSA, and PSA
WHO model	Lai, 2007	NR	100 years	Societal	3%	3%	Change in disease incidence	ECA
GHO	Bolin, 2006	4	20 years	Health‐care and societal	3%	3%	QALY	USA, MSA, and PSA
ACT	Stapleton, 1999	NR	Life‐time	Health service (UK NHS)	1.75%	Not required	Additional number of LY saved	USA
Decision analytical/simulation modelling	Brown, 2014	NR	Until age 65	NR	3.5%	NR	Increase in quit attempts	USA
	Cantor, 2015	Short term: quit or no‐quit. Long term: alive or dead	1 year or life‐time	Health‐care provider	3%	3%	Quit rates	USA and MSA
	Croghan, 1997	NR	Life‐time	NR	0%, 3%, 5%	Not required	Abstinence rates	USA
	Halpern, 2007a	Continued cessation, relapse, resume smoking, continued smoking	10 years	NR	NR	3%	Quit rates	NR
	Hill, 2006	NR	6 months	Texas government	NR	Not required	% individuals not smoking at 6 months	USA and MSA
	Nohlert, 2013	NR	Until age 85	Societal	3%	3%	Abstinence rates	USA, MSA, and PSA
	Ong, 2005	NR	1 year	NR	3%	Not required	Sustained quitters generated	MSA and PSA
	Shearer, 2006	NR	NR	Government	NR	Not required	Continuous abstinence rates	MSA
	Stapleton, 2012	NR	Life‐time	Health service	3.5%	1.5–3.5%	Abstinence rates	Various possible
Dynamic/static modelling (n = 3)	Feenstra, 2005	1	75 years	Societal	4%	4%	Abstinence rates	USA and MSA
	Ranson, 2002	NR	NR	NR	3.0–10.0%	3.0–10.0%	Number of deaths averted	ECA
	Van Genugten, 2003	Current or former smoker. Lung cancer, CHD, stroke, and COPD	Period 1998–2050	NR	NR	NR	Total number of life‐years lost as the sum of the remaining life expectancy at the age of death	MSA
SmokingPaST Framework (n = 1)	O'Donnell, 2011	NR	NR	NR	NR	NR	Quit attempts	NR
Most reported		Not reported (n = 15), 1 (n = 3)	Life‐time (n = 7)	Health‐care system/payer (n = 10)	Not reported (n = 8), 3% (n = 8)	3% (n = 8)	Quit/abstinence rates (n = 13)	USA (n = 6) or combinations with USA (n = 7)

^aThis refers to the states considered in the model and may include: (1) never smoker, current smoker, former smoker; (2) never smoker, current smoker, ex‐smoker, death; (3) current smoker, former smoker, death; (4) current smoker, recent quitter, long‐term quitter; (5) no morbidity, chronic obstructive pulmonary disease (COPD) or lung cancer, coronary heart disease (CHD) or stroke first event, CHD or stroke subsequent event, death from CHD/stroke, death from COPD/lung cancer, death (all cause); (6) no current morbidity, asthma exacerbation, CHD or stroke: post first event, COPD or lung cancer, CHD or stroke: post subsequent event, death (CHD or stroke), death (COPD or lung cancer), death (all cause).

^bUncertainty analysis: USA = univariate sensitivity analysis; MSA = multivariate sensitivity analysis; ECA = extreme case analysis; PSA = probabilistic sensitivity analysis; DSA = deterministic sensitivity analysis; NRT = nicotine replacement therapy; NYTCP = New York Tobacco Control Program; SES = socio‐economic status; MI = minor limitations; SC = ; NR = not reported; QALY = quality adjusted life years.

Several (18 of 30) studies based on Markov models provided sufficient information on transition or health states used in the model. The most frequently used transition states were current smoker, former smoker or death, while health states included asthma exacerbation, coronary heart disease (CHD), stroke, chronic obstructive pulmonary disease (COPD) and lung cancer. In decision‐tree models (n = nine of 64) the most reported transition states were quit attempt or no quit attempt, often combined with success to quit or failure to quit.

The majority of the Markov models used a life‐time horizon (n = 22 of 30) while decision‐tree models considered a time between 1 and 50 years. Most of the studies based on other models lacked sufficient information, or reported a time‐horizon of 50 years. Most evaluations used a health‐care and/or payer perspective (n = 50 of 64). Twelve of 64 used a societal perspective. The reported primary measure of effectiveness in all models was quit rate or its variants (e.g. continuous abstinence rates).

The majority of the studies (n = 55 of 64) performed sensitivity analyses to account for uncertainties in their estimates. Markov model‐based studies performed mainly both univariate and probabilistic sensitivity analyses, decision‐tree models used univariate sensitivity analyses often in combination with multivariate sensitivity analyses (n = five of nine), and the other models (n = 25 of 64) conducted univariate sensitivity analyses (n = 13 of 25).

Quality assessment and transferability

Of the 64 included studies assessed for quality, 15 were excluded based on the first criteria (no health‐care perspective), 12 based on the second (no cost benefit or cost–utility analysis) and 24 on the final criteria (having major limitations). As shown in Table 3, 13 of 64 studies were then classified as having minor limitations, 35 as having potentially serious limitations and 16 as having very serious limitations.

Table 3.

Results of the quality assessment.

Classification	Studies
Minor limitations	Annemans, 2015; Annemans, 2009; Athanasakis, 2012; Bolin, 2006; Bolin, 2008; Bolin, 2009b; Boyd, 2009; Cornuz, 2003; Guerriero, 2013; Hoogendoorn, 2008; Howard, 2008; Over, 2014; Stapleton, 1999
Potentially serious limitations	Ahmad, 2005a; Ahmad, 2005b; Bae, 2009; Bauld, 2011; Bolin, 2009a; Brown, 2014; Cantor, 2015; Chevreul, 2014; Cornuz, 2006; Feenstra, 2005; Fiscella, 1996; Halpern, 2007b; Heitjan, 2008; Hill, 2006; Hojgaard, 2011; Hurley, 2008; Igarashi, 2009; Linden, 2010; Levy, 2002; Nohlert, 2013; Ong, 2005; Pinget, 2007; Shearer, 2006; Simpson, 2013; Song, 2002; Stapleton, 2012; Taylor, 2011; Tran, 2002; Van Baal, 2007; Vemer, 2010a; Vemer, 2010b; Von Wartburg, 2014; Warner, 1996; Welton, 2008; Xenakis, 2009
Very serious limitations	Bertram, 2007; Croghan, 1997; Dino, 2008; Halpern, 2007a; Knight, 2010; Lai, 2007; Lal, 2014; Levy, 2006; McGhan, 1996; Nielsen, 2000; Olsen, 2006; Ranson, 2002; Van Genugten, 2003; Xu, 2014; Jackson, 2007; O'Donnell, 2011

Table 4 provides an overview of the scoring per question on the EURONHEED checklist for the 13 studies judged as having sufficient quality including the summary scores. The studies’ total scores varied between 57 and 87% and the scores of the transferability checklist from 50 to 97%.

Table 4.

Results of the European Network of Health Economic Evaluation Databases (EURONHEED) checklist.

1 = yes, 0.5 = partially, 0 = no/no information, NA = not Applicable	Annemans, (2015)	Annemans, (2009)	Athanasa‐kis, (2012)	Bolin, (2006)	Bolin, (2008)	Bolin, (2009b)	Boyd, (2008)	Cornuz, (2003)	Guerriero, (2013)	Hoogen‐doorn, (2008)	Howard, (2008)	Over, (2014)	Stapleton, (1999)
Q1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q2	0	1	1	1	1	1	0	1	1	1	1	1	1
HT1	0.5	0	0.5	0.5	1	0	1	1	1	1	0.5	1	0.5
HT2	0.5	0	0.5	0.5	0.5	1	0	1	1	1	0.5	1	0.5
SE1	0.5	0.5	1	1	1	0	1	1	0.5	0	0	1	1
SE2	0.5	1	1	1	1	1	1	0.5	1	1	1	1	1
P1	1	1	1	0.5	0.5	1	1	1	1	1	1	1	1
SP1	1	1	1	1	1	1	1	1	1	1	1	0.5	1
SP2	0.5	0.5	0.5	1	1	1	1	0	0.5	1	0.5	1	0
SP3	0	0.5	0.5	NA	1	NA	0.5	NA	0	0.5	0.5	NA	0
SP4	0	0	0	1	1	0.5	0	0.5	1	0.5	0.5	NA	0
M1	0.5	0.5	0.5	0.5	0.5	1	1	NA	1	1	1	NA	0.5
M2	1	1	1	1	1	1	1	1	1	1	1	0.5	NA
E1	NA	NA	NA	0.5	1	1	0	NA	0.5	NA	NA	NA	1
E2	NA	NA	NA	NA	1	1	0.5	NA	0.5	NA	NA	NA	1
E3	0	0	0	0	0	0	NA	0.5	NA	0	0	0	NA
E4	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA
E5	1	0.5	0.5	1	1	1	1	1	1	1	1	1	1
E6	0	0	0	0	0	0	0	0	0	0	0	0	1
E7	NA	NA	NA	0.5	0.5	1	0	NA	1	1	NA	0	0
B1	1	1	1	1	1	1	1	1	1	1	1	1	1
B2	0	0	0	0.5	0.5	0	0	NA	0.5	NA	1	0	NA
B3	1	1	1	0.5	0.5	0	0	NA	0.5	NA	0	0	NA
B4	0	0	0	NA	NA	NA	NA	NA	NA	NA	0	0	NA
B5	1	0.5	1	1	1	0	1	0	1	1	1	0	0.5
C1	1	0.5	0.5	1	1	1	1	0.5	0.5	1	1	1	1
C2	0.5	0.5	0.5	1	1	0.5	1	1	1	1	1	0	1
C3	1	1	1	1	0.5	0	1	1	0.5	1	1	0	1
C4	1	1	0.5	1	0.5	0	1	1	1	1	1	1	1
C5	0.5	0.5	1	0.5	1	1	1	0.5	1	1	1	1	1
C6	0	0	0	0.5	1	1	1	0.5	0.5	1	1	0	1
C7	1	1	1	1	1	1	1	1	1	1	1	1	1
C8	0.5	0.5	0.5	0	1	1	1	1	1	1	1	1	1
C9	1	1	1	1	1	1	1	1	1	1	1	1	1
C10	NA	NA	NA	NA	NA	0.5	NA	1	NA	NA	NA	NA	NA
C11	1	1	1	1	1	0	0	1	0.5	1	1	0	0
D1	1	1	1	1	1	1	1	1	1	1	1	1	1
D2	1	1	1	1	1	1	1	NA	1	1	1	1	NA
D3	1	1	1	1	1	1	0	1	1	1	1	1	0.5
D4	1	0	05	0.5	0.5	0.5	0	0	0	0.5	0.5	0	0
S1	0	0	0	0	0	1	0.5	0.5	0	1	1	0.5	0
O1	0	0	0	1	0	1	1	1	1	1	1	0	0
Summary scoresa (%)
Totalb	61	57	64	74	79	67	70	77	76	87	78	59	69
Transferabilityc	60	50	63	73	81	80	88	75	81	97	90	67	66

Full items of the EURONHEED checklist are described in Supporting information, Table S4. Items comprising the transferability subchecklist are shown in bold type.

Average of the total summary score: 71%; average of the transferability summary score: 75%.

^aSummary scores were calculated using the formula as in EURONHEED checklist: $\frac{1}{n-x}{\sum }_i\mathit{Si}\times 100$ .

^bTotal summary score, number of questions = 42.

^cTransferability summary score, number of questions = 16.

The average score per section presented as the percentage of the total score are shown in Fig. 2. The average score per section was 0.69 (range = 0.35–0.92). The sections that scored below the average (69%) were: health technology assessment study population, effectiveness, benefit measure, variability and generalizability.

Figure 2.

Percentage of total score per section. Calculated as the average of the% of total score of subitems. [Colour figure can be viewed at wileyonlinelibrary.com]

Discussion

Key findings

Markov‐based state transition models with QALY as the outcome measure were the most frequently used technique in evaluating the cost‐effectiveness of smoking cessation interventions. However, the majority of the studies were reported poorly, making it hard to assess their transferability using the existing checklist‐based method. Where such assessment was possible, studies showed a wide variation in transferability scores, driven mainly by the method of selecting populations, assessing effectiveness and outcomes and estimating variability and generalizability of their own findings.

Relative transferability

The EURONHEED method assumes that without a quality score it would be impossible to transfer a study to another setting 9, 32, 95. Therefore, the explicit assessment using this method resulted in some studies being more favourable candidates than others. However, on average, all studies lacked in some attributes for full transferability. One of the main differences between a high score and a low score is how differently the studies scored on the questions on costs. For example, Annemans et al. (2009), with a score of 0.50, addressed most of the cost questions only partially, whereas Hoogendoorn et al. (2008), with a score of 0.97, did so fully. Therefore, costs are important determinants of the transferability assessment 9. Our review also highlighted other determinants; namely, selection of study population, intervention and comparator descriptions, effectiveness and benefit measures and variability/generalizability analyses—all scoring below the overall average score. Without a threshold, it was not possible to rank the assessed studies on their relative transferability, and this will be explored further below.

Comparison to current literature

Several systematic reviews are available on the cost‐effectiveness of smoking cessation 22, 23, 24, but only one systematic review looking at model‐based economic evaluations 20. Most of the studies included in their review used the Markov model with long‐term time horizons, included comparable health states and reported the similar measures of effectiveness and outcomes as ours, and common weaknesses included poor reporting of the modelling details. However, a key difference from our review is that they did not build on their findings to evaluate the extent to which such models could be transferable from the original context to others, for wider benefits 9, 10, 17. In areas outside smoking cessation, Korber has evaluated physical activity interventions for their transferability 96. Consistent with our findings, she also found that a very few included studies explored variability from place to place and discussed caveats regarding the generalizability of results, ‘leading to a wide variation in the transferability of the study results ranging from “low” to “very high” with everything in between’ 96. Another study 97 found that population and methodological characteristics were poorly reported—a finding that echoes our own results on the weaknesses of the models.

Implications of this review

Despite the availability of several guidelines on how to conduct and report adequately on economic evaluations 29, 31, there is still a considerable variation in the quality of published economic evaluations in smoking cessation. Arguably, this may limit the use of such evidence in other contexts. Some authors argue that the factors affecting the perception of applicability (the process question) and transferability (the outcome question) together might be broader than the factors associated with external validity 13. Notwithstanding this difference, the EURONHEED method relies heavily upon the quality of reporting to ascertain transferability 32. Therefore, such scores can be limited in use by the end‐users for two reasons. First, a poorly constructed model could have been reported well scoring high on the transferability scale and vice versa. Secondly, without a threshold score, it is hard to judge a study or to rank and compare across the studies. Nixon et al. 32 argue that the EURONHEED score should, rather, be used as a general guide in making decisions, but also note that the explicit assessment of transferability using this method will introduce an educational element, helping researchers to improve the design, conduct and reporting of future studies.

This review highlights the educational element noted above. Transparency in the model building and subsequent analysis and results, which can be captured by the quality of reporting, can enhance our understanding of the underlying process and outcome questions. However, a robust method would require more analyses based on the model outputs (as opposed to the checklists), backed up by the perceptions of actual stakeholders (including decision makers) as to what is relevant, adaptable, valid and transferable to them 13, 16. The European study on Quantifying Utility of Investment in Protection from Tobacco (EQUIPT) 98 provides some promise to that end by encompassing both model‐based analyses (e.g. on the parameter importance and variability) and the analysis of the stakeholder views (e.g. on the importance of interventions and intention to use economic evidence in policymaking) 99, 100, in addition to the systematic reviews based on the published models such as this. Although the final results of the EQUIPT study are yet to be published, this comprehensive framework appears to provide the end‐users with an understanding of a key transferability attribute—what changes in the economic model would make it transferable to their own settings and why 15.

This review also reiterates the already identified challenge in terms of the way in which economic evaluations in broader public health are designed, conducted and reported 101. The finding that only one‐fifth of the included study met quality classification for transferability implies that policymakers, researchers and journal editors need to work together in enhancing the quality of new economic evaluations and making it more transferable. The guidelines used by economic evaluation community and journals such as this are helpful to that end 102. However, such guidelines should also emphasize the need for the authors to assess and report transferability of their models to the new contexts. This would ensure that future studies could consider adding model‐based analysis of transferability on to the checklist‐based evaluation, backed up by, where possible, analysis of the views of stakeholders.

Limitations

A major limitation of this review has been the limitation embedded in the existing method of transferability assessment 9, 32. Future research may overcome this limitation by adopting a comprehensive assessment as discussed above. In addition, limiting the search to English language only might have excluded some studies. However, we identified more model‐based economic evaluations than a previous similar review 22. The use of three quality criteria 31 for inclusion of studies in the transferability assessment could potentially have introduced some bias, as it was based on the overall assessment, as opposed to some standard checklists such as those by Drummond 103 or Philips 104. However, the variety of items included in our data extraction form as outlined in the best practice guidelines 102 were very similar to the Drummond or Philips checklists, implying the possibility of such bias to be minimal. Finally, exclusion of low‐/middle‐income countries to reduce study heterogeneity could have limited this review in its primary focus (i.e. evidence transferability to less‐affluent countries).

Conclusion

Existing economic evaluations in smoking cessation vary in quality, resulting mainly from the way in which they selected their populations, measured costs and effects and assessed the variability and generalizability of their own findings. All studies lacked one or more key study attributes for full transferability. A robust design, coupled with comprehensive reporting of key study attributes, could make economic evaluations transferable to a new context.

Declaration of interests

None.

Funding

S.P. and P.K.'s time in this research was funded partly by the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 602270 (EQUIPT).

Supporting information

Table S1 Search strategy.

Table S2 Exclusion criteria.

Table S3 List of high‐income countries available at: http://data.worldbank.org/about/country‐and‐lending‐groups

Table S4 EURONHEED checklist.

Berg, M. L. , Cheung, K. L. , Hiligsmann, M. , Evers, S. , de Kinderen, R. J. A. , Kulchaitanaroaj, P. , and Pokhrel, S. (2017) Model‐based economic evaluations in smoking cessation and their transferability to new contexts: a systematic review. Addiction, 112: 946–967. doi: 10.1111/add.13748.