How do we determine the impact of e-cigarettes on cigarette smoking cessation or reduction? Review and recommendations for answering the research question with scientific rigor

Abstract

Aims:

To propose a hierarchy of methodological criteria to consider when determining whether a study provides sufficient information to answer the question of whether e-cigarettes can facilitate cigarette smoking cessation or reduction.

Design:

A PubMed search through February 1, 2017 was conducted of all studies related to e-cigarettes and smoking cessation or reduction.

Settings:

Australia, Europe, Iran, Korea, New Zealand, United States.

Participants/Studies:

91 articles.

Measurements:

Coders organized studies according to six proposed methodological criteria: 1) examines outcome of interest (cigarette abstinence or reduction), 2) assesses e-cigarette use for cessation as exposure of interest, 3) employs appropriate control/comparison groups, 4) ensures that measurement of exposure precedes the outcome, 5) evaluates dose and duration of the exposure, and 6) evaluates the type and quality of the e-cigarette used.

Findings:

Twenty-four articles did not examine the outcomes of interest. Forty did not assess the specific reason for e-cigarette use as an exposure of interest. Twenty articles did not employ prospective study designs with appropriate comparison groups. The few observational studies meeting some of the criteria (duration, type, use for cessation) triangulated with findings from three randomized trials to suggest that e-cigarettes can help adult smokers quit or reduce cigarette smoking.

Conclusions:

Based on the proposed criteria, few studies claiming to address the effect of e-cigarettes on smoking cessation or reduction are of sufficient quality to inform the scientific question of interest. Studies with stronger measures and methods are needed to better inform the question of e-cigarette use for smoking cessation or reduction.

INTRODUCTION

The 2014 Surgeon General’s Report on the Health Consequences of Smoking states that death is “overwhelmingly caused by cigarettes and other combusted tobacco products,” and that use of e-cigarettes and other innovative nicotine delivery products is likely to be beneficial to public health if current smokers who are unable to quit can eventually switch completely to these products (1, 2). Studies show that e-cigarettes could benefit the health of current smokers because they are substantially less harmful (3–5), more affordable (6, 7), and able to deliver nicotine efficiently (8–10), thus satisfying cravings and reducing nicotine withdrawal (9, 11). Although experimental use of e-cigarettes among adult smokers has increased and could encourage smoking cessation on a broader scale than nicotine replacement therapies,(12–14) it is not yet clear whether e-cigarettes can effectively facilitate smoking cessation at the population level.

A number of reviews and opinion pieces have examined the evidence on the impact of e-cigarettes on cigarette smoking cessation (15–29). Reviews have come to different conclusions, possibly because they differ in the types of studies included, the samples included in those studies, and the measures used to assess e-cigarette use. Reviews reporting positive results note that e-cigarettes may be helpful to some smokers in quitting smoking (15, 26), and that e-cigarettes with nicotine are more effective in helping smokers to quit than those without nicotine (19, 23, 24). Equivocal reviews note that no conclusion can be drawn from the existing data (16, 21, 27–29), or that e-cigarettes are no more effective for smoking cessation than existing cessation products (18, 22). One review reporting negative effects on smoking cessation noted that the odds of quitting smoking were lower in e-cigarette users than non- e-cigarette users (17). Four reviews included meta-analyses, with three reporting a positive effect of e-cigarette use on smoking cessation (19, 23, 24), and one reporting a negative effect (17). All reviews highlight the low quality of existing evidence as a limitation, largely focusing on the lack of randomized controlled trials (RCTs) addressing the safety and efficacy of e-cigarettes for smoking cessation. While some reviews have evaluated the quality of evidence based on a specified set of criteria (16, 17, 19, 23), none of the reviews have addressed all of the following key issues: (a) whether an adequate exposure to e-cigarette use occurred to test the smoking cessation hypothesis (i.e., not trial use on one or two occasions, but use for sufficient duration, intensity and frequency to affect cessation); (b) whether e-cigarettes were used with the intention of smoking cessation; and (c) whether e-cigarettes were used on the most recent quit attempt proximal to the outcome measurement time. As a result, reviews to-date have typically combined uninformative or methodologically weak studies with well-designed studies, resulting in a lack of clarity and contradictory conclusions.

Rigorous standards are needed to evaluate the evolving evidence base. These standards would differentiate informative from uninformative studies and decrease the likelihood of a false impression of the evidence based on an accumulation of uninformative studies on this question. The purpose of this paper is to propose a hierarchy of methodological criteria to consider when determining whether a study on e-cigarettes and smoking cessation provides sufficient information to answer the question of whether e-cigarettes can help current smokers quit or reduce smoking. We then categorize existing studies reporting on e-cigarette use and smoking cessation according to these criteria. Finally, we provide cautions and recommendations for designing high quality randomized and observational studies to address questions of the efficacy and effectiveness of e-cigarette use for smoking cessation.

METHODS

We conducted a systematic review of published scientific literature on e-cigarettes indexed in PubMed through February 1, 2017. The study protocol has been published elsewhere (30), as has the detailed presentation of the inclusion and cataloguing of articles in this overarching project (31). The search strategy consisted of the following keywords: “e-cigarette*” OR “electronic cigarette” OR “electronic cigarettes” OR “electronic nicotine delivery” OR “vape” OR “vaping.” Eligible studies were experimental studies, quasi-experimental studies, observational studies (including case control, cohort and cross-sectional studies), case reports, case series, qualitative studies and mixed methods studies that claimed to evaluate the impact of e-cigarette use on abstinence from traditional cigarettes or the reduction in number of traditional cigarettes consumed. Four coders (AV, SF, AG, LC) evaluated abstracts for inclusion and extracted data into a structured coding spreadsheet; one coder (AV) double-checked all data extraction. During our initial review of these studies in 2014, the study team identified the elements critical to answering the scientific question and developed a set of six questions requiring a yes or no response:

1. Does the study examine and adequately measure the primary outcome of interest (i.e., cigarette smoking abstinence or cigarette smoking reduction)?

2. Does the study assess e-cigarette use specifically for smoking cessation or reduction as the exposure of interest (i.e., e-cigarettes were explicitly used with the intention to quit smoking)?

3. Does the study use an appropriate design with control or comparison groups to address the potential impact of e-cigarette use on smoking cessation or reduction?

4. Does measurement of the exposure (i.e., e-cigarette use) precede measurement of the final outcome (smoking cessation or reduction)?

5. Does the study evaluate the dose and the duration of the exposure to determine degree of adherence and adequate delivery of active ingredients for a sufficient time period to be a reasonable test of a cessation hypothesis?

6. Related to criterion # 5, does the study evaluate the type and quality of e-cigarette product used (e.g., its efficiency and reliability at delivering nicotine and other subjective experiences thought to aid smoking cessation - such as chemosensory/sensorimotor satisfaction or appeal as an alternative to smoking)?

In essence, these criteria are similar to those considered when thinking about the balance between internal validity and external validity. This is often a tension in moving from individual safety and efficacy studies to community or population based studies. Such a tension is unavoidable, but must be managed, as is done when considering the strengths and weaknesses of both randomized controlled studies and observational (“real world”) studies.(32) Questions four through six relate to an assessment of whether the study precisely measured the exposure of interest as an independent variable that, when used for cessation, delivers the “active ingredients” known to facilitate cessation or reduction. These six items were used as a hierarchy to organize studies according to their methodological strengths. Studies that did not address one or more of the questions on this list fell out of the hierarchy and were considered less informative compared to those that met most of the more fundamental criteria, or ideally, met all six criteria. Figure 1 illustrates this hierarchy of methodological considerations.

Figure 1. Flowchart of included articles in the hierarchy of methodological considerations.

NOTE: Articles included towards the bottom of the flowchart have the strongest methodologies and provide the best evidence for the potential impact of e-cigarette use on cigarette smoking cessation or reduction.

RESULTS

Criterion 1: Studies must examine the outcome of interest (i.e., cigarette smoking abstinence or reduction)

To provide information regarding whether e-cigarettes can be used as an effective tool for smoking cessation, a study must appropriately define and assess the outcomes of interest (cigarette smoking reduction and cessation). We considered smoking abstinence and cigarette reduction to be the most relevant and highest-quality outcomes. However, definition of smoking abstinence and reduction varied by study. We identified 91 articles related to e-cigarette use and cigarette cessation (Table 1). Twenty-four of these articles were excluded because they addressed other outcomes, including e-cigarette nicotine concentration during quitting (33), reasons for e-cigarette use (34), use of e-cigarettes as a quit method (35–45), correlation of e-cigarette use with expectancies (46, 47), quit attempts (48–50), intention to quit (51), e-cigarette use in opioid-dependent smokers (52), cessation of e-cigarette use (53), other outcomes associated with quitting smoking (54, 55), and use of e-cigarettes during periods of forced abstinence (56).

Table 1.

List of studies included in the review and assessment of study methodologies

Article	Study design	Outcome of interest?	Assessed Reason for Use: Cessation?	Appropriate study design?	Measurement: Exposure precedes outcome? (Timing)	Measurement: Assessed Dose of ENDS Use? (Dose)	Measurement: Assessed ENDS Product Type?
Bullen (2013) (57)	Randomized controlled trial	Yes	Yes	Yes	Yes	Yes	Yes
O’Brien (2015) (60)	Randomized controlled trial	Yes	Yes	Yes	Yes	Yes	Yes
Caponnetto (2013) (58)	Randomized controlled trial	Yes	Yes	Yes	Yes	Yes	Yes
Tseng (2016) (61)	Randomized controlled trial	Yes	Yes	Yes	Yes	Yes	Yes
Vickerman (2017) (101)	Longitudinal study with comparison group	Yes	Yes	Yes	Yes	No	Yes
Zawertailo (2017) (115)	Longitudinal study with comparison group	Yes	Yes	Yes	No	No	No
Shi (2016) (116)	Longitudinal study with comparison group	Yes	Yes	Yes	No	No	No
Adriens (2014) (59)	Randomized controlled trial	Yes	Yes	No	No	Yes	Yes
Pearson (2015) (81)	Longitudinal study with comparison group	Yes	Yes	No	No	No	No
Vickerman (2013) (80)	Longitudinal study with comparison group	Yes	Yes	No	No	No	No
Etter (2014) (63)	Longitudinal study with no comparison group	Yes	Yes	No	Yes	Yes	Yes
Polosa (2015) (70)	Longitudinal study with no comparison group	Yes	Yes	No	Yes	No	Yes
Stein (2016) (105)	Longitudinal study with no comparison group	Yes	Yes	No	Yes	Yes	Yes
James (2016) (106)	Longitudinal study with no comparison group	Yes	Yes	No	Yes	Yes	Yes
Nolan (2016) (102)	Longitudinal study with no comparison group	Yes	Yes	No	Yes	Yes	Yes
Goniewicz (2013) (84)	Cross-sectional survey	Yes	Yes	No	No	Yes	Yes
Dawkins (2013) (85)	Cross-sectional survey	Yes	Yes	No	No	Yes	Yes
Tackett (2015) (86)	Cross-sectional survey	Yes	Yes	No	No	Yes	Yes
Rutten (2015) (87)	Cross-sectional survey	Yes	Yes	No	No	Yes	No
Brown (2014) (91)	Cross-sectional survey	Yes	Yes	No	No	No	No
Zhu (2013) (90)	Cross-sectional survey	Yes	Yes	No	No	No	No
Allem (2015) (107)	Cross-sectional survey	Yes	Yes	No	No	No	No
McQueen (2016) (112)	Cross-sectional survey	Yes	Yes	No	No	No	No
Pechacek (2016) (113)	Cross-sectional survey	Yes	Yes	No	No	No	Yes
Beard (2016) (117)	Cross-sectional survey	Yes	Yes	No	No	No	No
Filippidis (2016) (118)	Cross-sectional survey	Yes	Yes	No	No	No	No
Caponnetto (2011) (100)	Case series	Yes	Yes	No	No	No	Yes
Hatsukami (2017) (123)	Randomized controlled trial	Yes	No	Yes	No	No	No
Biener (2014) (72)	Longitudinal study with comparison group	Yes	No	Yes	Yes	Yes	No
Brose (2015) (73)	Longitudinal study with comparison group	Yes	No	Yes	Yes	Yes	No
Al-Delaimy (2015) (78)	Longitudinal study with comparison group	Yes	No	Yes	Yes	No	No
Borderud (2014) (75)	Longitudinal study with comparison group	Yes	No	Yes	Yes	No	No
Choi (2014) (77)	Longitudinal study with comparison group	Yes	No	Yes	Yes	No	No
Grana (2014) (74)	Longitudinal study with comparison group	Yes	No	Yes	Yes	No	No
Prochaska (2014) (76)	Longitudinal study with comparison group	Yes	No	Yes	Yes	No	No
Hitchman (2015) (79)	Longitudinal study with comparison group	Yes	No	Yes	No	Yes	Yes
Adkison (2013) (82)	Longitudinal study with comparison group	Yes	No	Yes	No	No	No
Polosa (2016) (119)	Longitudinal study with comparison group	Yes	No	Yes	Yes	Yes	No
Manzoli (2016) (120)	Longitudinal study with comparison group	Yes	No	Yes	Yes	Yes	No
Polosa (2016b) (121)	Longitudinal study with comparison group	Yes	No	Yes	Yes	Yes	No
Zhuang (2016) (122)	Longitudinal study with comparison group	Yes	No	Yes	Yes	No	No
Caponnetto (2013) (64)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	Yes
Nides (2014) (68)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	Yes
Polosa (2014) (67)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	Yes
Polosa (2011) (65)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	Yes
Polosa (2014) (66)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	Yes
Berg (2014) (62)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	No
Polosa (2016) (103)	Longitudinal study with no comparison group	Yes	No	No	Yes	No	Yes
Pratt (2016) (104)	Longitudinal study with no comparison group	Yes	No	No	Yes	Yes	Yes
Farsalinos (2013b) (89)	Cross-sectional survey	Yes	No	No	No	Yes	Yes
Siegel (2011) (88)	Cross-sectional survey	Yes	No	No	No	Yes	Yes
Christensen (2014) (94)	Cross-sectional survey	Yes	No	No	No	No	No
Lee (2014) (93)	Cross-sectional survey	Yes	No	No	No	No	No
Popova (2013) (92)	Cross-sectional survey	Yes	No	No	No	No	No
Wagener (2014) (98)	Clinical laboratory study	Yes	No	No	Yes	No	Yes
Manzoli (2015) (83)	Longitudinal study with comparison group	Yes	No	Yes	Yes	Yes	No
Grace (2015) (69)	Longitudinal study with no comparison group	Yes	No	No	Yes	No	Yes
Pacifici (2015) (71)	Longitudinal study with no comparison group	Yes	No	No	Yes	No	Yes
Fraser (2015) (96)	Cross-sectional survey	Yes	No	No	No	Yes	Yes
Gallus (2014) (95)	Cross-sectional survey	Yes	No	No	No	Yes	Yes
Dutra (2014) (114)	Cross-sectional survey	Yes	No	No	No	No	No
Lechner (2015) (97)	Cross-sectional survey	Yes	No	No	No	Yes	No
Hiscock (2015) (111)	Cross-sectional survey	Yes	No	No	No	No	No
Etter (2016) (110)	Cross-sectional survey	Yes	No	No	No	No	No
Andler (2016) (108)	Cross-sectional survey	Yes	No	No	No	Yes	No
Busch (2016) (109)	Cross-sectional survey	Yes	No	No	No	No	No
McRobbie (2015) (99)	Clinical laboratory study	Yes	No	No	Yes	Yes	Yes
Campagna (2016) (54)	Randomized controlled trial	No	Yes	Yes	Yes	Yes	Yes
Cibella (2016) (55)	Randomized controlled trial	No	Yes	Yes	Yes	Yes	Yes
Pulvers (2014) (39)	Cross-sectional survey	No	Yes	No	No	Yes	No
Pepper (2014) (34)	Cross-sectional survey	No	Yes	No	No	No	No
Pokhrel (2013) (41)	Cross-sectional survey	No	Yes	No	No	No	No
Ramo (2015) (40)	Cross-sectional survey	No	No	No	No	No	No
Farsalinos (2013) (33)	Cross-sectional survey	No	No	No	No	Yes	No
Pokhrel (2014) (35)	Cross-sectional survey	No	No	No	No	No	No
Pokhrel (2014) (36)	Cross-sectional survey	No	No	No	No	No	No
Stein (2015) (52)	Cross-sectional survey	No	Yes	No	No	No	No
Beard (2015) (37)	Cross-sectional survey	No	Yes	No	No	No	No
Harrell (2015) (46)	Cross-sectional survey	No	No	No	No	Yes	Yes
Kinnunen (2015) (51)	Cross-sectional survey	No	No	No	No	No	No
Heydari (2015) (38)	Cross-sectional survey	No	No	No	No	No	No
Cho (2016) (49)	Cross-sectional survey	No	No	No	No	No	No
Lippert (2016) (48)	Cross-sectional survey	No	No	No	No	No	No
Camenga (2016) (42)	Cross-sectional survey	No	Yes	No	No	Yes	No
Huerta (2017) (50)	Cross-sectional survey	No	No	No	No	No	No
LeVault (2016) (44)	Cross-sectional survey	No	Yes	No	No	No	No
Spears (2016) (45)	Cross-sectional survey	No	Yes	No	No	No	No
Thrul (2017) (43)	Cross-sectional survey	No	Yes	No	No	No	No
Jorenby (2017) (56)	Clinical laboratory study	No	No	Yes	Yes	Yes	Yes
Copp (2015) (47)	Clinical laboratory study	No	No	No	No	No	Yes
Silver (2016) (53)	Case series	No	No	No	No	Yes	No

Sixty-seven remaining articles reported the impact of e-cigarette use on abstinence from cigarettes or on the reduction in number of cigarettes consumed (57–123). Of these, six were from five RCTs (57–61, 123), nineteen were from longitudinal observational studies with a comparison group (72–83, 101, 115, 116, 119–122), two were from clinical laboratory studies (98, 99), fifteen were from longitudinal observational studies without a comparison group (62–71, 102–106), twenty-four were from cross-sectional surveys (84–97, 107–114, 117, 118), and one was a case series (100).

Criterion 2: Studies must assess e-cigarette use for smoking cessation or reduction as the exposure of interest

For observational studies, it is crucial to confirm that participants are using e-cigarettes for the purpose of cessation. In addition to quitting, people use e-cigarettes for many reasons, e.g., because they are cheaper than cigarettes, they are less harmful, or out of curiosity (34). If smokers are not using e-cigarettes as part of a cessation attempt, then it is unclear that e-cigarette use should be expected to result in quitting. It is possible that use of e-cigarettes may motivate desire and attempts to quit at some unspecified time in the future. It is also possible that e-cigarettes could reduce the desire to quit, for example, by allowing a person to use e-cigarettes in settings where smoking is prohibited, thereby reducing the pressure to quit. No studies have been designed to answer this question; we therefore confine our attention to studies where motivation to quit is clearly tied to use of e-cigarettes. This issue is borne out in studies on non-standard use of nicotine replacement therapy (i.e., use for purposes other than quitting smoking)(124, 125) which show no impact of NRT use on cessation (124). Pearson et al. addressed this issue by asking, “What quit methods have you used in the past 3 months?” (81). Participants who used an e-cigarette as a quit method were classified as “exposed” and those who did not were classified as “unexposed,” regardless of other e-cigarette use.

For RCTs where participants are assigned to use e-cigarettes or abstain and are followed for the cessation outcomes of interest, the primary treatment indication is for smoking cessation. Treatment assignment makes assessing reasons for use unnecessary in an RCT since they are likely to be balanced across the treatment and control groups through the randomization. Reasons for use are therefore unlikely to confound the relationship between e-cigarette exposure for cessation and cessation outcomes in an RCT.

Of the remaining 67 articles, six were from RCTs (57–61, 123); in one of the RCTs, e-cigarettes were assigned with other non-combustible products and thus, the article was excluded at this stage (123). Of the other 61 studies in the hierarchy, 39 did not assess the reason for e-cigarette use as an exposure and were excluded at this stage (62, 64–69, 71–79, 82, 83, 88, 89, 92–99, 103, 104, 108–111, 114, 119–122).

Criterion 3: Studies must use an appropriate design with control or comparison groups and measures to address the potential impact of e-cigarette use on smoking cessation or reduction with minimal confounds

Apart from RCTs, the strongest epidemiologic studies examining whether e-cigarette use leads to smoking abstinence or cigarette reduction are longitudinal study designs, where exposure precedes outcomes and with appropriate comparison groups (i.e., smokers with similar characteristics who were not using e-cigarettes to quit). Of the 27 remaining studies, 20 did not have appropriate study designs (59, 63, 70, 80, 81, 84–87, 90, 91, 100, 102, 105–107, 112, 113, 117, 118). Eleven of these studies were cross-sectional (84–87, 90, 91, 107, 112, 113, 117, 118), and one was a case series (100). Additionally, eight longitudinal studies (59, 63, 70, 80, 81, 102, 105, 106) were excluded at this point. One study randomly assigned participants to e-cigarette use or control during the initial lab phase of the study, but then provided the control group with e-cigarettes during the follow-up period (59). Since there was no longer an unexposed control group during the phase of the study in which the smoking cessation outcomes were observed, this study was excluded. Five studies were longitudinal, but lacked comparison groups (63, 70, 102, 105, 106). Two additional studies were longitudinal with appropriate comparison groups (80, 81); however, in their assessment of the association between e-cigarette use and smoking cessation outcomes, the exposure and outcome were assessed at the same follow-up time point. Thus, while the parent study design was longitudinal, analyses associating e-cigarette use and smoking cessation in these two studies were cross-sectional.

After considering whether studies assessed the outcome of interest, e-cigarette use for smoking cessation as an exposure, and study design, only seven articles – two from the same parent RCT – remained (57, 58, 60, 61, 101, 115, 116).

Criteria 4–6: Studies must precisely measure the exposure of interest (i.e., e-cigarette use)

In order to precisely measure the exposure of interest (e-cigarette use), studies should: 1) establish temporality by ensuring that the exposure (i.e., e-cigarette use) preceded the outcome (Criterion 4); 2) measure dose and duration of e-cigarette use (Criterion 5); and 3) assess the e-cigarette product type (Criterion 6). Four of the seven remaining articles met all six criteria (57, 58, 60, 61). The other three articles did not establish temporality (115, 116), or capture dose or duration of e-cigarette use in sufficient detail at baseline to reasonably test a cessation hypothesis (101).

The four articles from three RCTs meet all six criteria to be considered as addressing the scientific question of interest in an appropriately rigorous manner (57, 58, 60, 61). These three studies are consistent in their findings that e-cigarettes can help cessation in adult smokers – either through abstinence or cigarette reduction – regardless of motivation to quit smoking. Bullen et al.’s result (57) of 7-day point prevalence abstinence of 21.1% (nicotine e-cigarette) and 21.9% (placebo e-cigarette) at six months in smokers motivated to quit are in line with meta-analyses of the impact of smoking cessation medications, including nicotine replacement therapy (NRT), presented in the 2008 update to the Public Health Service (PHS) guidelines for Treating Tobacco Use and Dependence. In the PHS guidelines, the combined results of 32 RCTs of short-term (6–14 weeks) use of the nicotine patch among smokers interested in quitting, which found six-month abstinence of 23.4% (95% CI: 21.3–25.8), and six trials of use of the nicotine inhaler (24.8%) (126). Caponnetto et al.’s results (58) are also similar to findings from meta-analyses of the impact of smoking cessation medications among smokers not willing to quit (126). Specifically, the 8.7% of smokers using placebo and nicotine-containing e-cigarettes and the 11% using nicotine-containing e-cigarettes who quit at one year in this study is similar to the 8.4% abstinence rate among nicotine replacement users in five RCTs among smokers unmotivated to quit (126). Tseng et al. (61) found that young adult daily smokers interested in reducing their cigarette consumption experienced significantly greater reductions in cigarettes per day from baseline when using a nicotine-containing e-cigarette compared to a placebo e-cigarette at 3-week follow-up (p = 0.03).

DISCUSSION

Of 91 articles that claim to examine the relationship between e-cigarette use and smoking cessation or reduction, only four articles from three RCTs meet all six recommended criteria. These three studies suggest that e-cigarettes are effective in helping adult smokers to quit or to reduce their cigarette consumption, and that rates of smoking cessation with e-cigarettes are similar to rates of cessation with NRT. The criteria outlined in this article draw on “design thinking”(127) to articulate the study design elements required to answer the scientific question of interest and differentiate informative from uninformative evidence. They serve as a screening tool to identify the appropriate studies for comparison; as in a standard systematic review process, additional evaluation is needed to determine the risk of bias in these studies and potential impacts of study limitations on the outcomes reported. Our findings highlight the following methodological concerns with existing studies: first, the majority of studies that claim to address the relationship between e-cigarette use and smoking cessation or reduction do not provide evidence to answer that question; second, poor study design limits inferences about the impact of the exposure on a subsequent outcome, particularly in comparison to an unexposed group; and third, exposure measures in many studies are insufficient for assessing dose, duration (e.g., ever used an e-cigarette), product type, or reasons for use.

It is important to be cautious in drawing conclusions from the existing studies of the effect of e-cigarette use on smoking cessation or reduction. There is need for improvement in methods and measures across the board, and RCTs should not automatically be considered a gold standard if they deliver the treatment poorly or suffer from inadequate internal validity like poor treatment adherence or differential drop-out.(128) Future studies should use, for example, improved measures of exposure (e.g., frequency of use, duration, product type, product features, e-liquid used, nicotine concentration (72, 73, 79)) and 7-day point-prevalence abstinence at 6-month follow-up as the outcome measure per the PHS guidelines (129) to facilitate direct comparison to established cessation treatments. Additionally, non-randomized studies must be scrutinized with respect to the appropriateness of comparison groups used and assessment of the most plausible confounders relevant to the question at hand (i.e., beyond demographics) to rule out other likely sources of bias. Such covariates are critical to address the potential impact of e-cigarette use on smoking cessation with some confidence that the study has ruled out likely “third variable” competing explanations. Without randomization of exposure to e-cigarette use, nicotine-dependent individuals may be more likely to try multiple cessation treatments (including e-cigarettes), and may be more likely to fail at quitting smoking because of their higher nicotine dependence (130). E-cigarette users who have successfully quit smoking would also be excluded from most observational studies (131). Similar issues have been raised in observational studies about the efficacy of NRT for smoking cessation. RCTs have unequivocally established NRTs efficacy, but this effect is not substantiated in some observational studies (132, 133). This does not imply that the treatments are ineffective overall, but rather that other, extra-treatment factors need to be taken into account when smokers try to quit, such as insufficient treatment dose to address the needs of dependent smokers.

Our conclusion that the majority of existing observational studies do not constitute reliable scientific evidence is consistent with established practice for quantitative reviews. The Institute of Medicine’s Standards for Systematic Reviews notes that “the likelihood of selection bias, recall bias, and other biases are so high in certain clinical situations that no observational study could address the question with an acceptable risk of bias” (p. 113) (134). Therefore, the IOM recommends that observational studies be used to complement or supplement findings from RCTs in systematic reviews. The Cochrane Handbook for Systematic Reviews notes further that non-randomized studies which use “different study designs (or which have different design features)…should not be combined in a meta-analysis” (p. 422) (135). Two Cochrane reviews examined data from both RCTs and cohort studies, but conducted limited meta-analyses using data from only the two RCTs where the designs and populations were deemed sufficiently similar to compare (19, 24). The other two published meta-analyses included heterogeneous studies from RCTs, longitudinal, and cross-sectional designs in their pooled analyses (17, 23). There are three key issues related to combining results from non-randomized studies with respect to adjustment for confounding. First, effect estimates and standard errors from non-randomized studies do not correct for imbalance in the exposed and unexposed groups (as in randomized studies). Second, non-randomized studies on the same topics are likely to control for different confounders in their analyses; this creates further heterogeneity in the included studies. Third, adjustment for confounding in individual studies – and in pooled analyses – may yield a more precise estimate, but does not reduce bias. The Cochrane Handbook warns: “meta-analyses of studies that are at risk of bias may be seriously misleading. If bias is present in each (or some) of the individual studies, meta-analysis will simply compound the errors, and produce a ‘wrong’ result that may be interpreted as having more credibility” (p. 247) (135).

Limitations of our study include the restriction of our search to articles published in PubMed that include e-cigarette related language in the title or abstract. This may underestimate the number of studies published on e-cigarette use and cessation. More generally, the constantly evolving nature of e-cigarette products, and especially the heterogeneity and poor consumer acceptability of “first generation” products, make an accurate, evidence-based evaluation of the effectiveness of e-cigarettes as a quitting tool quite challenging.

Despite the large number of studies claiming to address the question of whether e-cigarettes are an effective smoking cessation tool, only three existing studies address the topic according to our proposed criteria. In line with guidance from the IOM and the Cochrane Collaboration, we caution against the use of meta-analysis for observational studies where the studies included are at sufficient risk of bias to produce invalid, yet precise, results. More research – especially independent, high quality RCTs with appropriate control groups and more rigorous “real world” observational studies that fit the causal question are needed to further determine whether and how e-cigarettes can be an effective cigarette cessation or harm reduction aid. This should be aided in the U.S. by the recent availability of a standardized research e-cigarette (https://www.drugabuse.gov/funding/supplemental-information-nida-e-cig). Studies to-date that do not meet at least minimal criteria for scientific rigor from both an internal and external validity perspective may be misleading and add to confusion about the state of the existing evidence rather than provide the kind of information needed to inform policy.

WHAT THIS STUDY ADDS.

Many existing studies reporting on e-cigarettes and smoking cessation are limited by methodological flaws and are uninformative with respect to the key question of the effect of e-cigarettes on smoking cessation or reduction. This study proposes a set of methodological criteria by which to identify studies that address the impact of e-cigarettes on smoking cessation or reduction. Based on the proposed criteria, few studies claiming to address the effect of e-cigarettes on smoking cessation or reduction are of sufficient quality to inform the scientific question of interest.