A Source of Bias in Studies of E-Cigarettes and Smoking Cessation

Kalkhoran et al.¹ aim to assess the evidence for use of e-cigarettes as an effective aid to cigarette smoking cessation using data from the multiwave Population Assessment of Tobacco and Health (PATH) Study,² a nationally representative U.S. cohort study. As they note, randomized controlled trials (RCTs) would provide the gold standard of causal evidence regarding the impact of e-cigarette use on smoking cessation but few have been conducted and none within the United States. Despite the strengths of RCTs, findings from RCTs are not without limitations. By design, RCTs use controlled settings with a selective population that is rarely representative of the population of interest, which limits generalizability of results to the diverse population of U.S. smokers. As the authors note, the evidence from a large prospective observational study such as the PATH Study can overcome this generalizability issue by “demonstrating the effect of e-cigarettes under real world conditions.” However, the challenge remains of assuring that exposure groups are representative of the population of interest. Kalkhoran et al.’s analysis defined three exposure groups (daily e-cigarette use, nondaily e-cigarette use, and no e-cigarette use), each of which included smokers who were not attempting to quit; we argue that this inclusion is a critical obstacle to making a valid inference about the effectiveness of e-cigarettes on cigarette smoking cessation.

In its 2018 report on e-cigarettes,³ the National Academies of Science Engineering and Medicine (NASEM) addressed potential levels of evidence regarding the efficacy of e-cigarettes for smoking cessation. The report noted important threats to the validity of conclusions from a prospective observational study such as the PATH Study. They noted the importance of studying smokers who want to quit in this recommendation: “An optimal prospective observational study design would identify and follow a large cohort of smokers who want to quit or are making a quit attempt, assess e-cigarette exposure in detail before the smoking cessation outcome is assessed, biochemically confirm self-reported tobacco abstinence, and adjust for multiple potential confounding factors associated with e-cigarette use and with smoking cessation.” Fortunately, the PATH Study design and survey questions allow investigators to follow these recommendations. Although the Kalkoran paper did note the NASEM report, their analysis did not restrict the sample to smokers who made a quit attempt during the study. As a result, they compared cessation in exposure groups that were unbalanced in their motivation to quit cigarettes, which will have biased the results, as we explain below.

Smoking cessation is the most frequently reported reason for e-cigarette use; ⁴ thus, smokers who use e-cigarettes could be more likely to make a quit attempt and, thus, more likely to make a successful quit attempt than smokers who do not use e-cigarettes. Looking at PATH Study data, Coleman et al.⁵ reported that, among wave 1 dual users of cigarettes and e-cigarettes, 72% reported one of the reasons for their use of e-cigarettes was to help quit smoking cigarettes. Unfortunately, as Kalkhoran et al. note, the PATH data do not allow the timing of e-cigarette use and the quit attempt to be sorted out in the sample they chose; e-cigarettes may be taken up as part of a quit attempt, or a quit attempt may follow prolonged e-cigarette use. In both cases, e-cigarettes would be associated with quit attempts. We can report that 52.9% of wave 1 cigarette smokers who were daily e-cigarette users reported a quit attempt prior to wave 2 compared to only 40.7% of smokers who did not use e-cigarettes at wave 1. Thus, among Kalkhoran’s daily e-cigarette group, 30% more made a quit attempt compared to the no e-cigarette use reference group. Given this, the authors cannot rule out that the higher rate of prolonged quitting observed at follow-up among daily e-cigarette users arose because this group simply included more smokers who made a quit attempt than the reference group of those who did not use e-cigarettes. However, there is no evidence that e-cigarettes caused the greater rate of attempted quitting. An alternative hypothesis is that those who were more likely to quit successfully were also more likely to choose an e-cigarette as a cessation aid. Thus, to address their hypothesis that e-cigarette use increased successful cessation, Kalkhoran et al. should have restricted their sample to smokers who made a quit attempt as recommended in the NASEM report.

Two other published analyses of the PATH Study demonstrate how ignoring a smoker’s quit attempt status introduces bias that can exaggerate the impact of e-cigarette use on cessation. Similar to Kalkhoran’s study, Berry et al.⁶ included all non-e-cigarette users who did not make a quit attempt in their reference group and used logistic regression to control for nicotine dependence and a few other confounders. They reported a sevenfold increase in the odds of cessation from daily e-cigarette use compared to no e-cigarette use. Using the same data set, Benmarhnia et al.⁷ (not referenced in Kalkhoran article) restricted their sample to smokers who made a quit attempt and reported a much lower cessation risk difference of 0.06 (equivalent to an odds ratio of 1.5) for use of e-cigarettes to aid cessation. These dramatically different results suggest that the choice of analytic sample is of extreme importance and can lead to highly exaggerated results suggesting that daily e-cigarette use is strongly associated with successful cessation. We believe this is the case in the Kalkhoran study.

Analyses of the prospective effect of e-cigarette use on smoking cessation should follow the precepts outlined in the NASEM report. In particular, analyses should be restricted to smokers who are trying to quit in order to draw valid conclusions about the ability of e-cigarettes to help smoking cessation. As Kalkhoran’s study did not adhere to these precepts, we suggest that their results are most likely biased away from the null and need to be interpreted with great caution.

Funding

This work was funded by NIH grant 1R01CA234539.