Response to Brown et al “Does the offer of e-cigarettes benefit smoking cessation among unselected smokers?”

To the editor,

Brown and colleagues articulate an interpretive issue regarding our recent 5-arm workplace-based smoking cessation trial published in the New England Journal of Medicine (NEJM) (1). Specifically, they are concerned with the analytic approach and wording that we used to generate and summarize our conclusions regarding the difference in the rate of cessation between the e-cigarette and usual care study arms in our 5-arm trial.

A priori, we specified eight of the many possible between-arm contrasts in this 5-arm trial design as being of primary interest: the two financial incentives against each other, each against e-cigarettes and free cessation aids, e-cigarettes against free cessation aids, and each of these latter two arms against usual care. The trial was powered and designed using a frequentist approach to support these multiple contrasts using the Holm sequential method (2). We found that 3 of the 8 contrasts differed using a Holm-adjusted p-value of 0.05. The comparison in question, e-cigarettes versus usual care, was not one these three.

We agree with the authors that this approach is not in any way Bayesian. However, their preference for a Bayesian analysis and interpretation, in contrast to our frequentist approach, does not mean that our analysis was incorrect, nor that our interpretation represented an “error.” That aside, we generally agree with the perspective that p-value-based dichotomies for or against the effect of an intervention are suboptimal. Thinking about results on a continuum from weak to strong has obvious merits. However, a Bayes factor exists on a numerical continuum as well, and the author’s reported value of 2.05 requires interpretation just as do our adjusted p values for this same contrast (0.20 for the total sample and 0.27 for the engaged cohort). The authors state that they interpret this Bayes factor of 2.05 as “weak evidence of there being an effect of e-cigarettes of similar size as reported in the Cochrane review on the topic (3).”

We do not disagree with this presentation of our results, but rather note that it stems from one of two acceptable statistical philosophies. One adheres to somewhat arbitrary p values and recognizes the importance of adjusting for multiple comparisons so as to restrain the chances of making false-positive conclusions. The other adheres to somewhat arbitrary categorizations of Bayes factors and takes the anti-conservative approach of ignoring the issue of multiple comparisons. Fortunately, from the practical perspective of guiding the development of workplace-based cessation programs, one’s interpretive philosophy does not seem to matter. This is because our study found that offering financial incentives was both more effective and far more cost-effective than offering e-cigarettes. For these reasons, we stand by our original conclusions, and note that regardless of the statistical approach, there is no evidence to suggest that e-cigarettes ought to be offered in workplace-based cessation programs.