Is Good Enough Good Enough? E-Cigarettes, Evidence, and Policy

In 2017, under the leadership of Scott Gottlieb, the US Food and Drug Administration (FDA) opened the door to considering e-cigarettes as a harm-reduction product. In January 2020, the agency signaled its intention to continue to chart that path, writing,

[T]he potential for [electronic nicotine delivery systems] to act as a substitute for cigarettes, thereby encouraging smokers to seek to switch completely away from combustible cigarettes, may be dependent, in part, upon the product having acceptability and abuse liability more comparable to a cigarette.^1(p20)

Yet what has always remained unclear is what evidentiary standard the FDA will apply as it reviews products and evaluates public health benefit.

EVALUATING EVIDENCE

Systematic reviews and meta-analyses are intended to give clarity by looking across studies, aggregating findings from the strongest, weeding out the weakest. Nonetheless, how different nations and organizations have evaluated this question of what counts as appropriate evidence has depended heavily on the context and the primary concerns of review bodies.

In 2018, for instance, back-to-back systematic reviews from National Academics of Science, Engineering, and Medicine (NASEM) and Public Health England (PHE) took a very different view of what observational studies and randomized controlled trials offered. For NASEM, randomized controlled trials were the most conclusive evidence that could speak to pressing policy questions. Observational studies, whether cohort or cross-sectional, suffered from the messiness of real life. For PHE, although randomized controlled trials were important, observational studies were invaluable precisely because they captured the lived experience.²

Two years later, we have two more contributions that seek to draw policy conclusions from an even broader literature with great variation in quality.

In this issue of the journal, Wang et al. (p. 230) make a contribution with a meta-analysis that leans heavily toward observational studies (55 out of 64 studies). The observational studies, they concluded, were not associated with cessation. In contrast, they found that randomized controlled trials that provided free e-cigarettes were “significantly associated” with quitting compared with conventional therapy.

A 2020 Cochrane review, updated from 2016, which leaned more heavily toward randomized controlled trials (26 out of 50 studies), was cautiously optimistic. Its authors concluded,

There is moderate‐certainty evidence that ECs [e-cigarettes] with nicotine increase quit rates compared to ECs without nicotine and compared to [nicotine replacement therapy]. Evidence comparing nicotine EC with usual care/no treatment also suggests benefit, but is less certain.^3(p2)

There have been important contextual shifts between 2018 and 2020. We have seen the rise and fall of panic over EVALI (acute e-cigarette or vaping product use-associated lung injury). After an alarming rise between 2017 and 2019, the 2020 National Youth Tobacco Survey found that the proportion of high schoolers who report vaping in the past 30 days has dropped considerably. But we have also seen an all-out assault on e-cigarettes, which sets the stage for how we evaluate the accumulating evidence.

What stands out in the 2020 landscape of systematic evidentiary reviews is that despite ongoing debate around a number of issues—youth uptake, the implications of dual use, the collateral harms from vaping—is that we have modest evidence from both observational studies and randomized controlled trials that e-cigarettes do contribute to smoking cessation.

But if the landscape and the evidence are shifting, consistent with the past is that we are unlikely to see a change in the debate over what counts as evidence when it comes to meta-analysis and systematic review. In 2018, Villanti et al. argued that no systematic reviews to date have addressed the most pressing use issues that contribute to variations in findings. Likewise, they underscore the Cochrane Handbook for Systematic Reviews cautions that meta-analyses can compound the confusion when they combine studies reliant on “different study designs.”^4(p399)

Observational studies can measure very different things, making it challenging to identify studies that can be analyzed as a coherent whole. Meta-analysis is a technique meant to draw broad conclusions from studies that are similar in design. For example, it can be problematic to compare studies that solicited volunteers to studies that observe individuals who began using e-cigarettes on their own.

The chief difference between this latest review and the Cochrane review involves the studies included. The Cochrane reviews were concerned about the selection bias in longitudinal studies, which generally do not capture smokers who successfully quit using e-cigarettes. Rather, they focus on current smokers, with different levels of motivation to quit, who are trying e-cigarettes to stop. As the authors explained in 2016, exclusion of treatment successes “is likely to show a low treatment effect, even for treatments that are highly effective.”^5(p19) As a result, they concluded, “In future versions of this review, we will no longer include this group of studies.”^5(p19)

Wang et al. describe the observational studies they selected as exhibiting “substantial heterogeneity.” The authors also underscore that the evaluation of e-cigarettes is made more complex by variations in quality, design, nicotine formulation and concentration, flavoring, and a complex patchwork of local, national, and global regulation.

We would expect some differences in study selection based on the specific questions driving the analysis. Although each review asks a spectrum of questions—Wang et al. were concerned about the effect of daily e-cigarette use on cessation and the impact of free e-cigarettes while the Cochrane review also asked about unwanted side effects—at the heart of both was the same fundamental question: do e-cigarettes help people to stop smoking?

The main difference, then, is that Wang et al. included many of those studies and aimed to control for confounding with a sensitivity analysis. One author has underscored the value of this kind of analysis in response to previous critiques of selection bias (see https://bit.ly/2J8xBrF and Kalkhoran and Glantz⁶). The Cochrane reviews excluded observational studies that did not include smokers who had successfully quit using e-cigarettes.

For those primarily concerned about potential unintended consequences of vaping to those who do not currently use e-cigarettes, precaution warrants setting the evidentiary bar low. For those primarily concerned about reducing the immediate harms to smokers while minimizing unintended consequences, moderate evidence is good enough. For some, then, this most recent analysis will not meet even a low evidentiary bar. For others, it will exceed that bar. Still others will call the evidence from the updated 2020 Cochrane review into question. In other words, debates about quality of the evidence should not mask competing values about the risks of inadvertent harms of e-cigarettes, whether to smokers themselves or to nonsmoking or nonvaping bystanders. Evidence and value judgments about which harms are most concerning and what represents an acceptable trade continue to intersect.

SO WHERE DOES THIS LEAVE US?

The key question from a policy perspective is not only “Do e-cigarettes contribute to smoking cessation?” but also “What level of evidence is good enough to accept e-cigarette harm reduction?” As we debate this, it is important to remember that, as powerful as systematic reviews and meta-analyses can be, any evidentiary evaluation and summation is only as good as the data that go in. And even when the strictest standards are applied, the strong effects found in well-designed studies will be diluted.

A new study by Mendez and Warner considers 360 best- and worst-case scenarios related to vaping, smoking, and health. In only three of their models did the harms marginally outweigh the benefits. Nonetheless, they conclude, “e-cigarettes represent a meaningful if thus far modest public health contribution and could represent a more substantial one [emphasis added].”^7(p6) An effect need not be robust to produce consequential population health contributions.

We cannot, as a result, think of evidence apart from the policy that can amplify or minimize potential impact. Wang et al. argue that we do not have enough evidence to support e-cigarettes as a consumer product. Rather, based on their findings that cost and daily use are associated with quitting, they suggest that e-cigarettes may have a role as prescription therapy for smokers.

Both the specific conclusions of this study and the broader landscape of moderately strong and, from a population impact perspective, good-enough evidence suggest we should be thinking bigger in terms of public health policy. Smart policies consistent with the FDA’s Comprehensive Plan for Tobacco and Nicotine Regulation can widen, rather than restrict, access to those who desperately need an alternative to deadly, combustible tobacco products while still restricting access to nonsmoking youths, in particular.