Problematic Assessment of the Impact of Vaporized Nicotine Product Initiation in the United States

Levy et al.¹ recently published a decision theoretic-based analysis of the potential public health impact of vaporized nicotine products (VNPs), such as electronic cigarettes. We appreciate that the authors addressed this topic of growing public health importance in the United States. However, we are concerned about the validity of their model and, therefore, its conclusions.

Levy et al. fail to adhere to many well-established scientific best practices of forecasting.^2,3 Levy et al. could—although did not—assess the accuracy of their short-term predictions. Such an assessment reveals that their model is inaccurate.

Their model focuses on the birth cohort of 1997 aging over time. This cohort perspective enables validation of model prediction for years in which observable data are available. Their model predicted 12.6% of males and 8.5% females in the 1997 birth cohort would be current cigarette smokers at age 18, which occurred in the year 2015. Yet, in reality, only 6.6% of 18-year old males (95% CI, 2.2% to 11.1%) and 3.6% of 18-year old females (95% CI, 0.4% to 6.7%) were current every day or some day cigarette smokers based on the 2015 National Health Interview Survey. These short-term prediction errors produced by the Levy et al. model will propagate substantially in the long term.

Levy et al. also fail to estimate the overall uncertainty in their ultimate quantity of interest: the number of years of life gained or lost from VNP use. Their model is a function of dozens of parameters, each with its own sampling and stochastic uncertainty. Uncertainty in the years of life gained or lost from VNP use is far more than the sum of uncertainty in each parameter. The sensitivity analysis performed, which varies one parameter and holds all other parameters constant, grossly underestimates overall uncertainty and confers a false sense of confidence on the true public health impact of VNP use.

A problematic major assumption Levy et al. make involves harm reduction associated with long-term use of VNPs. They set this harm reduction as 95% for exclusive VNP use compared to exclusive cigarette use. However, scientific evidence is rapidly accumulating that VNPs are far from benign. For example, e-cigarette aerosols carry high levels of aldehydes (eg, formaldehyde) that affect cardiovascular function and high levels of fine particles that accelerate heart disease.^4,5 E-cigarette users experience equivalent reductions in vascular function (eg, vitamin E levels and flow-mediation dilatation) as cigarette smokers.⁶ Furthermore, e-cigarette use suppresses immune and inflammatory-response genes in nasal epithelial cells similar to cigarette smoking.⁷ Although Levy et al. vary the harm reduction associated with long-term use of VNPs to 75% in their sensitivity analysis, an even smaller harm reduction may be more realistic and ought to have been considered.

In summary, forecasting models such as that of Levy et al. can be useful to the Food and Drug Administration and other regulatory agencies. Yet, forecasting models must adhere to well-established scientific best practices. Failing to utilize the best available scientific evidence, producing inaccurate short-term forecasts, and omitting uncertainty in the ultimate quantity of interest greatly reduces the utility of any forecast.

Funding

This work was supported by National Cancer Institute at the National Institutes of Health (R21-CA197912 to SS, R01-CA077026 to JS, R01-CA140150 and R21-CA185767 to BP, and R01-CA190347 to JPP).

Declaration of Interests

None declared.