Fostering transparency in e-cigarette research synthesis: The utility and limitations of methodological hierarchies

SUMMARY STATEMENT

Where the evidence meets the criteria for informing the scientific question, we can begin to synthesize findings, even if only a few studies address our question of interest. The key work of our hierarchy – and of criteria like ours – is to determine which studies are informative in this regard.

We appreciate the comments we received in response to our article proposing a methodological hierarchy to determine which studies should be considered to evaluate the impact of e-cigarettes on smoking cessation or reduction (1). As noted by Robson and McNeill (2), the hierarchy we proposed resulted in the exclusion of any observational studies from the small list of informative studies on this topic. We agree that strong observational studies are essential and designed our “criteria” to enable those studies to be included in informing the research question. Unfortunately, at the time of our publication, none of the published observational studies met the six conditions we identified a priori. We agree with Robson and McNeill (2), as well as Weier (3), that regardless of the evidence, many people are using e-cigarettes to try to quit smoking cigarettes. Research on the experience of vapers is essential to informing how e-cigarettes may be used in treatment and identifying key targets for measurement in future studies. However, our goal in describing the hierarchy was to identify the elements of a study that would allow us to draw causal inferences about the effect of e-cigarettes on smoking cessation. As noted by Weier (3), triangulation with qualitative and population-level studies will be needed to assess the convergence of evidence on this topic. Since our search was conducted, population-level studies have been published documenting the relationship between increased frequency of e-cigarette use (4, 5) and increased prevalence of e-cigarette use in the U.S. and U.K. (6, 7) and rates of smoking cessation. There have also been studies showing no association between e-cigarette use and smoking cessation (8, 9). While none of these studies would have met all of our criteria, our criteria can open discussion on how these studies can be evaluated, the design or measures needed to improve inferences drawn from them, and how their findings fit with the strongest studies.

We also agree with Green and Hilton (10) on a call for criteria to evaluate studies on the relationship between e-cigarette use and smoking initiation. As in the case of e-cigarettes and smoking cessation, the conceptual model underlying the research question, the exposures and outcomes of interest, the assumptions of the mechanisms involved, and in this case, where randomized studies are infeasible, the key confounding factors must be articulated in order to determine how to evaluate the evidence.

There are pros and cons to identifying a minimal set of such standards and a tension between a shorter list of criteria that would be more flexible and a longer list that requires greater rigor. If the criteria are too flexible, we may include studies that are uninformative while making them look at least minimally acceptable. Where we landed, however, requires that studies be highly rigorous, especially with respect to collecting detailed information to provide insight into the “dose” of an e-cigarette used (i.e., frequency and duration of use) and, assuming that nicotine is an active part of the intervention, whether nicotine was adequately delivered by the device. These items align with our calls for better measurement of e-cigarette use in all studies, including national surveillance (11). Importantly, the goal of this work is to foster transparency in e-cigarette research and advance rigorous, informative studies that guide effective treatments and policies to reduce smoking.