“Tips From Former Smokers” Can Benefit From Considering All Available Data

We agree¹ that “Tips from Former Smokers” (Tips) has been incredibly important and has likely helped many smokers kick the habit. We also agree that campaign evaluations should be based on all available data.

For these same reasons, however, our study² should not be discounted simply because the findings suggest Internet searches for cessation and the health risks featured in Tips declined between the first and second years of the campaign. Moreover, careful scrutiny of the most relevant peer-reviewed works, including those cited by the commenters, yields findings similar to our original analysis of search queries (Figure 1).^2–6

Figure 1. Peer reviewed evidence regarding Tips, 2012-2013.

The graph shows the change in 1-800-Quit-Now calls during Tips 2012⁴ and Tips 2013⁶ compared to a similar period for 2011 as derived from the initial Tips 2012 study;⁴ the results of our study regarding Google searches for cessation (e.g., “quit smoking”) during Tips 2012 and Tips 2013 compared to the same period for 2011, but with the analyses for Tips 2013 combined into a single estimate;² the changes in smokefree.gov visits during Tips 2012⁴ and Tips 2013⁶ compared to the same period for 2011 as also derived from the initial Tips 2012 study.⁴ These reanalyses used the entire window (e.g., start and end dates of Tips) to specify the portions of the time series used in the analysis, however, the results yield the same conclusions if we only analyze a subset of weeks with the highest advertisement volume for Tips 2013 as derived from the commenters’ figure. Changes in self-reported quit attempts were derived from responses before and after Tips for Tips 2012³ and differences in quit attempts between respondents recalling exposure versus not recalling exposure to Tips in a cross-sectional study for Tips 2013,⁵ both after controlling for confounders in a multiple logistic regressions. All reanalyses relied on a relative measure comparing the difference in the observed and counterfactual outcomes, subsequently divided by the counterfactual, indicating a percent change (e.g., Tips 2012 quit attempts was ~(34%-31%)/31%). This makes the analyses across the diverse metrics comparable on a similar scale.

The commenters’ Lancet study found self-reported quit attempts significantly increased from 31% to 34% during the first year of Tips,³ along with increases in 1-800-Quit-Now calls and Smokefree.gov visits.⁴ This news was rightly celebrated and consistent with these findings; our results demonstrated a concurrent increase in cessation-related Internet searches (19%, 95% CI=14%, 26%).²

But, looking beyond the campaign’s first year, we demonstrated another important finding that traditional metrics also demonstrate: Tips’ appeal may be waning. First, self-reported quit attempts were not statistically significantly higher among those reporting exposure to the campaign during Tips 2013 after adjusting for covariates in multiple logistic regressions,⁵ mirroring what we found for cessation searches in 2013. Second, even though 1-800-Quit-Now calls increased during Tips 2013, this increase was significantly (p0.001) lower (by a factor of nearly half) than the increase during Tips 2012.^4,6 Our results reflected similar patterns, with significantly fewer searches during Tips 2013 than Tips 2012, including for health risks in both repeated advertisements and new advertisements.

There was one caveat. Smokefree.gov visits increased substantially more during Tips 2013 than Tips 2012 and this merits recognition, but also investigation of the cause so it can be exploited.⁶

That our search-based analytics were largely confirmed by traditional metrics underscores their probable validity for understanding cessation help seeking and interest in health risks. Admittedly, the commenters rightly acknowledge our method had limitations, as all methods do. For example, survey responses change as a function of the survey itself in addition to the intervention being studied.⁷ We have pioneered many new methods for search query surveillance,⁸ including a state-of-the-art revision to Google Flu Trends (the basis of the commenters’ critique of search data).⁹ This work and numerous validation studies elsewhere (including those for tobacco-related behaviors)^10,11 suggests the commenters’ concerns do not nullify the use of search query surveillance for understanding Tips.

It is widely accepted that media campaigns have the greatest appeal when they offer new content with outcomes spiking early and declining as content is repeated.¹² Based on all available data, Tips appears to be following a similar pattern.

Our findings do not diminish how Tips has helped smokers or our support for mass media campaigns. We remain enthusiastic about what could be next for Tips and how search query surveillance, along with careful analysis of all available data, can inform future evaluations.