Exploring the “Active Ingredients” of an Online Smoking Intervention: A Randomized Factorial Trial

Abstract

Introduction:

Research needs to systematically identify which components increase online intervention effectiveness (i.e., active ingredients). This study explores the effects of 4 potentially important design features in an Internet-based, population-level smoking intervention.

Methods:

Smokers (n = 1,865) were recruited from a large health care organization, regardless of readiness to quit. Using a full factorial design, participants were randomized to 1 of the 2 levels of each experimental factor (message tone [prescriptive vs. motivational], navigation autonomy [dictated vs. not], e-mail reminders [yes vs. no], and receipt of personally tailored testimonials [yes vs. no]) and provided access to the online intervention. Primary outcomes were self-reported 7-day point-prevalent smoking abstinence and confirmed utilization of adjunct treatment (pharmacotherapy or phone counseling) available through the health plan at 1 year. Outcomes were also assessed at 2 and 6 months and were examined among all enrolled participants (intent-to-treat [ITT]) and all who viewed the intervention (modified ITT).

Results:

At 1 year, 13.7% were abstinent and 26.0% utilized adjunct treatment. None of the contrasting factor levels differentially influenced abstinence or treatment utilization at 12 months. In the modified ITT sample, smokers receiving testimonials were less likely to use adjunct treatment at 6 months (odds ratio = 0.54, 95% confidence interval = 0.30–0.98, p = .04).

Conclusions:

None of the design features enhanced treatment outcome. The negative effect observed for testimonials is provocative, but it should be viewed with caution. This study offers a model for future research testing the “active ingredients” of online interventions.

INTRODUCTION

The number of studies examining the effectiveness of online smoking cessation programs has exploded in recent years. However, as recent reviews demonstrate, the evidence base for these programs among adult smokers is mixed (Berg, 2011; Civljak, Sheikh, Stead, & Car, 2010; Hutton et al., 2011; Myung, McDonnell, Kazinets, Seo, & Moskowitz, 2009). Internet-based cessation programs can be effective compared to control intervention (An et al., 2008; Brendryen & Kraft, 2008; Haug, Meyer, & John, 2011; Nath Simmons, Heckman, Fink, Small, & Brandon, 2013; Swartz, Noell, Schroeder, & Ary, 2006), but they are not always (Dezee, Wink, & Cowan, 2013; Japuntich et al., 2006; Oenema, Brug, Dijkstra, de Weerdt, & de Vries, 2008; Pisinger, Jorgensen, Moller, Dossing, & Jorgensen, 2010). This is not surprising given the diversity of these programs, including variation in their informational content, presentation style, content delivery methods, and use of supplemental intervention and outreach.

After more than a decade of research, it is time to move beyond asking if online smoking cessation programs are effective and begin to systematically identify what specific content and design features make these programs more effective. Only with this level of understanding will we be able to consistently develop effective online smoking cessation programs in the future.

Collins and colleagues (2011) detailed a disciplined methodological approach for creating effective behavioral interventions, called the Multiphase Optimization Strategy (MOST; Collins, Dziak, & Li, 2009; Collins, Murphy, Nair, & Strecher, 2005; Collins, Murphy, & Strecher, 2007). This strategy uses sequential screening, refining, and confirming experiments to ultimately build and validate an “optimized” intervention. In a prior screening experiment, we found offering smokers highly tailored “success stories” (aka, testimonials) and using a personalized message source (research study team vs. participants’ health insurer), each increased smoking abstinence rates at six months among smokers ready to quit smoking and who were provided nicotine replacement therapy (Strecher et al., 2008).

In this screening study, we examined the independent and combined effects of four design features, or potential “active ingredients.” Unlike our prior study, this intervention was designed for smokers at all pre-action stages of readiness to quit smoking. Most smokers are not ready to quit in the next month (Campbell et al., 2013; Fu et al., 2011; Prochaska & Velicer, 1997; Velicer et al., 1995; Velicer, Redding, Sun, & Prochaska, 2007; Wewers, Stillman, Hartman, & Shopland, 2003), so it is important to create online interventions that are effective on a population level, not just among the minority ready to take action.

The four design features (aka, factors) were as follows: message tone (prescriptive vs. motivational), navigation autonomy (dictated vs. not), e-mail reminders (yes vs. no), and inclusion of testimonials (yes vs. no). Each of these factors, discussed in detail elsewhere (McClure et al., 2012, 2013), was chosen either based on empirical or theoretical evidence for its effects on smoking cessation or because its treatment effects were unclear. For example, research suggests interventions grounded in the principles of motivational interviewing can be effective across a range of health risk behaviors, including smoking cessation (Burke, Arkowitz, & Menchola, 2003; Heckman, Egleston, & Hofmann, 2010; Hettema & Hendricks, 2010; Hettema, Steele, & Miller, 2005; Lundahl & Burke, 2009; Vasilaki, Hosier, & Cox, 2006). Dictating content or content order based on readiness to quit may also increase treatment effectiveness by making treatment information more salient to smokers; and recently published research suggests dictating navigation through a health-focused Web site can increase exposure to the intervention content and health knowledge (Crutzen, Cyr, & de Vries, 2012), each of which could mediate behavior change. Testimonials can “transport” readers, influence their beliefs (Green & Brock, 2000), and as a result, lead to smoking behavior change (Strecher et al., 2008). Finally, periodic e-mail reminders may encourage return Web visits (Greaney et al., 2012; Schneider, van Osch, Schulz, Kremers, & de Vries, 2012), increase treatment exposure, and promote greater behavior change. In short, each of the chosen factors represents a potential “active ingredient” for an online smoking cessation intervention.

We hypothesized that content written in a motivational tone and nondictated navigation autonomy would each promote greater program engagement, increase use of adjunctive treatment services (counseling and/or pharmacotherapy available through participants’ health insurance), and result in greater long-term abstinence. These hypotheses preceded more recent evidence that suggests dictating navigation increases treatment exposure (e.g., Crutzen et al., 2012). Similarly, no data were available on the effects of proactive e-mail prompts to suggest whether or not this would encourage greater treatment use or cessation. We hypothesized that these prompts could encourage return visits and program exposure, thereby increasing the odds of quitting smoking and use of adjunctive treatment, but they could also be an annoyance and undermine treatment participation or behavior change. Finally, based on our own prior research demonstrating a relationship between personally tailored testimonials and smoking abstinence among smokers who were ready to stop smoking (Strecher et al., 2008), we sought to explore whether the same would be true when the sample included smokers at differing stages of readiness to quit, when pharmacotherapy was not provided, and when smoking was assessed at longer term follow-up. Findings from this study add to the still nascent literature informing the optimal design of Internet-based behavior change programs, in general, and for smoking cessation, in particular.

METHODS

The Questions about Quitting (Q²) study recruited participants from Group Health, a large, nonprofit health plan in the U.S. Pacific Northwest. Study materials were approved by the institutional review boards at Group Health Research Institute and the University of Michigan. The study is registered with clinicaltrials.gov (NCT00992264). Data were collected between May 2010 and November 2012. Study methods have been described in detail (McClure et al., 2012, 2013), but relevant details are summarized here.

Enrollment Procedures

Adult likely smokers were identified from automated health plan records and mailed a study invitation letter. Interested smokers were directed to the study Web site and given a unique log-in access code. Individuals were screened for eligibility online. Those eligible provided consent, completed a baseline assessment, were randomized, and then viewed the study welcome page. This page described the intervention program, based on individuals’ assigned program features, and how to immediately access the program.

Eligibility criteria were as follows: 18 years or older, a current Group Health member, smoked lifetime 100 cigarettes, smoked in the last seven days, averaged at least 5 cigarettes/day, were not using treatment to stop smoking, had access to the Internet for personal use, were willing to check their e-mail at least once a week, were comfortable reading and writing in English, had no visual impairments preventing computer use, and were comfortable using a computer and the Internet.

Screening Experiment Design

In MOST, screening experiments identify key intervention characteristics that independently or in combination influence the intervention outcome (Baker et al., 2011; Collins et al., 2005, 2007, 2011). Once important characteristics are identified, their ideal dose/exposure can be explored. Finally, important components are combined in an “optimized” intervention, which is then compared against a control or standard of care to establish the efficacy of the optimized product.

Consistent with MOST, we conducted a two-level, full factorial experiment to screen for optimum design features. Half of participants were randomized to each contrasting level of the four experimental factors, resulting in 16 arms (see Table 1). Randomization was automatic following the baseline survey and was stratified by readiness to quit smoking. All participants were exposed to an active intervention, but they were blinded to group assignment.

Table 1.

Randomization Arms

Arm	Message tone	Navigation	Proactive e-mails	Testimonials
1	Prescriptive	Dictated	Yes	Yes
2	Prescriptive	Not dictated	Yes	Yes
3	Prescriptive	Dictated	No	Yes
4	Prescriptive	Not dictated	No	Yes
5	Prescriptive	Dictated	Yes	No
6	Prescriptive	Not dictated	Yes	No
7	Prescriptive	Dictated	No	No
8	Prescriptive	Not dictated	No	No
9	Motivational	Dictated	Yes	Yes
10	Motivational	Not dictated	Yes	Yes
11	Motivational	Dictated	No	Yes
12	Motivational	Not dictated	No	Yes
13	Motivational	Dictated	Yes	No
14	Motivational	Not dictated	Yes	No
15	Motivational	Dictated	No	No
16	Motivational	Not dictated	No	No

Intervention

The Q² intervention was organized into three core content areas, each designed for smokers at different stages of readiness to quit smoking (those not ready to quit, those ready to quit, and those already quit; see sample screenshots in McClure et al., 2013). The core content contained motivational or action-oriented information for quitting smoking tailored to each person’s interest in quitting smoking, gender, smoking history, self-efficacy, and other baseline characteristics. Additional supplemental content was included in a “Special Features” section. Supplemental content was not tailored and addressed a range of health topics of relevance to smokers, but was not necessarily focused on smoking.

Experimental Factors

Each person’s intervention was similar, but varied somewhat based on the assigned experimental factor levels. Each factor is summarized in the following subsections and discussed in more detail elsewhere (McClure et al., 2012, 2013).

Message Tone

Intervention content was written in either a prescriptive or a motivational tone. Prescriptive messaging was didactic, advised smokers to quit smoking, and informed them how to achieve this goal. Motivational messaging was designed to reflect the key principles of motivational interviewing (express empathy, develop discrepancy, roll with resistance, support autonomy, and self-efficacy; Miller & Rollnick, 2002). Motivational messages recognized smokers’ potential ambivalence about quitting and their autonomy in making decisions about if, how, and when they would quit smoking.

Navigation Autonomy

Participants were randomized either to freely navigate the site or were required to view content in a pre-specified, or dictated, order based on their baseline readiness to quit smoking. After viewing the content deemed most relevant to their baseline readiness to quit, these individuals were then free to navigate the site at-will.

Proactive E-mails

Participants were randomized to receive a weekly e-mail reminder or not. E-mail messages were standardized across all individuals and encouraged participants to return to the Q² Web site to view the supplemental content.

Testimonials

Participants were randomized to receive a personalized testimonial in each of the three core content areas or to not receive testimonials. These testimonials were designed to promote self-efficacy for quitting and were tailored on each individual’s stage of change, level of nicotine dependence, prior use of pharmacotherapy, depression history, perceived risks and benefits of quitting smoking, and their self-efficacy for quitting.

Assessment and Measures

Participants completed online assessments at baseline and 2-, 6-, and 12-months postenrollment. Primary outcomes were assessed at one year, but interim timepoints allowed us to explore whether the contrasting factor levels differentially affected the main outcomes over time. Participants received $20 after completing the baseline survey and $10 for completing each follow-up survey. To encourage participation at one-year follow-up, five participants were randomly chosen from among the 12-month respondents to receive a $100 gift card.

Primary study outcomes were a self-report of no smoking, even a puff, for the last seven days (seven-day point-prevalent abstinence [PPA]) and verified treatment utilization (either pharmacotherapy or counseling) at 12 months. Per convention, nonresponders at follow-up were considered smokers. As part of the intervention, all participants were encouraged to consider use of pharmacotherapy and enrollment in the health plan-sponsored phone counseling program. Each of these treatments is covered for health plan members. Automated pharmacy records and insurance claim records were used to assess use of pharmacotherapy (nicotine replacement, varenicline, or bupropion). For bupropion or wellbutrin, only prescriptions associated with an indication of nicotine dependence or smoking cessation were included. Utilization of the health plan-sponsored phone counseling program was assessed using treatment enrollment records.

Secondary outcomes included self-reported seven-day PPA at two and three months, 30-day PPA, and verified use of either pharmacotherapy or the covered phone counseling program, each assessed separately at 2-, 6-, and 12-months postenrollment. Self-reported utilization of any treatment (pharmacotherapy, counseling, or other) was also assessed, but the results are not presented because they did not differ from the findings based on the more methodologically rigorous verified treatment utilization reported in this paper.

Additional baseline measures included demographics, cigarettes per day; use of other tobacco products; the Fagerström Test for Nicotine Dependence (Heatherton, Kozlowski, Frecker, & Fagerström, 1991); readiness to quit (characterized as in the next 30 days, in the next six months but not in next 30 days, or not in the next six months); motivation and self-efficacy for quitting smoking, each assessed on a 10-point Likert scale (1 [not at all] and 10 [very]); perceived importance of quitting smoking, assessed on a 10-point Likert scale (1 [not at all] and 10 [very]); presence of a smoking spouse or partner (yes/no); self-reported depression history (yes/no; McClure et al., 2009); where smoking is allowed in the home (anywhere, certain areas, outside, not allowed); need for cognition (Cacioppo, Petty, & Kao, 1984); and literacy (Morris, MacLean, Chew, & Littenberg, 2006). Finally, at 2-month follow-up, participants rated the program across a range of qualitative descriptors (e.g., supporting, preachy, respectful, personalized, etc.), each assessed on a 7-point scale ranging from “not at all” to “extremely.” Participants who recalled viewing the testimonials rated how helpful the testimonials were, how much they related to them, whether they were believable, whether the stories were written about actual former smokers, or whether the stories were based on former smokers’ experiences but written by someone else. Each item was rated on a 7-point scale ranging from “not at all” to “a lot” or “strongly disagree” to “strongly agree.”

Statistical Power and Analytic Samples

Power calculations assumed a 12-month PPA rate of 15% (see rationale in McClure et al., 2012). The study was designed to have 85% power detect a 5% difference between each contrasting factor level for each primary outcome using a two-sided test to compare two sample proportions and when significance was set to 0.05. This 5% difference in quit rate was chosen a priori to represent the minimum clinically meaningful difference between treatment conditions. That is, if abstinence or treatment utilization differed by less than 5%, one could argue the two treatment conditions are not meaningfully different enough to deem one as clinically superior. However, differences of 5% could have a significant impact if implemented on a population level.

Analyses were conducted in two analytic samples. Our primary analysis was an intent-to-treat (ITT) analysis, using all participants randomized. We also conducted a modified ITT analysis in which only participants exposed to the treatment content by the assessment timepoint were included. Treatment exposure was determined as exposure to any of the three core content areas as determined by automated tracking data. Participants who had viewed intervention content prior to each follow-up were included in the modified ITT sample at that timepoint. Individuals who left the health plan prior to study enrollment and failed to re-enroll were excluded from analyses of verified treatment utilization because these individuals were no longer eligible to access the provided pharmacotherapy or counseling.

Analyses

Descriptive statistics were used to characterize the ITT and modified ITT study samples. Preliminary analyses explored the relationship between baseline covariates and intervention engagement and were used to determine which covariates were adjusted for in the modified ITT analyses. Main effects and two-way interactions for each of the randomized factors were assessed using descriptive statistics and logistic regression. All logistic regression models included main effects of the four contrasting levels, all two-way interactions, and the following baseline covariates: readiness to quit (stratification factor for randomization), education, White-non Hispanic race/ethnicity, age, and sex. Logistic regression models performed in the modified ITT subsamples also controlled for literacy, nicotine dependence, the number of cigarettes per day, self-reported depression history, motivation to quit smoking, need for cognition, self-efficacy, perceived importance of quitting, and whether or not the participant’s partner smoked. These covariates were included because preliminary analyses indicated the associations between them and whether or not an individual was included in the modified ITT sample were strong enough that imbalance may have occurred between the randomized factor levels. As a result, unadjusted factor comparisons estimates might have been biased. Tests of significance for randomized factors were taken from multivariate logistic regression models unless otherwise specified. Because the purpose of the current screening experiment is more exploratory than a traditional randomized clinical trial, we did not control for multiple comparisons, but instead report all odds ratios (ORs) and 95% confidence intervals (CIs). Analyses were conducted using Stata Version 12 (StataCorp, 2011).

RESULTS

Recruitment, Intervention Exposure, and Retention

Recruitment flow is presented in Figure 1. A total of 1,865 participants were enrolled and randomized. Of these, 1,261 (67.6%) viewed the intervention. Twenty-seven (1.4%) left the health plan prior to randomization and were excluded from treatment utilization analyses. Most intervention exposure occurred within the first two months. Follow-up survey completion rates were 68.6% (n = 1,279) at 2 months, 63.3% (n = 1,180) at 6 months, and 72.4% (n = 1,350) at 12 months; however, survey participation was not required for inclusion in primary outcome analyses.

Figure 1.

Recruitment flow and allocation.^aTwo participants were not randomized due to a server error and were excluded from the analytic sample.^bHalf of participants were randomized to each level of each experimental factor using a two-level full factorial design, so all participants received an active intervention.^cIntent-to-treat analyses of main outcomes included complete case analyses with no missing data.^dAnalyses of treatment utilization were restricted to participants who were enrolled in the health plan for at least one day during the study period, excluding 27 people from the total sample for these analyses only.^eModified intent-to-treat analyses were restricted to persons who had viewed the intervention by each timepoint. Analyses of main outcomes included complete case analyses with no missing data.

Participants

Demographics are presented in Table 2 for all enrolled participants (ITT sample) and those exposed to the intervention (modified ITT sample) by Month 12. Participants who did not view any intervention content (n = 604) differed from those who did. They were slightly younger (42.7 years vs. 44.9 years, p = .002), less educated (p < .001), less likely to be ready to quit in the next month (38.2% vs. 46.3%; p = .001), had lower motivation for quitting (M score 7.1 vs. 7.6; p = .001), rated quitting smoking as less important to them (M score 7.3 vs. 7.8; p < .001), had lower need for cognition scores (M score 80.4 vs. 84.4, p < .001), and had lower health literacy (p = .03).

Table 2.

Baseline Characteristics of Enrolled Participants

	All participants, n = 1,865a	Modified ITT sample, n = 1,261b
Female, n (%)	1,178 (63.2%)	798 (63.3%)
White, non-Hispanic, n (%)	1,534 (82.3%)	1,048 (83.1%)
Education category, n (%)
High school or less	524 (28.1%)	329 (26.1%)
Some college	944 (50.6%)	631 (50.1%)
College degree or higher	396 (21.2%)	300 (23.8%)
Employed, n (%)	1,287 (69.0%)	882 (69.9%)
Married/partnered, n (%)	1,052 (56.4%)	711 (56.4%)
Readiness to quit, n (%)
In next 30 days	815 (43.7%)	584 (46.3%)
In next 6 months, but not in next 30 days	816 (43.8%)	537 (42.6%)
Not thinking of quitting	234 (12.6%)	140 (11.1%)
Years smoked, M (SD)	24.9 (14.2)	25.5 (14.2)
Age, M (SD)	44.2 (14.7)	44.9 (14.6)
Nicotine dependence (FTND), M (SD)	4.2 (2.2)	4.2 (2.2)
Motivation for quitting, M (SD)	7.4 (2.5)	7.6 (2.4)
Self-efficacy for quitting, M (SD)	5.4 (2.6)	5.4 (2.6)
Importance of quitting, M (SD)	7.6 (2.6)	7.8 (2.5)
Need for cognition score, M (SD)	83.1 (17.8)	84.4 (17.9)
Positive history of depression, n (%)	1,268 (68.0%)	866 (68.7%)
Spouse or partner smokes, n (%)	531 (28.5%)	342 (27.1%)
Use of other tobacco products, n (%)	175 (9.4%)	113 (9.0%)
Cigarettes per day, M (SD)	15.4 (7.4)	15.6 (7.5)
Health literacy, n (%)c
Never	1,310 (70.2%)	910 (72.2%)
Rarely	139 (23.0%)	264 (20.9%)
Sometimes	52 (8.6%)	69 (5.5%)
Often	7 (1.2%)	12 (1.0%)
Always	6 (1.0%)	6 (0.5%)

Note. FTND = Fagerström Test of Nicotine Dependence; ITT = intent-to-treat; SD = standard deviation.

^aComplete data were available on all baseline outcomes, with one missing value each for race and education.

^bParticipants confirmed exposed to the intervention by 12-month follow-up.

^cFrequency report needing help reading instructions, pamphlets, or other written material from a doctor or pharmacy.

Smoking Abstinence

Overall, 13.7% (n = 256) reported seven-day PPA at 12 months. Abstinence rates, ORs, and 95% CIs for main effects are reported in Tables 3 and 4. None of the experimental factor levels significantly altered the odds of quitting smoking at any timepoint. For example, in the ITT sample (Table 3), 13.0% of participants randomized to the prescriptive tone were abstinent at 12 months compared to 14.5% of individuals randomized to the motivational tone (OR = 0.88, 95% CI = 0.51, 1.51). As the CI includes 1.00, we cannot conclude a difference between the contrasting levels. Findings were similar and nonsignificant for 30-day PPA (data not shown).

Table 3.

Main Effects of Factor Levels on Outcomes in the Intent-to-Treat Sample

	7-day PPA (N =1,865)	Adjunct treatment utilization (N = 1,838)a	7-day PPA (N =1,865)	Adjunct treatment utilization (N = 1,838)a
	N	OR (95% CI) % abstinent	N	OR (95% CI) % abstinent
2 months
Tone		1.20 (0.62–2.32)		1.23 (0.72–2.12)
Prescriptive	932	7.8	920	12.9
Motivational (reference)	933	7.5	918	11.1
Testimonial		0.94 (0.48–1.86)		0.66 (0.37–1.16)
Yes	933	7.1	926	11.0
No (reference)	932	8.3	912	13.0
Navigation		0.74 (0.37–1.49)		0.72 (0.41–1.26)
Dictated	934	6.8	920	11.4
Autonomous (reference)	931	8.5	918	12.6
Proactive e-mails		0.76 (0.38–1.52)		0.81 (0.46–1.42)
Yes	933	7.0	916	10.6
No (reference)	932	8.4	922	13.4
6 months
Tone		0.97 (0.52–1.81)		0.99 (0.63–1.57)
Prescriptive	932	10.1	920	19.2
Motivational (reference)	933	10.8	918	18.4
Testimonial		1.65 (0.91–3.00)		0.67 (0.42–1.08)
Yes	933	10.0	926	18.1
No (reference)	932	10.9	912	19.5
Navigation		1.53 (0.84–2.79)		0.86 (0.54–1.36)
Dictated	934	11.5	920	18.9
Autonomous (reference)	931	9.4	918	18.7
Proactive e-mails		0.75 (0.40–1.43)		0.66 (0.41–1.06)
Yes	933	9.1	916	16.7
No (reference)	932	11.8	922	20.9
12 months
Tone		0.88 (0.51–1.51)		0.94 (0.62–1.43)
Prescriptive	932	13.0	920	26.1
Motivational (reference)	933	14.5	918	25.8
Testimonial		1.12 (0.66–1.91)		0.70 (0.46–1.07)
Yes	933	13.8	926	25.3
No (reference)	932	13.6	912	26.6
Navigation		1.19 (0.70–2.02)		0.78 (0.51–1.18)
Dictated	934	14.5	920	26.0
Autonomous (reference)	931	13.0	918	25.9
Proactive e-mails		0.75 (0.43–1.30)		0.67 (0.44–1.03)
Yes	933	13.3	916	23.7
No (reference)	932	14.2	922	28.2

Note. PPA = point-prevalent abstinence; OR = odds ratio; CI = confidence interval.

^aTwenty-seven participants were excluded for not being enrolled in the health plan during the study period and for having no access to the provided adjunct treatment.

Table 4.

Main Effects of Factor Levels on Outcomes in the Modified Intent-to-Treat Samples

	7-day PPA		Adjunct treatment utilizationa
2 months	Total N = 1,175	OR (95% CI) % abstinent	Total N = 1,157	OR (95% CI) % utilizing treatment
Tone		1.13 (0.51–2.52)		1.18 (0.62–2.24)
Prescriptive	605	9.6	596	15.3
Motivational (reference)	570	8.9	561	14.4
Testimonial		0.63 (0.27–1.47)		0.51 (0.26–1.03)
Yes	574	8.5	569	13.4
No (reference)	601	10.0	588	16.3
Navigation		0.63 (0.27–1.47)		0.69 (0.35–1.35)
Dictated	602	8.5	594	13.5
Autonomous (reference)	573	10.1	563	16.3
Proactive e-mails		0.80 (0.35–1.82)		1.17 (0.61–2.24)
Yes	596	8.9	582	13.6
No (reference)	579	9.7	575	16.2
6 months	Total N = 1,237	OR (95% CI) % abstinent	Totala N = 1,219	OR (95% CI) % utilizing treatment
Tone		0.79 (0.38–1.66)		1.10 (0.63–1.93)
Prescriptive	638	11.8	629	22.1
Motivational (reference)	599	12.5	590	21.2
Testimonial		1.07 (0.52–2.18)		0.54 (0.30–0.98)*
Yes	612	11.1	607	20.1
No (reference)	625	13.1	612	22.4
Navigation		1.49 (0.74–2.97)		0.87 (0.49–1.52)
Dictated	642	13.2	634	21.1
Autonomous (reference)	595	10.9	585	22.2
Proactive e-mails		0.76 (0.36–1.58)		0.95 (0.54–1.67)
Yes	622	11.2	608	20.2
No (reference)	615	13.0	611	23.1
12 months	Total N = 1,261	OR (95% CI) % abstinent	Totala N = 1,243	OR (95% CI) % utilizing treatment
Tone		0.82 (0.43–1.57)		0.95 (0.57–1.57)
Prescriptive	650	14.9	641	27.8
Motivational (reference)	611	15.7	602	28.2
Testimonial		0.81 (0.42–1.56)		0.66 (0.39–1.11)
Yes	630	14.9	625	27.0
No (reference)	631	15.7	618	29.0
Navigation		1.26 (0.66–2.37)		0.74 (0.44–1.23)
Dictated	654	16.5	646	26.9
Autonomous (reference)	607	14.0	597	29.1
Proactive e-mails		0.72 (0.37–1.40)		0.88 (0.53–1.46)
Yes	636	15.4	622	26.5
No (reference)	625	15.2	621	29.5

Note. PPA = point-prevalent abstinence; OR = odds ratio; CI = confidence interval.

^aTwenty-seven participants were excluded for not being enrolled in the health plan during the study period and for having no access to the provided adjunct treatment.

^* p < .05.

No two-way interactions between factors significantly influenced smoking abstinence in either analytic sample. Data are available in the Supplementary Tables 1 and 2.

Treatment Utilization

Twenty-six percent (n = 478) of participants utilized the provided adjunct treatment (pharmacotherapy or counseling) at one year. Main effects of the factor levels on verified treatment utilization in the ITT and modified ITT samples are reported in Tables 3 and 4. No statistically significant main effects were observed in the ITT sample (Table 3). In contrast, among participants exposed to the intervention at six-month follow-up (modified ITT sample), a significant main effect of testimonials on treatment utilization was observed (OR = 0.54, 95% CI = 0.30–0.98, p = .04). Persons assigned to receive the testimonials were less likely to have used adjunct treatment during the first six-month postenrollment (20.1% utilized treatment vs. 22.4%). OR’s at 2 and 12 months were similar, but nonsignificant (2-month OR = 0.51, 95% CI = 0.26–1.04, p = .06; 12-month OR = 0.66, 95% CI = 0.39–1.11, p = .12). No other main effects were observed in the modified ITT samples (see Table 3).

No main effects were observed for the effect of each factor level on use of either pharmacotherapy or the covered phone counseling program, each examined separately (data not shown). Finally, no two-way interactions between any of the factors were significant (see Supplementary Tables 1 and 2).

Testimonials

Overall, 343 people (36.8% of those randomized to testimonials in the ITT sample and 54.4% of those randomized to testimonials in the modified ITT sample) saw at least one of the three testimonial pages. Persons who received the testimonials and recalled seeing them at two months rated them as believable (M = 5.4, SD = 1.3), moderately relatable (M = 4.6, SD = 1.6), and moderately helpful (M = 4.2, SD = 1.7) on a 7-point Likert scale. Most agreed that they were written by actual former smokers (M = 5.3, SD = 1.4) and disagreed with the statement that the stories were based on smokers’ experiences but written by someone else (M = 3.4, SD = 1.7). Overall qualitative program ratings were similar between persons randomized to the testimonials and those who were not (data not shown).

DISCUSSION

None of the contrasting design features studied increased abstinence or adjunct treatment utilization at one year. In fact, the modified ITT results suggest the testimonials may have had a slight negative effect on subsequent utilization of the available adjunct pharmacotherapy and counseling. This pattern was observed at 2, 6, and 12 months, but only statistically significant at six months. Promoting adjunct treatment is potentially important for online interventions because the evidence clearly demonstrates that smoking cessation rates are increased by the use of counseling and/or pharmacotherapy (Fiore et al., 2008). Population-based, online interventions may have limited effects on abstinence if they do not effectively promote uptake of these treatments. Our study offered a methodologically rigorous way to examine intervention effects on this outcome as participants had similar access to the counseling and pharmacotherapy through their health insurance, eliminating access as a barrier, and we could verify treatment use based on automated records. The results were unchanged when we examined self-reported utilization of any available treatment (data not presented).

Contrary to expectations, the results suggest that the tailored testimonials could dissuade people from seeking additional treatment. Although provocative, caution must be used in drawing this conclusion. First, we did not adjust for multiple comparisons so the significant finding at six-month follow-up could be spurious. Second, only about half of those who viewed the intervention (and about a third of all participants) saw the testimonials. If this exposure had been greater, more confidence could be placed in the association between the testimonials and treatment utilization. But even focusing on those people who were randomized to the testimonials and remembered seeing them shortly after exposure, the qualitative ratings offer no indication the testimonials had a negative effect on participants’ overall view of the program, which could have had an indirect effect on subsequent treatment use. Moreover, there was nothing in the testimonial content that should directly account for a negative effect on adjunct treatment utilization. Each testimonial was written in a question and answer format and provided advice and encouragement for how to deal with common issues from the perspective of a former smoker (e.g., How did you get ready to quit?, What were the first two weeks like after you quit?, What did you struggle with after you quit smoking?). Content neither recommended nor dissuaded use of particular therapies. Finally, even if confidence is placed in the overall pattern of lower utilization rates at 2, 6, and 12 months among people assigned to the testimonials in the modified ITT samples, the absolute difference in utilization rates were only from 1%–3%, calling into question the true significance of these findings.

Further research is warranted to understand what role testimonials can play in online smoking cessation programs. We recommend this research look closely at the effects of the specific testimonial content and not simply at whether their presence is impactful. Additionally, future research should explore how to increase exposure to online intervention content. It is striking that one third of participants failed to view any of the intervention content, even though it was immediately accessible following enrollment. In some cases, participants may have been fatigued by the enrollment process, which took 20min or more for eligibility screening, consent, and baseline data collection. Even so, they failed to return later, even when prompted to do so. As we found, those who failed to view the intervention differed in potentially clinically meaningful ways from those who viewed the intervention. We encourage future researchers to routinely report metrics of online program exposure, as well as evaluate strategies for maximizing initial and ongoing program engagement.

This study has a number of strengths. We used a methodologically rigorous design, a large sample, and automated data to verify adjunct treatment utilization and exposure to the online intervention. Additionally, results were analyzed using both ITT and modified ITT samples and our conservative primary outcomes, which imputed missing values, allowed us to include all participants in the analytic samples. It is also worth noting that the overall observed abstinence rate (13.7% at one year) compares favorably to long-term cessation rates associated with minimal self-help interventions (8.5%), quitline counseling (12.7%), and counseling alone (14.6%; Fiore et al., 2008). This supports our belief that the Internet holds promise as a low-cost, population-level strategy for intervening with smokers and suggests the Q² program itself likely had an overall effect.

Study limitations are also noted. As mentioned previously, many participants failed to view the intervention, highlighting the importance of tracking and reporting program exposure in studies testing the effectiveness of online interventions. Additionally, follow-up retention rates were lower than ideal (68.6% at 2 months, 63.3% at 6 months, and 72.4% at 12 months) but higher than reported in other recent online smoking intervention studies (Pike, Rabius, McAlister, & Geiger, 2007; Richardson et al., 2013; Smit, de Vries, & Hoving, 2012; Smit, Hoving, Cox, & de Vries, 2012; Stoddard, Augustson, & Moser, 2008; Swartz et al., 2006). This did not impact our treatment utilization outcomes, but it could have resulted in lower abstinence rates than actually occurred among participants because missing participants were counted as smokers. Finally, it is important to note that the study was under-powered to detect if the observed effects on abstinence or treatment utilization were statistically significant because the observed differences were all less than 5%, the a priori determined threshold used to determine sample size for this trial. Again, this threshold was based on what we believed to be the minimal effect we would consider to have clinical relevance. We recognize that this cutoff of 5% is subjective and that lower thresholds may still be relevant on a population level. Nevertheless, it is difficult to argue for the clinical relevance of the observed effects, which were fairly small. At one year, the largest difference between factor levels in abstinence rates observed in the modified ITT sample was only 1.5%. Even looking across the earlier timepoints, the observed differences did not exceed 2.5% for abstinence and 3.1% for treatment utilization. Given these results, we cannot conclude that any of the experimental factor comparisons had a sufficient differential effect to warrant selecting one factor level over the other. Similarly, the small point estimates and observed differences make it difficult to confidently glean insight from the direction and pattern of results over time beyond concluding that there was not strong evidence supporting implementation any of the experimental factor levels over the others, at least as they were operationalized in this trial.

We should also note that while the primary intent of this study was to understand if each of the proposed design features had meaningful differential effects on smoking cessation or treatment utilization, a secondary intent was to explore whether or not factor level effects varied among individuals. For example, is motivational tone preferential for some smokers while a prescriptive tone is preferential for others? This is a critical line of inquiry to advance the field of personalized treatment tailoring. Presentation and discussion of these results is beyond the scope of this paper, but we do not want to leave readers with the impression that our treatment design advocates a “one size fits all” approach for online intervention. On the contrary, one of the chief advantages of Web-based interventions is that they can be both targeted (i.e., matched to groups of people) and tailored (i.e., adapted for individuals) and in so doing, potentially have a greater effect. This work adds to the still nascent research base needed to inform this work in the future.

CONCLUSIONS

It is time to stop simply asking if Internet-based smoking cessation programs are efficacious and begin to systematically identify the “active ingredients,” which make these programs more effective. Although none of the design features examined in this study enhanced outcomes, this itself is an important outcome. The results also illustrate the potential for design decisions to undermine treatment effects. This study serves as a cautionary tale and a model for others interested in this line of inquiry.

SUPPLEMENTARY MATERIAL

Supplementary Tables 1 and 2 can be found online at http://www.ntr.oxfordjournals.org

FUNDING

This research was funded by the National Cancer Institute (R01 CA138598 to JM).

DECLARATION OF INTERESTS

None declared.