Online Community Use Predicts Abstinence in Combined Internet/Phone Intervention for Smoking Cessation

Abstract

Objective

To estimate the causal effects of online community use on 30-day point prevalence abstinence at 3 months among smokers randomized to combined Internet+Phone intervention for smoking cessation.

Method

Participants were N=399 adult smokers in the Internet+Phone arm of The iQUITT Study, a randomized trial of Internet and proactive telephone counseling for smoking cessation. All participants accessed a web-based smoking-cessation program with an established online community and received telephone counseling. Automated tracking metrics of passive (e.g., reading posts, viewing profiles) and active (e.g., writing posts, sending messages) community use were extracted at 3 months. Self-selected community use defines the groups of interest: None, Passive, and Both (passive+active). Inverse probability of treatment weighting corrected for baseline imbalances on demographic, smoking, and psychosocial variables. Propensity weights estimated via generalized boosted models were used to calculate Average Treatment Effects (ATE) and Average Treatment effects on the Treated (ATT).

Results

Patterns of community use were: None=145 (36.3%), Passive=82 (20.6%), and Both=172 (43.1%). ATE-weighted abstinence rates were: None=12.2% (95% CI=6.7–17.7); Passive=25.2% (95% CI=15.1–35.2); Both=35.5% (95% CI=28.1–42.9). ATT-weighted abstinence rates indicated even greater benefits of passive community use by non-users.

Conclusions

More than one third of participants who received telephone counseling and used the community both passively and actively achieved abstinence. Participation in an established online community as part of a combined Internet+phone intervention has the potential to promote short-term abstinence. Results also demonstrated that information and support that originate in the community can serve as a resource for all users.

For the estimated 42.1 million adults in the United States who currently smoke cigarettes (Agaku, King, Dube, Centers for Disease, & Prevention, 2014), telephone quitlines and Internet cessation programs provide broadly accessible support to quit. There is strong evidence for the effectiveness of telephone counseling interventions (Stead, Hartmann-Boyce, Perera, & Lancaster, 2013). However, standalone quitlines are becoming increasingly rare, as most telephone counseling vendors also provide Internet programs and other adjunct services (Croyle; North American Quitline Consortium, 2014). The Internet is the first place many look for health information, offering unparalleled reach to the millions of smokers who search online for quit smoking information each year (Fox & Duggan, 2013). Internet cessation programs are scalable, available day and night, and capable of delivering the core elements of evidence based cessation treatment (Bock, Graham, Whiteley, & Stoddard, 2008). A number of systematic reviews have documented their effectiveness (Civljak, Stead, Hartmann-Boyce, Sheikh, & Car, 2013; Hutton et al., 2011; Shahab & McEwen, 2009), especially programs that are tailored and interactive.

While multi-modal cessation interventions are known to be more effective than single modality approaches (Fiore, Jaén, Baker, & Tobacco Use and Dependence Guideline Panel, 2008), evidence regarding the effectiveness of combined Internet/phone programs is limited and findings have been mixed. Several observational studies have documented higher abstinence rates among users of both Internet and phone services (Puckett et al., 2015; Zbikowski, Hapgood, Smucker Barnwell, & McAfee, 2008). Three randomized trials of combined Internet/phone interventions have been conducted, two among cigarette smokers (Graham et al., 2011; Swan et al., 2010) and one among smokeless tobacco users (Danaher et al., 2015). In the COMPASS Trial, Swan and colleagues (Swan et al., 2010) randomized smokers from a large health system to receive varenicline in conjunction with Internet, phone, or a combined Internet/phone intervention. Intent-to-treat (ITT) analyses – with missing outcomes imputed as smoking – revealed that abstinence rates were uniformly high across all three arms, with the early advantage noted for phone at 3 months attenuated at the 6-month endpoint. In The iQUITT Study, Graham and colleagues (Graham et al., 2011) randomized smokers recruited through search engines to a basic Internet, enhanced Internet, or enhanced Internet plus telephone counseling intervention. The combined Internet/phone arm outperformed the other two arms at the initial 3-month follow-up, roughly doubling the odds of abstinence, and maintained its advantage throughout the remaining 15-month of follow-up. In the smokeless tobacco trial, Danaher and colleagues (Danaher et al., 2015) randomized participants recruited online to Internet-only, Phone-only, Internet+Phone, or a self-help print control arm. ITT analyses showed that all three interventions outperformed the control, but that the combined Internet+Phone intervention did not outperform either the Internet-only or the Phone-only arms at both 3 and 6 months.

A distinguishing feature of Internet cessation interventions is the opportunity to join an online social network. It is well established that “offline” social networks have direct effects on health behaviors (Berkman, 1995; Christakis & Fowler, 2008; Kawachi & Berkman, 2001), and a recent meta-analysis found a positive effect of online social networking interventions on health behavior change (Laranjo et al., 2015). A number of Internet studies have suggested that individuals who participate in online social networks achieve higher abstinence rates, even after controlling for a range of baseline covariates (An et al., 2008; N. K. Cobb, Graham, Bock, Papandonatos, & Abrams, 2005; Richardson et al., 2013; Schwarzer & Satow, 2012). In secondary analyses of The iQUITT Study, we found that among smokers randomized to the enhanced Internet arm, those that participated in the online community were more likely to be abstinent at the 3 month follow-up compared to non-community users using analytic methods to control for possible selection bias (Graham, Papandonatos, Erar, & Stanton, 2015). Passive engagement in the online community (i.e., “lurking”/reading) exerted the strongest effect on quitting –roughly doubling the odds of abstinence over no community use – with only small incremental benefit observed for those who also actively participated (i.e., posting). These findings suggest that online community engagement may be an important element of Internet cessation treatment.

The role of online community participation in a combined Internet/phone intervention has not been investigated to date. Given the proven effectiveness of telephone counseling, and the relatively intensive nature of this kind of personalized coaching, it is unknown whether the additional information and support from an online community would continue to play a significant role in promoting abstinence in the same way that has been observed in Internet-only studies, or whether any such benefit would be dwarfed by a phone intervention. These analyses aim to replicate and extend our earlier work, by exploring the effect of online social network engagement on abstinence in the context of a combined Internet/phone intervention for smoking cessation, while still controlling for between-subject differences in the propensity to utilize the various treatment components. Disentangling the unique contribution of Community engagement on smoking abstinence, over and above any benefits conferred by telephone counseling and non-Community website utilization, has practical implications for the large number of payers and providers that have already adopted combined Internet and phone services. Furthermore, identifying the key ingredients of these broad-reach, scalable interventions is critical to advancing the evidence base (Civljak et al., 2013) and maximizing their public health impact.

Method

Ethical Approval

The study protocol received human subject protections approval from the Georgetown University institutional review board.

Design

The parent study (The iQUITT Study, NCT00282009) was a 3-arm randomized trial conducted between March 2005 and November 2008 that enrolled a total of 2,005 participants (Graham et al., 2011). The parent trial compared an interactive smoking cessation website with an online social network (Enhanced Internet) alone and in conjunction with proactive telephone counseling (Enhanced Internet plus Phone) to a static Internet comparison condition (Basic Internet). The current analyses focus on the combined Internet/phone arm, which yielded abstinence rates of 19.0%, 19.7%, 21.5%, and 19.6% at 3, 6, 12, and 18 months post-randomization, respectively (Graham et al., 2011). Given that abstinence rates remained stable throughout the study, and that use of the Internet intervention and completion of telephone counseling calls occurred primarily during the first 3 months of the trial (C. O. Cobb & Graham, 2014), we decided to focus our analyses on the initial 3-month follow-up. The sample for the current study comprised N=399 participants that logged in to the website at least once and completed one or more telephone counseling calls during the first 3 months of the study. Self-reported smoking status at 3 months was available on 82% (N=329), with missing outcomes imputed as smoking per study protocol.

Participants

Participants were current smokers aged 18 and older in the United States who smoked 5 or more cigarettes per day. Recruitment was conducted via Internet search engines: individuals who used the terms “quit(ting) smoking”, “stop(ping) smoking”, or “smoking” in a search engine query and who clicked on a link to the cessation website being evaluated (www.quitnet.com) were recruited to participate in the parent study (Graham, Bock, Cobb, Niaura, & Abrams, 2006). Eligibility screening and informed consent were conducted online. The baseline telephone survey was administered by trained research assistants within 48 hours of consent. Following the baseline assessment, participants were randomized (stratified by gender and baseline motivation to quit) using a random number generator. Participants were emailed a link to their assigned Internet intervention, and received a proactive counseling call within 48 hours.

Interventions

Participants had free access to the premium service of QuitNet, a widely used web-based smoking cessation program (N. K. Cobb et al., 2005) with a large online social network (N. K. Cobb, Graham, & Abrams, 2010). Communication in this network occurred through asynchronous channels such as private internal e-mail (“Q-Mail”) and one-to-many messaging in threaded forums, or through synchronous channels such as chat rooms. Users were able to self-affiliate into clubs (user-initiated mini-sites, complete with dedicated forums), and buddy lists allowed individuals to keep track of their friends. Social influence regarding cessation was conveyed through profile pages, journals (similar to a blog), anniversary lists, and testimonials. QuitNet maintained a complete transactional history of all events during the trial period.

Participants were also offered five proactive telephone calls in a relapse-sensitive schedule (Zhu et al., 1996) delivered by professional telephone counselors from National Jewish Health in Denver, CO. National Jewish Health is a nonprofit academic medical center that provides telephone counseling services to state quit lines, health plans, and employer groups. Study counselors followed the same counseling and quality monitoring protocols in place at the National Jewish quitline operation. They established a supportive working relationship with participants focused on skill building in preparation for a quit attempt and to prevent relapse. Counselors prompted and reinforced use of QuitNet during each call based on real-time summary data regarding a participant’s use of the Web site (e.g., number of visits, date of last login, feature utilization) available through a web-based interface that integrated the two treatment modalities. Telephone counselors recorded the number of completed calls.

Assessment Procedures

The baseline assessment consisted of demographic, smoking, psychosocial, and medical history measures. The follow-up assessment was conducted by research assistants who could not be masked to treatment assignment, but who did not provide any form of intervention. Participants were paid $25 for completing the 3-month survey by phone ($15 for online).

Measures

“Treatment” variable

The treatment variable of interest in this study was engagement in the online community. Metrics of both passive and active community engagement were extracted from the QuitNet database. Passive engagement metrics included: number of people a user received Q-Mail from, number of Forum messages read, number of Club messages read, and number of Testimonials viewed. Active engagement metrics included: number of people to whom a user sent QMail, number of Forum posts, number of uses of Chat, and number of Testimonials written. Community use patterns were defined as follows: None refers to participants who never visited the community; Passive refers to participants with values of 1 or higher on any of the passive community use metrics and values of 0 on all active community use metrics; Both refers to study participants who had values of 1 or higher on any passive and any active community use metrics.

Predictors of “Treatment”

To account for the self-selected use of the online community, we constructed a propensity score model (McCaffrey, Ridgeway, & Morral, 2004) that included the following measures based on their demonstrated association with smoking behavior and/or engagement with smoking cessation treatment.

Demographic variables included gender, age, race, ethnicity, education, employment status, marital status, and household income were assessed at baseline using standard items. Participants also indicated how frequently they communicated online via blogs, bulletin boards, chat rooms, or instant messaging.

Baseline smoking variables included daily smoking rate, other tobacco use, age of first smoking experience, age at onset of daily smoking, the number of quit attempts in the past year, desire to quit (1=not at all, 10=very much), confidence in quitting (1=not at all, 10=very), motivation to quit (Prochaska, DiClemente, & Norcross, 1992), and number of smokers in the house. The Fagerström Test for Nicotine Dependence (Heatherton, Kozlowski, Frecker, & Fagerstrom, 1991) and the short forms of Smoking Situations Confidence and the Smoking Temptations Inventories (Velicer, Diclemente, Rossi, & Prochaska, 1990) were administered. Participants were asked about past-year use (at baseline) and past 3-month use (at follow-up) of nicotine replacement therapy (patch, gum, lozenge, spray, inhaler), prescription cessation medications (Zyban/bupropion), behavioral treatments (pamphlet/book, individual counseling, group counseling, telephone counseling), alternative quit methods (e.g., acupuncture, hypnosis), and “other” methods (recoded into pharmacological, behavioral, or alternative treatments).

Psychosocial variables included the Cohen Stress Scale (Cohen, Kamarck, & Mermelstein, 1983), the 10-item Center for Epidemiologic Studies Depression Scale (Andresen, Malmgren, Carter, & Patrick, 1994), a modified version of the Partner Interaction Questionnaire (Cohen & Lichtenstein, 1990; Graham, Papandonatos, et al., 2006), the Weight Concern Scale (Borrelli & Mermelstein, 1998), number of other people living in the household, and the Social Network Index, a measure of social integration that assesses participation in various social relationships (Cohen, Doyle, Skoner, Rabin, & Gwaltney, 1997).

Medical history items included a history of tobacco-related medical conditions (e.g., high blood pressure, cancer, heart disease, bronchitis). Body mass index was calculated from height and weight. Participants were asked if they drank alcohol (yes/no) and an item from the Drug Abuse Screening Test (Skinner, 1982) assessed drug use (yes/no).

Outcome

The primary outcome was self-reported 30-day point prevalence abstinence, defined as the proportion of subjects not smoking for the previous 30 days at 3 months post-randomization. Self-reported smoking status is an accepted measure in Internet cessation trials (Civljak et al., 2013). Participants not reached at the 3-months were coded as smokers.

Predictors of outcome

Website use and telephone counseling during the first 3 months of the study were examined as predictors of outcome, over and above community engagement levels. We controlled for use of the non-Community aspects of the website in order to parse out the unique contribution of Community engagement on abstinence. Website utilization was broken into use of static content vs. interactive features. Static content included: 1) a Quitting Guide with evidence-based cessation information; 2) a Medication Guide about US FDA-approved medication and other products; 3) a searchable database of Frequently Asked Questions (FAQs) submitted by community members over the life course of the site; and 4) a list of cessation resources in the US, searchable by zip code. Interactive features included: 1) a Quit Date Wizard to assist smokers in choosing a quit date; 2) a Medication Wizard that provided individually tailored medication recommendations; 3) a Medication Plan that enabled users to set/update their chosen quitting medication; 4) expert tools that included an assessment of nicotine dependence (Fagerstrom & Schneider, 1989), a ‘‘Why Do You Smoke’’ questionnaire, and an assessment of readiness to change (Prochaska et al., 1992); 5) a dynamic quitting calendar with cessation strategies tailored by quit date; 6) a personalized Journal for self-monitoring progress; 7) expert counseling available through private messages or public forums; and 8) testimonials for users to share their personal experiences with quitting. The number of counseling calls completed with staff from National Jewish was used as a measure of phone utilization. All counts were transformed to the square root scale.

Statistical Analyses

We began by examining differences in baseline characteristics by level of online community use among study participants (Table 2). One-way ANOVA was used to test for differences in continuous variables exhibiting low-to-moderate skewness. Rank-based Kruskal-Wallis ANOVA was used for highly skewed variables. Chi Square tests were used to test for differences in categorical variables, one factor level at a time. Fisher exact tests were employed when individual cell counts dropped below five participants.

Table 2.

Baseline covariates by level of community engagement (None, Passive, Both) in the analytic sample (N=399).

	Overall (N=399)	None (N=145)	Passive (N=82)	Both (N=172)	P-value
Demographic Variables
Women; N (%)	211 (52.9)	63 (43.4)	42 (51.2)	106 (61.6)	.005
Age (years); mean (SD)	37.3 (11.2)	37.2 (12.1)	38.3 (11.5)	36.9 (10.4)	.646
Race, White; N (%)	350 (87.7)	124 (85.5)	71 (86.6)	155 (90.1)	.434
Ethnicity, Hispanic; N (%)	10 (2.51)	3 (2.07)	1 (1.22)	6 (3.49)	.598a
Education, some college or more; N (%)	318 (79.7)	115 (79.3)	65 (79.3)	138 (80.2)	.974
Employment, Full time; N (%)	271 (67.9)	99 (68.3)	60 (73.2)	112 (65.1)	.435
Household income, less than $50,000; N (%)	219 (55)	79 (54.5)	44 (53.7)	96 (56.1)	.921
Marital status, married/ cohabitating; N (%)	247 (61.9)	94 (64.8)	47 (57.3)	106 (61.6)	.532
No. of people in household; mean (SD)	2.88 (1.51)	2.92 (1.44)	2.65 (1.25)	2.96 (1.66)	.485a
Used Internet for more than 5 years; N (%)	333 (83.5)	118 (81.4)	66 (80.5)	149 (86.6)	.328
Online communication, at least daily; N (%)	164 (41.1)	48 (33.1)	30 (36.6)	86 (50.0)	.006
Smoking Variables
Cigarettes per day; mean (SD)	20.9 (10.2)	20.8 (11.3)	20.9 (9.6)	21.0 (9.4)	.589a
FTND; mean (SD)	5.28 (2.35)	5.08 (2.39)	5.41 (2.41)	5.38 (2.29)	.453
First puff age (years); mean (SD)	14.1 (3.55)	13.9 (3.40)	14.6 (3.79)	14.0 (3.56)	.352
Daily smoking age (years); mean (SD)	17.1 (3.91)	17.0 (4.55)	17.3 (3.51)	17.0 (3.50)	.872
No. of quit attempts past year; mean (SD)	2.87 (6.27)	3.33 (4.81)	2.16 (2.61)	2.83 (8.27)	.370a
Desire to quit (1–10); mean (SD)	9.15 (1.19)	9.12 (1.31)	9.02 (1.07)	9.23 (1.16)	.432
Confidence in quitting (1–10); mean (SD)	6.24 (2.27)	5.89 (2.34)	6.32 (2.56)	6.49 (2.02)	.060
Smoking Temptations Inventory; mean (SD)	3.96 (0.51)	3.93 (0.52)	3.95 (0.57)	3.98 (0.46)	.576
Smoking Situations Confidence Inventory; mean (SD)	2.8 (0.60)	2.7 (0.60)	2.72 (0.55)	2.92 (0.60)	.001
Other types of tobacco use; N (%)	33 (8.29)	14 (9.66)	11 (13.4)	8 (4.68)	.047
Smokers in house; N (%)	70 (17.5)	29 (20.0)	13 (15.9)	28 (16.3)	.620
Motivation to quit (next 30 days); N (%)	349 (87.5)	120 (82.8)	67 (81.7)	162 (94.2)	.002
No. of quit methods past year; mean (SD)	2.4 (1.96)	2.3 (2.05)	2.6 (1.80)	2.5 (1.96)	.305a
Psychosocial Variables
CES-D; mean (SD)	9.69 (5.63)	9.90 (5.87)	9.27 (5.46)	9.73 (5.52)	.719
Cohen Stress Scale; mean (SD)	6.27 (3.25)	6.47 (3.57)	6.10 (3.31)	6.18 (2.94)	.638
Weight Concern Scale; mean (SD)	4.73 (2.43)	4.54 (2.48)	4.83 (2.46)	4.83 (2.38)	.530
PIQ – Positive subscale; mean (SD)	9.7 (2.3)	9.58 (2.36)	9.67 (2.29)	9.82 (2.26)	.638
PIQ – Negative subscale; mean (SD)	5.67 (4.16)	5.49 (4.28)	6.16 (3.69)	5.58 (4.28)	.481
Medical History
Body Mass Index; mean (SD)	26.9 (5.9)	26.8 (5.5)	27.2 (5.7)	26.9 (6.4)	.911
Alcohol use (yes); N (%)	283 (70.9)	98 (67.6)	63 (76.8)	122 (70.9)	.338
Drug use (yes); N (%)	59 (14.8)	30 (20.7)	12 (14.6)	17 (9.88)	.026
History of depression; N (%)	120 (30.1)	34 (23.4)	25 (30.5)	61 (35.5)	.067
History of respiratory disease; N (%)	63 (15.8)	22 (15.2)	10 (12.2)	31 (18.0)	.476
History of high blood pressure; N (%)	60 (15.0)	25 (17.2)	14 (17.1)	21 (12.2)	.388

^aP-values calculated using Fisher's Exact test for categorical variables with low cell counts and Kruskal-Wallis test for skewed continuous variables.

FTND: Fagerström Test for Nicotine Dependence; CES-D: Center for Epidemiologic Studies, Depression Scale; PIQ: Partner Interaction Questionnaire.

Propensity score modeling

Differences in crude abstinence rates by community use group do not have a causal interpretation, as they fail to account for self-selection into a particular online community engagement level (Stuart, Marcus, Horvitz-Lennon, Gibbons, & Normand, 2009). However, they provide a useful baseline against which to judge the magnitude and direction of selection bias in our study, and have been included in our summary output. Correction for selection bias can be accomplished in two steps via propensity score modeling. Step one focuses on estimating the probability of selecting a particular community engagement level conditional on baseline covariates, also known as the propensity score. In step two, Inverse Probability of Treatment Weighting corrects raw abstinence rates for participants’ differential propensity to participate in the community, essentially recreating a randomized experiment (Lunceford & Davidian, 2004). The proof relies on the fact that the true propensity score is also a balancing score, such that the distribution of these covariates is the same across participants having the same propensity score, irrespective of their actual community engagement level (Rosenbaum & Rubin, 1983). Of note, estimated scores may differ from true ones and, thus, one may fail in practice to achieve the desired covariate balance. In this paper, we have chosen to estimate propensity scores non-parametrically using Generalized Boosted Models (GBM) as implemented in the GBM package (Ridgeway, 2014), thus gaining robustness to possible propensity model misspecification.

More than one weighting scheme is possible, each leading to different causal estimates (McCaffrey et al., 2013). One reweighting approach focuses on estimating Average Treatment Effects (ATEs) for all subjects, regardless of their actual treatment condition (Imbens, 2004). For example, the ATE of passive versus no participation is the difference in abstinence rates of the entire sample had it participated in the community passively versus not at all. Estimating this treatment effect requires weighting the members of the Passive and None groups, so that their covariate distributions resemble that of the whole sample and, hence, each group has similar propensity to participate in the community at any given engagement level. The abstinence rates in these two weighted samples then serve as estimates for the difference in abstinence rates that would have been observed in the entire sample had it engaged passively as opposed to not at all. Although multinomial modeling capabilities for analyzing multivalued treatments have recently been added to the GBM package, modeling each treatment level separately leads to improved covariate balance (McCaffrey et al., 2013). Thus, we fit separate GBMs for participating in the online community at each observed engagement level.

A second reweighting approach is based on estimating Average Treatment effects on the Treated (ATTs) separately by treatment group (Imbens, 2004). Whereas ATEs are bidirectional, ATTs are unidirectional estimates of the benefit conferred by one community use level rather than another. For example, the ATT of passive versus no participation is the difference between (i) the actual abstinence rate of the Passive group, and (ii) the abstinence rate of this same group, had none of its members participated in the online community at all. Estimating this effect requires weighting the None group, so that its covariate distribution resembles that of the Passive group; the abstinence rate in the weighted None group then serves as an estimate for the abstinence rate of the Passive group under the None treatment condition. Although this rebalancing could be accomplished using ratios of previously-obtained ATE weights, when multivalued treatments are involved, it is recommended that separate GBMs be fit de novo to each pair of groups involved in the comparison of interest, ignoring remaining groups (McCaffrey et al., 2013). Unit weights are assigned to individuals in the target group, while those in the comparison group have their abstinence outcomes weighted by the odds that they would have exhibited the target behavior. Individuals in the comparison group with covariate values common in their group alone are down-weighted, whereas those with covariate values common in the targeted group are up-weighted.

ATE and ATT estimates agree when treatment effects are constant across study participants. Otherwise, ATT estimates allow one to make group-specific inferences, and to gauge the effect of moving study participants from one study group to another (e.g., by encouraging passive community use among non-users). However, findings may not apply to all subjects involved in a pairwise comparison, as near-zero propensity scores imply that certain subjects might be unlikely to engage in a different type of community use. For example, ATE results would apply to None subjects only if their covariate profile suggested that they could be encouraged to engage in the community, as evidenced by non-zero engagement probabilities in the corresponding propensity score model. Further, None subjects willing to engage only in a passive manner would contribute to the None vs. Passive ATT comparison, but not to the None vs. Both ATT comparison. Whereas an estimate of the effective sample size for a pairwise comparison is always available in matched analyses, since mismatched individuals are simply dropped from the comparison, no such direct estimates are available under IPTW. However, near-zero propensity scores correspond to near-infinite weights, inflating their variance. Hence, the coefficient of variation of the weights can be used to obtain an estimate of the information loss due to a mismatch in the underlying propensity score distributions.

For both ATE and ATT analyses, individual GBM fits were weighted combinations of up to 10,000 trees of depth two, capturing both main effects and 2-way interactions in model covariates (McCaffrey et al., 2004). A shrinkage parameter of 0.01 was used for smoothing, and minimization of the mean of the Kolmogorov-Smirnov test statistic was used for selecting the number of trees providing the best covariate balance across groups.

Weighting and balance checking

Propensity scores were visually inspected to assess covariate overlap and the need for weight trimming (Lee, Lessler, & Stuart, 2011), aided by the balance assessment diagnostics of the TWANG package (Ridgeway, McCaffrey, Morral, Ann, & Burgette, 2014). Improvements in covariate balance were assessed based on change in absolute standardized bias measures, quantities similar to effect size measures (Harder, Stuart, & Anthony, 2010). For continuous covariates, these were between-group mean differences before and after weighting, divided by the unweighted standard deviation of the full sample (ATE) or the target engagement group (ATT). For categorical variables, separate standardized bias measures were calculated for each covariate level, based on between-group differences in proportions. In addition, differences in spread were assessed by looking at the between-group ratio of variances before and after weighting. Significance testing was avoided in assessing covariate balance (Ho, Imai, King, & Stuart, 2007); rather, absolute bias measures smaller than .25 standard units and variance ratios in the interval [4/5, 5/4] were deemed indicative of successful balancing.

Both ATE and ATT weights include in their denominator estimates of the probability that participants would have selected the level of community use characteristic of their group. Individuals with very low propensity to use the community at the observed level can unduly influence the analyses. Hence, both sets of weights were visually inspected, in case they required trimming (Lee et al., 2011).

Outcome modeling

GBM-based propensity scores were inverted, fed as weights into the SURVEY package (Lumley, 2014) and used to estimate differences in ATE- and ATT-weighted abstinence rates by community engagement group. Baseline confounders that remained imbalanced were controlled for via regression adjustment. We also controlled for individual differences in website and telephone utilization, post-treatment covariates that could not be included in the propensity score. After a preliminary screen for lack of sample variability or multicollinearity issues, a backward elimination procedure was used to identify significant predictors of abstinence.

Results

As shown in Table 1, 24.1% (N=163) of N=675 randomized participants never logged in to the website and, thus, never accessed the community, while 26.8% (N=181) did not complete any telephone counseling calls. Given the small size of the latter group, we decided not to model its propensity to engage online, focusing our efforts instead on the 80.8% of the original sample (N=399) that logged into the website at least once and completed at least one counseling call.

Table 1.

Community use patterns by Internet and telephone counseling treatment utilization.

	Randomized Sample	Phone calls completed = 0	Phone calls completed≥1
Internet + Phone arm	N=675	N=181	N=494
# (%) participants with website logins = 0	163 (24.1%)	68 (37.6%)	95 (19.2%)
# (%) participants with website logins≥1	512 (75.9%)	113 (62.4%)	399 (80.8%)
# (%) participants with Community Use = None	206 (40.2%)	61 (54.0%)	145 (36.3%)
# (%) participants with Community Use = Passive	104 (20.3%)	22 (19.5%)	82 (20.6%)
# (%) participants with Community Use = Passive + Active	202 (39.5%)	30 (26.5%)	172 (43.1%)

NB: Community use summaries are calculated solely among participants logging into the website.

Characteristics of the analytic sample (N=399) are shown in Table 2 and include: average age of 37.3 years (SD=11.2), 52.9% female, 87.7% White, 61.9% married/cohabitating, 79.7% with at least some college education, 87.5% motivated to quit in next 30 days, daily smoking rate of 20.9 cigs/day (SD=10.2), 2.9 quit attempts in the past year (SD=6.3), and 48.6% communicated online via blogs, chat rooms, online bulletin boards daily or almost daily.

Predictors of Telephone Counseling Use (among website users)

For purely descriptive purposes, we provide herein a list of variables showing significant imbalances between telephone counseling users and non-users (absolute standardized bias ≥.25); full details are given in Supplemental Table 1. Among participants that logged into the website at least once in the first three months of the study (N=512), those that completed one or more counseling calls (N=399) were older (37.3±11.2 vs. 33.9±9.4 years, p<.01) and more likely to report annual household income of $50,000 or higher (45% vs. 37.2%, p<.05) compared to those who completed no counseling calls. They were also heavier smokers (20.9±10.2 vs. 18.1±8.1 cigs/day, p<.05), had higher scores on the Smoking Temptations Scale (3.96±0.51 vs. 3.81±0.52, p<.05), and reported using more quit methods in the past year (2.44±1.96 vs. 1.97±1.61, p<.05).

Patterns of Community Engagement

In the analytic sample, 145 (36.3%) never visited the community, 82 (20.6%) engaged passively, and 172 (43.1%) engaged both passively and actively. Self-selection into one of three levels of online community use defines the 3 groups of interest in this study (i.e., None, Passive, Both). Among the N=399, 12 participants that used the community actively, with no passive use; these were included in the Both group given their small number, but potentially informative data.

Propensity Score Modeling

Table 2 shows between-group differences in baseline participant characteristics used as potential predictors of online community use at 3 months in our propensity model. Statistical significance at p<.05 was attained for gender, smoking abstinence self-efficacy, stage of motivational readiness to quit smoking, having other smokers in the household, other tobacco and drug use, and prior blog use. Participants in the Both group were mostly female, more likely to be thinking of quitting within 30 days, had the lowest rates of other tobacco and drug use, highest self-efficacy scores, and highest rates of online communications. Participants in the Passive group were the least likely to have other smokers in the household. Although remaining baseline variables did not show significant between-group differences, they were included in the GBM model following recommendations to be over-inclusive at this stage (Austin, 2011).

Weighting and Balance Checking

Estimated probabilities of observing the actual level of community use in each group were in the .20–.98 range, safely bounded away from zero. Therefore, there was no need to trim ATE weights limited to the range 1.02–4.99 across groups. As a result, information losses for ATE analyses due to weight variability were below 13% for all three groups defined by community use. Reflective of more limited overlap in the propensity score distributions, weights were more variable (0.02–3.94) for ATT analyses, resulting in large reductions in the effective sample size that ranged from 23%–48%. Again, no weight trimming was deemed necessary.

Balance diagnostics for the variables in Table 2 showed that all absolute standardized bias measures fell below the target 0.25 standard units after weighting (Supplemental Figures 2–4), although variance ratios in the [4/5, 5/4] range were harder to achieve. After weighting, imbalances no longer remained among quit methods at follow-up.

Outcome Modeling: Unadjusted Analyses

Table 3 presents between-group differences in abstinence rates that have not been adjusted for website and phone utilization as yet. They are depicted graphically in Figure 1.

Table 3.

Abstinence rates by community use level and pairwise ATEs and ATTs before and after propensity score weighting.

	None	Passive	Both	Difference*
	Rate (95% CI)	Rate (95% CI)	Rate (95% CI)	ΔRate (95% CI)	p-value
Unweighted Analyses	.124 (.070, .178)	.244 (.151, .337)	.355 (.283, .426)
Passive vs. None	--	--	--	.120 (.012, .227)	.028
Both vs. None	--	--	--	.231 (.141, .320)	<.001
Both vs. Passive	--	--	--	.111 (-.007, .228)	.064
ATE Weighted Analyses	.122 (.067, .177)	.252 (.151, .352)	.355 (.281, .429)
Passive vs. None	--	--	--	.130 (.015, .244)	.026
Both vs. None	--	--	--	.233 (.141, .325)	<.001
Both vs. Passive	--	--	--	.103 (-.022, .228)	.105
ATT weighted to match "None"	.124 (.070, .178)	.279 (.159, .399)	.350 (.266, .433)
Passive vs. None	--	--	--	.155 (.023, .287)	.021
Both vs. None	--	--	--	.226 (.126, .325)	<.001
ATT weighted to match "Passive"	.121 (.058, .184)	.244 (.151, .337)	.327 (.231, .423)
None vs. Passive	--	--	--	−.123 (−.235, −.010)	.032
Both vs. Passive	--	--	--	.083 (.050, .217)	.221
ATT weighted to match "Both"	.143 (.064, .223)	.230 (.125, .335)	.355 (.283, .426)
None vs. Both	--	--	--	−.211 (−.318, −.105)	<.001
Passive vs. Both	--	--	--	−.124 (−.252, .003)	.055

^*Differences are calculated as 1^st group – 2^nd group, where pairs are expressed as "1^st group vs. 2^nd group".

Notes: ATE weighted estimates represent estimated abstinence rates if the whole population received the corresponding treatments. ATT weighted estimates represent estimated abstinence rates if the matched population received corresponding treatments.

Figure 1.

3-month abstinence rates by level of community engagement: results from unweighted, ATE-weighted, and ATT-weighted analyses.

Unweighted analyses

Overall 30-day point prevalence abstinence at 3 months in the analytic sample was 24.8% (95% CI=20.6–29.1). As seen in Table 3, abstinence rates rose monotonically with increased levels of community use, from 12.4% among None to 24.4% among Passive to 35.5% among Both. Three-month abstinence was less likely among None compared to Passive (p=.028) and Both (p<.001); no statistically significant differences in abstinence emerged between Passive and Both (p=.064).

ATE-weighted analyses

ATE-weighted abstinence rates differ from observed rates due to a reweighting scheme that brings group-specific covariate distributions closer to those of the overall sample, raising the likelihood of community use in the None group and diminishing it in the Both group. Table 3 shows that under this scheme abstinence rates fell slightly to 12.2% (95% CI=6.7–617.7) in the None group; increased to 25.2% (95% CI=15.1–35.2) in the Passive group; and remained unchanged at 35.5% (95% CI=28.1–42.9) in the Both group. These results translate into statistically significant increases of 13 percentage points (95% CI=1.5–24.4, p=.03) due to passive community use and 23.3 percentage points (95% CI=14.1–32.5, p<.001) due to combined passive+active use compared to no use. No statistically significant differences in abstinence emerged between Passive and Both groups (p=.11).

ATE findings are bidirectional and apply to the entire sample. For example, the 10.3 percentage points decrease in abstinence had the entire sample participated in the community passively instead of both passively and actively is equal in magnitude and opposite in sign to the increase in abstinence had the entire sample participated both passively and actively instead of passively alone. As a result, only 3 out of 6 possible pairwise comparisons are shown in Table 3.

ATT-weighted analyses

ATT-weighted abstinence rates are also shown in Table 3, and differ from observed rates due to a reweighting scheme that brings the covariate distributions of community use groups closer to that of a specific group, rather than the sample as a whole. They supplement ATE findings and permit unidirectional predictions for all 6 pairwise comparisons.

Had participants in the None group used the community passively, their abstinence rates would have increased from 12.4% to 27.9% (95% CI=15.9–39.9), representing a statistically significant improvement of 15.5 percentage points (95% CI=2.3–28.7, p= .02); had they participated both passively and actively, abstinence rates would have risen further to 35.0% (95% CI=26.6–43.3), representing a statistically significant improvement of 22.6 percentage points (95% CI=12.6–32.5, p<.001) over the 12.4% rate actually observed.

Had participants in the Passive group not used the community at all, their abstinence rates would have dropped from 24.4% to 12.1% (95% CI=5.8–18.4), a statistically significant 12.3 percentage point decline (95% CI=1.0–23.5, p=.03); in contrast, had they also participated actively, their abstinence rates would have increased to 32.7% instead (95% CI=23.1–42.3), representing a non-significant 8.3 percentage point increase in abstinence rates (p=.22).

Had participants in the Both group only used the community passively, their abstinence rates would have dropped from 35.5% to 23.0% (95% CI=12.5–33.5), a 12.4 percentage point decrease that did not reach statistical significance (p=.06). In contrast, had they not used the community at all, abstinence rates would have fallen to 14.3% (95% CI=6.4–22.3), a significant drop of 21.1 percentage points (10.5–31.8, p<.001).

Outcome Modeling: Adjusted Analyses

Table 4 presents estimates of pairwise differences between the 3 groups expressed in the odds ratio (OR) scale. Four sets of ORs are presented: a) unadjusted for website and phone use, b) adjusted for phone use, c) adjusted for website use, and d) adjusted for both website and phone use. Although one would have expected p-values for unweighted analyses (Model 1) in Table 4 to agree with those for the differences in proportions in Table 3, small differences were observed due to the fact that Wald tests are not transformation-invariant. However, the overall pattern of unadjusted findings was identical between the two tables.

Table 4.

Association between community use and smoking abstinence after regression adjustment for both website and telephone utilization.

Models*	Model 1		Model 2		Model 3		Model 4
Odds ratios**	OR (95% CI)	p-value	OR (95% CI)	p-value	OR (95% CI)	p-value	OR (95% CI)	p-value
Unweighted Analyses
Passive vs. None	2.28 (1.12, 4.61)	.022	2.00 (0.98, 4.11)	.058	1.89 (0.92, 3.89)	.084	1.70 (0.81, 3.53)	.158
Both vs. None	3.88 (2.16, 6.95)	<.001	3.12 (1.71, 5.68)	<.001	2.68 (1.42, 5.06)	.003	2.24 (1.17, 4.30)	.016
Both vs. Passive	1.70 (0.94, 3.08)	.078	1.56 (0.85, 2.85)	.153	1.42 (0.77, 2.61)	.267	1.32 (0.71, 2.46)	.383
ATE Weighted Analyses
Passive vs. None	2.43 (1.16, 5.08)	.019	2.04 (0.96, 4.34)	.065	2.08 (0.97, 4.43)	.059	1.77 (0.82, 3.83)	.145
Both vs. None	3.97 (2.16, 7.27)	<.001	3.16 (1.68, 5.96)	<.001	2.88 (1.51, 5.50)	.001	2.38 (1.22, 4.65)	.011
Both vs. Passive	1.64 (0.88, 3.05)	.122	1.55 (0.82, 2.95)	.181	1.39 (0.74, 2.63)	.311	1.34 (0.70, 2.59)	.378
ATT weighted to match "None"
Passive vs. None	2.73 (1.26, 5.93)	.011	2.29 (1.03, 5.09)	.043	2.16 (0.94, 4.99)	.070	1.78 (0.75, 4.24)	.190
Both vs. None	3.80 (2.05, 7.03)	<.001	3.12 (1.62, 6.03)	<.001	3.01 (1.53, 5.90)	.001	2.61 (1.28, 5.31)	.008
ATT weighted to match "Passive"
Passive vs. None	2.34 (1.07, 5.11)	.032	2.05 (0.92, 4.56)	.081	1.89 (0.84, 4.30)	.127	1.60 (0.66, 3.74)	.277
Both vs. Passive	1.51 (0.77, 2.94)	.227	1.37 (0.70, 2.69)	.364	1.21 (0.60, 2.44)	.593	1.11 (0.54, 2.27)	.777
ATT weighted to match "Both"
Both vs. None	3.28 (1.60, 6.71)	.001	2.59 (1.23, 5.44)	.012	2.46 (1.15, 5.26)	.021	2.02 (0.92, 4.43)	.079
Both vs. Passive	1.84 (0.94, 3.60)	.076	1.71 (0.86, 3.39)	.124	1.54 (0.77, 3.07)	.222	1.47 (0.73, 2.96)	.280

^*Model 1: Naïve model (only community use in the model); Model 2: adjusted by sqrt(#of calls); Model 3: adjusted by sqrt(# of FAQs read); Model 4: adjusted by both sqrt(# of calls) and sqrt(# of FAQs read).

^**ORs are calculated as 1st group/2nd group, where pairs are expressed as "1st group vs. 2nd group".

This concordance allowed us to focus on the question of whether the between-group differences in abstinence observed between different levels of community engagement were deflated after adjustment for website and phone use. Specifically, we sought to determine whether the observed differences in quit rates by community engagement may have been an artifact of other treatment utilization, namely use of other elements of the website and/or phone call completion. Preliminary analyses of all website utilization metrics showed that FAQs were the aspect of website use most strongly associated with abstinence. Therefore, in adjusted outcome models that controlled for website utilization (Models 2 and 4 in Table 4), FAQs were entered as the sole website use metric. Phone use was modeled as number of calls completed. Square root transformations captured nonlinear utilization effects on smoking outcome.

Phone utilization adjustment

Adjusting for phone utilization alone (Model 2) deflated all Passive vs. None comparisons noted in Model 1. In unweighted, ATE-weighted, or ATT-weighted estimates with Passive as the target group, the odds of abstinence in the Passive group were estimated at about 2X that of the None group at the same level of phone support. However, ATT-weighted estimates with None as the target group revealed a slightly stronger 2.3X odds ratio that remained statistically significant (p=.04). Additionally, the Both vs. None comparison remained very highly significant (p<.001), whether one used unweighted, ATE-weighted, or ATT-weighted estimates with None as the target group, with the odds of abstinence in the Both group estimated at about 3X that of the None group at the same level of phone support. When ATT-weighted estimates were employed with Both as the target group the odds ratio was deflated to 2.6X, but remained statistically significant (p=.01). The Both vs. Passive comparison was not statistically significant under any weighting scheme, with odds ratios in the range of 1.4X-1.7X consistent with slightly better outcomes in the Both group.

Website utilization adjustment

Adjusting for website utilization (Model 3) led to even stronger deflation of the Model 1 Passive vs. None comparisons than adjustment for phone utilization, with none of the relevant odds ratios attaining statistical significance. In contrast, the Both vs. None comparison remained highly significant (p<.01) whether one used unweighted, ATE-weighted, or ATT-weighted estimates with None as the target group. When ATT-weighted estimates were employed with Both as the target group, the odds ratio was deflated to 2.6X, but remained statistically significant (p=.01). The Both vs. Passive comparisons were not statistically significant under any weighting scheme, with odds ratios in the range of 1.2X-1.5X consistent with slightly better outcomes in the Both group.

Phone and website utilization adjustment

Joint adjustment for phone and website utilization (Model 4) deflated all pairwise comparisons other than that of Both vs. None under unweighted, ATE-weighted, and ATT-weighted analyses with None as the target group. Odds ratios ranged from 2.2X-2.6X, consistent with better outcomes in the Both group.

FAQ use and counseling calls were both positively associated with abstinence. Irrespective of weighting scheme, the magnitude of their effects varied little across models 2–4, indicating that the benefit from website utilization was quite separate from that conferred by telephone counseling. To the extent that FAQ use and community engagement were strongly correlated (Supplemental Table 2), weighting for differential propensity to engage in the community should also have corrected for selection bias associated with FAQ use. However, propensity to seek out telephone support appears to have had distinct determinants from those driving community engagement. This would suggest that telephone counseling effects in our models have a purely associational interpretation, and should be interpreted with caution, whereas effects of FAQ use may well have a causal interpretation (Supplemental Table 3).

Discussion

This study replicates and extends earlier work demonstrating that engagement in an online community for smoking cessation is associated with abstinence using analytic methods that control for the possibility of self-selection bias (Graham et al., 2015). In this study of current smokers randomized to a combined Internet+Phone intervention, ATE-weighted analyses with missing outcomes coded as smoking revealed short-term abstinence rates of 12% among non-community users, 25% among passive users, and 36% among passive+active users. Increases in abstinence associated with passive and passive+active community use over no use attained statistical significance, but the additional benefit observed for combined passive+active over passive use alone did not. Findings suggest that passive use alone (“lurking”) yields clear benefits for users of a combined Internet+Phone smoking cessation intervention. The magnitude of these quit rates among participants who used the community (25%–36%) is noteworthy, especially given that the trial did not involve pharmacotherapy.

The present findings mirror results from our earlier study that examined community use among participants that only had access to an Internet intervention (Graham et al., 2015). In that study, ATE-weighted analyses with missing outcomes coded as smoking found that passive community use significantly increased abstinence rates from 4% to 15% compared to no community use at all, and that combined passive+active use yielded only a small incremental benefit over passive use alone, further increasing abstinence rates to 20%. In both analyses, we observed that quit rates increased with greater community engagement.

Additional analyses that adjusted pairwise differences among community use groups for website and phone utilization revealed a potentially important consideration for Internet smoking cessation interventions: controlling for website utilization attenuated the effect of online community use on abstinence to a greater degree than controlling for telephone counseling utilization. In interpreting this finding, it is important to remember that study participants were first and foremost Internet users, recruited through search engines. Phone counseling for many participants may have been an “alternate” treatment modality that they may not have sought out of their own volition. Previous research noted that a “web-based intervention is most attractive if the offer to use is made by web, whereas a phone-based intervention is more likely to be used if the offer is made over the phone” (p. 288) (Balmford, Borland, Benda, & Howard, 2013). The fact that 27% of participants never completed any counseling calls is consistent with this notion. The role of online community use in a combined Internet/phone program where users enroll by phone remains an open question. Quitline callers may differ from Internet users on baseline characteristics and receptivity to Internet interventions, among others.

Website utilization was represented in these models by the FAQs variable, which emerged as the strongest predictor of abstinence among both static website content elements and dynamic website features in preliminary analyses. FAQs were questions originally posed by users that were answered by an Expert. What this suggests is that the information obtained from other members of the site may have served as a valuable resource to website users who chose not to participate in the community. These findings point to the importance of making user-generated content broadly available to all program users, and for telephone counselors to encourage use of this feature, especially for participants who are reluctant to visit the community.

Several limitations should be noted. First, our analyses may actually underestimate the impact of online community use on abstinence based on our decision to control for non-Community website use. Use of the static and interactive elements of the website were clearly associated with Community use. We wanted to identify the unique contribution of online community engagement, but in doing so may have attenuated the effect of our exposure variable of interest. Second, our analyses do not address potential self-selection bias associated with follow-up attrition, and are conditional on a minimal level of both website and phone use. Third, analyses focus on short-term (3-month) abstinence. Future research should address longer-term effects of participation in an online community. Fourth, propensity score modeling addresses within-study imbalances on variables like demographic characteristics, but does not address imbalances between a study sample and the general population. While our original study sample mirrored the general population of smokers on some characteristics (Graham et al., 2011) and was more diverse than smokers who were recruited but not enrolled (Graham, Bock, et al., 2006), generalizability is limited to smokers seeking Internet cessation treatment. Further, given that our data were drawn from a randomized trial of treatment seeking and highly motivated smokers who had access to an already established and vibrant online community, our findings may not be generalizable to the broader pool of users of publicly available cessation websites (Wanner, Martin-Diener, Bauer, Braun-Fahrlander, & Martin, 2010) or to those with a less established community. Fifth, propensity score adjustment cannot achieve balance over unmeasured confounders that are uncorrelated with variables included in the model specification. While we have attempted to be as comprehensive as possible in considering potential predictors of community engagement, our analyses draw on data available from the parent trial; there may well be unmeasured variables that we could not account for in our propensity model. Finally, reverse temporal ordering may be possible: abstinence could have preceded community engagement (e.g., individuals turning to the community to celebrate abstinence). This is an empirical question to be examined in future research.

In conclusion, online community engagement appears to have had beneficial effects on abstinence, even in the context of proactive telephone counseling. More than one third of participants that received telephone counseling and participated in the community both passively and actively were classified as abstinent in unweighted analyses. Propensity score weighting showed that the effects of combined passive+active community use versus no use were not diminished in ATE-models controlling for website or phone usage, suggesting that this treatment effect is real and not simply an artifact of self-selection bias. Taken together, these analyses suggest that combined Internet+phone interventions that include an online community should encourage users to browse the community for smoking cessation advice and/or participate actively in the manner best suited to their individual needs and preferences for information and support. Moreover, community content should be made broadly accessible through features such as FAQs, so that all users have the opportunity to benefit from the “wisdom of the crowds.”