Baseline Characteristics and Generalizability of Participants in an Internet Smoking Cessation Randomized Trial

Abstract

Background

The potential for sampling bias in Internet smoking cessation studies is widely recognized. However, few studies have explicitly addressed the issue of sample representativeness in the context of an Internet smoking cessation treatment trial.

Purpose

To examine the generalizability of participants enrolled in a randomized controlled trial of an Internet smoking cessation intervention using weighted data from the National Health Interview Survey (NHIS).

Methods

A total of 5,290 new users on a smoking cessation website enrolled in the trial between March 2012–January 2015. Descriptive statistics summarized baseline characteristics of screened and enrolled participants and multivariate analysis examined predictors of enrollment. Generalizability analyses compared demographic and smoking characteristics of trial participants to current smokers in the 2012–2014 waves of NHIS (n=19,043), and to an NHIS subgroup based on Internet use and cessation behavior (n=3,664). Effect sizes were obtained to evaluate the magnitude of differences across variables.

Results

Predictors of study enrollment were age, gender, race, education, and motivation to quit. Compared to NHIS smokers, trial participants were more likely to be female, college educated, daily smokers, and to have made a quit attempt in the past year (all effect sizes 0.25–0.60). In comparisons with the NHIS subgroup, differences in gender and education were attenuated while differences in daily smoking and smoking rate were amplified.

Conclusions

Few differences emerged between Internet trial participants and nationally representative samples of smokers, and all were in expected directions. This study highlights the importance of assessing generalizability in a focused and specific manner.

Tobacco use continues to be the leading cause of preventable death in the United States (US), causing 480,000 premature deaths among adults and nearly $289 billion in total economic burden each year (1). Smokers make up 17.8% of the population, an estimated 42.1 million adults (2). Smoking is unevenly distributed throughout the population: males, younger adults, multiracial individuals and American Indian/Alaska Natives, those with high school education or less, and those living below the poverty level smoke at higher rates than other subgroups (2). Though smoking rates are comparable between Whites and African Americans, African Americans are more likely to suffer from smoking-related mortality and morbidity and are less likely to quit (3). Decreasing overall smoking prevalence and reaching subgroups at greater risk for smoking and/or smoking-related illness will require broad-reach, population-based cessation approaches.

There is wide recognition that Internet interventions are a highly promising delivery system for tobacco dependence treatment (4–8). Internet use is now at 87% of all adults in the US (9), and has steadily risen through all segments of the US population (10), including subgroups at disproportionate risk for smoking. The Internet is used by 87% of males, 97% of young adults aged 18–29, 81% of African Americans, 76% of those with high school degree or less, and 77% of those with income < $30,000 per year (10). Nearly two thirds of US adults turn to the Internet for information about health concerns (11) and 6–9% of Internet users – an estimated 10 million adults – search for quit smoking assistance online each year (12). Taken together, these data demonstrate the potential for Internet interventions to be an important cessation resource to reach and engage a sizable and diverse population of smokers.

The Internet is well-suited to deliver the core elements of evidence-based cessation treatment recommended by national tobacco dependence treatment guidelines (4), including practical counseling (problem solving/skills training), social support from peers and experts, and information about pharmacotherapy. Internet treatment can be accessed 24/7 and used as long as a smoker desires. Use is not restricted by geographical location and costs are often minimal or nonexistent to users through coverage by employers, health plans, or states, making it an ideal resource for those who do not have easy access to other cessation services. The costs of Internet programs are relatively fixed, resulting in greater efficiency as use increases (6, 13). Several large-scale randomized trials have shown quit rates of 18–20% at 12 months (14, 15) and meta-analyses and systematic reviews support the effectiveness of tailored and interactive interventions (5, 7, 8, 16). Even at modest levels of effectiveness, the broad reach, sustainability, and cost efficiency of Internet interventions may translate to a large population impact (17) in reducing smoking prevalence (3).

There are unique and often “thorny” methodological challenges in evaluating the impact of Internet interventions (18–23). The possibility of sampling bias in Internet-based studies is one such issue that has been discussed in detail (24, 25). Sampling bias may occur based on differences between individuals who have access to and use the Internet and those who do not (26–28). Sampling bias may also result from differences between smokers who seek cessation assistance online and those who seek assistance by other means (8). While sampling bias may exist for Internet-recruited samples, the risk for bias is not believed to be any greater or less than samples recruited for “traditional” clinical trials (29, 30). Indeed, all types of clinical trials may be subject to concerns about generalizability to the extent that they recruit participants who may be more motivated, willing to participate in a research study, or open to new treatment. Clinical trials have also been criticized for underrepresentation of racial and ethnic minorities and participants with lower levels of education or income (31–34). Internet-recruited samples have been shown to be more demographically diverse (35) compared to samples recruited using non-Internet methods. A systematic review of Internet-based randomized controlled trials (36) noted that the Internet greatly increases geographic and demographic reach and that Internet-based trials may actually increase access to populations that are difficult to reach via traditional recruitment methods (e.g., rural adults, individuals not seeking medical attention). It is not yet clear, though, if Internet-based trials can manage to overcome the failure in clinical trials to recruit underrepresented participants. Glasgow (37, 38) and others (39–43) have written about the importance of explicitly evaluating the generalizability of recruited participants in studies of behavior change, including smoking, against a specific target population. To date, few Internet smoking cessation randomized trials have reported such analyses (30, 44).

Research is needed that explicitly examines the similarities and differences between participants in Internet cessation trials and a clearly defined target population to which results of a given trial are intended to generalize. We describe here the baseline characteristics of a sample of smokers who voluntarily registered on a publicly-available web-based smoking cessation program and agreed to participate in a randomized controlled cessation treatment trial via the Internet. To address questions of generalizability, we compare our study sample to the general population of smokers in the weighted National Health Interview Survey (NHIS) and to a more refined subgroup of NHIS current smokers based on Internet use and cessation behavior. The objectives of this study were to: (a) Describe the baseline characteristics of smokers in an Internet smoking cessation randomized trial; and (b) Examine the generalizability of the enrolled sample using data drawn from a nationally representative dataset.

Method

Study Overview

The overarching goal of this randomized trial is to test the individual and combined effects of two potentially complementary strategies to improve adherence to the evidence-based elements of cessation treatment. The two strategies are a social network intervention (SN) and access to an initial course of free nicotine replacement therapy (NRT). The study uses a 2×2 randomized, controlled factorial design to compare the effectiveness of both strategies against a web-based control (WEB). The study randomized participants to (a) WEB, (b) WEB+SN, (c) WEB+NRT, or (d) WEB+SN+NRT with repeated measures at baseline, 3, and 9 months post enrollment. The primary outcome of the trial is 30-day point prevalence abstinence at 9 months. The study protocol, including power analysis and sample size considerations, has been described in detail elsewhere (45). The current analyses examine the characteristics of individuals who completed study eligibility screening and the final enrolled sample. This trial was reviewed and approved by the Western Institutional Review Board.

Eligibility Criteria and Sample Recruitments

Study participants were current smokers who were recruited into the trial immediately following registration on BecomeAnEX.org, a free, publicly-available Internet cessation program. BecomeAnEX has been promoted through a national multi-media campaign since 2008 (46), with more recent promotional activities during the recruitment of this trial focused on paid search advertising targeting current smokers. There were no additional marketing strategies specific to the trial; new registrants who were US residents, current smokers, and age 18 or older were invited to participate in a research study immediately after website registration. Interested participants completed online eligibility screening. Eligibility criteria were current smoking status, age 18 and older, and no contraindications for NRT use. Eligible participants provided informed consent, completed an online baseline survey, and were randomized to treatment using a computer algorithm. The full process of study enrollment was automated by a web-based clinical trials management system.

Consistent with the National Institute of Health standards for clinical research, the study proposed specific goals for planned enrollment of women and minorities based on the distribution of race/ethnicity among current smokers in the United States using data from the National Health Interview Survey (47). According to 2007 NHIS data (the most current at the time of study conception), 8.79% of US smokers were Latino. By race, the majority of smokers in the US were White (83.81%) followed by Black/African American (11.97%), Asian (2.3%), American Indian/Alaska Native (1.68%), and Native Hawaiian/Other Pacific Islander (0.24%). Although smoking prevalence differs for men and women within each race/ethnicity category, we aimed to recruit an equal number of men and women. Once planned enrollment goals were met during the study, recruitment was turned off for specific subgroups based on gender/race. Potentially interested individuals were informed that the study was no longer recruiting and were directed back to the cessation website. Figure 1 shows the trial CONSORT diagram for recruitment and enrollment.

Figure 1. Study CONSORT Diagram.

WEB: Web-based control; SN: Social network intervention; NRT: Nicotine replacement therapy

Measures

These analyses focus on data collected during eligibility screening and baseline assessment. All measures administered in the trial have been described in detail elsewhere (45).

Sociodemographic characteristics and health status

Information was obtained about age, gender, race, ethnicity, education level, employment status, and marital status. Participants also rated their current health status (1=Poor to 5=Excellent), indicated whether they had ever had an illness caused or made worse by smoking, and whether they had been advised by a healthcare provider to quit smoking in the past year. Frequency of Internet and social media use was measured using questions from the Pew Research Center (48).

Smoking history

Current smoking and smoking history were assessed using a brief standard questionnaire which covered smoking frequency and rate, use of other tobacco products, confidence and desire to quit smoking, motivation to quit smoking (49), nicotine dependence (50), and quitting history, including the number of past year quit attempts and quit methods.

Comparison Survey

The questionnaires and datasets for the National Health Interview Survey (NHIS) (51) were reviewed to identify variables that corresponded to measures administered in the trial. This dataset was selected for several reasons: it included the largest pool of current smokers, had data collection periods that overlapped with the trial enrollment period (2012–2014), and had the most number of overlapping questions with the trial compared to other national health-related surveys that were considered (e.g., TUS-CPS, HINTS).

NHIS is an annual survey conducted by the National Center for Health Statistics of the Centers for Disease Control and Prevention. The main objective of the NHIS is to monitor the health of the US population through the collection and analysis of data on a broad range of health topics. NHIS is a cross-sectional household interview survey of the civilian non-institutionalized US population selected based on a stratified, multistage probability sampling design (52–54). Sampling and interviewing are continuous throughout each year. NHIS currently oversamples African Americans and Hispanics, and the data are collected through in-person household interviews conducted by interviewers employed and trained by the US Census. The included data were collected using a computer assisted personal interviewing (CAPI) mode, in which the NHIS questionnaire was administered using a laptop computer with interviewers entering responses directly into the computer during the interview. Combined data from multiple years of the Sample Adult Core of NHIS, which contains one randomly selected “sample adult” from each household, were used for analyses. The household response rate across all three years ranged from 73.8%–77.6% (average 75.7%) while interview rate ranged from 79.7%–81.7% (average 80.6%) among the eligible sample adults (52–54).

To examine generalizability, we selected the following variables that corresponded to questions administered in the trial: sociodemographic variables (gender, age, race, ethnicity, education, employment, and marital status), smoking frequency, cigarettes smoked per day, lifetime smoking of at least 100 cigarettes, quit attempt in the past year (yes/no), use of the Internet to search for health-related information (yes/no), and having received advice from a healthcare provider about quitting smoking.

Statistical Analyses

First, descriptive statistics were used to summarize baseline characteristics of screened and enrolled participants. Baseline characteristics are presented for the full enrolled sample as well as for each treatment arm. Next, demographic differences were examined between eligible participants that enrolled and those that did not enroll. A multivariable logistic regression model was constructed to identify predictors of full enrollment among those that were screened. One-way ANOVA and chi-square tests were then used to test for differences between the intervention arms among those enrolled for continuous and categorical variables, respectively. For categorical variables with individual cell counts below five, Fisher exact tests were employed.

To assess generalizability, we conducted a series of analyses. We began by comparing the enrolled study sample to the full sample of current smokers from NHIS. These analyses address the question of generalizability to nationally-representative samples of US adult smokers. Next, we selected a specific subgroup of current smokers to examine more fine-grained questions of generalizability. This subgroup was defined not only by current smoking status and age as in the first set of analyses, but also by whether they had searched for health information on the Internet and made at least one quit attempt in the past year.

National estimates were obtained using the sample weights provided with the data to correct for non-response and non-coverage biases to the extent possible. However, since NHIS has a complex sampling design, estimates of standard errors obtained under the assumption of simple random sampling are generally invalid. To allow the computation of standard errors that account for the stratified multistage sampling design, NHIS provides public use strata and primary sampling unit variables that are used to specify the design structure for estimation. While the standard errors associated with estimates obtained with these types of large surveys can generally be very small, they tend to be large if subgroups with smaller sample sizes are being analyzed. Combining multiple years of NHIS requires adjustment of the sample weights, which was done by dividing each by the number of survey years combined, following the NHIS guidelines (52–54).

Generalizability was assessed by comparing estimates from the study sample with the weighted estimates from NHIS. Odds ratios and differences in proportions were calculated, along with corresponding 95% confidence intervals, which account for the uncertainty in both the weighted survey estimates and the study sample estimates. Confidence intervals were computed via normal approximation for log-odds ratios and differences of two proportions, respectively. Corrected standard errors associated with the survey estimates were used in the calculation of the confidence intervals. Note that the confidence intervals are reported to provide a measure of uncertainty around the estimates. However, due to large sample sizes, these intervals may indicate statistically significant differences that are actually not meaningful. Therefore, we provide effect sizes to emphasize magnitudes of the differences, instead of their statistical significance. Cohen’s h is an effect size measure, suggested by Cohen (55) to obtain effect sizes for differences in proportions. It is calculated as the difference between arcsine-transformed proportions ( $\mathrm{h}=2arcsin\sqrt{p_1}-2arcsin\sqrt{p_2}$). In addition to providing comparisons independent of the sample size, Cohen’s h allows us to employ a rule of thumb criterion to assist in interpreting the effect sizes. Following Cohen’s guidelines (55), an effect size of 0.2 (and lower) is considered “small,” while 0.5 and 0.8 represent “medium” and “large” effect sizes, respectively. In our analyses, we used 0.2 as a soft threshold to identify medium and large differences to examine further in the original scale. All analyses were performed in R 3.2.0 (56). The “survey” package (57, 58) was used to obtain weighted estimates and the associated standard errors.

Results

Study Participants

Recruitment was conducted between March 2012 and January 2015. During the recruitment period, N=20,670 new registrants of BecomeAnEX.org were screened for the study: 11,022 were eligible, and 5,290 enrolled (48% of eligible) and were randomized to one of the four intervention conditions. The multivariable model predicting study enrollment showed increased odds of enrollment among older age groups, females, non-Hispanic Whites, those with at least several years of college education, and those who reported being ready to quit in the next 6 months. Additional details are available in Supplemental Table 1.

Baseline characteristics of enrolled participants are shown in Table 1. The enrolled sample was primarily female (3,217; 60.8%), White (4,237; 80.1%), non-Hispanic (4,996; 94.4%), and average age was 41.8 years (SD=13.2). The participants were regular Internet users, with 93.5% (4,948) reporting daily Internet use. Participants were daily smokers (5,091; 96.2%) who smoked an average of 16.6 cigarettes per day (SD=8.5). The majority (4,389; 83.0%) reported being ready to quit in the next 30 days; 16.6% (879) were planning to quit in the next 6 months. There were no significant differences on any sociodemographic or smoking history variables between the intervention arms.

Table 1.

Baseline Characteristics of Eligible (n=5,732) and Enrolled (n=5,290) Study Participants.

	Eligible (not enrolled) Sample	Enrolled Sample	WEB	WEB + SN	WEB + NRT	WEB + SN + NRT
	N=5732	N=5290	N=1337	N=1323	N=1313	N=1317
Sociodemographic Variables
Age (years), Mean (SD)	N/A	41.8 (13.2)	42.4 (13.3)	41.9 (13.1)	41.6 (13.0)	42.4 (13.3)
Age (years), range	N/A	18–82	18–81	18–80	18–77	18–82
Female, n (%)	3202 (55.9)	3217 (60.8)	822 (61.5)	809 (61.1)	788 (60.0)	798 (60.6)
Race, n (%)
American Indian or Alaskan Native	163 (2.8)	114 (2.2)	34 (2.5)	18 (1.4)	29 (2.2)	33 (2.5)
Asian	175 (3.1)	82 (1.6)	14 (1.0)	25 (1.9)	17 (1.3)	26 (2.0)
Black or African American	1086 (18.9)	802 (15.2)	216 (16.2)	193 (14.6)	209 (15.9)	184 (14.0)
Native Hawaiian or Other Pacific Islander	66 (1.2)	55 (1.0)	13 (1.0)	14 (1.1)	14 (1.1)	14 (1.1)
White	4242 (74.0)	4237 (80.1)	1060 (79.3)	1073 (81.1)	1044 (79.5)	1060 (80.5)
Hispanic, n (%)	436 (7.6)	294 (5.6)	80 (6.0)	82 (6.2)	61 (4.6)	71 (5.4)
Education, n (%)
Some high school or Less	532 (9.3)	247 (4.7)	66 (4.9)	57 (4.3)	59 (4.5)	65 (4.9)
High school graduate	1565 (27.3)	1200 (22.7)	319 (23.9)	279 (21.1)	293 (22.3)	309 (23.5)
Some college or technical school	2550 (44.5)	2631 (49.7)	643 (48.1)	672 (50.8)	658 (50.1)	658 (50.0)
College graduate	1085 (18.9)	1212 (22.9)	309 (23.1)	315 (23.8)	303 (23.1)	285 (21.6)
Employed full-time, n (%)	--	2402 (45.4)	597 (44.7)	614 (46.4)	608 (46.3)	583 (44.3)
Marital Status, Has a Partner, n (%)	--	2735 (51.7)	695 (52.0)	714 (54.0)	665 (50.6)	661 (50.2)
Internet Use
Daily Internet use, n (%)	N/A	4948 (93.5)	1260 (94.2)	1228 (92.8)	1229 (93.6)	1231 (93.5)
Daily use of Internet to communicate with others, n (%)	N/A	2978 (56.3)	733 (54.8)	746 (56.4)	742 (56.5)	757 (57.5)
Smoking Variables
Cigarettes per day, Mean (SD)	N/A	16.6 (8.5)	16.5 (8.4)	16.5 (8.3)	16.6 (8.4)	16.7 (8.7)
Smoked at least 100 cigarettes in lifetime, n (%)	N/A	5205 (98.4)	1316 (98.4)	1298 (98.1)	1294 (98.6)	1297 (98.5)
Daily smoking, n (%)	N/A	5091 (96.2)	1294 (96.8)	1267 (95.8)	1261 (96.0)	1269 (96.4)
Use other tobacco products (every day/some days), n (%)	N/A	235 (4.5)	53 (3.9)	68 (5.2)	64 (3.9)	50 (3.8)
Want to quit, Mean (SD)	N/A	4.6 (0.6)	4.6 (0.6)	4.6 (0.6)	4.6 (0.6)	4.6 (0.6)
Confidence to quit, Mean (SD)	N/A	3.3 (1.1)	3.4 (1.0)	3.3 (1.1)	3.3 (1.1)	3.3 (1.1)
Readiness to Quit
Next 30 days, n (%)	4712 (82.2)	4389 (83.0)	1110 (83.0)	1100 (83.1)	1091 (83.1)	1088 (82.6)
Next 6 months, n (%)	953 (16.6)	879 (16.6)	222 (16.6)	218 (16.5)	216 (16.5)	223 (16.9)
Not thinking of quitting, n (%)	67 (1.2)	22 (0.4)	5 (0.4)	5 (0.4)	6 (0.5)	6 (0.5)
Quit attempts in past year, Mean (SD), range	N/A	3.1 (11.1)	3.6 (16.8)	2.8 (5.9)	3.2 (10.9)	2.9 (7.8)
Number of quit methods used past year*, Mean (SD)	N/A	1.6 (1.8)	1.6 (1.9)	1.6 (1.8)	1.5 (1.8)	1.6 (1.8)
Fagerström Test for Nicotine Dependence, Mean (SD)	N/A	5.3 (2.3)	5.2 (2.3)	5.2 (2.2)	5.3 (2.3)	5.3 (2.3)
Health Status
General health (Excellent/Very Good), n (%)	N/A	1608 (30.4)	397 (29.7)	434 (32.8)	399 (30.4)	378 (28.7)
Illness from smoking (yes), n (%)	N/A	3351 (63.3)	854 (63.9)	836 (63.2)	818 (62.3)	843 (64.0)
Doctor advice to quit (yes), n (%)	N/A	3283 (62.1)	835 (62.5)	822 (62.1)	806 (61.4)	820 (62.3)

N/A: data not available;

^*Among those who reported 1 or more quit attempts in the past year, n=3810

Comparison of Study Participants to NHIS Current Smokers

Table 2 shows the comparison of study participants to NHIS current smokers (N=19,043), and to the subset of current adult smokers who searched for health information on the Internet and made at least one quit attempt in the past year (N=3,664). There were no differences between the study sample and NHIS current smokers on race or marital status. Statistically significant differences in ethnicity, employment, and age all had small effect sizes below the 0.2 threshold; slightly above the 0.2 threshold were differences in the proportion smoking 20 cigarettes or more (h=0.25) and having received cessation advice from a healthcare provider (h=0.23), both of which were higher among trial participants. Differences in gender, education, daily smoking, and quit attempts had medium effect sizes. Compared to the NHIS sample, study participants were more likely to be female (60.8% vs. 45.3%), to have at least some college education (72.6% vs. 44.7%), to be daily smokers (96.2% vs. 77.4%), and to have made a quit attempt in the past year (68.5% vs. 47.4%). There were no statistically significant differences with large effect sizes.

Table 2.

Study sample compared to NHIS samples (pooled 2012, 2013 2014 data).

Characteristic	Enrolled study sample (N=5,290)	NHIS current adult smokers (Unweighted N=19,043)				NHIS current adult smokers who searched for health info on the Internet AND made at least one quit attempt in the past year (Unweighted N=3,664)
	% (95% CIa)	Weightedb % (SE)	Study/NHIS OR (95% CIc)	Study - NHIS Δ% (95% CId)	Study - NHIS Effect size (Cohen’s h)	Weightedb % (SE)	Study/NHIS OR (95% CIc)	Study - NHIS Δ% (95% CId)	Study - NHIS Effect size (Cohen’s h)
Gender; Female	60.8 (59.5, 62.1)	45.3 (0.6)	1.9 (1.8, 2.0)	15.5 (13.8, 17.2)	0.31	55.3 (1.2)	1.3 (1.1, 1.4)	5.5 (2.8, 8.1)	0.11
Ethnicity; Hispanic	5.6 (5.0, 6.2)	10.3 (0.3)	0.5 (0.5, 0.6)	−4.7 (−5.6, − 3.8)	−0.18	8.7 (0.6)	0.6 (0.5, 0.8)	−3.1 (−4.5, − 1.8)	−0.12
Race; White	80.1 (79.0, 81.1)	81.0 (0.5)	0.9 (0.8, 1.0)	−0.9 (−2.3, 0.5)	−0.02	82.2 (0.8)	0.9 (0.8, 1.0)	−2.1 (−4.0, − 0.1)	−0.05
Employed	57.0 (55.7, 58.3)	59.1 (0.5)	0.9 (0.9, 1.0)	−2.1 (−3.7, − 0.4)	−0.04	64.2 (1.2)	0.7 (0.7, 0.8)	−7.2 (−10.0, −4.5)	−0.15
Education; Some college or more	72.6 (71.4, 73.8)	44.7 (0.6)	3.3 (3.0, 3.6)	27.9 (26.2, 29.6)	0.58	60.0 (1.5)	1.8 (1.5, 2.0)	12.7 (9.4, 15.9)	0.27
Married or living with partner	51.7 (50.4, 53.0)	53.1 (0.6)	1.0 (0.9, 1.0)	−1.4 (−3.1, 0.4)	−0.03	55.0 (1.5)	0.9 (0.8, 1.0)	−3.3 (−6.5, − 0.2)	−0.07
Every day smoking	96.2 (95.7, 96.7)	77.4 (0.5)	7.5 (6.4, 8.7)	18.8 (17.8, 19.9)	0.60	71.6 (1.1)	10.2 (8.5, 12.1)	24.7 (22.5, 26.8)	0.73
1+ quit attempts past year	68.5 (67.2, 69.7)	47.4 (0.5)	2.4 (2.3, 2.6)	21.1 (19.5, 22.7)	0.43	100.0 (0.0)	N/A	N/A	N/A
Cigarettes per day; 20 or more	40.9 (39.6, 42.2)	28.9 (0.5)	1.7 (1.6, 1.8)	12.0 (10.3, 13.6)	0.25	19.5 (0.9)	2.9 (2.5, 3.2)	21.3 (19.2, 23.5)	0.47
Received cessation advice	62.1 (60.7, 63.4)	50.6 (0.5)	1.6 (1.5, 1.7)	11.5 (9.8, 13.2)	0.23	61.9 (1.2)	1.0 (0.9, 1.1)	0.2 (−2.5, 2.9)	0.004
Age group (screening)
18–24	10.4 (9.6, 11.2)	12.8 (0.5)	0.8 (0.7, 0.9)	−2.4 (−3.6, − 1.2)	−0.08	17.0 (1.4)	0.6 (0.5, 0.7)	−6.6 (−9.4, − 3.8)	−0.19
25–44	45.3 (43.9, 46.6)	40.2 (0.5)	1.2 (1.2, 1.3)	5.1 (3.4, 6.7)	0.10	48.8 (1.2)	0.9 (0.8, 1.0)	−3.5 (−6.2, − 0.9)	−0.07
45–64	40.1 (38.8, 41.4)	37.9 (0.6)	1.1 (1.0, 1.2)	2.2 (0.5, 3.9)	0.05	31.0 (1.3)	1.5 (1.3, 1.7)	9.1 (6.3, 11.9)	0.19
65+	4.3 (3.8, 4.9)	9.1 (0.3)	0.5 (0.4, 0.5)	−4.8 (−5.6, − 4.1)	−0.19	3.2 (0.4)	1.3 (1.0, 1.8)	1.0 (0.1, 2.0)	0.06

^a95% CI’s obtained using approximate Chi-square CI’s for single sample proportions;

^bPoint estimates and corresponding standard errors were obtained from the corresponding subgroup of the pooled 2012–2014 NHIS data using the appropriate weighted design;

^c95% CI’s were obtained via normal approximation for log odds ratios;

^d95% CI’s were obtained via normal approximation for differences of two proportions.

Comparison of Study Participants to NHIS Subset of Current Smokers

Comparing the study sample to the NHIS subset of current smokers who searched for health information on the Internet and made at least one quit attempt in the past year amplified differences in the proportion of every day smokers and heavy smokers. Study participants were more likely to be every day smokers (96.2% vs. 71.6%, h=0.73) and to smoke 20+ cigarettes per day (40.9% vs; 19.5%, h=0.47) compared to the NHIS subset. All other differences were attenuated, falling at or below the 0.2 threshold.

Conclusion

This study presents the baseline sociodemographic and smoking history characteristics of smokers screened for and enrolled in an Internet smoking cessation randomized trial. Somewhat older, female, non-Hispanic Whites, with higher educational attainment and employment were more likely to enroll in the study after initial screening. The majority of enrolled participants were daily smokers planning to quit in the next 30 days, though approximately 17% were less motivated to quit, indicating they were thinking of quitting sometime in the next 6 months. On average, trial participants were in their early 40s but age ranged from 18 through early 80s. Nearly a third of participants had a high school education or less, and more than half reported employment other than full time (i.e., part-time, unemployed, retired, student). Trial participants were frequent Internet users, more than half of whom used the Internet to connect with others through social channels (e.g., blogs, instant messaging) daily. More than 60% reported having an illness caused or made worse by smoking and having received a doctor’s advice to quit smoking. Taken together, these characteristics describe a sample of largely middle-aged, Internet savvy, motivated treatment seekers with smoking-related health concerns who had been counseled by a healthcare provider to quit smoking in the past year.

To address the question of generalizability of the enrolled sample, we first examined comparisons to a nationally representative sample of current smokers. Given the large sample size of both datasets, we employed standardized effect sizes to compare the magnitudes of differences rather than relying solely on statistical significance. Modest effect sizes were observed for differences in gender, education, daily smoking, and quit attempts. Trial participants were more likely to be female and to have at least some college education, and to be daily smokers that had made a quit attempt in the past year. Given that women and adults with at least some college education are more likely to look for health information online (59, 60), the modest effect sizes observed for these differences are to be expected. It is also not surprising that, compared to the general population, smokers who seek to enroll in an Internet treatment study – or for that matter, any treatment study – would be more likely to smoke every day and to smoke more intensely. Deeper involvement with smoking and more unsuccessful efforts to quit, perhaps coupled with early signs of smoking-related illness, may prompt smokers’ interest in cessation assistance. Trial participants were also more likely to be concentrated in the 45–64 year age range, which is consistent with the notion that, on average, those most interested in cessation have had considerable cumulative lifetime experience with smoking and are now ready to make changes.

When the national sample was narrowed to those who search for health information on the Internet and made at least one quit attempt in the past year, differences between the samples in gender and education were attenuated while differences in daily smoking and smoking heaviness were amplified. In other words, as trial participants were matched more closely on Internet use behavior and motivation to quit, only daily smoking and heavy smoking distinguished the samples. These results dispel the notion that smokers recruited via the Internet may be lighter smokers in general, and therefore would on this basis somehow react differently to treatments compared to smokers who enroll in clinical trials in healthcare or other settings.

These results also highlight a critical issue: it is important to ask the right question when gauging the extent to which the findings of any trial may be generalizable. Asking “Is this sample representative?” is misguided unless the target population is clearly specified (e.g., representative of the US population?; representative of the population of smokers in the US?). However, even asking “Is this sample representative of all smokers?” may be misguided since no single cessation treatment strategy should be expected to reach, attract and engage all smokers, and no single clinical trial recruitment approach should be expected to generate a sample that is broadly representative of all smokers. The Internet is simply one channel to deliver evidence-based cessation treatment using existing infrastructure where (millions of) smokers are located (61). Rather, we argue that the right question to ask with regards to generalizability must reference the target population used for a specific trial (e.g., “Is this sample of treatment seeking smokers recruited through the Internet generalizable to other Internet using smokers who are motivated to quit?”) In addition, we caution researchers and reviewers of Internet-based trials to consider the limitations of a specific trial appropriately and thoughtfully. Statements like “The results cannot be generalized to all people pursuing smoking cessation” and “Generalizability may be a concern because of recruitment by self-selection” – while technically true – are overly simplistic and not unique to Internet studies.

This study has several limitations that should be considered. First, as with all Internet smoking cessation studies, we studied a sample of convenience – individuals who found their way to, and took the time to enroll via, the BecomeAnEX website. Therefore, our findings and generalizable population implications of the study should be viewed in this context. Second, different results may have been obtained had we selected a different dataset as a comparator in lieu of the National Health Interview Survey. Our selection of NHIS was driven by the available overlapping metrics and the pool of current smokers for comparison. Other options for researchers to consider include the Behavioral Risk Factor Surveillance Survey and the Health Information National Trends Survey. Third, although the trial enrolled more than 5,200 participants and is large in comparison to other Internet-based cessation trials (5), the final enrolled sample represents a subset of all users on the BecomeAnEX website. This is an inherent challenge in conducting any type of clinical trial where drop off occurs at each step of the enrollment process (i.e., eligibility screening, informed consent). Our final enrollment rate is comparable to prior studies that have utilized similar enrollment methods (14) and similar or higher than several other large-scale trials (15, 62, 63). Finally, these analyses do not consider abstinence rates, which are beyond the scope of the present paper but forthcoming in analyses focused on the main outcomes of the trial. It will be important to examine population (non-treatment) quit rates in a matched national sample to gauge the effectiveness of the interventions, but this will require statistical approaches that are still under development (64).

We conclude that, with some caveats noted above, smokers enrolled into treatment studies via the Internet may not differ largely from smokers enrolled into other treatment studies: they are likely to be middle-aged smokers with considerable cumulative lifetime involvement with smoking who are facing up to the reality that they are at higher risk of smoking-related illnesses if they continue to smoke. They are likely to be highly motivated to quit, with a history of failed quit attempts. White, more highly-educated women are perhaps more likely to be represented in these studies than other demographic subgroups, but this kind of selection effect may simply reflect the population most likely to seek health information on the Internet. The difference observed in this study may not be dissimilar to those related to other recruitment methods (e.g., smokers recruited in healthcare settings may be more likely to suffer from smoking-related illnesses compared to a general population of smokers). Like traditional clinical trials, however, Internet-based studies are less likely to attract large numbers of underrepresented minority and low-SES participants. While there has been ample research on recruitment of underrepresented groups to traditional clinical trials (65) and to smoking cessation programs in general (66), relatively few studies have focused on recruiting underrepresented groups to Internet cessation programs (67–71). Given the demonstrated effectiveness of Internet cessation programs, this is an important area worthy of more extensive inquiry. We hope that the methods employed in these analyses provide a useful example for others conducting web-based research to explicitly examine questions of generalizability in their work. Addressing these questions directly and appropriately will serve to more clearly elucidate the potential population impact of Internet cessation interventions. We encourage a more refined consideration of the question of generalizability and sample representativeness, particularly when it comes to Internet interventions.