Feasibility of biochemical verification in a web-based smoking cessation study

Abstract

Background and Aims

Cogent arguments have been made against the need for biochemical verification in population-based studies with low-demand characteristics. Despite this fact, studies involving digital interventions (low-demand) are often required in peer review to report biochemically verified abstinence. To address this discrepancy, we examined the feasibility and costs of biochemical verification in a web-based study conducted with a national sample.

Methods

Participants were 600 U.S. adult current smokers who registered on a web-based smoking cessation program and completed surveys at baseline and 3 months. Saliva sampling kits were sent to participants who reported 7-day abstinence at 3 months, and analyzed for cotinine.

Results

The response rate at 3-months was 41.2% (n=247): 93 participants reported 7-day abstinence (38%) and were mailed a saliva kit (71% returned). The discordance rate was 36.4%. Participants with discordant responses were more likely to report 3-month use of nicotine replacement therapy or e-cigarettes than those with concordant responses (79.2% vs. 45.2%, p=.007). The total cost of saliva sampling was $8280 ($125/sample).

Conclusions

Biochemical verification was both time- and cost-intensive, and yielded a relatively small number of samples due to low response rates and use of other nicotine products during the follow-up period. There was a high rate of discordance of self-reported abstinence and saliva testing. Costs for data collection may be prohibitive for studies with large sample sizes or limited budgets. Our findings echo previous statements that biochemical verification is not necessary in population-based studies, and add evidence specific to technology-based studies.

1. Introduction

Accurate measurement of smoking status is imperative to evaluate the effectiveness of cessation interventions. Self-report surveys can underestimate smoking status (1), especially in high-demand trials where misreporting is more likely. High-demand trials are those involving populations with social pressures to quit (e.g., pregnant smokers (2), or those with smoking-related diseases (3-5)), and in intensive interventions with frequent contact that may create social desirability among participants to report success (6, 7). In such trials, biochemical verification is recommended. However, in low demand cessation trials such as studies with no face-to-face contact (8) and large population-based trials that require minimal interaction with study staff or effort by participants (7, 9, 10), rates of discordance between self-reported and biochemically verified abstinence have been shown to be small in magnitude (7, 10) and comparable across intervention conditions. For these kinds of trials, biochemical verification has been determined to be unnecessary (7, 9).

Despite cogent arguments against the need for biochemical verification in low-demand, population-based studies, trials of digital smoking cessation interventions delivered via the Internet or text message are often required in peer review to include biochemically verified abstinence as a primary outcome. Several recent trials have included such measures (11-14), while others that have relied on self-report have noted the lack of biochemical verification as a limitation of their findings (15-23). The emphasis on biochemical verification during peer review may be because digital interventions are newer than traditional face-to-face or telephonic approaches, and the perception that such studies should include more conservative metrics of abstinence to demonstrate their effectiveness. However, trials of digital interventions are often ill-suited for biochemical verification. Participants are usually recruited from the general population (vs. special populations) and have minimal direct interaction with study staff, and interventions are often self-guided and less intensive (24-26). Given the broad, national reach and scalability of digital interventions, such trials also often involve large, geographically diverse samples, posing additional challenges for biomarker collection.

The aim of this study was to address the discrepancy between recommendations regarding biochemical verification in population-based cessation trials and what we have observed to be the state of the science for digital cessation interventions. We also sought to add to the scant literature about the actual conduct of biochemical verification among participants whose entire study experience (recruitment, intervention, follow-up) occurred via the Internet. Specifically, we report the response rate, discordance rate, costs, and characteristics that distinguish those with discordant vs concordant results to inform the discussion about the utility of biochemical verification in trials of digital interventions.

2. Methods

2.1 Study Sample

Participants were new registrants on BecomeAnEX.org, a free, publicly available smoking cessation website run by Truth Initiative since 2008 (27, 28). Participants were recruited to a study that involved completing a baseline survey and 3-month follow-up assessment of smoking status. Recruitment information included details about biochemical verification and availability of incentives for completing each phase of the study. Eligibility criteria were age 18 or older, US residence, and smoking cigarettes “every day” or “some days”. There were no other inclusion/exclusion criteria.

2.2 Intervention

All participants had access to BecomeAnEX, which was developed in collaboration with Mayo Clinic (27). Consistent with national guidelines (29), BecomeAnEX helps users develop problem-solving and coping skills to quit smoking, educates about pharmacotherapy and assist users in selecting a cessation medication, and facilitates social support through a large online social network of current and former smokers.

2.3 Procedure

Study oversight was provided by Chesapeake IRB. Recruitment occurred immediately following registration. Users completed eligibility screening, provided informed consent, completed a baseline survey, and confirmed their e-mail address, all within 24 hours of registration. The entire enrollment process was conducted online, automated by a proprietary clinical trials management system.

The follow-up survey was sent via e-mail 3 months after enrollment. Non-responders were sent email reminders at 3 and 6 days. Due to budget constraints, no other methods of follow-up data collection (e.g., phone, mail) were employed. Participants reporting 7-day point prevalence abstinence (ppa) were overnight mailed a swab saliva collection kit (SalivaBio Oral Swab from Salimetrics^® (30)) within 1 business day of survey completion. Depending on the timing of follow-up survey completion (e.g., end of day Friday), participants received the collection kit within 1-4 calendar days. Collecting saliva cotinine within 7 days is recommended as a reasonable interval to verify smoking status (9). Mailed saliva cotinine sampling is an accurate method to verify smoking status (31-33) and is most practical in studies with geographically diverse participants. Study staff notified participants by e-mail and text message when kits were mailed and delivered to encourage timely return. The kit included instructions to avoid foods high in sugar, acidity, or caffeine immediately before sample collection, and to wait at least 10 minutes after rinsing the mouth with water before collecting saliva. Participants were asked to document the presence of oral disease or injury, and to record past 12-hour consumption of alcohol, caffeine, nicotine, and medications. This form was sent back with the saliva sample, which was stored in a dedicated freezer until it was overnighted to Salimetrics^® (30) for cotinine analysis. Cotinine levels of less than 15 ng/mL were determined to be concordant with self-reported abstinence (9). Participants were compensated $20 for the baseline survey, $50 for the 3-month survey, and $25 for returning the saliva sample (Amazon giftcodes sent via email).

2.4 Measures and Analysis

The baseline survey assessed sociodemographic characteristics (age, gender, race, ethnicity, education, employment, marital status, income); smoking history (current cigarettes per day, desire and confidence to quit, nicotine dependence (34)); quitting history (number of quit attempts, quit methods); and curiosity about using e-cigarettes to quit. The 3-month survey assessed current smoking (30-day ppa, 7-day ppa); smoking and quitting history since enrollment; and smoking-related interactions with health professionals. Cotinine level and data on past 12-hour use of nicotine were included in analyses.

Bivariate tests were used to examine differences between participants with cotinine levels above or below the 15 ng/mL threshold on baseline characteristics, cessation-related behaviors, and smoking outcomes. Between group differences were examined with two-sample t-tests for normally distributed continuous and ordinal variables, Mann-Whitney U test for skewed continuous variables, and two-sided Chi-square or Fisher's exact tests for categorical variables. All analyses were completed using SPSS v.21 (35).

3. Results

3.1 Study Sample

From October 2015 to March 2016, 600 participants were enrolled. Participants were primarily white (81.5%), female (73.0%), and had some college education (72.2%). Most were 25-44 years old (42.2%) or 45-64 years old (42.8%); about half were employed full time (50.2%), had a spouse/partner (49.5%), and reported income higher than $30,000/year (56.8%). Smoking rate at enrollment was 18.2 cigarettes per day (SD=9.8), and 74.5% reported smoking their first cigarette within 30 minutes of waking. Participants reported an average of 2 quit attempts in the past year (Interquartile range (IQR)=0-3.0): the most common quit methods were “willpower/cold turkey” (55.5%) and medication (nicotine replacement therapy products and prescription medicines) (40.3%). One third (36.5%) of participants had used e-cigarettes in the past year to quit smoking, 15.8% reported past 30 day use of e-cigarettes, 77.8% reported being curious about using them to quit, and 40.7% planned on trying them to quit.

3.2 Smoking Outcomes and Biochemical Verification

The 3-month follow-up rate was 41.2% (n=247): 93 participants reported 7-day ppa (37.7%) and were mailed a saliva collection kit, and 66 (71.0%) of those participants returned their saliva samples. All samples returned were usable to test for cotinine. On average, participants returned saliva samples one week after receipt (SD=4.4 days). Among returned samples, 24 had cotinine level at or higher than 15 ng/mL, yielding a discordance rate of 36.4%. Of the 66 participants that returned samples, 9 did not provide past 12-hours nicotine use data and 6 had cotinine levels at or higher than 15 ng/mL. Of the 57 participants who provided past 12-hours nicotine use data, 12 reported using a nicotine product and had cotinine levels 15 ng/mL and higher.

3.3 Comparison of Concordant and Discordant Participants

There were no baseline differences between those who did and did not return the saliva sample. At 3-months, participants who did not return a saliva sample were less likely to have reported 30-day ppa (59.3% vs. 81.8%, p=.022). Among those who returned a sample, participants with discordant results differed on baseline levels of education, employment, marital status, time to first cigarette, and interest in using e-cigarettes to quit from participants with concordant cotinine results (Table 1). At 3-months, discordant participants were less likely to report 30-day ppa, had a higher number of quit attempts, and were more likely to have used nicotine replacement therapy or e-cigarettes (Table 2).

Table 1. Baseline characteristics by 3-month follow-up survey completion and biochemical verification outcome.

	Completed follow-up	Self-reported 7-day ppa	Returned saliva kit	Cotinine level
				≥15 ng/mL	<15 ng/mL	p-value
Sample size (n)	247	941	66	24	42
Demographic variables
Age group, years, %						.609
18-24	7.7	5.3	6.1	4.2	7.1
25-44	49.4	50.0	47.0	54.2	42.9
45-64	38.9	42.6	43.9	41.7	45.2
65+	4.0	2.1	3.0	0.0	4.8
Gender, Female, %	71.7	72.3	72.7	70.8	73.8	.794
Race, White, %	84.2	85.1	81.8	83.3	81.0	1.0002
Ethnicity, Hispanic, %	2.8	2.1	3.0	0.0	4.8	.5302
Education, Some college or more, %	80.2	79.8	83.3	66.7	92.9	.0132
Employment, full time, %	54.3	61.7	57.6	37.5	69.0	.013
Has spouse/partner, %	53.8	59.6	62.1	45.8	71.4	.039
Income ≥ 30K, %	61.1	57.4	59.1	45.8	66.7	.098
Smoking variables
Cigarettes per day, mean (SD)	17.6 (9.43)	16.0 (7.54)	16.0 (7.51)	17.1 (6.68)	15.4 (7.95)	.375
Time to first cigarette, ≤ 30 min, n (%)	74.5	70.2	68.2	83.3	59.5	.046
Want to quit, mean (SD)	4.6 (.66)	4.7 (.62)	4.7 (.70)	4.8 (.38)	4.6 (.82)	.235
Confident to quit, mean (SD)	3.3 (1.11)	3.6 (1.15)	3.5 (1.17)	3.3 (1.27)	3.7 (1.10)	.211
Quit attempts in past year, median (IQR)	1.0 (0-3.0)	2.0 (0-4.0)	2.0 (0-4.0)	1.5 (0-4.0)	2.0 (0-4.0)	.886
Quit methods in past year, %3, 4
Willpower/Cold turkey	53.4	61.7	60.6	66.7	57.1	.446
Behavioral	16.2	21.3	19.7	16.7	21.4	.7552
Medication	39.3	39.4	34.8	37.5	33.3	.733
Alternatives	21.9	21.3	21.2	20.8	21.4	.955
E-cigarettes	36.4	40.4	40.9	50.0	35.7	.256
Past 30-day e-cig use	19.8	21.3	18.2	29.2	11.9	.0862
Curious about e-cigs to quit5, %	78.1	76.6	77.3	83.3	73.8	.374
Plan on trying e-cigs to quit5, %	39.7	43.6	43.9	70.8	28.6	.001

¹One participant who self-reported 7-day ppa was mistakenly not sent a saliva kit.

²At least 1 cell had expected cell <5, reported Fisher's exact test p-value

³Includes participants who reported at least 1 quit attempt in the past year

⁴Willpower/cold turkey: cold turkey, prayer; Behavioral: telephone counseling, Internet interventions, book or pamphlet, individual counseling, group counseling; Medication: nicotine patch, gum, spray, lozenge, inhaler, or bupropion and varenicline; Alternative: switching to other tobacco products, cutting down or switching brands, acupuncture, hypnosis, other alternative therapies, any “other” open ended response.

⁵Responded “probably” or “definitely”

Table 2. Self-reported data by 3-month follow-up survey completion and biochemical verification outcome.

	Completed Follow-up	Self-reported 7-day ppa	Returned saliva kit	Cotinine level
				≥15 ng/mL	<15 ng/mL	p-value
Sample size (n)	247	941	66	24	42
Smoking outcomes, %
Continuous abstinence	19.0	50.0	54.5	41.7	61.9	.112
30-day abstinence	28.7	75.5	81.8	62.5	92.9	.0062
Number quit attempts, median (IQR)	2.0 (1.0-4.0)	1.5 (1.0-4.0)	1.0 (1.0-3.3)	3.0 (2.0-5.5)	1.0 (1.0-2.0)	<.001
Quit methods in past 3 months, % 3, 4
Willpower/Cold turkey	78.8	74.2	72.3	70.8	73.2	.839
Behavioral	37.4	39.3	38.5	33.3	41.5	.516
Medication	57.0	57.3	53.8	62.5	48.8	.284
Alternatives	33.6	28.1	26.2	33.3	22.0	.314
E-cigarettes	39.9	33.3	24.6	45.8	12.2	.002
Used any NRT or E-cigs	59.9	61.7	57.6	79.2	45.2	.007
Health professional advice, %
Advised to quit	45.3	38.3	42.4	45.8	40.5	.868
Advised to use medication	53.6	63.9	71.4	72.7	70.6	1.0002
Asked about e-cigarettes	14.6	16.1	15.9	31.3	7.1	.082
Saliva Kit Return and nicotine use data
# days to return kit, mean (SD)	---	---	7.8 (4.4)	8.0 (4.2)	7.8 (4.5)	.882
Did not return data on nicotine use in the past 12-hours, n (%)	---	---	9 (13.6)	6 (25.0)	3 (7.1)	.0632
Provided data on nicotine use in the past 12-hours, n (%)	---	---	57 (86.4)	18 (75.0)	39 (92.9)
Reported nicotine use in the past 12-hours, n (%) 5	---	---	12 (21.1)	12 (66.7)	0	<.001

¹One participant who self-reported 7-day ppa was mistakenly not sent a saliva kit.

²At least 1 cell had expected cell <5, reported Fisher's exact test p-value

³Includes participants who reported at least 1 quit attempt in the past year

⁴Willpower/cold turkey: cold turkey, prayer; Behavioral: telephone counseling, Internet interventions, book or pamphlet, individual counseling, group counseling; Medication: nicotine patch, gum, spray, lozenge, inhaler, or bupropion and varenicline; Alternative: switching to other tobacco products, cutting down or switching brands, acupuncture, hypnosis, other alternative therapies, any “other” open ended response.

⁵Percentage of those who provided data on nicotine use in the 12-hours preceding the saliva cotinine sample collection

3.4 Costs of Biochemical Verification

The total cost for biochemical verification was $8280, which included the kit and two-way overnight shipping costs for 93 participants that reported abstinence, plus commercial laboratory testing fees and incentives for the 66 participants that returned samples. The cost per sample was $125. These costs do not include staff time in preparing/shipping kits, communicating with participants, processing returned kits, delivering incentives, and communicating with the testing lab.

4. Discussion

We examined the feasibility and utility of verifying self-reported smoking abstinence via saliva sampling in a study conducted entirely via the Internet. Biochemical verification was time and cost intensive, and yielded a small number of usable samples due to low response rates, concordant use of other nicotine products, or missing data regarding use of other nicotine products. We cannot conclude that the discordance rate was due to misreporting since a quarter of discordant participants did not provide data about other nicotine product use. Discordant participants were more likely to be interested in using e-cigarettes to quit at baseline and more likely to report actual use of e-cigarettes to quit at 3-months. We agree with Johnson et al. (14) that guidelines for biochemical verification need to be refined to account for the use of e-cigarettes and other nicotine-containing products.

Four limitations should be noted. First, the response rate was low despite financial incentives, perhaps due to a lack of additional telephone follow-up which has yielded higher response rates in our previous research (36). High levels of attrition are common in studies of digital interventions (24, 25), making our results potentially informative to others looking to estimate the number of cases available for biochemical verification. We also note that roughly a third of participants reporting 7-day ppa did not return a saliva sample, potentially because they knowingly misreported their smoking status. Second, our analyses were restricted to cotinine and did not consider other biomarkers such as anabasine that could be used in the presence of other nicotine products (12). The expense of anabasine assays would have been prohibitive for our study, and likely for others as well (9). Third, it is possible that some participants may have relapsed between completion of the follow-up survey and time of saliva collection. We do not believe this possibility would increase discordance rates in our study more than in other types of trials. Finally, our results may not be representative of smokers participating in other digital cessation interventions.

This research extends prior conclusions that “in large population, low-intensity intervention trials, biochemical verification is neither feasible nor necessary” (SRNT, p. 157 (9)) in the context of digital cessation interventions. Researchers weighing the value of biochemical verification for their own trials or during peer review should consider its impact on sample size and statistical power (9). We encourage researchers to think critically about the need to conduct biochemical verification digital intervention trials, and not simply to embark on such efforts for the appearance of rigor. As noted by West et al.,(8) outcome measures like the Russell Standard may be implemented without biochemical verification when there is no face-to-face contact with participants. Less costly methods such as contacting a significant other to verify smoking status (37), or implementing a “bogus pipeline” (38) may be useful alternatives. We urge peer-reviewers to avoid the reflexive response to dismiss the validity of studies simply because they lack biochemical verification.

Highlights.

• 93/600 participants abstinent at 3-mo mailed saliva kits and 71% returned samples

• 36.4% had cotinine levels above 15 ng/mL and had discordant results

• Biochemical verification yielded low number of usable sample and was cost-intensive

• Findings reiterate questionable utility of biochemical verification in this setting

Abbreviations

PPA

point prevalence abstinence