Pilot Randomized Controlled Trial of a Novel Smoking Cessation App Designed for Individuals With Co-Occurring Tobacco Use Disorder and Serious Mental Illness

Abstract

Introduction

High rates of tobacco use among people with serious mental illness (SMI), along with their unique needs, suggest the importance of developing tailored smoking cessation interventions for this group. Previous early-phase work empirically validated the design and content of Learn to Quit, a theory-based app designed for this population.

Methods

In a pilot randomized controlled trial, we compared the feasibility, acceptability, and preliminary efficacy of Learn to Quit versus QuitGuide, an app designed for the general population. All participants received nicotine replacement therapy and technical assistance. Daily smokers with SMI (N = 62) participated in the trial with outcomes assessed at weeks 4, 8, 12, and 16.

Results

Compared to QuitGuide, Learn to Quit participants had similar number of days of app use (34 vs. 32, p = .754), but larger number of app interactions (335 vs. 205; p = .001), longer durations of app use (4.24 hrs. vs. 2.14 hrs; p = .044), and higher usability scores (85 vs. 79, p = .046). At week 16, Learn to Quit led to greater reductions in cigarettes per day (12.3 vs. 5.9 for QuitGuide; p = 0.10). Thirty-day point prevalence abstinence was verified in 12% of Learn to Quit participants versus 3% of QuitGuide participants (odds ratio = 3.86, confidence interval = 0.41 to 36, p = .239). Changes in psychiatric symptoms and adverse events were not clinically significant between conditions.

Conclusions

This pilot trial provides strong evidence of Learn to Quit’s usability, feasibility, and safety. Preliminary evidence suggests the app may be efficacious. A randomized controlled efficacy trial is needed to test the app in a larger sample of smokers with SMI.

Implications

This study suggests that the Learn to Quit app is a feasible approach to deliver smoking cessation treatment in patients with co-occurring tobacco use disorder and SMI. This means that, if found efficacious, this technology could be used to deploy smoking cessation treatment to larger segments of this population, hence improving public health. Therefore, a randomized controlled trial should be conducted to examine the efficacy of this digital intervention.

Introduction

Tobacco use disorder shortens the lifespan of adults with a diagnosis of schizophrenia, bipolar disorder, and recurrent major depression (ie, serious mental illness [SMI]) by 25 years,¹ a comorbid condition that contributes to $100 billion in health care expenditures in the United States alone.² Thus, determining whether it is possible to deliver more effective and wider reaching smoking cessation interventions for this population is a public health priority.

Smartphone apps are a scalable technology that could provide the necessary skills and support for quitting in this population, of which 88.6% are mobile phone users, and 68% smartphone users.³ Given the substantial barriers to traditional treatment modalities in this population, smartphone apps provide an opportunity to improve public health by increasing access.⁴ While less personalized than face-to-face treatments, mobile technology has effectively delivered smoking cessation treatment in the general population.⁵ More specifically, it has shown promise in reducing tobacco dependence in underserved populations⁶ and treating psychiatric symptoms in patients with SMI.⁷

However, research has shown that apps developed for the general public have inadequate levels of usability and engagement in people with SMI.⁸ In our own work,⁹ we showed that depressive symptoms predicted low utilization with SmartQuit, a smoking cessation app developed for the general population.¹⁰ More directly, we evaluated a smoking cessation app developed by the National Cancer Institute (NCI) (ie, QuitPal) in a sample of patients with SMI and identified potential design requirements for smoking cessation apps for this group.¹¹ Recognizing the need to address these requirements, we developed Learn to Quit, a smoking cessation app based on Acceptance and Commitment Therapy (ACT)¹² and US Clinical Practice Guidelines¹³ that combines a simple design structure, gamification, behavior analytic principles, and storytelling to deliver smoking cessation content.¹⁴ Learn to Quit was tested in a mixed-methods study evaluating the preliminary usability and acceptability of the app, suggesting high levels of engagement, comprehension, and retention of app content.¹⁵

Building on this early-phase work, this paper reports the results of a pilot randomized controlled trial (NCT03069482) that tested the feasibility, acceptability, and preliminary efficacy of Learn to Quit compared to an active app control (NCI QuitGuide). We hypothesized that using Learn to Quit would result in higher levels of usability and app engagement compared to QuitGuide. Secondary outcomes of this pilot study included smoking reductions, quit attempts, and biochemically verified quit rates. Consistent with the goals of a feasibility trial,¹⁶ this pilot also evaluated the feasibility and safety of conducting a mobile health (mHealth) randomized controlled trial among patients with SMI.

Methods

Design

Pilot parallel randomized controlled trial with follow-up assessments at 4, 8, 12, and 16 weeks postrandomization.

Participants

Inclusion criteria were (1) current diagnosis of schizophrenia, schizoaffective, bipolar, or recurrent major depressive disorder based on the International Classification of Diseases (Tenth Revision); (2) self-reported daily smoking of five or more cigarettes per day (CPD) and a carbon monoxide (CO) breath test of more than 6 parts per million¹⁷; (3) desire to quit smoking in the next 30 days; (4) age 18 or older; (5) willing and medically eligible to use nicotine replacement therapy; (6) fluent in spoken and written English; (7) adherence to psychiatric treatment; and (8) stable housing. We excluded individuals who (1) had problematic alcohol or illicit drug use in the last 30 days, (2) had an acute psychotic episode or were unsafe to participate in the study, (3) were pregnant or had the intention to become pregnant, or (4) were currently receiving smoking cessation treatment.

Interventions

Learn to Quit

This Android app¹⁴ contains 28 theory-based modules that (1) focus on ACT to deliver smoking cessation knowledge and skills, (2) use key elements of the US Clinical Practice Guidelines (eg, setting up a quit date), and (3) provide psychoeducation and tips on adhering to nicotine replacement therapy. ACT for smoking cessation has shown to be as efficacious as well-established treatments in several clinical trials,^18–20 and it is an effective intervention for patients with SMI.^21,22 Delivery of this content was informed by principles of applied behavior analysis²³ (eg, fixed ratio of reinforcement to promote a steady uptake of app rewards or tokens) and the serious games literature (eg, the use of storytelling, “game challenges,” and the concept of “skills mastery”),²⁴ with the goal of increasing retention and comprehension of app content and promoting skills practice.

Learn to Quit Modules are divided into “lesson” and “skills” modules. Each day, participants can complete up to one “lesson” module that teaches smoking cessation content and one “skills” module that puts those skills into practice. Therefore, the entire program takes 14 days if modules are completed daily. Participants can also go back and review previously completed modules to enhance retention of content and practice smoking cessation skills. The app also has a daily “check-in” that asks participants about their mood, smoking urges, and how many cigarettes were smoked in a day. These check-ins can be completed at any time and as many times as needed. A more detailed description of the Learn to Quit app is available elsewhere.¹⁴

QuitGuide

This app developed by the National Cancer Institute delivers US Clinical Practice Guidelines for smoking cessation.¹³ Key QuitGuide content includes (1) psychoeducation about the health consequences of smoking; (2) features to track smoking habits, mood, and cravings; and (3) tips for quitting (eg, getting rid of cigarette ashtrays). More information about QuitGuide’s rationale and content can be found at www.smokefree.gov.

Combined Nicotine Replacement Therapy

All participants received an 8-week course of transdermal nicotine patches starting at 21 mg/24 h and a 1–2 week supply of nicotine lozenges (4 mg). Participants with lower levels of smoking at baseline (ie, 10 or less CPD) received an adjusted course of nicotine patches per package instructions. All participants were instructed to use the patches and lozenges on their quit date, and a study physician provided oversight of Combined Nicotine Replacement Therapy (C-NRT) dispensing and monitoring.

Procedures

We recruited participants through a variety of methods. These included identification of potential participants through electronic health records, coordination with primary care clinics, patient health portal invitations, and collaboration with smoking cessation programs and mental health clinics. Interested individuals who passed an initial phone screening were invited to an in-person intake/consent visit. During this visit, participants enrolled in the study and signed a Release of Information form that was used to contact their medical provider to confirm final eligibility. Participants’ psychiatric diagnoses and history of alcohol and drug use were collected from medical providers and/or medical records and confirmed with Mini International Neuropsychiatric Interviews.²⁵ Problematic alcohol or drug use was screened using the Drug Abuse Screening Test²⁶ and the Alcohol Use Disorders Identification Test.²⁷ Adherence to psychiatric treatment was self-reported by the participant and confirmed by the treatment provider. Eligible participants were invited to a second in-person visit in which they were randomized 1:1 to one of the interventions. Consistent with previous trials suggesting that psychotic symptoms are a predictor of low engagement with smoking cessation treatment in SMI,²⁸ we stratified participants based on having a primary psychotic or mood disorder.

Each participant received an Android smartphone device (Samsung Galaxy Express Prime) with access to a monthly phone, text, and data plan. Follow-up assessments were conducted preferably in-person. However, participants who were not able to attend an in-person visit were offered a telephone assessment or web survey to minimize the role of transportation as a barrier to participation. All participants received technical assistance on using the app and smartphone. We offered up to 4 telephone or in-person meetings that lasted between 15 and 30 minutes during the first month after being randomized to the intervention to assist with tasks such as setting up a Google Play account to download apps, logging into e-mail accounts, setting up their voice mailbox, etc.

Compensation included $110 and retention of the study smartphone for those who completed all measures. All procedures were approved by the Institutional Review Board at Duke University.

Measures

C-NRT Adherence and Safety

Adherence was measured with self-reported days of use of nicotine patches and nicotine lozenges at each follow-up assessment using a single item question used in previous addiction research.²⁹ Adverse events and serious adverse events were collected at each study visit, reviewed by study physician and the first author, and reported to the IRB and/or the National Institutes of Health when necessary.

Usability

App usability was measured with the System Usability Scale,³⁰ a valid and reliable 10-item questionnaire widely used by user-centered design researchers to measure perceived ease-of-use of a software system (eg, “I found the various functions of this app were well integrated”). Scores above 84.1 represent an A+ grade. Lower scores (<78.8) indicate levels of usability that span the B+ range (acceptable) to the F range (not acceptable).

User Engagement

Measures of app engagement were automated with Google Analytics to collect intensive longitudinal data. First, we tracked the total amount of daily app interactions used from randomization to week 16. For example, selecting a module in the Learn to Quit app and completing it accounted for two interactions. Likewise, self-monitoring mood levels in QuitGuide and selecting a tip to manage it accounted for one interaction. Note that the app interactions metric is more conservative than “screen clicks” which can be highly dependent upon app design (see Supplementary Table 1 for a description of how interactions were coded in our background analytics). Second, we measured daily duration of app use. We used the “event” function in Google Analytics to track how long (in seconds) the app was used on a given day. This metric discards idle time (ie, duration was not measured unless some element on the screen was clicked) and therefore is also a conservative metric of app engagement.³¹

Smoking Behavior Measures

We measured smoking history at baseline by collecting self-reported years of smoking and number of cigarettes smoked per day. Nicotine dependence at baseline was measured using the Fagerström Test for Nicotine Dependence.³² Smoking abstinence was measured at each timepoint with biochemically verified 7-day point prevalence abstinence (PPA) breath tests using the piCO+ Smokerlyzer with a breath CO cutoff <5 parts per million at the timepoint.³³ At trial endpoint, we also measured continuous 30-day PPA with CO verification. Change in smoking behavior was measured with self-reported CPD from baseline to trial endpoint (week 16). Quit attempts were defined as self-reported 24-hour smoking abstinence and measured at each timepoint.

Psychiatric Functioning Measures

Psychiatric functioning was measured at each timepoint with the global severity index and the anxiety and depression subscales of the Brief Symptom Inventory.³⁴ The global severity index provides a measure of overall psychiatric distress. Scores on each subscale of the Brief Symptom Inventory range from 0 to 4 with higher scores indicating more psychiatric symptoms. Positive and negative symptoms were measured with the Positive and Negative Syndrome Scale (PANSS).³⁵ The negative and positive symptoms subscales of the PANSS have scores that range between 7 and 49, with higher scores indicating more psychopathology.

Cognitive Functioning

We assessed cognitive functioning at baseline given its potential impact on app usability and engagement. For that purpose, we used the Brief Assessment of Cognition,³⁶ a cognitive task developed for patients with SMI that provides a global index of cognitive performance. Scores are presented in standardized units, with every unit increase indicating one standard deviation above the population average.

Smoking-Related Experiential Avoidance

Learn to Quit is a theory-based app that targets reductions in smoking-related experiential avoidance,¹⁴ a key process that has been linked to smoking cessation.³⁷ Therefore, we used the Avoidance and Inflexibility Scale³⁷ to measure this psychological process at each timepoint. The Avoidance and Inflexibility Scale is a 13-item scale with scores that range from 13 to 65, with higher scores indicating higher levels of smoking-related experiential avoidance.

Data Analytic Strategy

We extracted baseline characteristics by arm and examined them descriptively to evaluate the results of our stratified randomization. Specifically, the Learn to Quit arm had higher number of participants with employment disability, a schizophrenia spectrum disorder, and psychotic symptoms (Table 1). These factors, which are markers of SMI severity, have been theoretically linked to low engagement with smoking cessation treatment in patients with SMI²⁸ and may have favored the QuitGuide condition. Additionally, Learn to Quit participants had higher baseline levels of CPD. Therefore, we followed the simpler and conservative approach of not adjusting for these variables for our primary analysis and calculated an adjusted model only for continuous 30-day PPA at trial endpoint.

Table 1.

Baseline Characteristics

Variables	Learn to Quit	QuitGuide
N	33	29
Demographics
Age (M, SD)	46.1 (11.3)	45.6 (10.9)
Female	64% (21)	55% (16)
Ethnicity
Hispanic/Latino	0	0
Not Hispanic/Latino	100% (33)	97% (28)
Unreported	0	3% (1)
Race
White	49% (16)	55% (16)
Black/African American	42% (14)	38% (11)
Asian	3% (1)	0
American Indian or Alaska Native	0	3.5% (1)
Multiracial	6% (2)	3.5% (1)
Education
Secondary school or less	33% (11)	35% (10)
Some college	27% (9)	27.6% (8)
Associate’s degree	12% (4)	17.2% (5)
Bachelor’s degree or higher	27% (9)	20.7% (6)
Employment disability	58% (19)	31.0% (9)
Household income (<$35 000)	73% (24)	62.1% (18)
Smartphone ownership	83% (24)	90% (18)
Diagnostic features
Psychiatric diagnosis
Recurrent major depression	24% (8)	31% (9)
Bipolar I or II	45% (15)	52% (15)
Schizophrenia spectruma	30% (10)	17% (5)
With psychotic symptoms	45% (15)	24.1% (7)
Psychiatric and cognitive functioning	M, SD, Range	M, SD, Range
Brief Symptom Inventory
Global Severity Index	1.1, 0.7, 0–2.9	0.92, 0.6, 0.03–2.2
Anxiety	1.19, 0.9, 0–3.17	0.89, 0.7, 0–2.5
Depression	0.96, 0.8, 0–3	0.86, 0.5, 0–2.3
Positive and Negative Syndrome Scale
Positive symptoms	11.9, 3.5, 7–20	10.1, 2.8, 7–18
Negative symptoms	10.8, 3.6, 7–19	10, 3.7, 7–21
Years of mental health treatment	21.8, 14.3, 1–49	18.6. 10.8, 4–51
Lifetime hospitalizations	3.2, 3.1, 0–12	5.3, 10.3, 0–50
Brief Assessment of Cognition	−1, 1.3, −3.7 to 1.7	−0.9, 1.34, −5.2 to 1.2
Smoking
Cigarettes per day	21, 15.5, 5–60	14, 6.4, 5–30
Years smoking cigarettes	25.6, 12.9, 3–58	26.8, 11.3, 3–46
Nicotine dependencec	5.2, 2.6, 1–10	4.7, 2.3, 1–10
Smoking-related avoidanced	50, 8.9, 31–65	47.8, 7.5, 35–60

SD = standard deviation. Unless otherwise specified, data are presented as % (N).

^aSchizophrenia and Schizoaffective Disorders.

^bBrief Assessment of Cognition.

^cFagerström Test for Nicotine Dependence.

^dAvoidance and Inflexibility Scale.

Boxplots of app usability revealed that there were values that highly deviated from each group mean, affecting the accuracy of traditional ordinary least square methods.³⁸ Therefore, we conducted Yuen’s robust t tests with Wilcox’s robust Cohen’s d effect size method³⁹ to analyze this outcome. Reductions in CPD were extracted from a change score from baseline to week 16 and then regressed to each randomized condition in an unadjusted analysis of variance. Unadjusted models of change have been recommended by the literature.⁴⁰ However, as a secondary analysis and to assess for the robustness of this outcome, we calculated a model that adjusted for baseline levels of CPD. Intensive longitudinal data consisted of daily app interactions and duration of app use from day 1 to day 120 at the level of the individual (ie, 120 rows per individual per number of individuals). To account for both intrasubject and between-subject variability, we created multilevel models with two error terms, one for between-subject observations (u_0j), and one for between-subject observations nested within individuals over time (r_ij). The model’s dependent variable was app engagement (interactions or duration) and included arm assignment as the independent variable. The distributions of app interactions and duration of app use were negatively skewed and had large numbers of zeros (ie, the apps were not used on many days). Accordingly, we used the glmmADMB package in R to create multilevel zero inflated negative binomial models to analyze the effect of arm on each engagement outcome.⁴¹

The trial was not powered to reliably detect smoking abstinence outcomes; however, to aid interpretation, we calculated treatment arm differences in odds ratios and extracted 95% confidence intervals (CIs) at each timepoint. We conducted an intent-to-treat analysis, with all missing values considered as “smoking.” In cases where a treatment arm had 0% quit rates, we applied the Haldane–Anscombe correction.⁴² When applicable, treatment effects were converted to Cohen’s d to facilitate interpretation. All statistical analyses were conducted in R Language.

Results

From April 2017 to August 2018, we identified and enrolled a total of 92 participants in our trial (see Consort chart in Figure 1). In-person and clinical provider screenings confirmed eligibility for 76 patients, 63 of whom (83%) were randomized to the study intervention. One participant was withdrawn from the study postrandomization due to incompatibility of C-NRT with a new psychiatric medication prescribed by their provider. Therefore, the final randomized sample included 62 participants (Learn to Quit = 33; NCI QuitGuide = 29). See Table 1 for a summary of baseline characteristics.

Figure 1.

Participant flow diagram. EHR = Electronic Health Record; PI = Principal Investigator; SMI = serious mental illness.

Participant Safety

The Learn to Quit arm had a mean of 1.12 (standard deviation [SD] = 1.14) intervention “related” or “possibly related” adverse events (AEs) per participant, whereas the QuitGuide group had a mean of 1.24 (SD = 1.38). This difference was not statistically significant (t = −0.370, p = .712). A closer look at AEs indicated that 87% of them were linked to C-NRT use. There were two AEs related to smartphone use (advertisement pop-ups, general phone use issues), both of which were mild and resolved through technical coaching by the research staff. No serious AEs were determined to be related to the study intervention.

Trial Feasibility, App Usability, and App Engagement

The trial reached 102% of its enrollment target (92/90), retained 80% of all positively screened participants (61/76), and 98% of those who were randomized to the intervention (61/62). The usability scores of the Learn to Quit app were 7 points higher than QuitGuide, with an average in the A+ range for Learn to Quit compared to an average score in the B+ range for QuitGuide (Yuen’s t = 2.06, M_dif = 8.29, 95% CI = 0.13 to 16.4, p = .046).

Participants used the app for a similar amount of days in each condition (p = .754). However, background analytics of user engagement (total app interactions = 17,028; total duration of app use = 178.5 hours) indicated that number of app interactions and duration of app use were significantly higher in Learn to Quit compared to QuitGuide, with a risk ratio equivalent to a small Cohen’s d effect (d = 0.34) in number of interactions and a small Cohen’s d effect (d = 0.41) in duration of Learn to Quit use compared to QuitGuide (Table 2).⁴³ Note that due to a required a software update in the QuitGuide app (to Version 2.16), duration of app use was missing from 11 out of 29 participants in this arm. This software update, however, was not linked to participant characteristics and did not change the appearance, structure, or function of the QuitGuide app. Therefore, this mechanical failure was unrelated to participants themselves and the independent variable (ie, the functional components of the app), making the missing data Missing Completely At Random, and “ignorable” from a statistical perspective.⁴⁴

Table 2.

Primary and Secondary Outcomes of Learn to Quit’s Pilot Randomized Controlled Trial

	Learn to Quit	QuitGuide	Effect	p-value
Usability and engagement	M (SD)	M (SD)
System Usability Scale	85.2 (15.5)	78.4 (16.5)	Cohen’s da = 0.53	.046*
Days used	34.1 (27.7)	32 (24.5)	Cohen’s d = 0.08	.754
App interactionsb	335 (303)	205 (172)	RRc = 1.8 (1.2, 2.6)	.001*
App duration in hoursb	4.24 (4.29)	2.14 (1.72)	RR = 2.1 (1, 4.3)	.044*
Technical coaching (% Complete)	90.2% (3.6/4)	96.5% (3.9/4)	Cohen’s d = 0.4	.127
Smoking behavior	M (SD)	M (SD)	Cohen’s d
Reductions in CPD	12.3 (11.5)	5.9 (5.9)	0.68	.010*
Quit attempts	5.30 (4.57)	8.96 (6.18)	0.68	.009*
C-NRT adherenced	M (SD), Percent	M (SD), Percent
Days of patch use	21.3 (10.8), 36%	28.8 (12.2), 48%	0.13	.625
Days of lozenge use	6.5 (9.5), 92%	8.5 (10.2), 120%	0.03	.914
Smoking abstinencee			OR (95% CI)
30-day PPAf, week 16	12.1% (4/33)	3.4% (1/29)	3.86 (0.41, 36)	.239
30-day PPA, week 16 (adjustedg model)			8.76 (0.78, 97.8)	.077
7-day PPA, week 4	9.1% (3/33)	17.2% (5/29)	0.48 (0.10, 2.21)	.347
7-day PPA, week 8	18.2% (6/33)	0% (0/29)	13.9 (0.75, 259)	.077
7-day PPA, week 12	9.1% (3/33)	6.9% (2/29)	1.35 (0.21, 8.7)	.752
7-day PPA, week 16	12.1% (4/33)	3.4% (1/29)	3.86 (0.41, 36)	.239

CI = confidence interval; C-NRT = Combined Nicotine Replacement Therapy; CO = carbon monoxide; CPD = cigarettes per day; OR = odds ratio; PPA = point prevalence abstinence; ppm = parts per million; RR = risk ratio. Unless otherwise specified all statistical models for continuous variables used Analysis of Variance. OR indicates odds ratio in logistic regression for binary variables.

^aCohen’s d and p-values are based on robust estimation methods.

^bGroup means were extracted from averages of individual use over time and then from an average across individuals.

^cRR and corresponding 95% CI resulting from zero inflated negative binomial models of app engagement data.

^dPercent was calculated by dividing self-reported days of patch use from baseline to week 8 by 60 days (8 weeks), and days of lozenge use from baseline to week 8 divided by 7 days.

^eAll abstinence rates were biochemically verified (CO < 5 ppm).

^fPPA: point-prevalence abstinence rates.

^gThe model adjusted for baseline levels of CPD, employment disability, schizophrenia spectrum and psychotic symptoms.

*p values < .05.

A closer look at user engagement with the Learn to Quit app indicated that 61% of participants completed 100% of the modules (ie, 28), 76% completed at least 75% of the modules, and 85% completed at least 50% of the modules. The rate of completion of lesson versus skills modules was similar. More specifically, 67% of participants completed all lesson modules, and 61% all skills modules. Since engagement with Learn to Quit was encouraged through gamification and tokens, we examined repeated practice of app modules. Background analytics indicated that 52% of all modules were practiced at least twice, 38% at least three times, and 21% at least four times.

Smoking Reductions, Quit Attempts, and Smoking Abstinence

Participants randomized to the Learn to Quit app had statistically significant reductions in self-reported average CPD compared to QuitGuide from baseline to week 16 (b = 6.34, 95% CI = 1.63 to 11.05, p = .01). However, a secondary analysis adjusting for baseline levels of CPD indicated that the unadjusted analysis should be interpreted with caution (b = 2.01, 95% CI = −1.45 to 5,47, p = .260). Participants made more quit attempts in the QuitGuide arm compared to the Learn to Quit arm, and this difference was large and statistically significant (b = −3.66, 95% CI = −6.40 to −0.92, p = .009).

At week 16, we observed biochemically confirmed 30-Day PPA rates favoring the Learn to Quit app. In a secondary analysis of biochemically confirmed 30-Day PPA at trial endpoint, we calculated the results of a statistical model adjusting for baseline levels of CPD, employment disability, schizophrenia spectrum, and psychotic symptoms. The model generated an estimate that approached statistical significance (odds ratio = 8.76; 95% CI = 0.785 to 97.85; p = .077). Seven-day PPA rates at week 4 favored QuitGuide, with an 8% abstinence difference compared to Learn to Quit. However, in the remaining follow-up assessments (weeks 8, 12, and 16), abstinence rates were higher in the Learn to Quit versus the QuitGuide arm (Table 2). The diagnoses of the abstainers at week 16 were recurrent Major Depressive Disorder (n = 2) and bipolar disorder (n = 2) for Learn to Quit and recurrent MDD (n = 1) for QuitGuide.

C-NRT Adherence

Across arms, the patch was used 42% of all required days (8 weeks), whereas lozenges were used 106% of all required days (7 days). Between arms, there were no significant differences in nicotine patch and lozenge use. Out of the four participants in the Learn to Quit arm who successfully quit at the 16-week endpoint, two used the patch for 25 or more days for at least 1 month. The remaining two participants did not use the patch or used it only a few times. The participant in the QuitGuide arm who successfully quit at the 16-week endpoint used the patch for 20 days during 1 month.

Changes in Psychiatric Functioning

Differences between baseline and endpoint assessments indicated that there were small reductions in symptom severity, anxiety, and depression as measured by the Brief Symptom Inventory. Conversely, there were increases in positive symptoms and a statistically significant increase in negative symptoms as measured by the PANSS.

Between-group differences across psychiatric scales varied. The Learn to Quit arm had symptom reductions in 3 out of 5 psychiatric subscales, whereas the QuitGuide arm had symptom increases in 4 out of 5 symptom scales (Table 3). None of these between group differences were statistically significant.

Table 3.

Change in Psychiatric Functioning and Smoking Related Avoidance From Baseline to 4 Months Between Treatment Arms, and for the Full Sample

	Between Armsa			Full Sampleb
	Learn to Quit	QuitGuide	p-value		p-value
BSIc
Global Severity Index	−14.5% (0.57)	−2.2% (0.54)	.782	−8.93% (0.55)	.503
Anxiety	−3.5% (0.65)	10.7% (0.79)	.440	−6.26% (0.71)	.705
Depression	10.2% (0.83)	7.7% (0.64)	.618	−1.54% (0.74)	.920
PANSSd
Negative Symptoms	10.7% (4.3)	17.9% (4.3)	.945	14% (4.3)	.029*
Positive Symptoms	−2.1% (3.37)	12.2% (4.09)	.472	4.5% (3.6)	.460
AISe
Total Score	−13.4% (12.3)	−8.3% (13.6)	.644	−11% (12.8)	.019*

AIS = Avoidance and Inflexibility Scale; BSI = Brief Symptom Inventory; PANSS = Positive and Negative Syndrome Scale; SD = standard deviation.

^aAnalysis of variance adjusted for baseline levels of psychiatric functioning.

^b t-tests.

^cBrief Symptom Inventory (M, SD).

^dPositive and Negative Syndrome Scale (M, SD).

^eAvoidance and Inflexibility Scale (M, SD).

*p values < .05.

Changes in Smoking-Related Experiential Avoidance

Differences between baseline and endpoint measures of experiential avoidance suggest there were statistically significant reductions across arms. These reductions were descriptively larger in the Learn to Quit arm but did not appear to be statistically significant when compared to QuitGuide.

Discussion

Results from this pilot randomized controlled trial indicate that Learn to Quit—an app designed for patients with SMI—had larger effects on app usability and engagement compared to QuitGuide across different engagement metrics. The study demonstrated that recruiting and retaining patients with SMI in mHealth randomized controlled trials is safe and feasible. While not the primary outcome of the study, the trial showed that Learn to Quit has promising efficacy as a smoking cessation intervention, with higher and sustained abstinence rates at a 4-month follow-up, and promising smoking reductions compared to QuitGuide. Finally, reductions in smoking-related experiential avoidance were consistent with Learn to Quit’s theory-based process of change.

We found a few unexpected outcomes in this trial. First, Learn to Quit had higher levels of usability than QuitGuide as measured by our self-report measure of usability, the System Usability Scale. However, QuitGuide obtained higher usability scores than those reported in our previous work, which indicated that the app had “C minus” scores in this population.^14,15 Therefore, while this study had a larger sample than our previous studies, more research is needed to establish whether the QuitGuide app has acceptable usability levels in patients with SMI. Overall, however, background analytics of the QuitGuide app suggest that usability may be a necessary but not sufficient factor to increase sustained behavioral engagement with this app over time.

Second, despite higher abstinence rates in Learn to Quit at weeks 8, 12, and 16, abstinence rates at week 4 were higher for QuitGuide. While the small sample size of this pilot complicates the interpretation of these fluctuations in quit rates between groups over time, a possible explanation for this finding could be that QuitGuide’s design had a strong focus on taking immediate steps toward cessation. For example, QuitGuide encourages users to set up a quit date as soon as they start using the app. Conversely, Learn to Quit’s design does not give the option to set up a quit date until reaching module 9 (ie, “Committing to a quit date”), which in turn unlocks the “set a quit date” feature. In addition, QuitGuide’s homepage regularly encourages the user to seek abstinence (eg, the “I was Smokefree Today!” button). In our previous evaluation of the QuitPal app in patients with SMI,¹¹ participants expressed their desire to have tailored smoking cessation content that gradually offered smoking cessation skills. In addition, they also requested skills to manage psychiatric symptoms and content that was delivered in an understandable and engaging manner. Therefore, the absence of these features in the control app could explain this lack of sustained results, and in turn, Learn to Quit’s sustained outcomes.

Third, quit attempts for the overall sample were on average higher than the general population (M = 1.6; SD = 5),¹⁹ but consistent with a previous report which indicated that patients with a mental health diagnosis are comparatively more likely to attempt quitting than smokers without a mental health diagnosis.⁴⁵ In addition, quit attempts significantly differed across groups, but unexpectedly, they favored the QuitGuide arm. As described earlier, QuitGuide’s content includes a series of features on the homepage that frequently encourage users to make quit attempts, while the Learn to Quit app has a strong focus on smoking cessation preparation. Overall these results are consistent with Learn to Quit’s app concept of “Learn, Practice, and Play” which encourages smokers to put cessation on the side for a while and prioritize the establishment of a behavioral smoking cessation repertoire.

Fourth, there was a statistically significant increase in negative symptoms from baseline to trial endpoint on the PANSS scale. This increase in negative symptoms is not consistent with a previous large trial evaluating the safety of smoking cessation treatments in psychiatric patients.⁴⁶ However, this 1.72-point increase in negative symptoms (14% increase) was not clinically significant, falling below the 34% change from baseline required for a Minimally Clinically Important Difference in this scale.⁴⁷ Further, because of the small sample size of this pilot study, caution should be taken in interpreting this specific finding.

Finally, there were no differences between groups in adherence to C-NRT, with both arms having overall low levels of patch use. Low adherence to the nicotine patch and lozenges is a common finding in smoking cessation trials, with adherence rates as low as 20% among patients with mental illness.⁴⁸ Although C-NRT was not the sole focus of Learn to Quit, the app’s attempt to increase patch adherence did not seem to produce better results than the control app. This suggests that interventions or app features designed to facilitate C-NRT use within SMI populations represent an important future direction.

The abstinence rates from this trial (12% vs. 3%) are comparable to the abstinence rates found in a large multisite trial in the period after NRT was discontinued among psychiatric patients (13% vs. 8%).⁴⁶ Yet, this digital intervention has the potential to reach a greater proportion of the SMI population. In addition, this study is consistent with previous smoking cessation app research in different types of SMI populations. For example, Minami et al.⁴⁹ developed SMI-CM, a smoking cessation app for patients with mood disorders and tested its feasibility in a small sample. SMI-CM used a mindfulness and acceptance approach combined with contingency management to reinforce CO verified abstinence twice a day. Results suggested that the app was positively perceived by patients and led to promising smoking reductions. Similarly, Wilson et al.⁵⁰ conducted early-phase work to evaluate the Stay Quit Coach App in combination with contingency management in a small sample of patients with psychotic disorders. Two successive cohorts of patients were used to refine the intervention, which showed promising smoking reductions. This pilot work, using a larger sample and an active control, provides strong evidence about the possibility of developing usable and engaging mHealth technologies in patients with both psychotic and severe mood disorders, underscoring the potential of mHealth for dissemination of smoking cessation interventions in this group.

To our knowledge, this is the first trial that objectively measured duration of user engagement with a smoking cessation app. Previous trials have typically reported number of days of app use (ie, logins),⁵¹ which, as shown in our data, can present a limited picture of engagement with digital technology. Our trial showed that an analysis limited to number of days of app use may have concluded that both interventions were equally engaging. However, using these refined background analytics metrics, the trial found higher rates of engagement with the digital intervention throughout the 4-month period compared to the control condition. The presence of technical support during the first month and subsequent monthly contacts with participants may have contributed to these rates. While substantial app interactions and duration of app use may be important in the initial phases of a digital intervention, future studies might still benefit from a days of use metric to demonstrate sustained engagement with a digital technology over time and their potential relation to treatment outcomes. On the whole, the study showed that the use of a broad range of engagement metrics seems to be an important approach to better understand user engagement.

Our technical support approach was informed by the supportive accountability model,⁵² which may be particularly beneficial in SMI populations. The rates of completion of technical coaching sessions were high, which might be explained by the fact that all participants were expected to complete all technical coaching sessions as part of study participation. In addition, note that most participants owned a smartphone (see Table 1) and were familiar with smartphone technology. Given general trends in smartphone technology adoption, familiarity with smartphone technology is likely to increase in the future in this population. Therefore, future studies will need to determine whether Learn to Quit requires technical assistance or if it could instead be used as a standalone technology in this population.

Study accrual was as challenging as in other trials in this population.⁵³ However, the fact that we included participants with both mood and psychotic disorders and the convenience of enrolling in a mHealth intervention that required minimal in-person contact may have facilitated these rates (ie, 102%). Finally, retention rates were promising (98% postrandomization). Our staggered approach to enrollment (ie, one in-person visit to inform and orient participant, and a second in-person visit for randomization) led to the majority of study dropouts prior to randomization. This approach is consistent with previous behavioral health trials, which use this strategy to ensure the randomized sample is committed to participating in the study. Other factors, such as flexibility in our methods of follow-up assessment to address transportation barriers (ie, in-person, telephone, web survey), may have also contributed to this retention rate.

This pilot study had a number of limitations. First, our measure of C-NRT adherence relied on a single item of medication adherence that had been used in previous drug research.²⁹ However, given retrospective bias and deficits in cognitive functioning among patients with SMI,⁵⁴ a more objective approach (eg, collecting used patches) would have increased the reliability and validity of this measure. Despite this, the NRT adherence rates are not far off from previous reports in the literature, which suggest rates in patients with SMI as low as 20%.⁴⁸ Second, our method of stratification by primary psychotic or mood disorder did not successfully balance the presence of more severe psychiatric diagnoses across groups. This could be due to the fact that stratification was based on a psychiatric interview (Mini International Neuropsychiatric Interviews) that was used for cross-validation, which did not always capture the presence of psychotic symptoms in patients with ongoing psychiatric treatment (as opposed to reports based on providers’ clinical experience). Third, due to transportation barriers, 26% of the outcome measures were collected by telephone or survey. However, biochemical verification was obtained from all participants who self-reported abstinence at trial endpoint. These data suggest that remote collection of outcomes in this population is feasible and helps to increase retention outcomes. Finally, software changes in the QuitGuide app led to missing data on the duration of app use variable for about one-third of our sample. However, such mechanical failure was unrelated to participants themselves or to the nature of the intervention; therefore, from a statistical standpoint, this type of data missingness represented an “ignorable” threat to the validity of our analysis.⁴⁴ Further, our use of intensive longitudinal methodology which combines individual and group trajectories provided enough statistical power to detect differences between groups, increasing confidence in the obtained results.

This is the first pilot randomized controlled trial of a smartphone smoking cessation app designed for patients with SMI that demonstrates that mHealth technology can be potentially used to reduce tobacco use in this population. The trial demonstrated that a tailored theory-based smoking cessation app is more usable and engaging compared to a nontailored active intervention that rigorously controlled for time and attention. A randomized controlled trial powered to reliably detect smoking cessation outcomes is now needed to evaluate the efficacy of this novel digital intervention in a larger sample of the population.

Funding

Funding for this study was provided by the National Institute on Drug Abuse (R00 DA037276).

Declaration of Interests

Paolo Mannelli is a consultant to Alkermes Inc, Guidepoint Global, and has received research support from Alkermes Inc, and Orexo. Francis Joseph McClernon owns a company that provided smoking cessation consulting and market research services to GSK Consumer Health in the last 3 years.