Engagement with a digital therapeutic for smoking cessation designed for persons with psychiatric illness fully mediates smoking outcomes in a pilot randomized controlled trial

Abstract

Understanding the mechanisms of change of digital therapeutics is a critical step to improve digital health outcomes and optimize their development. Access to and engagement with digital content is arguably a core mechanism of change of these interventions. However, the mediational role of app engagement has been largely unexamined. To evaluate the mediational effect of engaging with a digital therapeutic for smoking cessation designed for adults with psychiatric disorders. Secondary analysis of a pilot clinical trial of 62 adults with serious mental illness who were randomized to receive either a tailored digital therapeutic (Learn to Quit) or a digital therapeutic for the general public (NCI QuitGuide). Engagement was captured using background analytics of app utilization, including (a) number of interactions with app content, (b) minutes/day of app use, and (c) number of days used. The main outcome was reductions in cigarettes per day from baseline to the four-month endpoint. Mediational analysis followed the Preacher and Hayes bootstrap method. Number of application interactions fully mediated reductions in cigarettes per day in the Learn to Quit application but not in QuitGuide (Average Causal Mediation Effect = .31, p = .02). Minutes/day of app use played an uncertain role, and number of days used was not a significant mediator. Results suggest that one of the mechanisms of action of the Learn to Quit device, engagement with theory-based content, functioned as intended. Future research of digital therapeutics should emphasize granular approaches to evaluating apps’ mechanisms of action.

Adults with serious mental illness who frequently interacted with a tailored smoking cessation smartphone application were more likely to reduce their cigarette smoking.

Implications.

Practice: Clinicians should recommend tailored digital interventions (e.g., smartphone apps) with clear evidence of a link between engagement and improved outcomes for individuals with psychiatric disorders.

Policy: Reducing barriers to adoption and implementation of effective and tailored digital therapeutics in persons with psychiatric disorders may have benefits for this population.

Research: Future research on digital therapeutics should examine the impact of more granular metrics of engagement on health outcomes.

Introduction

Digital health interventions, such as smartphone applications, are becoming more widely recognized as a critical component of mental health treatment. Accumulating evidence has demonstrated that smartphone applications are effective in treating mental health challenges, particularly anxiety [1] and depression [2], and eliminate several barriers involved in receiving traditional face-to-face care (e.g., transportation). In light of the clinical benefits of these tools and their potential to offer treatment more conveniently, expanding their use to populations that are difficult to engage, such as those with serious mental illness (SMI) [3, 4], maybe especially important to address this significant health inequity. This could be particularly important in the current era of the COVID-19 pandemic when face-to-face care is even less available [5, 6].

Smartphone applications developed for persons with SMI have demonstrated benefits in improving symptoms, community functioning, and motivation as well as reducing smoking [7–10]. Further, studies of smartphone applications for this population have generally reported high usability, feasibility, and satisfaction ratings [11]. Ben-Zeev and colleagues (2018) conducted a randomized trial of a smartphone-delivered intervention versus a face-to-face intervention in 163 persons with SMI and compared the engagement rates of participants in these two conditions. They found that participants randomized to the smartphone condition were more likely to fully engage (defined as using the smartphone app on 5/7 days in a week) than those assigned to the face-to-face intervention (defined as attending 60/90 min of a weekly session) during 8 weeks of treatment [12]. As such, smartphone applications appear to be valuable and usable for persons with SMI and may be especially effective for facilitating engagement.

While some psychological processes have been proposed as important mechanisms of change in psychotherapy (e.g., self-efficacy) [13], at its core, engagement with an intervention, or the simple act of dedicating one’s attention to address a behavioral problem, has been argued to be one of its foundational mechanisms of change [14–16]. Therefore, to promote initial and sustained uptake of these tools for persons with SMI, it is vital to evaluate the role of engagement in producing the desired health outcomes. But, this task is complicated by substantial heterogeneity in how engagement in smartphone applications is measured across studies [17]. Further, there are limitations of the more commonly used measures, namely the number of clicks or logins. Specifically, these measures may be helpful to initially characterize user behavior; however, they often fail to capture important information about how people are using the application (e.g., type of content accessed, sustained attention to application content; [18, 19]). For example, a person that quickly sorts through all of the various offerings available within an application may have a high number of clicks but may not have dedicated enough time to comprehend the app content or fully complete any app features. As such, functionally defining engagement within smartphone application studies is a critically important first step in assessing its impact on treatment outcomes.

The present study sought to examine the role of smartphone application engagement on outcomes in a sample of persons with SMI using a functionally defined measure of user engagement and additional metrics of user engagement with different levels of granularity. This paper is a secondary analysis of a pilot randomized trial that evaluated Learn to Quit, a novel smartphone application targeting tobacco use disorder that was tailored for the SMI population as compared to QuitGuide, a smartphone application developed by the National Cancer Institute (NCI) targeting smoking used in the general population [10]. The conceptual model that drove the development of this app argued that to effectively address tobacco use disorder among SMI populations, the contents of a digital therapeutic had to include (a) psychological content addressing co-morbid psychiatric symptoms, and (b) design elements that addressed known treatment engagement barriers in this population, such as low educational attainment and cognitive deficits [20]. Results from this trial demonstrated the promising effects of Learn to Quit as compared to QuitGuide on reductions in self-reported cigarettes per day at the 4-month endpoint [10]. These positive findings are notable given the alarmingly elevated rates of smoking and associated treatment challenges among those with SMI [21–25]. However, evaluating whether these reductions in cigarettes per day resulted from a functional and multidimensional approach to defining engagement with a smartphone application would validate the conceptual model that drove the design of this app, and inform future treatment development efforts of digital therapeutics for other underserved populations.

The specific goals of the present study were to evaluate the extent to which (a) number of functionally defined app interactions (i.e., content-based actions specific to the application), (b) minutes/day of app use, and (c) days of app use (i.e., number of days where participant logged in at least once), mediates the relationship between group assignment and reduction in cigarettes. Consistent with the mechanisms of change literature described above [14–16], we predicted that targeted and sustained attention to the app (which would be reflected in interactions and minutes but not days) would be necessary for the target behavior to change. Therefore, we hypothesized that the number of interactions and minutes/day of app use, but not days of app use, would mediate the relationship between group assignment and reduction in cigarettes at trial endpoint. Finally, an exploratory goal was to evaluate the extent to which (d) average duration of app interactions (i.e., not just the total amount of interactions but the amount of time spent engaging in them) mediated the relationship between group assignment and reduction in cigarettes.

Methods

Participants

Participants were identified through electronic health records and coordination with healthcare and smoking cessation programs. Participants were eligible if they: (a) had an ICD-10 diagnosis of an SMI (schizophrenia, schizoaffective disorder, bipolar disorder, or recurrent major depressive disorder), (b) self-reported smoking five or more cigarettes per day and had a carbon monoxide breath test of more than six parts per million, (c) wanted to quit smoking in the next 30 days, (d) were ≥18 years old, (e) willing and medically eligible to use nicotine replacement therapy, (f) were fluent in spoken and written English, (g) were adherent to psychiatric treatment, and (h) were living in stable housing. Individuals were excluded if they (a) had problematic alcohol or illicit drug use in the last 30 days, (b) had an acute psychotic episode or were unsafe to participate in the study, (c) were pregnant or had the intention to become pregnant, or (d) were currently receiving smoking cessation treatment.

Interventions

Participants received one of two smartphone applications: Learn to Quit or QuitGuide. Learn to Quit is a novel smartphone application that was specifically designed for persons with SMI and co-occurring tobacco use. This application is comprised of 28 modules that provide knowledge, skills, and recommendations for smoking cessation based on acceptance and commitment therapy and the US Clinical Practice Guidelines. Modules were designed to facilitate knowledge acquisition (“lessons”) and implementing the knowledge into daily life (“skills”). A thorough description of this application has previously been published elsewhere [20]. QuitGuide is a smartphone application developed by the NCI that includes health information about smoking, tracking tools (e.g., for cravings), and advice for quitting. QuitGuide information can be found at www.smokefree.gov.

In addition to one of the smartphone applications, all participants were provided with 8 weeks of nicotine replacement therapy (transdermal nicotine patch and lozenges), which was monitored by a study physician (see Vilardaga et al., 2019 for more details).

Measures

Only measures included in this secondary analysis are described in this section (see Vilardaga et al., 2019 for a full description of all measures collected in the RCT). App interactions were defined as the total number of interactions completed. Interactions were functionally defined by capturing engagement with content-based actions specific to each app and were pre-specified based on the presumed active ingredients of the applications. Interactions were also weighted such that starting, completing, or abandoning a module counted for the same amount of interactions regardless of the number of clicks required to complete those actions (Table 1). Minutes/day of app use was measured by capturing the number of seconds that participants spent using the different app features each day. We aggregated the total number of seconds per day of app utilization and converted it to minutes. Days of app use was measured by aggregating the total number of days that participants logged into the application. These three measures were automated and objectively measured with Google Analytics. Change in self-reported cigarettes per day from baseline to endpoint was examined as the primary smoking outcome. Finally, average duration of app interactions was calculated by dividing the total number of interactions by the total number of minutes of app use and used to evaluate the exploratory aim. As noted in the main outcome paper [10], minutes/day of app use data were missing for a small subset of participants randomized to QuitGuide (n = 11) due to a software update of this app and it was determined that the missing data were missing completely at random (MCAR).

Table 1.

| Sample of interaction coding in Learn to Quit and NCI QuitGuide apps

	Action	Number of clicks	Number of interactions
Learn to Quit
Learning modules and quiz	Completed learning module + quiz	16–27	Started: 1 Finished: 1 Completed quiz: 3 Total: 5
	Uncompleted learning module	Minimum: 4	Started: 1 Abandoned: 1 Total: 2
Skills modules	Completed skills module	9–19	Started: 1 Finished: 1 Total: 2
	Uncompleted skills module	Minimum: 4	Started: 1 Abandoned: 1 Total: 2
Check-in	Anytime “check-in”	1–5	Each item: 1 Total: up to 5
	End of day “check-in”	1–5	Each item: 1 Total: up to 5
NCI QuitGuide
Tracking	Tracking mood	4	1
	Tracking slips	5	1
	Tracking cravings	8	1
Resources	Quitting help	2	1
Statistics	View stats	4	1

A previous version of the above table was previously published in the supplementary material in Vilardaga R, Rizo J, Palenski PE, Mannelli P, Oliver JA, Mcclernon FJ. Pilot randomized controlled trial of a novel smoking cessation app designed for individuals with co-occurring tobacco dependence and serious mental illness. Nicotine Tob Res. 2019;22(9):1533–1542.

Procedure

All procedures were approved by the Duke University Institutional Review Board and all participants provided written informed consent. Participants were randomized 1:1 to Learn to Quit or QuitGuide and were stratified by diagnosis (psychotic disorder vs. mood disorder). All participants received a smartphone with a phone, text, and data plan as well as up to four telephone or in-person meetings with study staff to assist with smartphone-related tasks. Assessments were conducted at baseline and at four follow-up timepoints (4, 8, 12, and 16 weeks after randomization). Participants could earn $110 and retain their smartphones if they completed all measures.

Data analysis

Bivariate correlations were initially conducted among all engagement metrics and the primary outcome of change in self-reported cigarette use. Before mediation analyses, correlations were run between all variables of interest. A series of four causal mediation models examined (a) number of app interactions, (b) average minutes per day of app use, (c) total number of days of app use, and (d) average duration of app interactions as potential mediators between treatment group assignment (independent variable; IV) and change in self-reported cigarette use from baseline to four-month endpoint (dependent variable; DV). Given that this causal mediation analysis was embedded in an experimental design that used randomization to methodologically mitigate treatment effect confounders, no covariates were included in the models. For all four models, a linear model with the IV predicting the DV was conducted followed by a linear model with IV predicting DV, controlling for the mediator variable (see Fig. 1 for a visual representation of all the models). As a formal test of mediation, n = 1,000 Monte Carlo draws (i.e., bootstrap estimates) repeatedly simulated comparison between each linear model to test the significance of the mediator effect (i.e., indirect effect) based on the methods of Preacher and Hayes [26]. A direct effect was calculated as the effect of the IV on DV, adjusting for the mediator. A total effect was calculated by combing indirect and direct effects. Sensitivity analyses were conducted with significant mediators to assess the degree to which there may be bias due to residual confounding of the mediator-outcome effect [27, 28]. Mediation analyses were conducted with both treatment groups in the same analysis rather than stratifying analyses by treatment group to allow for direct comparison of the findings by treatment condition (i.e., to determine whether mediation occurred in one intervention vs. other), which is consistent with the parent study rationale and hypotheses. Analyses were conducted in R using the mediation [29] and lavaan [30] packages.

Fig 1.

Mediation models. This figure presents the results of all mediation models. Each model includes a predictor (treatment group), a mediation effect, and the smoking outcome (change in cigarette use). Arrows reflect the direction of relationships entered in the model. Numerical values along the arrow reflect the strength of relationships between variables using standardized relationship coefficients. Numerical values in parentheses along the arrow reflect the strength of relationship between the predictor and outcome before the mediation variable was added to the model. Single asterisks reflect values are significant at the p < .05 level while double asterisks reflect values are significant at the p < .01 value. Data were missing from 11 participants for models B and D.

Results

Demographics and clinical characteristics

As noted in Table 2, the sample was diverse (e.g., close to 50% was comprised of individuals from racial minority backgrounds) and was balanced in terms of levels of education and smoking history. The Learn to Quit arm had a higher number of individuals with schizophrenia spectrum disorders compared to the QuitGuide arm (i.e., 30% vs. 17%).

Table 2.

| Demographic, clinical, and engagement characteristics

Characteristics	Learn to Quit (n = 33)		QuitGuide (n = 29)
Categorical variables	n	%	n	%
Gender, n (%)  Female  Male	21 12	64% 36%	16 13	55% 45%
Race, n (%)  White  Black of African American  Asian  American Indian or Alaskan Native  Multiracial	16 14 1 0 2	49% 42% 3% 0% 6%	16 11 0 1 1	55% 38% 0% 3.5% 3.5%
Ethnicity, n (%)a  Hispanic or Latinx  Not Hispanic or Latinx	0 33	0% 100%	0 28	0% 100%
Primary diagnosis, n (%)  Schizophrenia spectrum disorder  Bipolar disorder  Major depressive disorder	10 15 8	30% 45% 24%	5 15 9	17% 52% 31%
Education level  Some or completed high school  Some college  Associate’s degree  Bachelor’s degree or higher	11 9 4 9	33% 27% 12% 27%	10 8 5 6	35% 27% 17% 21%
Continuous variables	M	SD	M	SD
Age (years)	46.1	11.3	45.6	10.9
Smoking information  Baseline cigarettes/day  Baseline years smoking cigarettes  Change (reductions) in cigarettes/day from Baseline to 16-week endpoint	21 25.6 12.3	15.5 12.9 11.5	14 26.8 5.9	6.4 11.3 5.9
Engagement metrics  Number of interactions  Number of days used  Minutes/day of app useb  Average duration of interactions (minutes)	335.5 34.1 228.07 1.6	303.9 27.7 247.68 0.8	212.7 32.0 129.38 2.3	170.5 24.5 102.99 1.1

^aOne participant in QuitGuide did not report ethnicity so percentage is calculated out of n = 28.

^bData missing from 11 participants.

Correlational analysis

All three primary engagement indicators were significantly and highly correlated with each other (rs range: .70–.89) and were all significantly and moderately correlated with cigarette reduction (rs range: .42–.50). The exploratory engagement indicator of the average duration of app interactions was significantly correlated with minutes/day of app use (r = −.25) but not with cigarette reduction or other engagement metrics (Table 3).

Table 3.

| Correlations among engagement and outcome variables

	1	2	3	4	5
1. Treatment group
2. Change in cigarette use	.39*
3. Number of app interactions	.30*	.50*
4. Minutes/day of app use	.28*	.42*	.87**
5. Days of app use	.05	.44*	.89**	.70**
6. Average duration of app interactions (minutes)	−.44*	.00	.03	−.25*	.16

**p < .01, *p < .05.

Causal mediation

Number of app interactions

The relationship between the treatment group and changes in cigarette use was mediated by total number of app interactions. As Fig. 1A illustrates, the relationships between treatment group and total application interactions (β = .28, p = .02) and total application interactions and change in cigarette use (β = .44, p < .01) were both significant. The relationship between the treatment group and change in cigarette use was no longer significant after total application interactions were included in the model (β¹ = .19, p = .10; β = .31, p = .02). The indirect effect (i.e., mediation effect) was significant (B = 2.41, 95% bootstrapped CI [0.27, 5.60], β = .12, p = .01), as was the total effect (B = 6.16, 95% bootstrapped CI [1.75, 10.79], β = .32, p < .01). Sensitivity analyses based on the correlated residuals method [31] suggested that the sensitivity parameter (ρ) would have to equal .40 (or explain 11% of the variance) for the true mediated effect to be zero.

Minutes/day of app use

In a subsample of participants with available data (n = 51), the relationship between treatment group and changes in cigarette use was not mediated by daily minutes of application use (Fig. 1B). The relationships between treatment group and daily minutes of application use (β = .24, p = .03) and daily minutes of application use and change in cigarette use (β = .38, p = .04) were both significant. The relationship between treatment group and change in cigarette use decreased in magnitude after daily minutes of application use were included in the model, and as expected neither relationships were significant (β¹ = .16, p = .13; β = .26, p = .08). The indirect effect had confidence intervals trending in the expected direction but inconsistently produced a significant effect in our bootstrap models. Conservatively we opted to report here the nonsignificant indirect effect (B = 1.96, 95% bootstrapped CI [−0.09, 4.84], β = .09, p = .06). The total effect was significant (B = 5.46, 95% bootstrapped CI [1.03, 9.93], β = .26, p < .01).

Days of app use

The relationship between treatment group and changes in cigarette use was not mediated by total number of days of application use (Fig. 1C). The relationships between treatment group and total number of days of application use (β = .07, p = .75) were not significant while the total number of days of application use and change in cigarette use (β = .42, p < .01) were significant. The relationship between treatment group and change in cigarette use was still significant after total application interactions were included in the model (β¹ = .28, p = .02; β = .31, p = .01). The indirect effect was not significant (B = 0.61, 95% bootstrapped CI [−1.28, 3.29], β = .03, p = .54), while the total effect was significant (B = 6.16, 95% bootstrapped CI [1.94, 11.00], β = .31, p < .01).

Exploratory mediation

Average duration of app interactions

In a subsample of participants with available data (n = 51), the relationship between treatment group and changes in cigarette use was not mediated by the average duration of interactions (Fig. 1D). The relationship between treatment group and average duration of interaction was significant (β = −.34, p = .03), while the average duration of interactions and change in cigarette use was not significant (β = .10, p = .64). The relationship between treatment group and change in cigarette use increased in magnitude after daily minutes of application use were included in the model (β¹ = .29, p = .02; β = .26, p = .08), indicating a suppression effect. The indirect effect was not significant (B = 0.72, 95% bootstrapped CI [−3.04, 2.69], β = .03, p = .61). The total effect was significant (B = 5.46, 95% bootstrapped CI [1.08, 9.86], β = .26, p = .01).

Discussion

The purpose of the present study was to evaluate the extent to which engagement with a digital therapeutic for smoking cessation mediates the relationship between group assignment and smoking reduction among persons with SMI who participated in a smoking cessation RCT. Results supported our hypothesis in that a functionally defined app engagement metric (number of functionally defined app interactions) but not days of app use fully mediated the relationship between group assignment and cigarette reduction. For minutes/day of app use, we chose to conservatively report the nonsignificant mediation model, which only provided preliminary evidence for partial mediation (i.e., trend level significance). However, in some of our bootstrapped models, this metric fully mediated the relationship between treatment assignment and cigarette reduction. These mixed results could be explained by the fact that this analysis included only a subsample of all participants. Finally, the interplay between app interactions and their duration was examined in an exploratory mediation analysis, suggesting that engaging users frequently was more important than engaging them for a longer time.

App interactions mediated the treatment group effect on reductions in smoking, suggesting that completing functionally defined interactions is a mechanism of change. Further, interactions fully mediated reductions in cigarettes per day in those randomized to the Learn to Quit application but not the QuitGuide application. While there is no clear consensus in the literature about how to interpret the size of confounder effects in mediation analysis [32], results of our sensitivity analysis indicated that the observed mediation effect of app interactions required a high correlation between residuals to invalidate the effect, and therefore that the mediated effect was unlikely due to an unobserved confounder. As such, this study suggests that Learn to Quit led to reduced smoking because of meaningful engagement with theory-based app content and not due to a more simple metric of number of days of app use. The result of this analysis also suggests that our early-phase research effort to develop a theory-based application and tailor it through a user-centered design approach in our target population [20, 33, 34] led to a digital therapeutic that functioned as intended. That is, changes in smoking outcomes were the result of access to and engagement with our tailored theory-based digital health content, and not with a non-tailored digital health control. Therefore, using the ORBIT model as a framework [35] may have been key to translate a novel concept into a promising health-related intervention [33].

To our knowledge, this is the first study to evaluate the mediational effect of multiple indicators of engagement on health outcomes in a sample of adults with SMI. As such, these findings have important clinical and methodological implications that can be applied within and beyond smoking cessation interventions for psychiatric populations.

This study highlights the value of emphasizing a granular approach to evaluating engagement with digital technologies to better understand their mechanisms of change. Focusing on a broad and single measure of user engagement (i.e., days of app use) was not a significant mediator in our study. It may be that the user-centered design and tailoring of the Learn to Quit application to the SMI population [20] elicited greater active participation and retention of the content, which facilitated smoking reduction, thereby explaining the lack of effects in the QuitGuide group. Although the number of days of app use was not a significant mediator in our study, it was significantly related to smoking reduction in the correlational analysis, which is consistent with prior literature on smartphone interventions for SMI [12]. Further, Zhang and colleagues (2019) examined relationships between three types of smartphone application use (learning, goal-setting, and self-tracking) and symptoms of depression and anxiety. They found that each type of use was associated with improved depression; however, learning and goal setting were only related to outcomes at moderate durations of use [36], thereby suggesting a more complex interplay between type and quantity of use. Taken together, there is a need to consider multiple aspects of engagement including functionally defined app interactions when designing and evaluating digital therapeutics for persons with SMI.

The differential results that emerged when considering more granular measures of user engagement versus more broadly defined metrics of usage underscore important methodological considerations for future work. There is limited consistency and agreement in the field surrounding how best to operationalize engagement in both digital and traditional face-to-face interventions for adults with psychiatric disorders [37, 38]. But, Short and colleagues (2018) have proposed using the FITT categories when measuring engagement, which includes frequency, intensity, time, and type [39]. Although this approach is often used in physical activity research [40], it can be applied to smartphone applications in that it allows for multiple measures of engagement including the number of logins (frequency), the number of application-specific actions taken (intensity), the length of time during each login (time), and the kind of actions taken such as active versus passive (type). Our results align with this model in that our measures of engagement spanned multiple domains (frequency, intensity, and time), which allowed for a more comprehensive and nuanced understanding of the impact of application use on smoking outcomes. Therefore, expanding the definition of engagement and examining its impact on health outcomes is critical to inform intervention design and evaluation.

This study had limitations that should be considered when interpreting the results. First, the sample size of this pilot trial was modest and was further reduced by some missing duration of use data (i.e., minutes/days of app use and average duration of app interactions were missing from 11 participants), thereby reducing power. Despite this, our use of a granular measure of user-engagement (i.e., continuous capture of user behavior during a 4-month period), provided enough power to conduct these analyses. Second, although there were no maximum number of interactions participants could complete in either smartphone application, Learn to Quit had a wider variety and greater number of screens than QuitGuide, which may have contributed to increased engagement in Learn to Quit. However, since cognitive deficits among individuals with SMI (e.g., working memory) have been associated with poor task performance [41], greater number of screens could have had the countereffect of reductions in treatment engagement. Therefore, consistent with the hypothesis of our original user-centered design work [20], the results suggest that it was the specific design features included in the app to address barriers to app use among SMI populations (e.g., mental health symptoms, low education attainment, and cognitive deficits) that mediated treatment outcomes, and not the sheer number of screens available in the app. Taken together, these findings highlight that providing tailored content that addressed those barriers was key to obtain the desired health benefit. Third, we focused on the quantity of interactions as our indicator of functional engagement and did not categorize the type of interactions, which precludes evaluation of the relative importance of different actions taken within the smartphone applications. Fourth, we explored the possible interplay between interaction frequency and interaction duration to better capture the relationship between these engagement indicators and outcomes. However, the results of this analysis may not be reliable due to the reduced sample size in one of the treatment arms. Fifth, despite randomization, treatment groups differed on baseline cigarettes/day, which may have influenced the causal mediation given the scope of change in cigarettes/day over the trial was larger in the Learn to Quit arm. Finally, due to the sample size of this pilot trial, this mediation model used smoking reductions—an important health outcome in this population [42]—and not smoking abstinence, which is typically considered a more robust health outcome.

In light of these limitations, future research should consider extending the present study with an even broader array of engagement indices (e.g., type) to simultaneously evaluate their impact on both smoking reductions and abstinence in a larger sample using multiple mediation analysis. Additionally, it would be valuable for future studies to examine how engagement with theory-based content might influence other key mechanisms of action (e.g., experiential avoidance) in behavioral interventions for smoking [13], possibly with the inclusion of a directed acyclic graph to illustrate the expected relationships between these multiple variables [43]. This type of analysis would provide further clarity on how the Learn to Quit intervention resulted in smoking reduction among persons with SMI.

The present study demonstrated that a functionally defined measure of engagement with a digital therapeutic significantly mediated the relationship between group assignment and smoking reduction in adults with SMI, suggesting that the purported mechanism of action of the digital device functioned as intended. As the first study of its kind, it adds to the literature on intervention engagement by supporting the need to consider more granular and functionally defined measurement constructs. Continued efforts to design, evaluate, and deliver effective digital therapeutics that adequately engage adults with psychiatric disorders are needed.

Compliance with Ethical Standards

Conflicts of Interest: All authors declare that they have no conflicts of interest.

Human Rights: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent: Informed consent was obtained from all individual participants included in the study.

Welfare of Animals: This article does not contain any studies with animals performed by any of the authors.

Transparency Statements

1. The study was pre-registered at clinicaltrials.gov (NCT03069482).

2. The analysis plan was not formally pre-registered.

3. De-identified data from this study are not available in a public archive. De-identified data from this study will be made available (as allowable according to institutional IRB standards) by emailing the corresponding author.

4. Analytic code used to conduct the analyses presented in this study are not available in a public archive. They may be available by emailing the corresponding author.

5. Materials used to conduct the study are not publicly available.