Adolescent dropout from brief digital mental health interventions within and beyond randomized trials

Abstract

Objective

Many adolescents struggle to access appropriate mental health care due to structural or psychological barriers. Although traditional barriers to participation (e.g., location, cost) are hypothetically reduced or removed in internet interventions, low retention reduces the likelihood that adolescents will receive the intervention dosage intended to produce beneficial effects. It is therefore key to determine what factors are associated with dropout in digital mental health interventions with adolescents both within and beyond the context of research studies.

Methods

We compare completion rates from two projects evaluating self-guided, online single-session mental health interventions (SSIs) for adolescents. One was a randomized controlled trial (RCT) in which participants were paid for participation. The other was a program evaluation project in which participants were not paid for participation. We additionally compare SSI completion rates across various demographic groups and across baseline hopelessness levels.

Results

There was a statistically significant difference in SSI completion status between the RCT (84.75% full-completers) and the program evaluation (36.86% full-completers), X² (2, N = 2436) = 583.5, p < 0.05. There were no significant differences in the baseline hopelessness scores across completion statuses in either study. There were no significant differences in full-completion rates across demographic groups in either project.

Conclusion

Adolescents may be more likely to complete a brief digital mental health intervention in a paid, research-based context than in an unpaid, naturalistic context. Additionally, it is possible that the brevity of SSIs reduces demographic disparities related to retention by minimizing the time required to complete an intervention.

Highlights

• •RCT participants were more likely to complete the interventions.
• •Full completion rates in teens were similar across demographic groups.
• •Incentives and intervention brevity may be key to improving retention in teens.

1. Introduction

Data from a nationally-representative study conducted in 2016 suggested that approximately 1 in 10 adolescents had a diagnosable mental disorder in the United States (Whitney and Peterson, 2019). Further, between 2 and 5% of adolescents may experience sub-threshold mental health concerns at any point (Bertha and Balázs, 2013). The onset of mental disorders peaks in adolescence relative to childhood and adulthood, suggesting it is a crucial window for intervention and prevention (Kessler et al., 2005). Unfortunately, adolescents face barriers that limit their ability to access or remain in traditional, face-to-face mental health services, including stigma, high costs, provider shortages, lack of time, and the need for caregiver consent (Cavazos-Rehg et al., 2020; Findling and Stepanova, 2018; Gulliver et al., 2010; Minor Consent to Medical Treatment Laws, 2013). Digital interventions, and particularly self-guided programs (i.e., those that do not require a therapist or coach), hold potential for being more cost-effective and less time- and resource-intensive than face-to-face, therapist-delivered interventions. Indeed, many youths already report using digital tools for their mental health. In a survey of adolescents and younger adults with mental health conditions, approximately one-third of participants reported using apps or online communities to manage their mental health condition (Cohen et al., 2021). It is therefore crucial to understand retention of youth in digital mental health interventions both within and beyond the context of research. Determining what factors make an adolescent more or less likely to complete a digital intervention may help to optimize the potential of our most inherently-accessible interventions.

Some research has attempted to parse predictors of attrition in digital mental health interventions for youth, but results have been mixed and inconclusive. A systematic review of digital mental health interventions for children and adolescents found that retention varied widely across studies (ranging from 15.79% to 100%), but was high (79%) on average (Liverpool et al., 2020). The interventions included in the review were largely disseminated in structured research settings, and retention was defined as completing the first follow-up assessment of a clinically-relevant outcome (e.g., depression symptom severity). The wide range in retention rates may have reflected the heterogeneity of the interventions, which varied in their mode of delivery (i.e., apps, websites), administration (i.e., self-guided, coached), and duration (some were structured weekly programs while others were accessed as needed). No analyses investigated other factors, such as clinical or demographic characteristics, that might have related to retention.

There are several aspects of engagement and retention in digital mental health interventions for adolescents that warrant investigation—particularly given the need to identify strategies for optimizing program engagement. Firstly, there is little research investigating whether demographic youth characteristics differentially relate to dropout. In a cross-examination of multiple research studies involving digital programs for either physical and mental health concerns, adults with racial and ethnic minority identities were less likely to engage in digital interventions at all, compared to White, non-Hispanic adults. When daily engagement rates were analyzed, racial and ethnic minority individuals were underrepresented in “high-utilization” categories, relative to White non-Hispanic individuals (Pratap et al., 2020). It is not clear whether these findings are generalizable to adolescents, however, as studies including youth were excluded from the review. Further, no studies to our knowledge have investigated whether gender and/or sexual minority identities relate to program dropout. Adolescents with Lesbian, Gay, Bisexual, Questioning, and other non-heterosexual sexual orientations (LGBQ+) as well as Transgender and Gender Diverse (TGD) adolescents often experience identity-based discrimination and stigma, increasing their vulnerability to mental health concerns and difficulty accessing care (The Trevor Project, 2021). As such, it is important to gauge whether digital mental health supports—which are theoretically more accessible and equitable than face-to-face interventions—are easily completable by youths most likely to face limited access to treatment.

Secondly, there is a need to investigate how the context in which an intervention is accessed may shape retention. It is possible that retention rates differ when interventions are completed in a structured research setting compared to interventions completed in a less-structured, naturalistic setting. One study of online youth mental health interventions compared retention rates when the intervention was completed in a school setting and guided by teachers and when the intervention was independently accessed by youth themselves through a URL, in a community setting. Adolescents in the school setting, who were under teacher supervision, were more likely to complete intervention modules than unsupervised adolescents in the community. It is unclear whether the school-based setting itself or the presence (versus absence) of teacher supervision drove this effect (Neil et al., 2009). Continued investigation of interventions disseminated in both research and routine care settings is necessary. Further, participant payment may influence retention. One review found that an unguided, app-based program offering monetary incentives for study participation had better retention (defined as a longer number of days between the first and last active task completed by the participants) compared to programs without incentives (Anguera et al., 2016; Pratap et al., 2020). Previous research supports that monetary incentives increase participants' motivation to partake in research (Bentley and Thacker, 2004; Dainesi and Goldbaum, 2014). It is unclear, however, whether these findings might generalize to adolescents with mental health needs.

Third, there are conflicting findings on whether clinically-relevant baseline characteristics affect retention. Neil et al. (2009)' paper suggested that baseline clinical characteristics can impact retention, but the effects differ between participants in a structured research setting and a naturalistic setting. In the structured, school setting, adolescents with lower pre-test anxiety scores were more likely to complete intervention modules, whereas in the naturalistic, community sample, adolescents with higher pre-test depression scores were more likely to complete intervention modules (Neil et al., 2009). More research is needed to elucidate the effect of psychological distress on retention in both structured research settings and naturalistic settings. Hopelessness, a transdiagnostic marker of psychological distress, may be particularly relevant to examine (Castellanos-Ryan et al., 2016). Individuals with higher levels of baseline hopelessness may be less likely to complete an intervention due to beliefs that it will not help them (Kuyken, 2004).

Research on contributors to adolescents' engagement with digital mental health supports promises to inform strategies for improving retention, ultimately increasing the likelihood that an adolescent will receive the intervention dosage intended to produce beneficial effects. Unfortunately, variations in digital intervention length complicate any direct comparisons of engagement and completion rates across different digital mental health interventions. Single-Session Interventions (SSIs), structured interventions that intentionally last only one session, may be ideal interventions to investigate predictors of digital intervention retention (Schleider et al., 2020b). The brevity of SSIs allows researchers to determine what factors may impact dropout outside of intervention length (which tends to be consistent across digital SSIs: between 20 and 30 min total; Dobias et al., 2021; Schleider et al., 2020a; Sung et al., 2021). Digital, self-guided SSIs have been demonstrated to reduce symptoms of depression and anxiety in adolescents with effects lasting up to 9 months, but completion rates vary widely, from 40%–100% (Dobias et al., 2021; Schleider et al., 2020a; Schleider and Weisz, 2018; Sung et al., 2021). It is not clear whether incentives, demographics, or psychological variables affect dropout in digital, self-guided SSIs for adolescents.

This secondary, exploratory analysis uses data collected from two previous projects, an open program evaluation of digital SSIs (Schleider et al., 2020a) and a randomized controlled trial (RCT) of the same digital SSIs (Schleider et al., 2021) to examine dropout in brief online interventions for adolescents. The aims of this study are as follows: 1) Determine whether intervention completion status differs as a function of participating in the program evaluation project (no incentives provided) or the RCT (monetary incentives provided); 2) Investigate whether intervention completion status differs as a function of baseline hopelessness levels for participants in the program evaluation project and the RCT, respectively; and 3) Investigate whether intervention completion status differs as a function of demographic variables (racial/ethnic identity, age group, TGD status, & LGBQ+ status) for participants in the program evaluation project and the RCT, respectively. To the authors' knowledge, this is the first study to investigate how factors such as demographics, incentives, or clinical characteristics impact dropout rates in brief digital mental health interventions for adolescents.

2. Methods

2.1. Participants and procedures: program evaluation project

No inclusion or exclusion criteria were utilized for participants in the program evaluation project, as the interventions were openly available to anyone who could access them online. For the purposes of this study, we excluded participants <11 years old or ≥ 18 years old from all analyses.

Data collection took place between September 19, 2019, and September 29, 2021. In the first phase of the project, participants were recruited through online methods such as social media advertisements. In December 2020, a partnership was formed with the City of San Antonio, Texas Health Department to recruit youths in the San Antonio area using community-based methods such as community outreach, social media-based advertising, and teen health clinic partnerships. Parent permission was not required to participate. All participants provided assent prior to taking part in the program by reading the project description and affirming their interest in participating. Before the project was initiated, all procedures were reviewed and deemed “exempt” and not human subjects research (as a fully anonymous program evaluation) by the University's Institutional Review Board. Procedures for the program evaluation were pre-registered prior to youths' participation (https://osf.io/e52p3).

Youth in the program evaluation project were not assigned to any conditions. Rather, they self-selected one or more of the SSIs they wished to complete. These SSIs included Project Personality and The ABC Project (both described below), and the Teen Goals Project (described elsewhere; Schleider et al., 2020a). For the purposes of this study, we only included participants who started Project Personality or the ABC Project. Participants were not paid for their participation. Additional information regarding the Program Evaluation Project can be found in (Schleider et al., 2020a).

2.2. Participants and procedures: randomized controlled trial

Participants were included in the RCT if they were between the ages of 13 to 16 and demonstrated depressive symptoms, as indicated by a score greater than or equal to 2 on the Patient Health Questionnaire-2 (Richardson et al., 2010).

Participants were recruited through paid social media-based advertisements. Parental permission was not required to participate. This requirement was waived by the University's Institutional Review Board. All adolescents provided assent prior to taking part in the study. The trial was registered on clinicaltrials.gov prior to recruitment of subjects (NCT04634903).

Participants in the RCT were randomized to 1 of 3 SSIs. These SSIs included Project Personality and The ABC Project (both described below), and a Supportive Therapy SSI (described elsewhere; Schleider et al., 2021). For the purposes of this study, we only included participants who started Project Personality or the ABC Project. Participants were eligible to be paid up to $20 USD in gift cards for completing the baseline survey (including pre-intervention and post-intervention self-report assessments) and the 3-month follow-up survey (including a limited battery of additional self-report assessments).

Data collection took place between November 19 and December 6, 2020. Additional information regarding the RCT and its primary results can be found elsewhere (Schleider et al., 2021).

2.3. Interventions

2.3.1. Project personality

Project Personality is an SSI focused on teaching participants about Growth Mindset (Schleider and Weisz, 2019). The content is self-administered and takes approximately 30 min to complete. It includes both psychoeducative components and behavioral components designed to target proximal indicators of mental wellbeing (e.g., perceived control). It includes both didactic and interactive components; participants learn about neuroplasticity, review research on the malleability of traits, and listen to testimonials from older teens about times when they “used growth mindset.” Participants also complete exercises such as writing a letter to someone younger about growth mindset. Project Personality is freely available online (http://www.schleiderlab.org/yes.html).

2.3.2. The ABC project

The ABC project is similar in structure to Project Personality but focuses on teaching participants about Behavioral Activation (Schleider et al., 2019). It is a self-administered SSI and takes approximately 30 min to complete. It includes psychoeducative components and behavioral components. Participants learn the rationale behind behavioral activation, complete a life values assessment, create an activity hierarchy, and write about the benefits and challenges of completing activities. The ABC Project is freely available online (http://www.schleiderlab.org/yes.html).

2.4. Measures

2.4.1. Demographics

Demographic information collected from participants across both the RCT and the program evaluation included racial identity, age group, TGD status, & LGBQ+ status.

2.4.1.1. Racial identity

Options for self-reported racial identity differed slightly between the RCT and the program evaluation. For the purposes of this analysis, options were combined as follows: White; “Black” or “Black/African-American;” “Latino/Hispanic” or “Hispanic/Latinx;” “Asian” or “Asian Including Asian Desi;” Other (Including “Multiracial,” “American Indian/Alaska Native,” “Native Hawaiian/Other Pacific Islander” or “Prefer not to answer.”).

2.4.1.2. Age group

Participants in the program evaluation project selected their age bracket (provided in ranges to maintain anonymity: 11 to 13, 14 to 16, 17 to <18). Participants in the RCT reported their exact age. For the purposes of this analysis, age collected in the RCT was re-coded to fit in the same age brackets as those in the program evaluation project.

2.4.1.3. TGD status

Participants were asked, “Do you identify with a gender that is different from your sex at birth?” If participants answered “Yes,” they were considered TGD. If they answered “No,” they were not considered TGD. Participants who answered “Yes” or “Unsure” were asked to report their gender identity. If participants answered “unsure” to the first question and also indicated that their gender identity was “Unsure,” they were not considered TGD. If they answered “unsure” to the first question and indicated that their gender identity was something other than “unsure”, they were considered TGD.

2.4.1.4. LGBQ+ status

Participants were asked, “How do you identify your sexual orientation?” Participants that indicated they were Heterosexual were not considered LGBQ+. Participants that indicated they were unsure/questioning were considered LGBQ+. Participants that selected “other” were considered LGBQ+ unless they specified that they were straight/heterosexual. Participants who selected any other option (including Asexual, Bisexual, Gay, Homosexual, Lesbian, Pansexual, and Queer) were considered LGBQ+. Participants who indicated they did not wish to respond were excluded.

2.4.2. Beck hopelessness scale (BHS) 4-item version

This 4-item measure asks participants to rate their agreement with a statement (e.g., “I feel that my future is hopeless and that things will not improve”) on a 4-point Likert scale from Absolutely Disagree to Absolutely Agree (Perczel Forintos et al., 2013). Scores were averaged across the four questions to create a mean score ranging from 0 to 3. Higher scores indicated higher levels of hopelessness. Participants completed this scale prior to beginning an SSI in both the program evaluation and the RCT. Because no other pre-intervention assessments were included in both the program evaluation and the RCT, the BHS-4 served as the present study's index of pre-SSI clinically-relevant distress. Previous studies have similarly used the BHS-4 as an index of pre-intervention psychological distress (Kuyken, 2004). Internal consistency in the program evaluation was α = 0.85. Internal consistency in the RCT was α = 0.84.

2.5. Data analysis

Data analyses were completed as pre-registered (https://osf.io/jbxdz). The RStudio Statistical Program was used to complete data analyses.

2.5.1. Completion status

Participants were characterized as either Full Completers, Post-Activity Non-Completers, or Pre-Activity Non-Completers. These groups were modeled after categories used to classify completion in a previous study investigating an SSI (Sung et al., 2021). Full Completers were participants who reached the end of the full SSI content, defined by submitting a response to the final activity. An activity was defined as an interactive program element that requires the participant to expend effort to complete. Examples include selecting an answer to a question, writing a sentence in response to a prompt, or creating an action plan. Activities did not include reading or continuing to the following page. Post-Activity Non-Completers were defined as participants who dropped out after completing at least one activity. Pre-Activity Non-Completers were defined as participants who dropped out before completing any activities.

2.5.2. Participant characteristics

We report descriptive statistics for the participants' demographic characteristics in both the program evaluation project and the RCT in Table 1, Table 2. Overall, 803 participants were included from the program evaluation project and 1633 participants were included from the RCT. We conducted analyses to determine whether there were statistically significant differences in the demographic characteristics and baseline BHS scores of participants in the program evaluation project and the RCT. For each of these tests, a p-value of <0.05 was considered statistically significant. If applicable, post-hoc tests were conducted using Bonferroni-corrected critical values.

Table 1.

Participant race/ethnicity characteristics and overall distribution across completion statuses.

		Race/ethnicity
Program evaluation (n = 803)		Black/African American (n = 38)	Asian (n = 42)	Latino/Hispanic (n = 217)	White (n = 294)	Other (n = 193)
Full completer	296 (36.86%)	15 (39.47%)	20 (47.62%)	68 (31.34%)	118 (40.14%)	67 (34.72%)
Post-activity non-completer	208 (25.90%)	10 (26.32%)	6 (14.29%)	65 (29.95%)	59 (20.07%)	64 (33.16%)
Pre-activity non-completer	299 (37.24%)	13 (34.21%)	16 (38.09%)	84 (38.71%)	117 (39.79%)	62 (32.12%)

		Race/ethnicity
RCT (n = 1633)		Black/African American (n = 109)	Asian (n = 144)	Latino/Hispanic (n = 202)	White (n = 900)	Other (n = 278)
Full completer	1384 (84.75%)	95 (87.16%)	126 (87.5%)	171 (84.65%)	759 (84.33%)	233 (83.81%)
Post-activity non-completer	78 (4.78%)	6 (5.5%)	7 (4.86%)	7 (3.46%)	44 (4.89%)	14 (5.04%)
Pre-activity non-completer	171 (10.47%)	8 (7.34%)	11 (7.64%)	24 (11.89%)	97 (10.78%)	31 (11.15%)

Table 2.

Participant Age Group, TGD Status, and LGBQ+ Status across completion statuses.

	Age group			TGD status		LGBQ+ status
Program evaluation (n = 803)	11–13 (n = 225)	14–16 (n = 449)	17 or older (n = 110)	Yes (n = 220)	No (n = 565)	Yes (n = 432)	No (n = 314)
Full completer	93 (41.33%)	160 (35.63%)	35 (31.82%)	76 (34.55%)	209 (36.99%)	161 (37.27%)	119 (37.89%)
Post-activity non-completer	63 (28.0%)	114 (25.38%)	27 (24.55%)	57 (25.91%)	149 (26.37%)	115 (26.62%)	74 (23.57%)
Pre-activity non-completer	69 (30.67%)	175 (38.98%)	48 (43.64%)	87 (39.55%)	207 (36.64%)	156 (36.11%)	121 (38.54%)

	Age group			TGD status		LGBQ+ status
RCT (n = 1633)	11–13 (n = 118)	14–16 (n = 1515)	17 or older (n = 0)	Yes (n = 524)	No (n = 1095)	Yes (n = 1299)	No (n = 326)
Full completer	98 (83.05%)	1286 (84.88%)	0	439 (83.78%)	933 (85.21%)	1102 (84.83%)	275 (84.36%)
Post-activity non-completer	11 (9.32%)	67 (4.42%)	0	33 (6.29%)	44 (4.02%)	66 (5.08%)	12 (3.68%)
Pre-activity non-completer	9 (7.63%)	162 (10.69%)	0	52 (9.92%)	118 (10.78%)	131 (10.08%)	39 (11.96%)

2.5.3. Analytic plan

To determine whether SSI completion status (full completer vs. post-activity non-completer vs. pre-activity non-completer) differed as a function of project type (program evaluation vs. RCT), we conducted a Pearson Chi-Square test. A p-value of <0.05 was considered statistically significant. If significant, we examined standardized residuals to determine which groups significantly contributed to the overall chi-square statistic. A Bonferroni adjustment to the z critical of 1.96 was made so that the critical value was adjusted to +/− 2.65.

To investigate whether SSI completion status differed as a function of baseline hopelessness levels, we first conducted a Levene Test to determine if the assumption of homogeneity of variance was met in the program evaluation project and the RCT. A p-value of <0.05 indicated the assumption was violated. We conducted either a Kruskal-Wallis H Test or an ANOVA depending on the result.

To determine whether SSI completion status differed as a function of demographic variables (racial identity, age group, TGD status, & LGBQ+ status), we conducted Pearson Chi-Square tests for the program evaluation project and the RCT. For each of these tests, a p-value of <0.05 was considered statistically significant. If applicable, we examined standardized residuals using Bonferroni-corrected critical values.

We additionally conducted a multinomial logistic regression analysis with completion status as the dependent variable and age group, racial identity, TGD status, LGBQ+ status, mean BHS score, and project (i.e., RCT vs. program evaluation) as independent variables.

3. Results

3.1. Participant characteristics

Overall, 2436 participants were included in the analyses. Participants' racial/ethnic identities were 6.08% Black/African American, 7.69% Asian, 17.34% Latino/Hispanic, 49.4% White, and 19.49% a race/ethnicity group other than those provided as options (examples include American Indian/Alaska Native or Native Hawaiian/Other Pacific Islander). Approximately 14.19% of participants were 11–13 years old, 81.26% were 14–16 years old, and 4.56% were 17. In total, 30.95% identified as TGD, and 73.0% identified as LGBQ+.

A Pearson's Chi-Squared test revealed a statistically significant difference in race/ethnicity groups between the RCT and the program evaluation project, X² (4, N = 2379) = 131.7, p < 2.2e-16. We examined the standardized residuals, as presented in Table 3. A Bonferroni adjustment to the z critical of 1.96 was made so that the critical value was adjusted to +/− 2.81 (p < 0.005). In the RCT, there was a significantly higher proportion of Asian participants (8.82%) and White participants (55.11%), and a significantly lower proportion of Latino/Hispanic participants (12.37%%) and participants in the “other” category (17.02%) than in the program evaluation project (5.36% Asian participants; 37.5% White participants; 27.68% Latino/Hispanic participants; 24.62% “other” participants).

Table 3.

Standardized residuals associated with Chi-Square tests between Study conditions.

	Program evaluation	RCT
Race/ethnicity
Asian	−2.99	2.99
Black/African American	−1.76	1.76
Latino/Hispanic	9.31	−9.32
Other	4.41	−4.41
White	−8.11	8.11
LGBQ+ status
Not LGBQ+	11.22	−11.22
LGBQ+	−11.22	11.22
Completion status
Full completer	−24.02	24.02
Post-activity non-completer	15.23	−15.23
Pre-activity non-completer	15.74	−15.74

A Pearson's Chi-Squared test with Yates' continuity correction revealed a statistically significant difference in TGD status between the RCT and the program evaluation project, X² (1, N = 2404) = 4.46, p = 0.0.035. However, when examining the standardized residuals, none exceeded the Bonferroni-corrected z critical of +/− 2.49 (p < 0.0125), suggesting that there were no statistically significant differences between the proportion of TGD participants in the program evaluation project and the RCT.

A Pearson's Chi-Squared test with Yates' continuity correction revealed a statistically significant difference in LGBQ+ status between the RCT and the program evaluation project, X² (1, N = 2371) = 124.79, p < 2.2e-16. We examined the standardized residuals, as presented in Table 3. Bonferroni adjustment to the z critical of 1.96 was made so that the critical value was adjusted to +/− 2.49 (p < 0.0125). There was a statistically significant difference between the proportion of LGBQ+ individuals in the program evaluation project (37.27%) and the RCT (84.83%).

A Shapiro-Wilk normality test was conducted to determine whether BHS scores across the program evaluation project and the RCT were distributed normally. Results indicated non-normality, W = 0.97, p < 2.2e-16. Consequently, a Wilcoxon Rank Sum test with continuity correction was conducted. It revealed a statistically significant difference in median BHS scores between the participants in the RCT (Mdn = 1.75) and participants in the program evaluation project (Mdn = 1.5), Z = −8.07, p = 7.22e-16.

3.2. Program evaluation project vs. RCT

A Pearson's Chi-Squared test revealed a statistically significant difference in participants' SSI completion status between the RCT and the program evaluation project, X² (2, N = 2436) = 583.5, p < 2.2e-16. We examined the standardized residuals, as presented in Table 3. A Bonferroni adjustment to the z critical of 1.96 was made so that the critical value was adjusted to +/− 2.65 (p < 0.008). In the RCT, there was a significantly higher proportion of full-completers (84.75%) and a significantly lower proportion of post-activity non-completers (4.78%) and pre-activity non-completers (10.47%) than in the program evaluation project (36.86% full-completers; 25.9% post-activity non-completers; 37.24% pre-activity non-completers).

3.3. Baseline hopelessness levels

Prior to testing whether completion status differed as a function of baseline hopelessness levels in the program evaluation project, a Levene test was conducted to determine whether the assumption of homogeneity of variance was met. Levene's test indicated unequal variances (F = 7.83, p = 0.0004), so a Kruskal-Wallis H test was conducted. The Kruskal-Wallis H test revealed no significant differences in the BHS scores across completion statuses in the program evaluation project, X² (2, N = 803) = 0.52, p = 0.77.

Similarly, a Levene test was conducted to determine whether the assumption of homogeneity of variance was met in the RCT. Levene's test indicated equal variances (F = 0.36, p = 0.69), so an ANOVA was conducted. The ANOVA revealed no significant differences in the BHS scores across completion statuses in the RCT, F(2,1630) = 0.09, p = 0.92.

3.4. Demographic variables

Approximately 39.47% of Black/African American participants, 47.62% of Asian participants, 31.34% of Latino/Hispanic participants, 40.14% of White participants, and 34.72% participants of another race/ethnicity were full-completers in the program evaluation project. A Pearson's Chi-Squared test revealed a statistically significant difference in race/ethnicity groups across completion statuses in the program evaluation project, X² (8, N = 746) = 17.69, p = 0.024. We examined the standardized residuals, as presented in Table 4. A Bonferroni adjustment to the z critical of 1.96 was made so that the critical value was adjusted to +/− 2.93 (p < 0.003). There was a statistically small proportion of individuals identifying as “White” who were categorized as post-activity non-completers (20.07% of white participants). Approximately 87.16% of Black/African American participants, 87.5% of Asian participants, 84.65% of Latino/Hispanic participants, 84.33% of White participants, and 83.81% participants of another race/ethnicity were full-completers in the RCT. A Pearson's Chi-Squared test revealed no significant differences in the race/ethnicity groups across completion statuses in the RCT, X² (8, N = 1633) = 3.87, p = 0.87.

Table 4.

Standardized residuals associated with Chi-Square test between race/ethnicity and completion status in the program evaluation project.

	Asian	Black/African American	Latino/Hispanic	Other	White
Full completer	1.5	0.36	−1.94	−0.67	1.53
Post-activity non-completer	−1.78	0.04	1.55	2.6	−2.94
Pre-activity non-completer	0.12	−0.39	0.52	−1.69	1.14

Approximately 41.33% of participants aged 11–13, 35.63% of participants aged 14–16, and 31.82% of participants aged 17 or older were full-completers in the program evaluation project. A Pearson's Chi-Squared test revealed no significant differences in the age groups across completion statuses in the program evaluation project, X² (4, N = 784) = 6.85, p = 0.14. Approximately 83.05% of participants aged 13 and 84.88% of participants aged 14–16 were full-completers in the RCT. A Pearson's Chi-Squared test did reveal a statistically significant difference in age groups across completion statuses in the RCT, X² (2, N = 1633) = 6.53, p = 0.038. However, when examining the standardized residuals, none exceeded the Bonferroni-corrected z critical of +/− 2.65 (p < 0.008), suggesting that there were no statistically significant differences in completion across age groups in the RCT.

In the program evaluation project, 34.55% of TGD participants and 36.99% of non-TGD participants were full completers. A Pearson's Chi-Squared test revealed no significant differences in TGD status across completion statuses in the program evaluation project, X² (2, N = 785) = 0.63, p = 0.73. In the RCT, 83.78% of TGD participants and 85.21% of non-TGD participants were full completers. A Pearson's Chi-Squared test revealed no significant differences in TGD status across completion statuses in the RCT, X² (2, N = 1619) = 4.2, p = 0.12.

In the program evaluation project, 37.27% of LGBQ+ participants and 37.89% of non-LGBQ+ participants were full completers. A Pearson's Chi-Squared test revealed no significant differences in LGBQ+ status across completion statuses in the program evaluation project, X² (2, N = 746) = 0.98, p = 0.61. In the RCT, 84.83% of LGBQ+ participants and 84.36% of non-LGBQ+ participants were full completers.A Pearson's Chi-Squared test revealed no significant differences in LGBQ+ status across completion statuses in the RCT, X² (2, N = 1625) = 1.95, p = 0.38.

3.5. Multinomial logistic regression

Results from the multinomial logistic regression are presented in Table 5. The context in which participants accessed an SSI (RCT versus program evaluation) significantly predicted whether they were a post-activity non-completer or a full-completer, b = −2.41, p < 0.001. That is, participants in the program evaluation were 11.11 times more likely than participants in the RCT to discontinue an SSI partway through the program (after completing some interactive portion), versus completing the SSI. The context in which participants accessed an SSI (RCT versus program evaluation) also significantly predicted whether they were a pre-activity non-completer or a full-completer, b = −2.09, p < 0.001, such that participants in the program evaluation were 8.33 times more likely than those in the RCT to discontinue an SSI before completing any interactive elements, versus completing the SSI. No other predictors (age group, race/ethnicity, TGD status, LGBQ status, or baseline BHS scores) significantly predicted participants' completion status.

Table 5.

Results from multinomial logistic regression.

	B (SE)	95% CI for odds ratio
		Lower	Odds Ratio	Upper
Post-activity non-completer vs. full completer
Intercept	−0.67 (0.22)
11–13 years old	0.09 (0.19)	0.75	1.09	1.61
17 years old or older	0.17 (0.29)	0.66	1.18	2.09
Asian	−0.29 (0.38)	0.39	0.75	1.46
Black/African American	0.22 (0.31)	0.67	1.24	2.29
Latino/Hispanic	0.33 (0.20)	0.94	1.39	2.06
Other	0.45 (0.19)	1.09	1.57	2.26
TGD	0.20 (0.16)	0.89	1.22	1.68
LGBQ	0.14 (0.17)	0.82	1.15	1.59
BHS Mean	−0.06 (0.09)	0.78	0.94	1.12
RCT	−2.41 (0.17)⁎⁎⁎	0.06	0.09	0.13
Pre-activity non-completer vs. full completer
Intercept	0.22 (0.18)
11–13 years old	−0.36 (0.18)	0.49	0.69	0.99
17 years old or older	0.13 (0.25)	0.69	1.13	1.86
Asian	−0.34 (0.25)	0.44	0.71	1.15
Black/African American	−0.37 (0.28)	0.40	0.69	1.20
Latino/Hispanic	0.17 (0.16)	0.86	1.18	1.62
Other	−0.08 (0.16)	0.67	0.92	1.26
TGD	0.12 (0.13)	0.87	1.13	1.46
LGBQ	−0.09 (0.14)	0.69	0.92	1.20
BHS Mean	−0.08 (0.07)	0.80	0.93	1.07
RCT	−2.09 (0.14)⁎⁎⁎	0.09	0.12	0.16

^⁎⁎⁎Bonferroni-adjusted p < 0.001.

4. Discussion

Digital mental health interventions may improve access to mental healthcare for youth. Identifying predictors of dropout from digital mental health interventions may inform intervention designs that meet youths' needs and optimize odds of program completion. Intervention developers may, for example, conduct user-centered design work with demographic groups or clinical populations that are most likely to drop out of interventions to identify intervention adaptations that better meet these populations' needs. Further, information on the effect of participant payment on dropout may help researchers determine how participant payment can be used to substantially improve intervention completion rates. Accordingly, this study examined whether demographic, incentive-related, and clinically-relevant factors related to adolescents' intervention dropout, in separate implementation contexts (an RCT versus a program evaluation project). Results indicated that adolescents' likelihood of completing online, single-session mental health interventions was significantly higher in a randomized-controlled trial than in an open-access program evaluation project. This pattern aligns with previous work demonstrating higher retention in research settings compared to naturalistic settings (Neil et al., 2009). Contrasting with previous research on digital interventions (e.g., in Pratap et al., 2020, racial and ethnic minority individuals were underrepresented in “high-utilization” categories, relative to White non-Hispanic individuals), no consistent or significant differences emerged in online, single-session mental health intervention full-completion rates by participants' racial, ethnic, or gender identities, nor by sexual orientation. Also contrasting with previous research, the results in the current study did not indicate that symptoms of distress at baseline affect dropout (Neil et al., 2009).

One explanation for the discrepancy in dropout between the RCT and the program evaluation project is that the monetary incentive provided to adolescents in the RCT increased adolescents' motivation to finish the intervention. If so, these results underscore the need to appropriately incentivize adolescents for their participation in research. These findings align with previous research indicating the benefit of monetary incentives on participant retention (Bentley and Thacker, 2004; Dainesi and Goldbaum, 2014; Pratap et al., 2020).

Alternatively, these results may reflect a broader discrepancy between structured research studies and unstructured, naturalistic studies. This finding could be the result of self-selection bias, given that participation in the RCT required participants to complete a consent form and agree to be contacted for further assessment. Participants who self-selected to participate in the RCT, therefore, may differ from participants who self-selected to participate in an open-access program evaluation (Larzelere et al., 2004; Mohr et al., 2017). It is also possible that the acts of agreeing to participate in a research study and completing informed assent procedures increased participants' motivation or commitment to completing the intervention. It may be helpful for future work on digital intervention non-completion to directly evaluate assent procedures and/or selection bias as predictors of digital intervention completion. Additionally, future research may investigate whether studies with larger or smaller incentives show respective decreases or increases in dropout rates.

In both the program evaluation project and the RCT, the proportions of full-completers were similar across all racial/ethnic groups. It may be that previous research indicating individuals in racial and ethnic minorities are less likely to engage in an intervention are not generalizable to adolescents. Alternatively, it may be that the length of previously-studied interventions accounts for the differences in engagement in individuals in racial and ethnic minorities. A key barrier that may keep individuals in minoritized racial and ethnic groups from participating fully in interventions is the cost of time, meaning that individuals in minoritized racial and ethnic groups may have less free time than their white peers due to systemic injustices (Miranda et al., 2015). In SSIs, this barrier is minimized, which could explain why consistent differences were not found. Future research should attempt to directly compare retention in brief intervention studies and longer intervention studies.

Additionally, the proportions of full-completers were similar irrespective of TGD status and LGBQ+ status, suggesting adolescents with sexual and gender minority identities were not at higher risk of dropout. This finding aligns with previous research demonstrating that adolescents with sexual and gender minority identities report positive attitudes towards brief online mental health interventions (McDanal et al., 2021). This result is encouraging, given the barriers to traditional (face-to-face) mental health treatment that TGD and LGBQ+ adolescents often face (Cwinn et al., 2021; Sims and Nolen, 2021; Gridley et al., 2016; Rees et al., 2021; White and Fontenot, 2019; Williams and Chapman, 2011).

Results suggested that symptoms of distress at baseline did not affect dropout, which could be a result of the brevity of SSIs. Loss of motivation is associated with internalizing symptoms, which suggests lengthier interventions may be more difficult for individuals with high internalizing symptoms to complete (Madjar et al., 2021). Intentionally brief interventions may reduce this barrier and equalize completion rates for individuals with higher symptoms and lower symptoms. A key limitation in this study was that only baseline hopelessness levels were examined. Both the RCT and the program evaluation project collected information on participants' baseline depression and anxiety symptoms, but the projects used different measures, making comparison between them inadvisable. Future research should directly investigate how depression and anxiety symptoms affect retention in brief digital mental health intervention studies.

A limitation of this study is that the RCT and the program evaluation project differed in various demographic and psychological variables. Participants in the RCT were more likely to identify as Asian or White than participants in the program evaluation. Participants in the RCT were also more likely to identify as LGBQ+ than participants in the program evaluation project. This may be due to differences in recruitment strategies between the two projects, specifically the community-based recruitment strategies used for the program evaluation project in San Antonio. Despite these differences, full-completion rates did not differ in the RCT or the program evaluation project by race, ethnicity, or LGBQ+ status. Participants in the RCT had a higher median baseline hopelessness score than participants in the program evaluation. This is expected, as elevated depressive symptoms was an inclusion criterion for participating in the RCT, whereas there were no inclusion criteria for participating in the program evaluation project. Nevertheless, baseline hopelessness score did not appear to relate to SSI completion in either the RCT or the program evaluation project.

4.1. Conclusion

Adolescents may be more likely to complete a brief digital mental health intervention in a paid, research-based context than an unpaid, naturalistic context. It is possible that completion rates across demographic groups in an intervention that is intentionally brief may be similar because of the reduced time and access barriers, relative to longer digital interventions and treatments requiring caregiver permission. Researchers and companies developing digital mental health interventions in the future may take these findings into account and develop and disseminate their interventions accordingly. Further research surrounding retention and adherence to digital mental health interventions will be valuable to continue our understanding of how to create and disseminate digital mental health interventions that improve adolescents' mental health.

Funding

This work was supported by the National Institutes of Health (DP5OD028123) and a Service Grant from the City of San Antonio, Texas Health Department.

Declaration of competing interest

KAC receives grant support from a Stony Brook University Graduate Research Fellowship. JLS receives grant support from the National Institutes of Health (DP5OD28123), the Klingenstein Third Generation Foundation, the American Psychological Foundation, the Society for Clinical Child and Adolescent Psychology, Limbix, Inc., and the Upswing Fund for Adolescent Mental Health. JLS has co-authored and received royalties from sale of a therapeutic workbook for adolescents, published by New Harbinger. JLS is under contract with Oxford University Press to co-edit a book on low-intensity mental health interventions for youth. JLS serves on the Scientific or Clinical Advisory Boards for Walden Wise, Inc. and Koko. The authors report no other financial conflicts.