Abstract
Introduction: Cognitive behavioral therapy (CBT) is an established treatment for depression, but its success is often impeded by low attendance. Supportive text messages assessing participants' mood in between sessions might increase attendance to in-clinic CBT, although it is not fully understood who benefits most from these interventions and how. This study examined (1) user groups showing different profiles of study engagement and (2) associations between increased response rates to mood texts and psychotherapy attendance.
Methods: We included 73 participants who attended Group CBT (GCBT) in a primary care clinic and participated in a supportive automated text-messaging intervention. Using unsupervised machine learning, we identified and characterized subgroups with similar combinations of total texting responsiveness and total GCBT attendance. We used mixed-effects models to explore the association between increased previous week response rate and subsequent week in-clinic GCBT attendance and, conversely, response rate following attendance.
Results: Participants could be divided into four clusters of overall study engagement, showing distinct profiles in age and prior texting knowledge. The response rate to texts in the week before GCBT was not associated with GCBT attendance, although the relationship was moderated by age; there was a positive relationship for younger, but not older, participants. Attending GCBT was, however, associated with higher response rate the week after an attended session.
Conclusion: User groups of study engagement differ in texting knowledge and age. Younger participants might benefit more from supportive texting interventions when their purpose is to increase psychotherapy attendance. Our results have implications for tailoring digital interventions to user groups and for understanding therapeutic effects of these interventions.
Keywords: telehealth, cognitive behavioral therapy, short messaging service, engagement, digital literacy
Introduction
Depression is a severe mental health disorder, which is currently the leading cause of disability worldwide.1 Psychological therapy such as cognitive behavioral therapy (CBT) is one of the most commonly used psychological treatments for depression and has been identified as an effective intervention.2 However, studies have reported that participants need to attend an adequate number of sessions, suggested to range from 6 to 12 to achieve improved mental health outcomes.3–5 Unfortunately, low- or nonattendance of psychotherapy is common: meta-analytical evidence shows that around one in five participants drop out of psychotherapy.6 Attendance is even lower in participants with low socioeconomic or ethnic minority status.7,8
Mobile technology, in particular short messaging service (SMS), has shown to increase session attendance and might thus boost the effectiveness of psychotherapy and other behavioral interventions.9 For instance, text-messaging in between psychotherapy sessions can increase self-awareness, skill building, and perceived support10 and has shown to increase the time that participants stay in psychotherapy.11 Furthermore, the data collected by these interventions can facilitate tracking of mental health of participants over time and aid the design of “just-in-time” interventions. For example, daily mood ratings collected by SMS might be used as proxies for depression scores12 and for predicting next day psychotherapy attendance.13 Furthermore, because texting is a simple and low-cost tool widely used across socioeconomic and demographic groups, it might be particularly feasible for increasing access to treatment for underserved populations.14
Although mobile health interventions show beneficial effects and have a potentially wide reach, they might not always reach their maximum effect. For instance, it has been suggested that low participant engagement with mobile interventions over time reduces their effectiveness.15 Although there is a shared view that engagement should be promoted,16 this field of study is still in its early stages. The relationship between increased user engagement in mobile interventions, such as text messaging, and clinical outcomes, such as in-clinic psychotherapy attendance, has not been widely examined.17 Similarly, the direction of the therapeutic effect of supportive texting interventions and psychotherapy attendance remains unclear. A higher engagement with the texting component might increase the likelihood that participants will attend in-clinic psychotherapy, or vice-versa, after attending a session, participants might be more likely to engage more in the texting component.
The main aims of our current study were therefore (1) to identify subgroups of texting and attending behavior during a supportive text-messaging intervention for group CBT (GCBT), called “Moodtext” and (2) to examine associations between increased texting engagement and higher GCBT attendance. We defined responsiveness differently for each aim to make the variables coherent for the different analyses. First, for aim 1, we define the total texting responsiveness as the total number of text messages that required a mood score that was responded to. For aim 2, which is a weekly analysis, we defined the previous week response rate as the percentage of texts responded to in the previous week.
Combined, the results of this study can potentially give more insight into which participants benefit most from mobile interventions added to psychotherapy and can help guide these interventions to be more specifically tailored to participant subgroups. Furthermore, this knowledge can increase our insight of the nature of relationships between increased engagement in technology and face-to-face group psychotherapy.
Methods
Participants
We included predominantly low-income participants served in a public urban hospital, who were referred by their primary care providers if they expressed qualitative depressive symptoms or screened positive for depression based on the 9-item Participant Health Questionnaire (PHQ-9).18 Participants were considered eligible for GCBT if they had a PHQ-9 score ≥10 at initial assessment. Participants with comorbid substance abuse disorders, psychosis, or grief as primary problem were ineligible. Participants were provided with a mobile phone if they did not previously own one. The total study lasted from January 2014 to May 2018. Phase 1 of the study (n = 35, until August 2016) was a nonrandomized controlled trial, in which participants were not compensated.11 Phase 2 was a naturalistic study, in which all participants received the text-messaging adjunct (n = 38) and received a $25 gift card for their participation. Participants in phase 1 and 2 did not differ in length of therapy (W = 675, p = 0.91) or mean number of sessions attended (W = 634.5, p = 0.74). The University of California, San Francisco IRB approved this study (No. 10-04985). Participants provided written informed consent.
In-Clinic Group Cognitive Behavioral Therapy
GCBT was offered once a week as a continuously running group in Spanish and English. The sessions were led by a licensed clinical psychologist and/or a licensed clinical social worker experienced in CBT and in treating low-income and Latino participants. Clinicians used the updated Building Recovery by Improving Goals, Habits, and Thoughts (BRIGHT) manual.19 The treatment manual was developed in English and Spanish for use in public sector settings and has been found to be an efficacious treatment for depression in this population.20 Participants were scheduled to participate for a duration of 16 weeks, with week 1 being the first week. Although some participants were allowed to continue to attend group psychotherapy after the 16-week mark if they were still symptomatic or wished to make up missed content, we focused the current analyses on the first 16 weeks offered to participants. For the majority of participants (n = 39), there was no psychotherapy offered during 1 (n = 21), 2 (n = 12), and 3 (n = 6) weeks of their cycle, due to a holiday or absence of the psychotherapist. We discarded these weeks from the current analyses to focus on the relationship between attending psychotherapy and weekly response rate.
Structure of Texting Adjunct
All participants received a daily automated text at a random time between 8 am and 9 pm asking to rate their mood on a scale of 1–9 and describe what they were doing or thinking. Participants were told that the text messaging was a method to help them practice CBT-based skills, and to let therapists know how their mood was throughout the week. Participants also received a second daily message reiterating the theme of that week's content and medication and appointment reminders.11 The texting was programmed to start during the first week of GCBT. We excluded participants for the current analyses who due to technical errors started receiving text messages >2 weeks after the first GCBT group (n = 7).
Analyses
To explore different overall user groups, we clustered participants only on measures of overall engagement: (1) total texting responsiveness (e.g., the number of texts responded to over the whole study period) and (2) in-clinic GCBT attendance. We then examined the relationship of texting with GCBT in clusters who showed some level of engagement. Specifically, to explore directions of relationships, we consider whether previous week response rate was associated with attendance and vice versa, if attendance is associated with higher response rate in the week after GCBT.
Analysis 1: clustering participant engagement
Text messages responsiveness over the entire study period and GCBT attendance rates were normalized. We used a K-means algorithm, an unsupervised machine-learning method, with a Euclidean metric for computing the distance between points and cluster centers. For every participant, the normalized total texting responsiveness and GCBT attendance scores were used in the algorithm. We clustered on these two features alone to find groups with similar patterns of overall engagement, that is, both total texting responsiveness and GCBT attendance. We used the silhouette score21 to guide us in picking the optimal solution (number of clusters), as this method is commonly used21 and more easily understood than many more complex metrics. The silhouette score is a measure of how cohesive clusters are relative to how well separated they are. Higher silhouette scores mean that observations are better matched with the assigned cluster.
To explore differences in participant characteristics between the identified clusters, we considered age, gender, PHQ-9 scores, texting knowledge (whether a participant indicated that they knew who to text at baseline), and preferred communication method (texting/calling) at baseline. We used ANOVA for normally distributed continuous variables (age), Kruskal–Wallis rank sum tests for continuous variables that were not normally distributed (PHQ-9 scores) and chi-square tests for categorical data (gender, texting knowledge and preferred communication method).
Analysis 2: relating response rate to text messages and attending in-clinic GCBT
To explore whether increased texting in the preceding week was associated with more likely in-clinic session attendance, we considered a logistic mixed-effects model. In contrast to logistic and linear regression, mixed-effects models accommodate the possible nonindependency of measurements which could happen, as in our case, with repeated measures coming from the same participant.22 Mixed models are able to address both (1) variation that is explained by the independent variables of interest—fixed effects and (2) variation that is not explained by the independent variables of interest—random effects. Therefore, mixed models allow you to systematically account for item-level variability (within subjects) and subject-level variability (within groups). All of the mixed-effects models that we consider include random intercepts to account for overall differences between individuals in the outcome.
We included a centered age variable, the previous week response rate, that is, the fraction of SMS responded to in the previous week, week of study participation as independent variables and weekly attendance (attended GCBT yes/no) as the outcome variable. Conversely, to explore whether attending GCBT was associated with increased response rate in the week following (% of texts responded to), we consider a mixed-effect linear regression model with response rate the subsequent week as the outcome. We considered GCBT attendance, time in study, centered age, and the interaction between age and attendance as independent variables. To explore the significance of random effects, we considered two mixed-effects logistic regression models: a fixed-effects+random intercepts model and a maximal model, as recommended by Barr et al.,23 with random intercepts and random slopes for week in study and response rate/weekly GCBT attendance. We included the data of participants who opted out of the texting until they stopped receiving texts.
Linear mixed-effects models were checked for model assumptions by visual inspections of residual plots. p-Values for the logistic model were obtained by asymptotic Wald tests and for the linear method by the Satterthwaite method.24 These analyses were carried out in R studio V. 1.1.423 using the Lme425 and LmerTest package.26 The BOBYQA optimizer27 was used for model convergence before modifying the random effects structure as suggested by prior work.23
Results
Participant Engagement
The final analyses included 73 participants. Participants were predominantly Spanish speaking (90%), female (75%), middle-aged, 51.5 ± 12.1, and most did not have a high school diploma (73%). The mean number of group sessions attended by a participant was 6.7 ± 4.7. Participants responded to a mean of 49.5 ± 35.6 messages during the whole study period. Eleven participants opted out of texting at some point during the study by texting “STOP” or “PARAR.” See Table 1 for other demographic and clinical characteristics.
Table 1.
Demographic and Clinical Characteristics
| TOTAL (N = 73) | UNENGAGED (N = 19) | MOSTLY MOBILE (N = 16) | MOSTLY LIVE (N = 10) | FULLY ENGAGED (N = 28) | |
|---|---|---|---|---|---|
| Total texts responded to during study (mean, SD) (clustering variable) | 49.5 (35.6) | 6.42 (7.6) | 58.88 (20.9) | 22.5 (16.9) | 83 (14.4) |
| GCBT groups attended during study (mean, SD) (clustering variable) | 6.7 (4.7) | 2 (1.1) | 2.88 (1.5) | 11.2 (2.6) | 10.46 (2.7) |
| Weeks before last session attended (drop-out) (mean, SD) | 9.7 (6) | 3.37 (3.4) | 5.56 (4.5) | 15.2 (0.9) | 14.36 (2.4) |
| Percentage of texts responded to out of received texts until study drop-out or end of studya | 51.5% | 9% | 56% | 26% | 79% |
| Age (mean, SD)* | 51.5 (12.1) | 52.8 (12.6) | 44.4 (10.6) | 59.2 (11.2) | 51.9 (11.3) |
| Spanish speaking, n (%) | 66 (90) | 16 (84.21) | 16 (100) | 10 (100) | 24 (85.71) |
| Female, n (%) | 55 (75) | 12 (63.2) | 13 (81.2) | 8 (80) | 22 (78.5) |
| Prefer to call, n (%)b | 37 (53) | 14 (73.6) | 9 (56.2) | 6 (6) | 8 (28.6) |
| Prefer to text, n (%)b | 15 (21.7) | 2 (10.5) | 2 (12.5) | 1 (10) | 10 (35.7) |
| Depends on the situation, n (%)b | 17 (24.6) | 3 (15.7) | 3 (21.4) | 3 (30) | 8 (28.6) |
| Don't know, n (%) | 2 (2.9) | 0 | 0 | 0 | 2 (7.1) |
| Full time or part-time employment, n (%) | 20 (30) | 7 (37) | 5 (27) | 1 (10) | 7 (26) |
| Education, less than high school (n) | 50 (73) | 13 (68) | 12 (75) | 9 (90) | 19 (67) |
| Participant knows how to text,cn (%)* | 50 (71) | 9 (50) | 13 (93) | 4 (40) | 24 (88) |
| PHQ-9 score (mean, SD) | 14.3 (5.7) | 16.1 (5.4) | 15.6 (6.3) | 11.5 (5.5) | 13.2 (5.5) |
| Previous therapy, n (%) | 37 (53) | 11 (57.89) | 5 (31.25) | 5 (50) | 16 (57.14) |
| On antidepressants, n (%) | 35 (51) | 12 (63.16) | 7 (43.75) | 6 (60) | 10 (35.71) |
Bold indicates statistically significant difference.
Note that this variable is dependent on how long participants kept receiving text messages (e.g., if someone quickly dropped out of the study or stopped receiving messages this number may be inflated).
Participants knows how to text refers to the question: “Do you know how to text?.”
Preferred method of contact refers to the question “In general, if someone needs to reach you, do you prefer that they call or that they text you?”
GCBT, Group Cognitive behavioral therapy; PHQ-9, 9-item Participant Health Questionnaire.
Analysis 1: Clusters of Participant Engagement
The silhouette method, a commonly used method to guide the selection of the number of clusters, identified K = 4 clusters as the optimal number of clusters. The score was close to other solutions (Supplementary Data), but K = 4 was the solution for which the maximum score was achieved. This clustering is also attractive, as it lends itself to clinical interpretation and coincides with our experience of high/low user groups. We thus decided to proceed with this solution before running statistical tests to maintain sound statistical results. The clusters were named according to the overall combination of total texting responsiveness and total attendance of participants: “Unengaged” (19 participants), “Mostly Mobile” (16 participants), “Mostly Live” (10 participants), and “Fully Engaged” (28 participants). These clusters represent various engagement patterns and possibly preferences. (see Fig. 1) See Table 1 for demographic and clinical characteristics by clusters.
Fig. 1.
Metrics for K-means clustering indicated four clusters of overall participant engagement with the study components. These clusters represent different patterns of texting and attending behavior during the study. Color images are available online.
Statistical differences in demographics between clusters
Clusters differed in mean age, F(3) = 3.6, p = 0.018. Post hoc Tukey Honest Significant Differences for multiple pairwise comparisons indicated that this was driven by differences in age between the Mostly Mobile and the Mostly Live cluster (44.0 vs. 59.0, p = 0.012). Furthermore, clusters differed in the number of participants that knew how to text at baseline (self-reported), (p = <0.001, Fisher exact test). Post hoc tests with Bonferroni–Holm correction indicated that this was driven by differences between the Unengaged and the Fully Engaged cluster (47.4% vs. 85.7%, p = 0.031), the Mostly Clinic and Mostly Mobile (40% vs. 81.25%, p = 0.035), and the Mostly clinic and Fully Engaged cluster (40% vs. 85.7%, p = 0.031). Clusters did not significantly differ in gender and preferred method for communication (texting or calling) (p's > 0.05). We did not examine other demographic variables because the sample was relatively homogenous (low income, low education, and mostly Spanish speaking).
Analysis 2: Mixed-Effects Models
We excluded participants who were in the Unengaged cluster, leading to the inclusion of 54 participants.
Likelihood of in-clinic GCBT attendance considering previous week response rate
The relationship between previous week response rate (percentage of texts responded to) and attendance (attended a sessions yes/no) was moderated by age (significant interaction of response rate with age). Previous week response rate from older individuals was less likely to indicate whether they would attend GCBT (Table 2a and Fig. 2). There was also a significant negative effect of time, indicating a decrease in weekly attendance over the course of the study (Table 2a) and a positive effect of age, indicating that higher age was associated with higher likelihood of attendance. This model accounted for individual variance in previous week response rate and time better accounted for overall variance than a model that only had random slopes (χ2 = 17.13, df = 5, p = 0.005). Because knowledge differed between clusters, we additionally explored a model including an interaction term between knowledge and previous week response rate (separately from the model with age, as these variables were correlated, r = 0.28, p < 0.001). There were no significant interactions or main effects of knowledge (p's > 0.05). Finally, we examined a separate model including the phase of study (randomized controlled trial or naturalistic). There was no significant evidence of a main or moderating effect of phase in study (p > 0.08).
Table 2.
Mixed Effects Models
| FIXED EFFECTS | 2A WEEKLY GCBT ATTENDANCE |
||
|---|---|---|---|
| ODDS RATIOS | CI | p | |
| Model 1: modeling weekly Group Cognitive Behavioral Therapy attendance | |||
| Intercept | 1.13 | 0.37–3.44 | 0.828 |
| Time | 0.92 | 0.86–0.98 | 0.015 |
| Age (centered) | 1.12 | 1.02–1.23 | 0.023 |
| Previous week responsiveness | 2.02 | 0.71–5.73 | 0.188 |
| Age × previous week responsiveness | 0.9 | 0.81–0.99 | 0.038 |
| FIXED EFFECTS | 2B RESPONSE RATE IN WEEK AFTER GCBT |
||
|---|---|---|---|
| ESTIMATES | CI | p | |
| Model 2: modeling response rate following Group Cognitive Behavioral Therapy | |||
| Intercept |
0.67 |
0.59–0.75 |
<0.001 |
| Age (centered) |
−0.01 |
−0.01–0.00 |
0.027 |
| GCBT attendance |
0.07 |
0.02–0.11 |
0.004 |
| Time | −0.15 | −0.26 to −0.04 | 0.009 |
Time represents week of study, in units of weeks. For model 2, the nonsignificant age × GCBT attendance term was removed.
Bold indicates statistically significant difference.
CI, confidence interval; GCBT, group cognitive behavioral therapy.
Fig. 2.
For illustrative purposes, we plot the marginal effects of the regression model. This plot shows the predicted probabilities for the response of the model of the association between previous week responsiveness and GCBT attendance by age (mean ± 1 SD). Confidence intervals are based on standard errors. Color images are available online.
Subsequent week response rate to text messaging considering in-clinic GCBT attendance at the start of the week
Attending a session of GCBT was associated with a higher level of responding to text messages the week following the GCBT session. Response rates decreased over time. There was no evidence of a significant moderating or main effect of age Supplementary Table S1. After removing the nonsignificant interaction term, we found that the main effect of age was significant (Table 2b). This indicates that overall, higher response rate to texts is related to a lower age. The model that accounted for individual variance in attendance and time in study was significantly better at accounting for overall variance than a model with only random slopes (χ2 = 85.02, df = 5, p = <0.001). There was no evidence of a significant moderating effect of knowledge of texting at baseline (p's > 0.08). Participants who were more familiar with texting at study entry were more responsive throughout the study (p = 0.006). Finally, in a separate model including the phase of study, there was no evidence of a moderating effect of study phase on the relationship of attendance with subsequent week response rates (p = 0.07).
Discussion
Summary of Principal Findings
Participants who attended weekly in-clinic GCBT and were enrolled in a supportive mood text-messaging intervention can be divided into four clusters of engagement (e.g., unengaged, mostly live, mostly mobile, and fully engaged). Between these clusters, age and knowledge of texting at study entry differed significantly. Furthermore, excluding the unengaged cluster, we found that the effect of texting on the probability of attendance depended on age. Previous week response rate from older individuals was less likely to indicate whether they would attend GCBT. Considering the reverse relationship, we found that participants who attended GCBT were more likely to be more responsive to text messages in the subsequent week, regardless of age.
Implications of Clusters
Identifying subgroups of participant engagement provides information on which types of participants benefit from text-messaging interventions added to GCBT. Importantly, a significant proportion of participants (about 25%) did not participate in either the texting intervention or in GCBT (the “unengaged” cluster), which is in-line with previous findings that a high proportion of users quickly abandon digital interventions17 and high attrition rates for psychotherapy.6
Our findings suggest that technological comfort is associated with digital engagement, which has been reported previously28 and merits further investigation. In the current study, participants who indicated that they did not know how to text initially (about 40% of the sample) were still invited to participate in the supportive text-messaging system. Research assistants showed these individuals how to text, but typically only did so during the baseline visit. Practitioners or researchers who integrate technology into their interventions could choose to only provide this intervention to those who are already familiar with texting. However, to be more inclusive and avoid a further widening of the digital divide,29 future work should explore incorporating additional assistance for those who are less comfortable with technology.
Directionality of the Relationship between Texting and In-Clinic Attendance
Previous work has emphasized that increasing engagement with digital interventions (text messaging, apps, or internet interventions) is likely associated with favorable key clinical outcomes,16 including psychotherapy attendance. However, it has not been examined rigorously enough if, and for which participants, this is the case.30
In this study, we find that a positive relationship between engagement in adjunct text-messaging interventions and GCBT attendance holds mostly for younger patients. We hypothesize that older adults might not need the extra incentive of engaging in the automated texting system to become more motivated to attend in-clinic therapy. This is supported by our finding that older age was independently associated with a higher probability of GCBT attendance, which was also found in previous work31 and with a lower level of texting response rate. Interestingly, we find evidence of the reverse relationship: GCBT attendance is associated with increased response rate to the texting adjunct in the subsequent week. Although younger age was independently associated with texting response rate, the relationship between attendance and increased response rate was not moderated by age. We hypothesize that weekly contact with the provider and group motivates participants to respond more to the mood text messaging, as they might associate these messages more with support from the provider.10
Alternatively, participants might use the mood messaging as a means of practicing concepts learned in-person, thus working to sustain treatment gains. This finding supports the notion of the importance of face-to-face contact to strengthen the potential of a therapeutic digital relationship between patient and provider.32,33 Furthermore, these results suggest that digital technology can not only increase engagement with face-to-face therapy (in certain participants) but also can emphasize how human input strengthens digital engagement. Our results highlight the complex nature of the relationship between increased engagement with digital health interventions and improved clinical outcomes. The therapeutic effect of technology added to in-person health care needs to be explored more in future work.
Limitations
We studied a relatively limited number of participants, particularly for the analysis that examined differences between the four clusters. Furthermore, focusing on an underserved population is the strength of our study, but our results might be specific to low-income participants served in a public hospital. Furthermore, we defined engagement as the response rate to text messages that required a mood rating. However, there are many ways of measuring engagement, some of which may yield different insights.15 Furthermore, we used the maximum silhouette score to guide the number of clusters used, instead of selecting a number ourselves (as that could be considered cherry-picking). However, there was only a slight absolute difference with other clustering solutions in this score (Supplementary Data). The number of clusters may merit further investigation in future studies. Finally, while the mixed-effects models consider lagged variables, it cannot fully be concluded that, for example, attending a GCBT session causes people to subsequently respond to more text messages. However, our results show interesting relationships and we have offered possible causes for these relationships that require further inquiry.
Future Directions
Particular focus should be given to determining how to engage participants who quickly abandon both the in-person and digital component of interventions. Furthermore, the challenge of how to make technology-based interventions more beneficial for older patients and those with limited tech comfort needs to be addressed. Future work might also benefit from assessing participants' psychological profiles and preferences for technology in more detail, to identify additional baseline factors that predict to what cluster of engagement participants will likely fall into. This may allow for the identification of targets for intervention, which could help to “nudge” participants, for example, from the unengaged to the fully engaged cluster. Furthermore, although still in the early stages, greater personalization of digital interventions, for instance by using machine learning methods to adapt content over time, might lead to higher effects, less drop-out, and more engagement with the intervention.34,35
Conclusion
We show that participants enrolled in a text-messaging adjunct for GCBT can be divided into different user profiles of study engagement. Further, we provide evidence of a bidirectional relationship between text-messaging response rate and GCBT attendance, which is, in part, moderated by age. Younger participants might benefit more from adjunct texting interventions when their purpose is to increase psychotherapy attendance. These findings emphasize how supportive digital intervention and face-to-face contact might both enhance each other's effectiveness. Our results also underline the importance of tailoring (supportive) digital health interventions to different users to avoid the risk of failure, in particular, to people of different ages and comfort levels with technology.
Supplementary Material
Acknowledgments
We thank Patricia-Avila Garcia, who helped with the recruitment, testing, and data collection of this study and Chris Karr, who helped develop and manage the automated text-messaging service.
Disclosure Statement
No competing financial interests exist.
Funding Information
Adrian Aguilera was supported by National Institutes of Mental Health—K23MH094442 and a Robert Wood Johnson New Connections Award.
Supplementary Material
References
- 1. World Health Organization. Depression and other common mental disorders: Global health estimates. World Health Organization, 2017 [Google Scholar]
- 2. Cuijpers P, Berking M, Andersson G, et al. . A meta-analysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Can J Psychiatry 2013;58:376–385 [DOI] [PubMed] [Google Scholar]
- 3. Forde F, Frame M, Hanlon P, et al. . Optimum number of sessions for depression and anxiety. Nurs Times 2005;101:36–40 [PubMed] [Google Scholar]
- 4. Hansen NB, Lambert MJ, Forman EM. The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychol Sci Pract 2002;9:329–343 [Google Scholar]
- 5. Nieuwsma JA, Trivedi RB, McDuffie J, et al. . Brief psychotherapy for depression: A systematic review and meta-analysis. Int J Psychiatry Med 2012;43:129–151 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Swift JK, Greenberg RP. Premature discontinuation in adult psychotherapy: A meta-analysis. J Consult Clin Psychol 2012;80:547–559 [DOI] [PubMed] [Google Scholar]
- 7. Westmacott R, Hunsley J. Reasons for terminating psychotherapy: A general population study. J Clin Psychol 2010;66:965–977 [DOI] [PubMed] [Google Scholar]
- 8. Chavira DA, Golinelli D, Sherbourne C, et al. . Treatment engagement and response to CBT among Latinos with anxiety disorders in primary care. J Consult Clin Psychol 2014;82:392–403 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Lindhiem O, Bennett CB, Rosen D, Silk J. Mobile technology boosts the effectiveness of psychotherapy and behavioral interventions: A meta-analysis. Behav Modif 2015;39:785–804 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Aguilera A, Berridge C. Qualitative feedback from a text messaging intervention for depression: Benefits, drawbacks, and cultural differences. JMIR mHealth uHealth 2014;2:e46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Aguilera A, Bruehlman-Senecal E, Demasi O, Avila P. automated text messaging as an adjunct to cognitive behavioral therapy for depression: A clinical trial. J Med Internet Res 2017;19:e148. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Aguilera A, Schueller SM, Leykin Y. Daily mood ratings via text message as a proxy for clinic based depression assessment. J Affect Disord 2015;175:471–474 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Bruehlman-Senecal E, Aguilera A, Schueller SM. Mobile phone-based mood ratings prospectively predict psychotherapy attendance. Behav Ther 2017;48:614–623 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Pew Research Center. Mobile Fact Sheet. Available at www.pewinternet.org/fact-sheet/mobile (last accessed February5, 2018)
- 15. Short CE, DeSmet A, Woods C, et al. . Measuring engagement in eHealth and mHealth behavior change interventions: Viewpoint of methodologies. J Med Internet Res 2018;20:e292. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Grady A, Yoong S, Sutherland R, et al. . Improving the public health impact of eHealth and mHealth interventions. Aust N Z J Public Health 2018;42:118–119 [DOI] [PubMed] [Google Scholar]
- 17. Pham Q, Graham G, Carrion C, et al. . A library of analytic indicators to evaluate effective engagement with consumer mHealth apps for chronic conditions: Scoping review. JMIR Mhealth Uhealth 2019;7:e11941. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 2001;16:606–613 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Hepner KA, Miranda J, Woo S, et al. . Building Recovery by Improving Goals, Habits, and Thoughts (BRIGHT). Santa Monica, CA: RAND Corporation, 2011 [Google Scholar]
- 20. Muñoz RF, Mendelson T. Toward evidence-based interventions for diverse populations: The San Francisco General Hospital prevention and treatment manuals. J Consult Clin Psychol 2005;73:790. [DOI] [PubMed] [Google Scholar]
- 21. Arbelaitz O, Gurrutxaga I, Muguerza J, et al. . An extensive comparative study of cluster validity indices. Pattern Recogn 2013;46:243–256 [Google Scholar]
- 22. Raudenbush SW, Bryk AS. Hierarchical linear models: Applications and data analysis methods. Vol. 1. Thousand Oaks, CA: SAGE Publications, Inc., 2002 [Google Scholar]
- 23. Barr DJ, Levy R, Scheepers C, Tily HJ. Random effects structure for confirmatory hypothesis testing: Keep it maximal. J Mem Lang 2013;68:10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Luke SG. Evaluating significance in linear mixed-effects models in R. Behav Res Methods 2017;49:1494–1502 [DOI] [PubMed] [Google Scholar]
- 25. Bates D, Maechler M, Bolker B, Walker S. lme4: Linear mixed-effects models using Eigen and S4. R Package Version 2014;1:1–23 [Google Scholar]
- 26. Kuznetsova A, Brockhoff PB, Christensen RHB. lmerTest package: Tests in linear mixed effects models. J Stat Softw 2017;82 [Google Scholar]
- 27. Powell MJ. The BOBYQA algorithm for bound constrained optimization without derivatives. Cambridge NA Report NA2009/06. Cambridge: University of Cambridge, 2009:26–46 [Google Scholar]
- 28. Norton BL, Person AK, Castillo C, et al. . Barriers to using text message appointment reminders in an HIV clinic. Telemed J E Health 2014;20:86–89 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Seifert A, Reinwand DA, Schlomann A. Designing and using digital mental health interventions for older adults: Being aware of digital inequality. Front Psychiatry 2019;10:568. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Yeager CM, Benight CC. If we build it, will they come? Issues of engagement with digital health interventions for trauma recovery. Mhealth 2018;4:37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Roseborough DJ, McLeod JT, Wright FI. Attrition in psychotherapy: A survival analysis. Res Social Work Pract 2016;26:803–815 [Google Scholar]
- 32. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: A model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13:e30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Torous J, Hsin H. Empowering the digital therapeutic relationship: Virtual clinics for digital health interventions. NPJ Digit Med 2018;1:16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Triantafyllidis AK, Tsanas A. Applications of machine learning in real-life digital health interventions: Review of the literature. J Med Internet Res 2019;21:e12286. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Bockting CL, Smid NH, Koeter MW, et al. . Enduring effects of Preventive Cognitive Therapy in adults remitted from recurrent depression: A 10 year follow-up of a randomized controlled trial. J Affect Disord 2015;185:188–194 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.