The engagement problem: A review of engagement with digital mental health interventions and recommendations for a path forward

Abstract

Purpose of the review

Digital mental health interventions (DMHIs) are an effective and accessible means of addressing the unprecedented levels of mental illness worldwide. Currently, however, patient engagement with DMHIs in real-world settings is often insufficient to see clinical benefit. In order to realize the potential of DMHIs, there is a need to better understand what drives patient engagement.

Recent findings

We discuss takeaways from the existing literature related to patient engagement with DMHIs and highlight gaps to be addressed through further research. Findings suggest that engagement is influenced by patient-, intervention- and systems-level factors. At the patient-level, variables such as sex, education, personality traits, race, ethnicity, age and symptom severity appear to be associated with engagement. At the intervention-level, integrating human support, gamification, financial incentives and persuasive technology features may improve engagement. Finally, although systems-level factors have not been widely explored, the existing evidence suggests that achieving engagement will require addressing organizational and social barriers and drawing on the field of implementation science.

Summary

Future research clarifying the patient-, intervention- and systems-level factors that drive engagement will be essential. Additionally, to facilitate improved understanding of DMHI engagement, we propose the following: (a) widespread adoption of a minimum necessary 5-element engagement reporting framework; (b) broader application of alternative clinical trial designs; and (c) directed efforts to build upon an initial parsimonious conceptual model of DMHI engagement.

Introduction

With mental health concerns reaching unprecedented prevalence (1, 2), there is urgent need for accessible, evidence-based psychiatric treatments. Digital mental health interventions (DMHIs) delivered via the internet and/or mobile apps offer a promising avenue for meeting this challenge. A number of reviews and meta-analyses have found that DMHIs are efficacious (3, 4). Additionally, they have the potential to remove many of the barriers that plague traditional psychiatric treatment (5–7). Most notably, DMHIs designed by professionals trained in evidence-based treatment can be much more widely available than treatment provided by individual clinicians at virtually no marginal cost. They also address other key issues with standard mental health treatment: they promote patient autonomy; they offer convenience (not requiring workday appointments or transportation); and they can be accessed at times when patients are most in need of support (which often do not align with when clinic appointments are scheduled).

Despite their promise, DMHIs have increasingly been demonstrated to have a major shortcoming: patient engagement with them is poor. When speaking about DMHI engagement, it is important to note that this concept refers to engagement in real-world conditions, such as when these interventions are implemented in routine care or commercial settings (e.g., sold via the Google Play or the Apple App Store). Engagement in efficacy trials, consisting of highly motivated users who seek out participation in a study and meet multiple eligibility criteria, is typically high, but does not reflect real-world conditions (8). This difference between engagement in more controlled efficacy trials and less controlled implementation settings is well illustrated in a study conducted by Gilbody et al. (9) in which two DMHIs (Beating the Blues and MoodGYM)—both with prior randomized controlled trials (RCTs) demonstrating strong efficacy (10, 11)—were implemented in routine care. Fewer than 20% of participants completed either of the assigned interventions. The issue of DMHI engagement is not specific to the interventions studied in Gilbody et al. (9). Indeed, studies using other DMHIs show similarly low rates of sustained engagement and intervention completion in real-world settings (12–16).

When discussing engagement, it is also important to note that some drop out from any treatment or technology is to be expected. After all, meta-analyses of psychotherapy suggest that around 20% of patients discontinue prematurely and these rates are even higher for treatment with psychotropic medication (17, 18). Similarly, average 30-day retention rates for mobile applications in general are under 6% (19). Even for exceptionally popular apps outside the healthcare domain, like Instagram and Twitter, 90-day retention rates are only 30–50% (20). It is important to note, however, that unlike standard consumer apps, DMHIs are clinical treatments where suboptimal engagement can have significant repercussions.

Thus, in the quest to understand DMHI engagement and to ascertain whether DMHIs are a viable solution to the unprecedented mental health concerns worldwide, several core questions emerge. First, at the patient-level, what characterizes patients who are likely to engage with these interventions? Second, at the intervention-level, what intervention components improve engagement? Third, at the systems-level, what aspects of the larger healthcare and social climate promote engagement? And fourth, in the context of research, how can we build an evidence base that allows us to address this problem?

Definitions of Engagement

Before examining these questions, it is important to clarify what is meant by “engagement.” Within psychiatry, the most common definitions of engagement relate to behaviors and this is where our discussion will focus. We offer an operational definition of behavioral engagement as use of the core components of a DMHI. Common metrics of behavioral engagement include uptake (i.e., downloading and using the intervention at least once), sustained use (i.e., remaining active in using the intervention for some period of time after downloading), and adherence/completion (i.e., using the intervention at the intended frequency for the intended duration). For DMHIs, these behavioral engagement metrics can readily be tracked on system backends, making engagement with DMHIs easier to gauge than engagement with other treatments such as psychotropic medication. This proposed operational definition of engagement is supported by significant research suggesting a relationship between behavioral engagement (i.e., usage) and clinical outcomes (21). It is important to note that engagement requirements differ across DMHIs. There is not one set amount of use that defines completion. Instead, this is defined in the context of each specific DMHI.

Engagement can also be defined more broadly. Many human-computer interaction researchers define engagement as capturing and maintaining the attention and interest of users and their temporal, emotional and cognitive investment with an intervention (22). Nahum-Shani et al. (23) discuss engagement with digital content (i.e., using the DMHI), engagement with notifications from the DMHI (i.e., reading and thinking about them), and engagement with non-digital tasks (e.g., behavioral activation exercises recommended by the DMHI) all as part of the multi-dimensional construct of engagement. Definitions like these are conceptually important as they capture the full and true meaning of DMHI engagement. They are also more difficult to reliably measure in the context of intervention trials and real-world implementations. Metrics such as number of clicks or active time spent in the DMHI may be reasonable proxies for cognitive engagement, but are far from exact. Some studies ask users for a self-report of their emotional reactions to content or whether they acted upon the intervention recommendations as a way to measure these more nuanced yet critical aspects of engagement. Additionally, self-report measures such as the User Engagement Scale (24, 25), Digital Working Alliance Inventory (26, 27) and the Unified Theory of Acceptance and Use of Technology scale (28) are validated options for measuring aspects of this broader definition of engagement. While such measures can be useful, backend measures of behavioral engagement are still typically most available and most feasible in the context of real-world implementation, where engagement is most likely to be a concern.

The literature on engagement also frequently discusses two other metrics. The first is study completion, which refers to completion of post-treatment assessments, not completion of the key components of the DMHI. While study completion could be argued to serve as a proxy of DMHI engagement, its applicability is limited to research contexts. It is primarily useful as a metric for calculating sample size requirements for efficacy and effectiveness trials, not as an indication of DMHI engagement. The second is patient interest in certain interventions or aspects of interventions. Understanding patient interests and differences in interest across different demographic subgroups is useful for designing effective interventions. However, it does not necessarily extend to actual in situ engagement with those interventions.

Patient-Level Engagement Considerations

Across medicine, differential rates of treatment engagement by patient population have been identified and vary by healthcare domain. A handful of demographic variables have consistently been associated with higher engagement with DMHIs. Specifically, women show higher levels of DMHI engagement than men (29–34). Similarly, several studies have found that higher education levels predict higher engagement (32, 35–37). Finally, a number of studies suggest that certain personality traits, specifically neuroticism, agreeableness and introversion, are associated with greater interest in using DMHIs (38, 39). Out of these, however, only neuroticism has been found to be associated greater usage (40).

There remains some ambiguity on the extent to which other patient-level variables like race and ethnicity, age and symptom severity are related to engagement. With regard to race and ethnicity, some studies suggest that DMHI interest and intervention usage is greater among racial and ethnic minorities (41–43). Yet, other studies indicate that white patients show higher engagement (29). This is an important area for future inquiry because racial and ethnic minority patients tend to engage with traditional outpatient mental health services less frequently (44). If interest and usage are high in these populations, a potential opportunity exists to focus on reaching them with evidence-based services using DMHIs. These conflicting findings raise important questions about the extent to which tailoring DMHIs for specific racial or ethnic groups may improve engagement.

Similarly, studies have drawn different conclusions about the relationship between age and engagement in DMHIs. Some studies have found that younger patients express higher interest or show more behavioral engagement, whereas other studies have shown the opposite (29, 43, 45–47). These seemingly conflicting findings could potentially be attributed to differences in age breakdowns across studies and a potential nonlinear relationship. That is, perhaps age at either extreme (e.g., younger than 30 and older than 50) is a risk factor for disengagement (48).

Finally, a number of studies suggest that individuals with more severe symptoms report greater interest in DMHIs (47, 49, 50). However, when it comes to actual usage, some studies suggest higher usage among those with more severe symptoms (29, 50), whereas other studies suggest that more severe symptoms are associated with lower usage (34, 51, 52). It is possible that some of these conflicting findings could be attributed to difficulty drawing conclusions across studies of patients with different diagnoses. For example, some research suggests that patients with depression, which is marked by lack of motivation, show lower engagement than patients with anxiety disorders (53, 54). It also could be that the relationship between symptom severity and engagement is not linear. That is, higher severity of symptoms may motivate engagement up to a point, but those with the most severe symptoms may be less engaged. These questions require further exploration with more refined classification of data.

Intervention-Level Engagement Considerations

An important attribute of any DMHI intervention is the extent to which it includes some approach or strategy to enhance engagement. To date, the most heavily researched intervention-level engagement strategy has been the addition of human support. Meta-analyses suggest a medium positive effect size of including human support versus no human support on the efficacy of DMHIs (55, 56). But the impact on engagement, and specifically engagement in real-world implementations of DMHIs, is less clear. Various studies suggest that human support increases engagement (57–59), but the finding is not universal. For example, Levin et al. (60) found that weekly coaching calls did not increase engagement relative to automated email prompts. Additionally, strategies for adding human support are widely varied—from less scalable options like weekly phone calls with a clinician to more scalable options like asynchronous communication with a health coach. The more scalable coach support protocols have been found to be effective at enhancing engagement in some studies (36, 61, 62), however other studies have found this style of coach support does not enhance engagement (63, 64). Thus, while human support appears to be a promising engagement strategy, results are not unequivocally favorable. The existing literature leaves many questions unanswered regarding the specific components of such support that drive engagement and the optimal dose of such support.

Gamification is another engagement strategy that has garnered recent interest. Gamification refers to incorporating principles from gaming into the DMHI. These include leveling up, winning points or virtual rewards, integration of short-term challenges, and use of imaginary settings or narratives (65). One of the key arguments for gamification has been that it could make interventions more fun or rewarding, and therefore, keep patients engaged for longer. Yet, while gamified interventions have often been found effective, few studies have actually evaluated how effective gaming intervention components are at improving engagement (66, 67). Users have expressed mixed interest in gamified interventions (68) and at least one recent meta-analysis suggested that gamified depression apps did not generate improved adherence or efficacy over depression apps without gamification (69).

The use of contingency management is also a frequently discussed intervention-level DMHI engagement strategy. Contingency management is a principle drawn from behavior therapy referring to reinforcing or rewarding behavior change. The most common types of rewards applied are monetary either in the form of cash or prizes, but other types of rewards can also be used. There is significant research showing that contingency management improves treatment outcomes in the context of health concerns like substance use, medication and treatment adherence and a range of health risk behaviors (70–72). Some recent studies have shown that these types of incentives also increase digital intervention adherence for health behaviors (73, 74). Applications of contingency management specific to engagement with DMHIs are limited, but early results are promising. For example, Boucher et al. (75) found that monetary incentives increased the regularity and volume of usage of a DMHI for depression and/or anxiety.

Finally, persuasive technology is another promising DMHI engagement strategy. Persuasive technology features that have been applied in DMHIs include text messages, push notifications, interactive features, opportunities for data visualization and tailoring/personalization of intervention content (76). Significant research supports use of persuasive technology for engagement. Survey and qualitative studies suggest that patients express a desire for personalized content (77–79). Additionally, use of reminders (80), interactivity (81, 82), tailored push notifications (83), and data visualization (84) have been found to increase engagement with digital interventions. Persuasive technology may even be as effective as human support in enhancing engagement. For example, Kelders et al. (85) found that engagement was equivalent when a DMHI was enriched with persuasive technology features (i.e., tailoring, personalized feedback) and when it was enriched with coach support.

Just-in-time adaptive interventions (JITAIs) are a promising application of persuasive technology where users are prompted to interact with specific intervention content based on contextual data collected via self-report or passive monitoring (86). JITAIs can be effective for mental health concerns (87, 88) and may enhance engagement by delivering intervention content when a user is most receptive to it. However, real-world implementations in mental health are still limited (89), and the impact on engagement has not been directly evaluated.

Systems-Level Engagement Considerations

Strategies for healthcare systems to seamlessly integrate DMHIs in the context of routine care, thereby supporting engagement, have not been a focus of DMHI research to date. However, studies suggest that weaving DMHIs into the fabric of existing primary or specialty care may be a particularly promising approach for enhancing engagement. Specifically, previous work has shown that patients endorse greater interest in interventions recommended by their care team than those not accompanied by such a recommendation (90, 91). Additionally, referral from a healthcare provider is associated with lower DMHI attrition (92). These findings are consistent with research suggesting that social influence, defined as the extent to which important others support a given behavior, positively impacts technology adoption (28).

The research on barriers and facilitators of DMHI adoption and sustained use suggests a number of organizational variables that merit attention. Both Borghouts et al. (48) and Graham et al. (93) provide excellent overviews of this literature. Findings regarding several systems-relevant barriers and facilitators offer a starting point for determining what engagement strategies may be worth evaluating at the systems-level. Specifically, systems-relevant barriers identified in this literature include interoperability issues with other clinical systems, limited technical support, limited staff resources, cost/limited avenues for DMHI reimbursement, clinicians’ negative perspectives on DMHIs, and limited support from clinical leadership (48, 93). Anastasiadou et al. (77) found that both patients and providers perceived barriers related to the organizational environment as more prominent than patient-level and intervention-level barriers. These findings suggest value in drawing on allied fields like implementation science and its rich set of implementation frameworks to inform the study of DMHI engagement.

Implementation science has a long tradition of exploring multi-level strategies to support the uptake and sustainment of evidence-based interventions. For example, the Expert Recommendations for Implementing Change study systematically compiled input from stakeholders and published a list of implementation strategies and definitions of these strategies (94). This list offers a directory for identifying implementation strategies that could promote engagement with DMHIs at the systems level. Studies drawing on this strong body of implementation science literature that test application of various strategies in different contexts (e.g., primary care, community care, specialty care, marketplace) is an important next step in DMHI research.

Important Gaps for Research

There are a plethora of important areas for research into engagement with DMHIs, many of which we have noted in the previous sections. Below, we highlight three additional areas that we have not yet touched upon that are particularly important for the research community to address.

Adopting Standards for Reporting Engagement Metrics. A recent review (95) of DMHIs for depression indicated that consistency in reporting engagement metrics is alarmingly poor. Specifically, only 64% of studies reported the number of participants who used the DMHI at least once; only 23% of studies reported how many participants were still using the DMHI during the last week of the treatment period; and only 50% of studies reported the number of participants who completed the DMHI.

Unfortunately, we cannot attribute these results to difficulty measuring engagement because metrics for DMHI use, as noted above, are typically quantifiable on the system backend. We also cannot attribute them to the study of DMHIs being a new field because depression represents one of the most heavily researched clinical areas for DMHIs.

These results suggest that establishing reporting guidelines that specify the minimum necessary provision of information on engagement when publishing clinical trials of DMHIs is critical. As a starting point, Lipschitz et al. (95) suggest that a five-element standard of engagement reporting be adopted for all studies of DMHIs. This framework encompasses the following essential metrics:

1. Adherence criteria, defined as an explicit statement of what it means for participants to have used the DMHI as intended or met some minimum intervention use threshold. This could be defined in terms of content coverage (e.g., 80% of modules completed), frequency of use (e.g., use at least three times per week during the intervention period), or some other a priori threshold for intervention adherence.

2. Rate of uptake, defined as the percentage of participants randomized or referred to the DMHI who downloaded the intervention and used it at least once.

3. Level-of-use metrics, defined as both the total number of DMHI launches (i.e., average number of times used) and the total amount of time the DMHI was used (e.g., total minutes of use) during the intervention period.

4. Duration-of-use metrics, defined as the number of participants who used the app at least once per week every week of the intervention period unless less frequent use is identified as sufficient in the adherence criteria. Reporting the number of participants still using the DMHI in the final week of the intervention period or a survival analysis of time to last use are also particularly helpful duration-of-use metrics because they convey how long patients typically engage with the DMHI. When positive clinical outcomes are observed, these metrics also offer insight into the timeline for expected clinical improvement.

5. Number of intervention completers, defined as the number of participants who completed the intervention as intended per the specified adherence criteria.

Adopting this or some other minimum necessary reporting criteria is essential to move the field of DMHI engagement forward. Such reporting guidelines would allow for new insights into what constitutes sufficient engagement for clinical benefit; facilitate comparisons among DMHIs and between DMHIs and other treatment options; and offer benchmarks upon which further research must improve.

Considering Alternative Study Designs. To date, RCTs with parallel group designs have been the most common methodology in DMHI trials. But these only tell us whether an intervention package as a whole has a causal impact on outcomes of interest. Such trials are not designed to shed light on which components of an intervention impact the outcomes or when and how different intervention components should be applied.

Several other clinical trial designs offer data-driven strategies for answering questions related to treatment optimization for user engagement. As such, they provide efficient strategies for answering a number of questions related to engagement. The most widely used example is the factorial clinical trial, which involves packaging intervention components into various combinations such that the impact of each combination, as well as the main effect of each component itself (across combinations), can be evaluated (96, 97). Other less widely-used examples of trial designs include sequential multiple assignment randomized trials (SMARTs; 98, 99) and micro-randomized trials (MRTs; 100).

SMARTs (98, 99) offer insight into how components of a treatment package should be sequenced to optimize outcomes. They involve testing alternative treatments as a starting intervention and then identifying, at pre-specified points early in treatment, which individuals are responding/not responding to the initial intervention and randomizing both responders and non-responders to appropriate second-stage treatments. The causal effects of the initial treatment approach and the adapted treatment approach on an outcome of interest can then be evaluated. SMARTs could be used to help identify how to augment interventions or what intervention components to add for patients who exhibit poor engagement with DMHIs. For example, one possible first stage intervention in a SMART might be a self-guided DMHI with or without tailored motivational messaging (randomized at a 50–50 split). After two weeks, those participants in either condition showing insufficient engagement could be re-randomized to receive augmentation with coach support or to continue with the self-guided intervention. SMARTs offer a particularly compelling design for implementation studies because they allow an opportunity to test adaptive implementation strategies for clinics or specific patients who do not respond favorably to an initial approach (e.g., 101, 102).

MRTs (100) offer insight into whether intervention components of a DMHI have a proximal or near-term effect on engagement and when those components should be delivered to maximize that effect. They involve establishing a schedule of “decision points”: times at which a component of a DMHI might be delivered (e.g., multiple times per day). For example, a DMHI component may involve the delivery of motivational push messages to the patient via a smart device. At each decision point, participants would be randomly assigned to receive or not receive these motivational push messages. Then the proximal impact (i.e., over the next hour, day or week rather than the full intervention period) of the intervention component (i.e., motivational push messages) as well as the interaction between that proximal impact and context (e.g., time of day, location when the message was delivered) are evaluated.

Studies employing these designs offer untapped opportunities to better understand engagement and optimize interventions for engagement. For example, they offer opportunities to efficiently test the effects of tailoring intervention components to a given patient’s demographic characteristics (e.g., age, gender or race).

Conducting Studies that Facilitate Building a Theory of DMHI Engagement. At this point, there is not one widely accepted theory of what drives DMHI engagement. Probably the most widely applied theoretical models are the Technology Acceptance Model (103) and the Unified Theory of Acceptance and Use of Technology (UTAUT; 28). These models were developed and validated predominantly in the context of employee adoption of new information technologies, a considerably different context than the adoption of patient-facing, medical treatment technologies. Furthermore, the recently articulated Affect-Integration-Motivation and Attention-Context-Translation (AIM-ACT) framework has been proposed as an outline of psychological processes that dictate in-the-moment engagement with digital stimuli and may inform development of a broader theory on DMHI engagement (23). Finally, there is an expansive literature on behavior change theories related to treatment adherence (104). However, adherence to treatments like prescribed medication, for example, is also considerably different from adherence to DMHIs. Most notably, medication adherence is typically less cognitively demanding, less time consuming, and supported by more established efficacy data. While this literature-base provides a starting point for conceptualizing what drives adoption and sustained engagement with DMHIs, there is likely room to improve and hone these models.

The key will be to develop and validate theories that can inform intervention designers about constructs that are most robustly associated with DMHI engagement. To do so, we can look toward constructs that show strong evidence in both the technology adoption and treatment adherence literature. Several constructs already exhibit robust associations with both technology adoption and treatment adherence and could serve as a starting point for a parsimonious theory of DMHI engagement. These constructs include: social influence (beliefs among people who are important to the patient that he/she should engage in the new behavior; 105, 106, 107), facilitating conditions (environmental support for the new behavior; 108, 109), attitude (the balance of positive and negative feelings about the behavior; 110, 111–115), self-efficacy (beliefs in one’s ability to execute the new behavior; 111, 116–119), and habit strength (degree to which the behavior is an automatic part of one’s daily routine; 120, 121, 122). Initial theories specific to DMHI engagement could capitalize on these constructs as well as some of the predisposing characteristics discussed above and add additional constructs as new research emerges (see Figure 1).

Figure 1.

Parsimonious Conceptual Model of DMHI Engagement

Using this framework, some intervention characteristics, such as contingency management or gamification, could be conceptualized as variables that may produce higher engagement by shifting users’ attitudes toward the intervention (i.e., greater positive feelings about using the intervention). Other intervention characteristics, such as the integration of human support, could be conceptualized as enhancing engagement via social influence (in the case of involvement of a clinical team member with a prior relationship with the patient) or a facilitating condition (in the case of a newly assigned health coach specific to the intervention).

Taken together, DMHI engagement theory is an area ripe for innovation. Research can help evolve our understanding of theory by measuring constructs drawn from the cross-disciplinary theories and frameworks put forward above and evaluating associations between these constructs and DMHI usage data or other engagement metrics. This will be an essential part of building a science of DMHI engagement and improving the utility of these treatments.

Conclusions & Recommendations

DMHIs hold tremendous promise to transform psychiatric treatment by dramatically increasing access to evidence-based care. However, engagement is a critical issue that will likely determine the extent to which DMHIs become a mainstay of psychiatric treatment. Adequately addressing the issue of engagement will require acknowledging several key points. First, engagement is a multi-level issue. It must be addressed at the patient-, intervention- and systems-levels. Second, engagement is a multidisciplinary issue. Addressing it will require collaboration between clinicians, data scientists, human-centered design researchers, technologists, and organizational leaders. Third, while there are many aspects of engagement, at its core, the engagement problem is an implementation problem. Like other innovations and evidence-based practices that are the focus of implementation research, many DMHIs may have demonstrated efficacy in controlled settings, but in real-world settings they do not get and keep patients engaged enough to show sound clinical impact. Evaluating engagement requires studies in naturally-occurring, uncontrolled, routine care and marketplace settings. And finally, the engagement problem can only be solved by rigorous research. Specifically, researchers must address the engagement issue head-on by systematizing their reporting of engagement levels, considering alternative clinical trial designs rather than defaulting to RCTs with parallel group designs, and building a robust theoretical basis for evaluating the relationship between possible engagement-driving constructs and observed behavioral engagement. Only then will DMHIs have the potential to be the paradigm-changing force they could be in the treatment of mental health conditions.