Abstract
Mental health concerns are common, and various evidence-based interventions for mental health conditions have been developed. However, many people have difficulty accessing appropriate mental health care and this has been exacerbated by the COVID-19 pandemic. Digital mental health services, such as those delivered by mobile phone or web-based platforms, offer the possibility of expanding the reach and accessibility of mental health care. To achieve this goal, digital mental health interventions and plans for their implementation must be designed with the end users in mind. In this Review, we describe the evidence base for digital mental health interventions across various diagnoses and treatment targets. Then, we explain the different formats for digital mental health intervention delivery, and offer considerations for their use across key age groups. We discuss the role that the COVID-19 pandemic has played in emphasizing the value of these interventions, and offer considerations for ensuring equity in access to digital mental health interventions among diverse populations. As healthcare providers continue to embrace the role that technology can play in broadening access to care, the design and implementation of digital mental healthcare solutions must be carefully considered to maximize their effectiveness and accessibility.
Globally, mental health concerns are both common and undertreated1. In the United States, fewer than half of individuals in need of mental health treatment receive care2,3. Moreover, there are substantial variations in unmet treatment needs by geographic region, with more than 123 million Americans living in federally designated Mental Health Professional Shortage Areas4. For individuals who live in areas with mental health clinicians, there remain barriers to accessing mental health care. These barriers include, but are not limited to, stigma, cost, privacy concerns, a preference for dealing with problems independently and time constraints5,6. One study in the United States found that 62% of adults with a mental illness and 41% of adults with a serious mental illness did not receive any mental health care in the past year7. For individuals who do seek treatment, many do so in primary care medical settings in which pharmacotherapy is often used with inadequate dosing and follow-up care8.
To address these treatment gaps, a range of digital mental health interventions (DMHIs) have been developed over the past 25 years. These interventions have been developed to offer an alternative to traditionally delivered mental health interventions, and may be more accessible and acceptable to some individuals. We use DMHIs to refer to programmes that deliver psychological strategies and interventions via online and/or mobile platforms. We include telehealth services, in which mental healthcare services are delivered synchronously via a telephone or videoconference-based connection, under the umbrella term of DMHIs.
Many individuals and healthcare providers quickly adopted DMHIs to provide mental healthcare services while maintaining physical distancing protocols during the COVID-19 pandemic9,10. Organizational policies and state laws shifted to accommodate the need to deliver healthcare services at a distance, such as allowing clinicians to deliver remote services from home. Moreover, patients and providers alike became more familiar with the potential benefits of DMHIs11. Thus, the COVID-19 pandemic has hastened the adoption and development of some DMHIs (see BOX 1). Given the widespread utilization of digital tools, it is likely that DMHIs will continue to be used even as physical distancing protocols for infection control are no longer needed. Yet questions remain on how to best integrate DMHIs into future healthcare systems.
Box 1 |. Role of COVID-19 in expanding the availability of DMHIs.
The public health guidance related to the COVID-19 pandemic (for example, keeping physical distance from others and staying home) facilitated an expansion of digital mental health services171. This occurred, in part, because face- to-face services were halted in many areas of the world, and because pandemic-related stress prompted more individuals to seek mental health support172.
The most notable impact of the pandemic on mental health services was the rapid scale-up of virtual (that is, telephone-delivered and videoconference-delivered) mental health service provision. Studies of this scale-up to virtual mental health appointments indicate that the switch to virtual often took a few weeks173. Initially, as appointments switched to virtual, appointment counts decreased, and then increased substantially as time progressed173. Furthermore, virtual mental health services attracted a large percentage of patients for whom it was their first time using virtual health services, even in healthcare settings in which virtual services were previously offered174,175.
The COVID-19 pandemic also produced new opportunities for web-based and app-based digital mental health interventions (DMHIs) targeting pandemic-r elated distress. For example, an individually tailored COVID-19-focused Internet-based cognitive behavioural therapy (iCbT) programme demonstrated preliminary evidence of effectiveness at reducing stress, along with symptoms of depression and anxiety176. In the United States, a publicly available app targeting COVID-19 distress was developed and attracted nearly 50,000 users in a 5-month period, but clinical outcomes have not yet been published177.
More specifically, although many DMHIs show clinically meaningful improvements relative to controls when tested under tightly controlled research settings12, there is a large research to practice gap: many attempts to implement DMHIs in real-world settings have been unsuccessful and people do not engage with them13. This is not to say that all implementations of DMHIs have failed. Indeed, numerous Internet-based cognitive behavioural therapy (iCBT) programmes implemented in routine care settings, primarily outside the United States, have produced clinically meaningful effects, albeit to a lesser degree than in tightly controlled trials14. However, many existing DMHI programmes are not acceptable to the targeted end users and clinicians; in other words, they do not fit into clinicians’ workflows and are not aligned with how patients want to use personal technologies15. This is clearly problematic because if people are not accepting of DMHIs, and thus not accessing these interventions, they are not able to benefit from them. Without acceptability and subsequent adoption of interventions by healthcare providers and patients, efficacy of these interventions is meaningless from a public mental health standpoint.
In this Review, we aim to help inform the next steps for DMHIs that foster the widespread use of digital mental health services. First, we present an overview of DMHIs. We outline the treatment approaches that are used across mental health diagnoses, explain the range of ways in which these interventions are delivered and consider how interventions for individuals in key age groups vary in design, content and delivery methods. Then, we provide recommendations for integrating DMHIs into routine clinical care and promoting the accessibility of these interventions beyond the COVID-19 pandemic, with a particular focus on equity in access among diverse populations.
Therapeutic models
Digital mental health services are typically designed based on psychological treatments that have demonstrated efficacy in traditional, face-to-face mental health care. In this section, we discuss the treatment approaches commonly found in digital mental health services: cognitive behavioural therapy (CBT), acceptance and commitment therapy (ACT) and psychodynamic therapy (PDT). These therapeutic models were selected for review here due to their common use in face- to-face therapy and the frequency with which they have been used in the development of DMHIs. An overview of the therapeutic rationale and differences between therapeutic models in face- to-face and DMHI contexts is provided in FIG. 1.
We also review the literature on the efficacy and effectiveness of these approaches. Examples of recent meta-analyses on various types of DMHI for a range of treatment targets are provided in TABLE 1. This table is intended to provide a general overview and is not necessarily a comprehensive overview of all DMHI research to date.
Table 1.
Article | Number of studies included | Treatment target(s) | Population | Digital mental health intervention | Contrast | Effect size |
---|---|---|---|---|---|---|
Andrews et al., 2018 (REF.63) | 53 | Depression and anxiety | Adults | iCBT | iCBT vs treatment as usual | g = 0.38* |
iCBT vs waiting list control | g = 0.90* | |||||
iCBT vs face- to- face CBT | g = 0.14 | |||||
Guo et al., 2020 (REF.166) | 20 | Social anxiety disorder (SAD) | Adults | iCBT | iCBT vs control groups | g = −0.55* |
iCBT vs face- to- face CBT | g = −0.18 | |||||
Soh et al., 2020 (REF.167) | 33 | Insomnia | Adults | iCBT | iCBT vs control groups | d = 0.392* |
Wootten, 2016 (REF.168) | 18 | Obsessive- compulsive symptoms | Adults | Remote CBT (delivered by computer, telephone and videoconference) | Remote CBT vs control | g = 1.06* |
Remote CBT vs face- to- face CBT | g = −0.21 | |||||
Cervin and Lundgren, 2021 (REF.169) | 9 | Anxiety disorders | Children and adolescents | Technology- delivered CBT (tCBT; delivered by Internet, app, mobile phone or tablet computer) | tCBT vs control | OR = 4.73* |
Thompson et al., 2021 (REF.48) | 25 | Anxiety, depression, quality of life and psychological flexibility | Adults | iACT | iACT vs control groups on anxiety | g = 0.24* |
iACT vs control groups on depression | g = 0.38* | |||||
iACT vs control groups on quality of life | g = 0.27* | |||||
iACT vs control groups on psychological flexibility | g = 0.32* | |||||
Lindegaard et al., 2020 (REF.55) | 7 | Depression, anxiety and quality of life | Adults | iPDT | iPDT vs controls on depression (g = 0.46), anxiety (g = 0.20) and quality of life (g = 0.40) | g = 0.46* |
iPDT vs controls on anxiety | g = 0.20* | |||||
iPDT versus controls on quality of life | g = 0.40* | |||||
Linardon et al., 2019 (REF.80) | 66 | Depression, anxiety, stress, quality of life | Adults | Other: mobile apps | Smartphone apps vs controls on depressive symptoms | g = 0.28* |
Smartphone apps vs controls on generalized anxiety | g = 0.30* | |||||
Smartphone apps vs controls on stress | g = 0.35* | |||||
Smartphone apps vs controls on quality of life | g = 0.35* | |||||
Linardon et al., 2020 (REF.170) | 36 | Eating disorders | Adolescents and adults | Other: Internet- based, app- based and CD- ROM- based interventions | Digital interventions vs control conditions in reducing established risk factors and symptoms in prevention- focused trials | g values range from 0.19* to 0.43* |
Digital interventions vs control conditions in reducing established risk factors and symptoms in treatment- focused trials | g values range from 0.29* to 0.69* |
Hedge’s g effect sizes can be interpreted as 0.2 is a small effect, 0.5 is a medium effect and 0.8 is a large effect. Linardon et al.170 presented meta- analytic results for 35 outcomes; for brevity, results are summarized in ranges. Cervin and Lundgren169 presented an odds ratio (OR) rather than a Hedge’s g value in their published meta- analyses; the OR indicated that participants who received tCBT were approximately four times as likely as control participants to achieve remission from their primary anxiety disorder. CBT, cognitive behavioural therapy; DMHI, digital mental health intervention; iACT, Internet- based acceptance and commitment therapy; iCBT, Internet- based cognitive behavioural therapy; iPDT, Internet- based psychodynamic therapy.
Statistically significant result.
CBT, ACT and PDT are not the only frameworks on which DMHIs are based. Other therapeutic models are discussed in BOX 2.
Box 2 |. Other therapeutic approaches used in DMHIs.
Positive psychology interventions aim to increase well-being by cultivating positive feelings (such as gratitude), positive cognitions (such as ‘I can handle this’) and positive behaviours (such as performing acts of kindness). There is evidence that digital intervention programmes based within a positive psychology framework, which focuses on supporting patients to understand and recognize areas of growth, and work towards resilience and a greater sense of well-being, are effective at reducing depressive symptoms38,178.
Interpersonal therapy is a well-supported treatment for mental health concerns such as depression179. Interventions focus on improving the quality of a client’s interpersonal relationships and social functioning to help reduce their distress. Interventions focus on resolving problems in one of four key areas: interpersonal deficits, unresolved grief, life transitions or interpersonal disputes. Interpersonal therapy has been delivered in digital mental health intervention (DMHI) formats but is less effective than Internet-based cognitive behavioural therapy (iCBT)180.
DMHIs can also be designed to support couples. There is growing evidence for the efficacy of DMHIs based on integrative behavioural couple therapy181,182. Integrative behavioural couple therapy interventions focus on the emotional reactions of partners to the difficulties they encounter in their relationships. Therapeutic strategies include empathic joining around the problem (in which partners begin to empathize with one another about their problem), unified detachment from the problem (in which partners understand that they have co-created their current situation, and thus the problem) and building tolerance to some of the responses that the problem can trigger (such as becoming less blaming and more supportive while recognizing that the other partner is in pain).
Finally, there are numerous commercially available DMHIs, particularly mobile apps, that are not specifically based within a psychotherapeutic framework183. There is no research evidence for the effectiveness of such programmes.
Cognitive behavioural therapy.
CBT is a class of psychological interventions based on the premise that cognitive and behavioural factors contribute to the maintenance of emotional distress and behaviour problems16. For example, individuals might be taught to identify unhelpful thoughts and replace them with more helpful thoughts (a process referred to as cognitive restructuring) and might be prompted to modify maladaptive behavioural patterns. For example, an individual might be taught to identify an unhelpful thought such as ‘she did not say hi to me because she’s mad at me’ and replace it with a thought such as ‘she might not have seen me, or maybe she was in a hurry and did not have time to talk’. There are a wide variety of disorder-specific CBT protocols that address specific cognitive and behavioural maintenance factors (for example, biased thinking and avoidance) associated with different mental health disorders (for example, depression and anxiety)17,18 and behavioural health issues (for example, pain)19. CBT sessions with a therapist are typically held once a week for a few months. Meta-analyses of face- to-face CBT for various mental health disorders have found that CBT is either more effective than or comparably effective with other forms of psychotherapy20. Thus, CBT is often considered the gold standard or first-line treatment for numerous psychological disorders20.
CBT is the most researched family of face- to-face psychotherapies. It is also the most researched family of digital mental health programmes. Individual randomized trials and meta-analyses have shown that CBT delivered by telephone (T-CBT) is comparably efficacious with CBT delivered in a face- to-face setting21,22. There is also abundant evidence supporting the efficacy of iCBT, which is often delivered via interactive websites. Indeed, evidence from a meta-analysis and systematic review indicates that iCBT is comparably as effective as face- to-face CBT for depression and anxiety22,23.
There are well-studied commercially available iCBT programmes, such as Beating the Blues and SilverCloud, that are designed as online courses. In these types of iCBT programmes, users receive access to CBT-based psycho-educational content and are prompted to engage in the same types of exercises or activities that comprise traditional, face- to-face CBT. These iCBT programmes can look and feel similar to an online course, in which users work through lessons or modules and access interactive tools. For example, iCBT programmes typically guide users through cognitive restructuring and use interactive tools to prompt users to modify maladaptive behaviour patterns and engage in more adaptive behaviours. Combining iCBT and T-CBT in stepped care models, in which participants with depression begin with an iCBT programme and are ‘stepped up’ to T-CBT based on a lack of symptom change, have comparable efficacy with T-CBT alone, with lower overall costs24.
Mobile app-based CBT comes in various forms. Some apps guide users through modules similar to iCBT programmes, whereas others focus more explicitly on the practice of CBT-focused skills such as scheduling positive activities or identifying and challenging unhelpful thoughts. Mobile app-based CBT has a strong commercial presence, with thousands of apps tagged with ‘CBT’ on the Google Play and iTunes app stores. However, evidence to support the efficacy of CBT-based apps in real-world environments is limited. There are multiple reasons for this limited evidence base.
First, although both iCBT and app-based CBT are typically evaluated in research settings, there are many more commercially available apps that are accessible to the general public that have not been rigorously evaluated prior to public distribution. Engagement with apps is often suboptimal in real-life settings, perhaps due to the absence of concurrent human support, which is typically made available in a research trial25. Consequently, CBT exposure through app use in real-life settings is low26.
Second, high heterogeneity in methodologies (for example, the outcomes evaluated or whether a comparison group is included)27 and reporting practices for use and efficacy outcomes from app-based CBT intervention trials make it difficult to establish a clear evidence base28,29. One reason for heterogeneity in methodologies is that although randomized trial designs are the most rigorous approach for testing interventions30, such trials can be time-consuming and expensive, and technology evolves at a much faster pace, leading many groups to test effects in smaller, single-arm study designs31. However, trial designs and frameworks that encourage optimization without compromising rigour, such as those that evaluate intervention principles rather than specific interventions, are gaining favour31–34.
Third, with noted exceptions — for example, Woebot, IntelliCare, SuperBetter and WebMAP Mobile — the majority of CBT apps that demonstrate efficacy in research settings are not publicly available and only accessible to individuals enrolled in a research study35–39. Thus, data on these interventions in non-research, real-world settings are not available; consequently, effectiveness of their use in everyday, real-world environments cannot be assessed.
Acceptance and commitment therapy.
CBT is particularly well researched and prominent in the DMHI arena, but other evidence-based modes of psychotherapy have also been translated into DMHIs. ACT is a psychological intervention framework that focuses on mindfulness and acceptance strategies, along with commitment and behaviour change strategies, to increase psychological flexibility (defined as being in contact with the present moment and being able to engage in effective, flexible behaviour). There are ACT treatment protocols for a range of treatment targets including, but not limited to, depression40, anxiety41, psychosis42, eating disorders43, irritable bowel syndrome44 and grief45. Within ACT protocols, psychological flexibility is targeted through six processes thought to be relevant across a range of clinical diagnoses and conditions: acceptance (defined as active awareness and embrace of present experience), cognitive defusion (a technique used to help people cope with uncomfortable thoughts and feelings), being present (defined as ongoing, non-judgemental contact with events as they occur), the self as context (the idea that people are not the content of their thoughts and feelings but, rather, are the consciousness experiencing those thoughts and feelings), values (the identification of desired qualities or behaviours that give lift purpose) and committed action (defined as taking action in the direction of one’s valued life direction)46.
Research studies demonstrate that ACT can be competently delivered by telephone47 but there have not been large-scale randomized trials comparing telephone-delivered ACT with face- to-face ACT to determine the effectiveness of telephone-delivered ACT. Other ACT-based DMHIs, including Internet-based ACT (iACT), take the principles of ACT and translate them for an online learning environment. These are often structured similarly to iCBT programmes, in that participants receive access to psycho-educational models and supportive resources and tools (such as mindfulness audio recordings and reflective journaling entries). A meta-analysis on 25 studies of iACT interventions for adults that targeted depression, anxiety, quality of life and/or psychological inflexibility found small pooled effects for all outcomes post assessment that were maintained at follow-up time points48. Although these results indicate potential for iACT interventions, most of the control groups in these studies were wait-list controls, who receive no treatment. The lack of active controls limits the conclusions that can be drawn because wait-list controls offer a less rigorous test of an intervention’s effectiveness49. Furthermore, there was limited evidence of reliable, clinically significant effects48. Thus, further research is needed to evaluate how iACT interventions perform relative to active controls, who receive an alternative treatment, to determine the comparative effectiveness of iACT and, therefore, which type of treatment should be offered to patients.
Psychodynamic therapy.
PDT was developed from traditional psychoanalytic psychotherapy — an intensive psychotherapy in which patients are typically engaged in therapy sessions with a certified psychoanalyst several times a week over many years. PDT is a briefer form of therapy based on psychoanalytic theory of how the mind works. Patients engaged in PDT typically meet with their therapist once or twice a week for a shorter duration of time than in psychoanalytic therapy (such as a few months rather than years). Therapy sessions are focused on exploring aspects of the self that are not fully known, particularly how those aspects are manifested and influenced within the therapeutic relationship. In these sessions, patients might begin to recognize and explore recurring patterns in their lives to see how they avoid distress or develop defence mechanisms to cope; these insights might enable patients to change these patterns.
One review identified the following features as unique to PDT: a focus on affect and expression of emotion; exploration of attempts to avoid distressing thoughts and feelings via specific archetypes (for example, sublimation, a defence mechanism in which unacceptable urges are transformed into more socially acceptable behaviours); identification of recurring themes in the patient’s life; discussions of past experiences, with a focus on early life experiences; a focus on interpersonal relationships; a focus on the therapeutic relationship between patient and therapist; and an exploration of the patient’s fantasy life in which patients are encouraged to speak freely about whatever is on their minds during sessions, rather than following a more structured format50.
PDT has historically been less studied in research trials compared with CBT and ACT. However, a large body of research indicates that patients receiving PDT show greater improvements in symptoms than no-treatment control groups51,52. Other studies have found that patients receiving PDT have similar improvements to patients receiving CBT, although the evidence is somewhat mixed53.
The potential benefits of expanding PDT delivery through remote synchronous models (such as telephone and videoconference) include increased access for individuals who are homebound or do not live near psychodynamic therapists54. However, there are no published studies comparing remote synchronous PDT with traditionally delivered face-to-face PDT.
Internet-based PDT (iPDT) is structured similarly to iCBT and iACT programmes, with weekly text-based modules, homework assignments and e-mail support from a therapist or coach. iPDT differs from face-to-face PDT in a few ways. First, iPDT includes homework assignments whereas face- to-face PDT typically does not. Second, face-to-face PDT often focuses on a theoretical construct called transference, in which the patient’s unconscious conflicts associated with past relationships get triggered within current relationships, including the relationship with the therapist. In face-to-face PDT, issues surrounding transference are typically explored as they emerge, whereas iPDT is largely asynchronous (that is, any communication with a therapist/coach is not in real time). Although unconscious feelings towards the therapist/coach may still arise during iPDT, iPDT interventions are not designed to explicitly address transference.
There is a smaller and weaker evidence base for DMHIs based within a PDT framework relative to DMHIs based in CBT and ACT frameworks. A recent systematic review and meta-analysis of iPDT programmes55 found seven randomized control trials of iPDT programmes, which targeted depression56–59, anxiety disorders57–61 and work-related stress62. Compared with inactive control conditions (which included wait-list controls and unstructured support groups), iPDT produced small improvements in depression, anxiety and quality of life outcomes55 and the effect sizes were smaller than those found in meta-analyses of iCBT programmes63. Owing to the small number of studies on iPDT, further research is needed to more conclusively determine its efficacy.
Formats of DMHI delivery
When evidence-based psychotherapies are instantiated in digital platforms, they can be self-guided (for example, the user goes through the intervention independently), delivered with the support of a coach or therapist, or blended with face-to-face treatment.
Guided interventions.
Guided interventions are those that include human support as part of their delivery. Guided interventions have been delivered using telephone, video call and/or text message interactions. In studies of guided interventions, support has been delivered by mental health specialists (for example, therapists) as well as non-specialist professionals (that is, paid individuals without healthcare or counselling qualifications). Evidence from randomized controlled non-inferiority trials (that is, trials that test whether one intervention provides at least the same benefit to the patient as an established intervention) that compared technician-assisted iCBT (that is, iCBT delivered by a non-specialist professional) with clinician-assisted iCBT and with a wait-list control group suggests that non-specialist professionals are as effective as mental health specialists in delivering these interventions64,65. This is important because the demand for mental health services worldwide significantly outweighs the number and capacity of licensed mental health practitioners. Thus, the ability to task-shift the delivery of DMHIs to non-mental health practitioners or lay professionals can enable these tools to be delivered to more people in need.
The role of supporting therapists has been thoroughly examined in studies of iCBT. One study of iCBT therapist messaging found that iCBT therapists frequently encouraged, affirmed and guided their patients, and encouragement, affirmation and self-disclosures from the therapist were correlated with clinical improvement66. More recent research has further supported an association between clinical outcomes in iCBT and the content of what supporting therapists write in their messages to users (the primary form of communication in therapist-guided iCBT)67.
Meta-analyses have shown that guided interventions have demonstrated efficacy for common mental health problems such as depression68, post-traumatic stress disorder69, anxiety23 and insomnia70; the presence of a human guide has also been found to increase engagement with interventions in some studies22. Furthermore, meta-analytic reviews indicate that guided iCBT interventions perform as well as face-to-face CBT22. Guided interventions also have demonstrated efficacy for eating disorders and substance misuse, although effect sizes have generally been smaller relative to interventions for depression and anxiety71.
Because offering human support with a digital intervention requires substantial time and resources, growing attention is being directed to studying conversational agents, or chatbots, to simulate human support72. Chatbots use conversational artificial intelligence based on decision trees and advances in machine learning to communicate with users. There are several advantages to using chatbots compared with human support, including reduced reliance on humans, and therefore reduced costs, as well as increased capacity to provide immediate support because chatbots can respond at any time of day, immediately after a user’s message. Users have indicated that they feel supported by chatbots even when they know that they are communicating with artificial agents73. However, at present there are risks to these tools. Notably, chatbots can sometimes ‘get it wrong’ and offer unrelated, unhelpful or inappropriate responses to users’ inputs74. It is hypothesized that incorrect responses can negatively impact user engagement, and can be especially problematic for addressing users’ disclosure of high-risk behaviours of high relevance for DMHIs, such as non-suicidal self-injury or suicidality. Indeed, intervention designers must be attentive to the safety, privacy, liability and ethical considerations associated with using non-human support. It is also unknown how the therapeutic alliance (the relationship between the patient and provider and the degree to which they are engaged in collaborative, purposive work) is impacted by using chatbots75. To date, these technologies have not yet achieved capacity to simulate empathy relative to humans76. This has important implications for DMHIs because empathy predicts therapeutic alliance and is considered a non-specific factor that can positively impact psychotherapy treatment outcomes77–79.
Self-guided interventions.
Self-guided interventions, also referred to as unguided interventions, are interventions that are fully automated and do not involve human support as part of their delivery. With unguided interventions, users can download and access DMHI tools as they desire, and the interventions are designed to be delivered without someone guiding users through the programme. Consequently, unguided interventions have great potential for scale — they can be delivered to large populations of users because the availability of people to serve as guides is not a rate-limiting factor for the number of people who can be provided services. The ability of unguided interventions to function without human support also makes them cheaper to deliver compared with guided interventions. Unguided interventions have demonstrated efficacy relative to control conditions (including wait-list and active controls) for common mental health problems, such as depression and generalized anxiety disorder, but the effects are smaller than for guided interventions that include human support80. However, this difference between guided and unguided interventions may be moderated by symptom severity; in a large network meta-analysis of patient data from individuals with depression, the effects of guided interventions relative to unguided interventions were more pronounced for those with higher symptom severity (that is, individuals with moderate to severe depression benefit more from guided interventions)68. By contrast, those with mild or subthreshold depression experienced similar effects with guided and unguided interventions68.
Numerous commercially available DMHIs offer self-guided and guided versions of their programmes. In this model of contact by choice, users can choose their level of support by opting in and out of contact with a coach or mental health specialist. For example, the chatbot app Wysa offers users the ability to exchange messages with a therapist for an additional monthly fee81. No studies regarding the acceptability and effectiveness of this choice method have been published to date.
Blended approaches.
Blended approaches involve delivering DMHIs as part of face- to-face mental health interventions82,83. For example, a digital intervention could be used to facilitate self-monitoring (for example, symptom assessments) to guide treatment decisions, support skill practice between sessions (for example, as an alternative to worksheets) and/or enable additional communication streams between the therapist and the patient (for example, text messaging). Therapists have favourable perceptions of blended treatments, and the use of blended treatments is increasing84. In 2014, surveys revealed that mental health service stakeholders were more interested in implementing blended treatment than stand-alone Internet-based interventions 85; a subsequent survey of therapists in 2019 showed that blended treatments were perceived to have fewer disadvantages than stand-alone Internet-based interventions84. Trials testing blended approaches have been conducted. For example, two trials have demonstrated the efficacy of iCBT combined with synchronous chat with a therapist compared with a minimal-attention control (in which participants completed weekly mood assessments that were reviewed by a therapist) for depression in adolescents86,87. However, among people with psychosis, blended treatment for paranoia (comprising face-to-face sessions, a mobile app and usual care) did not outperform usual care alone (which typically involved antipsychotic therapy, contact with a mental health worker and outpatient psychiatric appointments)88.
Regarding the cost of care, a naturalistic study (in which participants received treatment through their standard healthcare setting) found that iCBT blended with face- to-face treatment was more costly to deliver and did not necessarily offer greater benefit compared with face- to-face treatment alone for people with anxiety or depression89. Further, many practising clinicians report using mobile apps with their patients in standard practice, but concerns about costs to the patient, privacy and data security are barriers to their use90–92. Continued research is needed to understand why and for whom blended care works. To this end, there are efforts to create instruments that help clinicians identify patients most appropriate for this treatment delivery approach83,93.
Considerations across the lifespan
DMHIs that are marketed for general audiences have typically been evaluated and/or developed with adults who are young (aged 18–40 years) or middle-aged (aged 40–60 or 65 years). However, multiple interventions are targeted towards specific age groups, such as youth (a term that may encompass children, adolescents and young adults up to age 25 years) or older adults (typically defined as those older than age 60 or 65 years). It is important to understand the design and effectiveness of DMHIs for youth and for older adults as these two groups have developmentally different needs and preferences.
DMHIs for youth.
DMHIs for youth have been heavily focused on computer (for example, CD-R OM), web (for example, secure Internet page) and mobile-based (for example, text messages or app) delivery94–96. For computer and web-based DMHIs, CBT has been the most commonly used treatment approach94,97. In children 6–18 years of age with depression and/or anxiety, iCBT has yielded medium intervention effects compared with wait-list control groups, but has not demonstrated a statistically significant benefit compared with face- to-face delivery of CBT97. However, iCBT programmes evaluated in published studies often rely heavily on psycho-education via text-based lessons, which is unlikely to be engaging to youth98.
Computer and web-based DMHIs for youth have also been based on other psychotherapies (for example, problem-solving therapy and attention bias modification training), with little to no effect compared with control conditions97. A small, but growing, body of evidence supports the efficacy of mobile DMHIs in improving paediatric health targets (for example, physical activity promotion or medication adherence)99. Less support has emerged for the efficacy of mobile DMHIs for mental health targets in youth100. Nevertheless, the mobile DMHI marketplace is continually growing101–103, raising concerns about the small evidence base for mobile DMHIs for youth (particularly children) compared with the number of commercially available apps96,97,104. Indeed, some mobile DMHIs are marketed to youth but have only been evaluated with adults. For example, Woebot, an app that uses a CBT-based conversational agent, is marketed for age 12 years and older. However, to date, this app has only demonstrated efficacy in decreasing depressive and anxiety symptoms in college students, and some evidence for efficacy in improving substance misuse behaviours and cravings in adults36,37. Frameworks that clinicians and caregivers can use to identify digital mental health tools for youth needs are being proposed to supplement the gap between established efficacy with adult populations and availability to youth105.
Youth face numerous barriers to traditionally delivered mental health services, including stigma, their family’s negative beliefs about mental health services and low mental health literacy106. For youth, DMHIs can be solely youth-facing36 or involve shared access with a caregiver107. Caregivers (often parents) are sometimes included as end users of DMHIs for youth due to the developmental needs of younger users who may require assistance with logistical aspects of an intervention, and findings that intervention outcomes improve when caregivers are involved97,99. Notably, despite this association between caregiver involvement and outcomes, parental consent to mental health treatment can also be a potential barrier to youth mental health treatment108. Thus, the effect of caregiver involvement in intervention efficacy might be dependent on the specific context, and DMHIs could offer a particularly important mental health option for youth who might otherwise not seek care due to barriers, including the need for parental consent109. DMHI outcomes for youth also improve with other adult support, such as from therapists and teachers97.
Design needs110,111 (for example, providing flexibility and an age-appropriate cognitive load) and frameworks specific to youth (for example, the Integrate, Design, Assess, Share Framework112) have been described elsewhere. Other broad design considerations for youth DMHIs include attention to language (for example, avoiding jargon and anticipating variability in health and general literacy)113,114; brevity (for example, youth often have a lower ability to maintain and manipulate information in their minds than do adults, so scrolling through text, particularly on smaller screens, can be challenging)115; and clear confidentiality and data sharing practices to increase feelings of safety about disclosing information in online spaces, particularly for youth from vulnerable populations116,117.
DMHIs for older adults.
DMHIs for older adults are associated with a large, strong research base. Interventions with an evidence base for older adults have primarily focused on computer and web-based interventions that rely heavily on psycho-education and skill-building, and often include some form of human support118,119. For example, therapist-guided iCBT programmes for older adults with depression and anxiety have demonstrated both efficacy (with large clinical improvements in depression or anxiety) and cost-effectiveness119,120. One study found equivalent benefits for anxiety symptoms from a blended form of ACT, which included a web-based intervention and four face-to-face sessions, and a traditional face- to-face, four-session CBT intervention; however, the blended ACT programme was associated with higher levels of patient satisfaction and well-being121. This suggests that interventions that include a combination of face-to-face and digital services may be of particular value for older adults.
It is important to note that older adults may not be as comfortable with technology, and some older adults view digital innovations as exclusionary rather than stimulating122,123. The digital divide between generations has been narrowing124,125, yet older adults might still lack experience with and access to digital tools, and might therefore face barriers to the effective use of technologies126. Indeed, many mobile health apps are not appropriately designed for and accessible to older adults, who often have low technical readiness and trust in technologies127,128. Although DMHIs for older adults might overcome some common barriers to traditionally delivered mental health services, such as cost, stigma and transportation issues, implementations of DMHIs for older adults should strategically target trust, buy-in and technical skill training or support to maximize the likelihood of success.
Towards greater and more equitable access
The more widespread use of DMHIs due to the COVID-19 pandemic is itself a research opportunity — as more people are using these services across healthcare settings, there are rapidly accumulating data on the acceptability and effectiveness of these services. As these data are analysed, we will be able to identify the barriers and facilitators to effective DMHI delivery. Early data generated during the COVID-19 pandemic has demonstrated that the rapid scale-up of telemedicine in the United States decreased mental service use by patients older than age 65 years, those who prefer a language other than English, and patients on Medicare and Medicaid (federal health insurance programmes available to individuals based on income, age and disability status). Furthermore, patient health visits for white patients increased, whereas visits decreased among other racial and ethnic groups129. Thus, the initial scale-up of telemedicine likely increased healthcare disparities for more vulnerable patient populations. It is clear that we need to do better. In this section, we outline opportunities to create more equitable access to mental health care. Due to the nuances of payment systems and social structures within the American healthcare system, this section focuses primarily on data from the United States. A brief global perspective on access is provided in BOX 3.
Box 3 |. Examples of access to DMHIs around the world High-income countries.
In high-income countries, digital mental health interventions (DMHIs) can improve access to mental health care and are being offered in both self-guided and guided formats. In addition, some countries have integrated DMHIs into their public health insurance models. For example, in Germany the Digital Healthcare act entitles all individuals covered by public health insurance to be reimbursed for their use of certain digital health applications. There is now a regulatory process to approve specific DMHIs for reimbursement184. In the United Kingdom, the Improving access to Psychological Therapies programme through the National Health Service (NHS) provides computerized cognitive behavioural therapy (CBT) and Internet-based cognitive behavioural therapy (iCBT) free of charge as a low-intensity treatment for depression and anxiety disorders185. The NHS offers an online library of mental health apps to guide patients to the appropriate apps. Finally, in Australia the Department of Health developed the Head to Health platform, which serves as a gateway to mental health resources, including DMHIs. Several DMHIs are available for free to Australians with a prescription and for a small fee without a prescription186.
Low and middle-income countries
In low and middle-income countries, stand-alone, scalable DMHIs could have a large impact on population mental health given the scarcity of mental health professionals in many countries187.
A recent review of DMHIs in low and middle-income countries identified 37 studies, with about half (N = 23) reporting clinical benefits to participants188. Sample sizes were small, and studies primarily focused on adults. Studies were primarily from East Asia, Central or Latin America and South asia188. There remains a need to identify the most effective DMHIs for these resource-limited settings and to integrate high-quality DMHIs into health systems in these areas.
Efforts towards greater and more equitable access to mental health care must be strategic in continued expansions of DMHIs. In the American healthcare system, in which most patients are on private health insurance plans, issues of payment and cost are front and centre. To ensure access to mental health care, it must be both available and affordable. The COVID-19 pandemic facilitated a rapid transition to remote care by relaxing rules for telehealth services (which were reimbursable under certain insurance plans only to individuals who lived in specific geographic areas or had specific diagnoses), and many states adopted telehealth parity through COVID-19 emergency orders. Telehealth parity guaranteed that clinicians would receive comparable payments for telehealth as for in-person services130. To integrate DMHI services into routine clinical practice and ensure that they continue to be available beyond the COVID-19 pandemic, telehealth parity laws should continue to expand and be made permanent.
There are numerous healthcare payment models within the United States, and many of these payment models create challenges for reimbursement for DMHIs other than synchronous telephone or videoconference appointments with clinicians. Organizations that operate outside fee-for-service models may be well positioned to offer DMHIs as part of their new standard of care. However, the majority of healthcare payments in the United States are tied to the fee-for-service model of health care; thus, care is not accessible to those with limited payment options. There is an urgent need to expand reimbursement for DMHI products, such as for mobile apps and web-based programmes, that can be provided with or without human support and for a wider range of clinicians who could support DMHIs131.
Much of the COVID-19 scale-up of telemedicine relied on videoconferencing in lieu of in-person appointments. Videoconferencing appears to be an effective and engaging method of delivering mental health care132,133. However, when videoconferencing is the only service available, patients can be left behind if they lack access to or comfort with technology capable of videoconferencing134. Using mobile phones as a delivery method might better engage vulnerable patient populations compared with computers or other technologies that are reliant on Wi-Fi to access DMHIs135. Indeed, there remain disparities in who is likely to adopt computer use and have home broadband access136. However, nearly all adults (96%) in the United States own a mobile phone (that is, a cellular phone that supports telephone calls and/or SMS messaging)137 and 95% of teenagers report that they either own or have access to a smartphone (that is, a cellular phone that can access the Internet and apps)138.
The bulk of evidence for DMHIs has focused on computer-based platforms. However, mobile phone-based interventions (and, specifically, interventions that do not require smartphone ownership and Wi-Fi access) are more likely to reach vulnerable patient populations. Moreover, despite the huge number of mental health apps, adoption is often poor26. SMS or text messaging-based interventions offer numerous potential benefits for expanding DMHIs and making DMHIs more accessible. First, text messages are more likely to be read than push notifications, which are often easy to ignore139. It is theorized that users might prefer text messages rather than having a potentially identifiable mobile app saved on their smartphone (for example, a specific app could identify the user as having depressive or anxiety symptoms)140. Second, academic clinical scientists are unlikely to develop, evaluate and disseminate apps with the same level of speed as found with industry-driven apps, but can often rapidly program and disseminate text message-based DMHIs141. These differences in end-user engagement and resources make the development of future apps targeted specifically for unique conditions and/or patient groups less feasible without industry support, whereas text message-based programmes may be more feasible. However, hidden costs of text messaging-based interventions need to be considered, as vulnerable patient populations are more likely than affluent individuals to be impacted by lapses in service or burdened by the cost of data packages or texting plans142,143.
Regardless of the technology used to deliver DMHIs, there is a clear need to involve representative end users and stakeholders in the design and development of these interventions to ensure that research samples are representative144. This is particularly important to consider when working towards more equitable access, as white individuals, and particularly white young to middle-age adult women, are typically over-represented in DMHI research trials145,146. To develop programmes that will be engaging for more diverse patient populations, stakeholders and end users from those populations need to be included in both the design and testing of the programme. Including both co-design elements and representative participation in clinical trials can enable early identification and remediation of problems with DMHIs for minoritized individuals.
Furthermore, DMHI programmes need to be developed based on both evidence-based treatments as well as theories that recognize the intersectional identities of members of underserved populations. Including a minority stress theory-based perspective is important. According to the minority stress model, having a minoritized status leads to increased exposure to stressors and this increased stress can lead to poor health outcomes147. For example, the HORIZONS intervention, which targeted sexual risk behaviours in African American adolescent females, incorporated both an evidence-based treatment148,149 and minority-based theory, and demonstrated effectiveness for reducing targeted risk behaviours150,151. Developing partnerships with public and private organizations serving traditionally underserved populations is key for leveraging the complementary strengths needed to design and deploy inclusive, culturally responsive and innovative mental health programmes152.
To best serve diverse populations of users, DMHIs need to be available in various languages. Adapting evidence-based DMHIs into a new language typically requires cultural adaptations, rather than a straight language translation. For example, in the adaptation of an iCBT programme for Colombian university students, the ‘Space from Depression’ programme was renamed ‘I can feel better’. This new name was considered a culturally suitable descriptor of improving mood that avoids the cultural stigma related to the label of ‘depression’153. When the Australian Mental Health eClinic was redesigned and adapted for Spanish speakers, participants identified the need for more tailored assessment tools, and greater integration with other Spanish language resources and communities154. Owing to differences in the length of written languages (for example, Spanish and French texts are approximately 20–30% longer than English texts), interface changes often need to be made to ensure that the layout of web-based and app-based programmes is appropriate and usable.
Finally, as discussed above, providing support in an automated manner or directly from a human supporter is valuable for engagement and clinical outcomes68–70. To make DMHIs more accessible and equitable, human support that is representative and promotes diversity must be included. Consistent with recommendations from the US Department of Health and Human Services’ Office of Minority Health, one way to improve mental health disparities is to increase clinical workforce diversity155. Thus, human support staff for DMHIs should include individuals with demographic membership that is in some way representative of the patients being served by the DMHIs. To move the field forward in promoting greater access to digital mental health care, we propose adopting the recommendations originally presented by Galàn et al.156 and adapted for a DMHI context (see BOX 4).
Box 4 |. Applying a racial justice lens to DMHI research.
The following recommendations, originally presented by Galàn et al.156 and adapted to digital mental health intervention (DMHI) contexts here, should be adopted to move the field forward in promoting greater access to digital mental health care.
Involve diverse community stakeholders
To combat long-standing racial injustices, researchers must engage members of underserved and minoritized communities in all stages of the research process and ensure they are benefitting from the work in a direct way.
For example, collaborators representing various backgrounds, including community members, should be involved in developing DMHIs and selecting the intervention targets. Who the DMHI is serving, how previous research has considered (or not considered) race and how that influences the questions being asked in the current research or content provided in the DMHI should be considered.
If research is being conducted in a hospital, in an academic institution or through a private company, collaborations with community members can help to address valid mistrust of these institutions among minority individuals.
Stakeholders from the community should also be involved to ensure that study findings are shared with the community (for example, through social media, infographics or newsletters).
Use inclusive research designs and recruitment procedures
Randomized controlled trials often use samples in which people with minoritized identities are under-r epresented189, which can limit the public health value of DMHI research141. Research frameworks and designs that mitigate selection bias and target inclusion of minoritized individuals and underserved communities should be considered.
For example, deploying interventions on technologies that participants readily have access to, meeting participants in preferred locations, such as local libraries or their homes, and outside standard work hours, and providing access to food during or after participation can encourage people with transportation barriers, who work full time or who lack private spaces to participate189. Compensation for participation in DMHI research should adequately reflect the investment of time and disruption to daily life (for example, childcare or transportation costs should be compensated).
The samples with whom measures or assessment tools were validated and how this impacts their use in other groups should be considered. For example, researchers should ensure that the scales used elicit valid and effective responses for all racial groups (for example, Likert scales have been shown to be culturally insensitive or meaningless to some racial groups)190. In addition to assessing deficits, strengths-based measures, such as psychological well-being and flourishing, should be included to detect the full effects of interventions.
Finally, the research team of individuals who will be interacting with participants should be diverse and inclusive to maximize the likelihood that a diverse range of individuals will participate.
Summary and future directions
Personal technologies, such as computers, tablets and mobile phones, continue to be ubiquitous. Thus, the potential roles of these technologies in supporting mental health and wellness are numerous and important. Here, we have described numerous DMHIs capable of producing meaningful changes in the lives of individuals struggling with various mental and behavioural health concerns48,63. We have highlighted that the evidence regarding clinical effectiveness is robust for remote, synchronous versions of evidence-based mental health treatments (for example, delivered by telephone or videoconference)21,157. Similarly, the evidence of treatment effectiveness is robust for many web-based interventions, such as iCBT68,70, and is growing for app-based and SMS-based interventions140,158. Many of these interventions produce larger effects when paired with some level of human support or guidance68, yet self-guided interventions can be effective and are valuable given their scalability68. However, there remain challenges in the field of digital mental health, and therefore areas where continued progress is needed.
First, future research should include representative populations and engage with appropriate stakeholders outside academia. Although numerous DMHIs have demonstrated efficacy in randomized controlled trials22,97, minoritized individuals are typically under-represented in these trials145,146. There are concerns that DMHIs believed to be efficacious may not be appropriate for or helpful to individuals from underserved communities159. To fully realize the potential of DMHIs, these interventions must be capable of eliminating, or at least narrowing, disparities in access to mental health care. Thus, to be confident in the effectiveness of DMHIs, they should be subject to further testing with diverse populations in real-world environments. Industry–academic collaborations might be particularly effective to this end because industry partners have access to design and development resources, and academic partners complement these resources by contributing their skills in psychological theory, research design and analysis.
Second, there are substantial areas for improvement regarding data privacy. There are legitimate concerns about privacy policies and problematic data sharing within digital mental health platforms that have limited their adoption160,161. Industry–academic partnerships must be thoughtfully built to maximize privacy and trust from potential DMHI end users. Some people trust universities as benevolent overseers of research data because university-sponsored research is subject to an ethics review. However, those who are less inclined to trust university systems because of past or ongoing wrongdoings might be more likely to engage with industry-focused projects and programmes.
Reviews of mobile mental health apps have found that the majority of commercially available apps do not include a privacy policy or terms of agreement, the reading level of existing written policies is frequently inaccessible to the general population and many of the existing privacy policies state that users’ information may be shared with third parties162. Thus, there are clear ethical concerns in the selection of digital mental health tools. To support ethical decision-making, Nebeker et al.163 developed a decision-making tool called the Digital Health Checklist. This checklist walks evaluators through four intersecting domains: access and usability; risks and benefits; privacy; and data management. These domains were identified as the key factors critical to decision-making, and can be useful for technology developers, ethics boards, clinical personnel and people considering participation in a digital health study163. To ensure ethical application of DMHIs, we recommend broadscale adoption of the Digital Health Checklist to guide DMHI selection.
Third, many DMHIs engage users in a structured manner. For example, many web-based programmes are module-based, in which users progress through modules based on time-based rules or completion-based rules. Videoconferencing programmes are synchronous, and sessions typically take place during pre-scheduled times. Many app-based programmes have tools that are continuously available to users, and users can access them at any time. However, DMHIs that are more responsive to the users’ context are being developed, such as just-in-time adaptive interventions164. Just-in-time adaptive interventions deliver intervention components to individuals in the right moments to support behaviour change. These interventions work by leveraging smartphone technologies such as sensing to deliver relevant in-context interventions in real time, and offer an increased likelihood that a user will receive the right intervention in the right place and time164. Micro-randomized trial designs (in which intervention options are randomly assigned at each relevant decision point to enable researchers to examine the causal effect of different intervention options) can be used to test just-in-time adaptive interventions and inform their optimization165.
Finally, continued expansion of DMHIs may benefit from increasing the automaticity of these tools. As we have indicated, DMHIs that include human support are often more effective than self-guided or automated treatments68. However, reliance on human support may limit the broad dissemination of these interventions because there are limits on how many individuals any one person can serve. It is therefore important for the field of digital mental health to continue exploring ways to encourage user engagement and support skill practice via DMHIs without human support. Continued progress in these areas will be of great value for increasing the scalability of DMHIs.
Acknowledgements
This work was supported by grants from the National Institutes of Health (NIH) (K01 DK116925; K08 MH112878; P50 MH119029). C.S.- S. is supported by a fellowship from the Cohn Family Foundation and an R3 funding mechanism supported by Rush University Medical Center and RTI International.
RELATED LINKS
Beating the Blues: https://www.beatingtheblues.co.uk/
Head to Health: https://www.headtohealth.gov.au/
IntelliCare: https://www.adaptive-health.com/intellicare
Online library: https://www.nhs.uk/apps-library/category/mental-health/
SilverCloud: https://www.silvercloudhealth.com/
SuperBetter: https://www.superbetter.com/
WebMAP Mobile: https://www.seattlechildrens.org/globalassets/documents/research/cchbd/webmap_mobile_app_flyer.pdf
Woebot: https://woebothealth.com/
Wysa: https://www.wysa.io/
Footnotes
Competing interests
E.G.L serves on the Scientific Advisory Board for Vault Health. A.K.G receives grant support from Adaptive Health, Inc.
Peer review information
Nature Reviews Psychology thanks Stefan Gold, who co-reviewed with Kerem Boege, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
References
- 1.Patel V & Prince M Global mental health. JAMA 303, 1976 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Merikangas KR et al. Service utilization for lifetime mental disorders in U.S. adolescents: results of the National Comorbidity Survey — Adolescent Supplement (NCS-A). J. Am. Acad. Child. Adolesc. Psychiatry 50, 32–45 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.SAMHSA. 2019 National Survey of Drug Use and Health (NSDUH) releases. SAMHSA; https://www.samhsa.gov/data/release/2019-national-survey-drug-use-and-health-nsduh-releases (2019). [Google Scholar]
- 4.Smith-East M & Neff DF Mental health care access using geographic information systems: an integrative review. Issues Ment. Health Nurs. 41, 113–121 (2020). [DOI] [PubMed] [Google Scholar]
- 5.Reardon T et al. What do parents perceive are the barriers and facilitators to accessing psychological treatment for mental health problems in children and adolescents? A systematic review of qualitative and quantitative studies. Eur. Child. Adolesc. Psychiatry 26, 623–647 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Andrade LH et al. Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychol. Med. 44, 1303–1317 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Walker ER, Cummings JR, Hockenberry JM & Druss BG Insurance status, use of mental health services, and unmet need for mental health care in the United States. Psychiatr. Serv. 66, 578–584 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Frank RG, Huskamp HA & Pincus HA Aligning incentives in the treatment of depression in primary care with evidence-based practice. Psychiatr. Serv. 54, 682–687 (2003). [DOI] [PubMed] [Google Scholar]
- 9.Zhou X et al. The role of telehealth in reducing the mental health burden from COVID-19. Telemed. J. E. Health. 26, 377–379 (2020). [DOI] [PubMed] [Google Scholar]
- 10.Goldman ML et al. Mental health policy in the era of COVID-19. Psychiatr. Serv. 71, 1158–1162 (2020). [DOI] [PubMed] [Google Scholar]
- 11.Ben-Zeev D The digital mental health genie is out of the bottle. Psychiatr. Serv. 71, 1212–1213 (2020). [DOI] [PubMed] [Google Scholar]
- 12.Mohr DC, Riper H & Schueller SM A solution-focused research approach to achieve an implementable revolution in digital mental health. JAMA Psychiatry 75, 113–114 (2018). [DOI] [PubMed] [Google Scholar]
- 13.Folker AP et al. Implementing Internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European Internet services. Internet Interv. 11, 60–70 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Etzelmueller A et al. Effects of Internet-based cognitive behavioral therapy in routine care for adults in treatment for depression and anxiety: systematic review and meta-analysis. J. Med. Internet Res. 22, e18100 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Borghouts J et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J. Med. Internet Res. 23, e24387 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hollon SD & Beck AT Cognitive and cognitive-behavioral therapies. Bergin Garfields Handb. Psychother. Behav. Change 6, 393–442 (2013). [Google Scholar]
- 17.Stewart RE & Chambless DL Cognitive-behavioral therapy for adult anxiety disorders in clinical practice: a meta-analysis of effectiveness studies. J. Consult. Clin. Psychol. 77, 595–606 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Waltman SH, Creed TA & Beck AT Are the effects of cognitive behavior therapy for depression falling? Review and critique of the evidence. Clin. Psychol. 23, 113–122 (2016). [Google Scholar]
- 19.Ehde DM, Dillworth TM & Turner JA Cognitive-behavioral therapy for individuals with chronic pain: efficacy, innovations, and directions for research. Am. Psychol. 69, 153–166 (2014). [DOI] [PubMed] [Google Scholar]
- 20.Hofmann SG, Asnaani A, Vonk IJJ & Sawyer AT The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cognit. Ther. Res. 36, 427–440 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Mohr DC et al. Effect of telephone-administered vs face- to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients: a randomized trial. JAMA 307, 2278–2285 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Carlbring P, Andersson G, Cuijpers P, Riper H & Hedman-Lagerlöf E Internet-based vs. face- to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn. Behav. Ther. 47, 1–18 (2018). [DOI] [PubMed] [Google Scholar]
- 23.Olthuis JV, Watt MC, Bailey K, Hayden JA & Stewart SH Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults. Cochrane Database Syst. Rev. 3, CD011565 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Mohr DC et al. A randomized noninferiority trial evaluating remotely-delivered stepped care for depression using Internet cognitive behavioral therapy (CBT) and telephone CBT. Behav. Res. Ther. 123, 103485 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Schueller SM, Tomasino KN & Mohr DC Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin. Psychol. Sci. Pract. 24, 27–45 (2017). [Google Scholar]
- 26.Torous J, Wisniewski H, Liu G & Keshavan M Mental health mobile phone app usage, concerns, and benefits among psychiatric outpatients: comparative survey study. JMIR Ment. Health 5, e11715 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Rathbone AL, Clarry L & Prescott J Assessing the efficacy of mobile health apps using the basic principles of cognitive behavioral therapy: systematic review. J. Med. Internet Res. 19, e399 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Hrynyschyn R & Dockweiler C Effectiveness of smartphone-based cognitive behavioral therapy among patients with major depression: systematic review of health implications. JMIR Mhealth Uhealth 9, e24703 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Huguet A et al. A systematic review of cognitive behavioral therapy and behavioral activation apps for depression. PLoS ONE 11, e0154248 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Schulz KF, Altman DG & Moher D & CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Int. J. Surg. 9, 672–677 (2011). [DOI] [PubMed] [Google Scholar]
- 31.Mohr DC, Cheung K, Schueller SM, Hendricks Brown C & Duan N Continuous evaluation of evolving behavioral intervention technologies. Am. J. Prev. Med. 45, 517–523 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Collins LM, Nahum-Shani I & Almirall D Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART). Clin. Trials 11, 426–434 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Collins LM, Kugler KC & Gwadz MV Optimization of multicomponent behavioral and bio-behavioral interventions for the prevention and treatment of HIV/AIDS. AIDS Behav. 20 (Suppl. 1), S197–S214 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Mohr DC, Lyon AR, Lattie EG, Reddy M & Schueller SM Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J. Med. Internet Res. 19, e153 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Mohr DC et al. IntelliCare: an eclectic, skills-based app suite for the treatment of depression and anxiety. J. Med. Internet Res. 19, e10 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Fitzpatrick KK, Darcy A & Vierhile M Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment. Health 4, e19 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Prochaska JJ et al. A therapeutic relational agent for reducing problematic substance use (Woebot): development and usability study. J. Med. Internet Res. 23, e24850 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Roepke AM et al. Randomized controlled trial of superbetter, a smartphone-based/Internet-based self-help tool to reduce depressive symptoms. Games Health J. 4, 235–246 (2015). [DOI] [PubMed] [Google Scholar]
- 39.Palermo TM, de la Vega R, Dudeney J, Murray C & Law E Mobile health intervention for self-management of adolescent chronic pain (WebMAP mobile): protocol for a hybrid effectiveness-implementation cluster randomized controlled trial. Contemp. Clin. Trials 74, 55–60 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Zettle R ACT for Depression: A Clinician’s Guide to Using Acceptance and Commitment Therapy in Treating Depression (New Harbinger, 2007). [Google Scholar]
- 41.Swain J, Hancock K, Hainsworth C & Bowman J Acceptance and commitment therapy in the treatment of anxiety: a systematic review. Clin. Psychol. Rev. 33, 965–978 (2013). [DOI] [PubMed] [Google Scholar]
- 42.Bach P, Hayes SC & Gallop R Long-term effects of brief acceptance and commitment therapy for psychosis. Behav. Modif. 36, 165–181 (2012). [DOI] [PubMed] [Google Scholar]
- 43.Manlick CF, Cochran SV & Koon J Acceptance and commitment therapy for eating disorders: rationale and literature review. J. Contemp. Psychother. 43, 115–122 (2013). [Google Scholar]
- 44.Ferreira NB, Gillanders D, Morris PG & Eugenicos M Pilot study of acceptance and commitment therapy for irritable bowel syndrome: a preliminary analysis of treatment outcomes and processes of change. Clin. Psychol. 22, 241–250 (2018). [Google Scholar]
- 45.Davis EL et al. Feasibility randomised controlled trial of a self-help acceptance and commitment therapy intervention for grief and psychological distress in carers of palliative care patients. J. Health Psychol. 25, 322–339 (2020). [DOI] [PubMed] [Google Scholar]
- 46.Hayes SC, Pistorello J & Levin ME Acceptance and commitment therapy as a unified model of behavior change. Couns. Psychol. 40, 976–1002 (2012). [Google Scholar]
- 47.Schimmel-Bristow A, Bricker JB & Comstock B Can acceptance & commitment therapy be delivered with fidelity as a brief telephone-intervention? Addict. Behav. 37, 517–520 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Thompson EM, Destree L, Albertella L & Fontenelle LF Internet-based acceptance and commitment therapy: a transdiagnostic systematic review and meta-analysis for mental health outcomes. Behav. Ther. 52, 492–507 (2021). [DOI] [PubMed] [Google Scholar]
- 49.Mohr DC et al. The selection and design of control conditions for randomized controlled trials of psychological interventions. Psychother. Psychosom. 78, 275–284 (2009). [DOI] [PubMed] [Google Scholar]
- 50.Blagys MD & Hilsenroth MJ Distinctive features of short-term psychodynamic-interpersonal psychotherapy: a review of the comparative psychotherapy process literature. Clin. Psychol. Sci. Pract. 7, 167–188 (2000). [Google Scholar]
- 51.Abbass AA, Hancock JT, Henderson J & Kisely S Short-term psychodynamic psychotherapies for common mental disorders. Cochrane Database Syst Rev. 2006, CD004687 (2006). [DOI] [PubMed] [Google Scholar]
- 52.Shedler J The efficacy of psychodynamic psychotherapy. Am. Psychol. 65, 98–109 (2010). [DOI] [PubMed] [Google Scholar]
- 53.Leichsenring F Comparative effects of short-term psychodynamic psychotherapy and cognitive-behavioral therapy in depression: a meta-analytic approach. Clin. Psychol. Rev. 21, 401–419 (2001). [DOI] [PubMed] [Google Scholar]
- 54.Saeed SA & Anand V Use of telepsychiatry in psychodynamic psychiatry. Psychodyn. Psychiatry 43, 569–583 (2015). [DOI] [PubMed] [Google Scholar]
- 55.Lindegaard T, Berg M & Andersson G Efficacy of Internet-delivered psychodynamic therapy: systematic review and meta-analysis. Psychodyn. Psychiatry 48, 437–454 (2020). [DOI] [PubMed] [Google Scholar]
- 56.Johansson R et al. Psychodynamic guided self-help for adult depression through the Internet: a randomised controlled trial. PLoS ONE 7, e38021 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Johansson R et al. Affect-focused psychodynamic psychotherapy for depression and anxiety through the Internet: a randomized controlled trial. PeerJ 1, e102 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Zwerenz R et al. Transdiagnostic, psychodynamic Web-based self-help intervention following inpatient psychotherapy: results of a feasibility study and randomized controlled trial. JMIR Ment. Health 4, e41 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Lemma A & Fonagy P Feasibility study of a psychodynamic online group intervention for depression. Psychoanal. Psychol. 30, 367–380 (2013). [Google Scholar]
- 60.Andersson G et al. Internet-based psychodynamic versus cognitive behavioral guided self-help for generalized anxiety disorder: a randomized controlled trial. Psychother. Psychosom. 81, 344–355 (2012). [DOI] [PubMed] [Google Scholar]
- 61.Johansson R et al. Internet-based affect-focused psychodynamic therapy for social anxiety disorder: a randomized controlled trial with 2-year follow-up. Psychotherapy 54, 351–360 (2017). [DOI] [PubMed] [Google Scholar]
- 62.Zwerenz R et al. Evaluation of a transdiagnostic psychodynamic online intervention to support return to work: a randomized controlled trial. PLoS ONE 12, e0176513 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Andrews G et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J. Anxiety Disord. 55, 70–78 (2018). [DOI] [PubMed] [Google Scholar]
- 64.Titov N et al. Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLoS ONE 5, e10939 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Robinson E et al. Internet treatment for generalized anxiety disorder: a randomized controlled trial comparing clinician vs. technician assistance. PLoS ONE 5, e10942 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Holländare F et al. Therapist behaviours in Internet-based cognitive behaviour therapy (ICBT) for depressive symptoms. Internet Interv. 3, 1–7 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Ewbank MP et al. Quantifying the association between psychotherapy content and clinical outcomes using deep learning. JAMA Psychiatry 77, 35–43 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.Karyotaki E et al. Internet-based cognitive behavioral therapy for depression: a systematic review and individual patient data network meta-analysis. JAMA Psychiatry 10.1001/jamapsychiatry.2020.4364 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Olthuis JV et al. Distance-delivered interventions for PTSD: a systematic review and meta-analysis. J. Anxiety Disord. 44, 9–26 (2016). [DOI] [PubMed] [Google Scholar]
- 70.Zachariae R, Lyby MS, Ritterband LM & O’Toole MS Efficacy of Internet-delivered cognitive-behavioral therapy for insomnia — a systematic review and meta-analysis of randomized controlled trials. Sleep Med. Rev. 30, 1–10 (2016). [DOI] [PubMed] [Google Scholar]
- 71.Taylor C, Graham AK, Flatt RE, Waldherr K & Fitzsimmons-Craft EE Current state of scientific evidence on Internet-based interventions for the treatment of depression, anxiety, eating disorders and substance abuse: an overview of systematic reviews and meta-analyses. Eur. J. Public Health 10.1093/eurpub/ckz208 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Miner AS, Milstein A & Hancock JT Talking to machines about personal mental health problems. JAMA 318, 1217–1218 (2017). [DOI] [PubMed] [Google Scholar]
- 73.Ho A, Hancock J & Miner AS Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot. J. Commun. 68, 712–733 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 74.Miner AS et al. Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Intern. Med. 176, 619 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 75.Miner AS et al. Key considerations for incorporating conversational AI in psychotherapy. Front. Psychiatry 10, 746 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 76.Morris RR, Kouddous K, Kshirsagar R & Schueller SM Towards an artificially empathic conversational agent for mental health applications: system design and user perceptions. J. Med. Internet Res. 20, e10148 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 77.Nienhuis JB et al. Therapeutic alliance, empathy, and genuineness in individual adult psychotherapy: a meta-analytic review. Psychother. Res. 28, 593–605 (2018). [DOI] [PubMed] [Google Scholar]
- 78.Priebe S, Conneely M, McCabe R & Bird V What can clinicians do to improve outcomes across psychiatric treatments: a conceptual review of non-specific components. Epidemiol. Psychiatr. Sci. 29, e48 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 79.Elliott R, Bohart AC, Watson JC & Greenberg LS Empathy. Psychotherapy 48, 43–49 (2011). [DOI] [PubMed] [Google Scholar]
- 80.Linardon J, Cuijpers P, Carlbring P, Messer M & Fuller-Tyszkiewicz M The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry 18, 325–336 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81.Inkster B, Sarda S & Subramanian V An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR MHealth UHealth 6, e12106 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 82.Fairburn CG & Patel V The impact of digital technology on psychological treatments and their dissemination. Behav. Res. Ther. 88, 19–25 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 83.Wentzel J, van der Vaart R, Bohlmeijer ET & van Gemert-Pijnen JEWC Mixing online and face- to-face therapy: how to benefit from blended care in mental health care. JMIR Ment. Health 3, e9 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.Schuster R, Topooco N, Keller A, Radvogin E & Laireiter A-R Advantages and disadvantages of online and blended therapy: replication and extension of findings on psychotherapists’ appraisals. Internet Interv. 21, 100326 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 85.Topooco N et al. Attitudes towards digital treatment for depression: a European stakeholder survey. Internet Interv. 8, 1–9 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Topooco N et al. Evaluating the efficacy of Internet-delivered cognitive behavioral therapy blended with synchronous chat sessions to treat adolescent depression: randomized controlled trial. J. Med. Internet Res. 21, e13393 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 87.Topooco N et al. Chat- and Internet-based cognitive-behavioural therapy in treatment of adolescent depression: randomised controlled trial. BJPsych Open 4, 199–207 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 88.Garety P et al. Effects of SlowMo, a blended digital therapy targeting reasoning, on paranoia among people with psychosis: a randomized clinical trial. JAMA Psychiatry 10.1001/jamapsychiatry.2021.0326 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 89.Kenter RMF et al. Costs and effects of Internet cognitive behavioral treatment blended with face- to-face treatment: results from a naturalistic study. Internet Interv. 2, 77–83 (2015). [Google Scholar]
- 90.Dragovic M et al. ‘Validated, easy to use and free’: top three requests for mobile device applications (‘apps’) from mental health consumers and clinicians. Adv. Ment. Health 18, 106–114 (2020). [Google Scholar]
- 91.Patoz M-C et al. Patient and physician perspectives of a smartphone application for depression: a qualitative study. BMC Psychiatry 21, 65 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 92.Marshall JM, Dunstan DA & Bartik W Clinical or gimmickal: the use and effectiveness of mobile mental health apps for treating anxiety and depression. Aust. N. Z. J. Psychiatry 54, 20–28 (2020). [DOI] [PubMed] [Google Scholar]
- 93.Kip H, Wentzel J & Kelders SM Shaping blended care: adapting an instrument to support therapists in using emental health. JMIR Ment. Health 7, e24245 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 94.Ebert DD et al. Internet and computer-based cognitive behavioral therapy for anxiety and depression in youth: a meta-analysis of randomized controlled outcome trials. PLoS ONE 10, e0119895 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 95.Rooksby M, Elouafkaoui P, Humphris G, Clarkson J & Freeman R Internet-assisted delivery of cognitive behavioural therapy (CBT) for childhood anxiety: systematic review and meta-analysis. J. Anxiety Disord. 29, 83–92 (2015). [DOI] [PubMed] [Google Scholar]
- 96.Grist R, Porter J & Stallard P Mental health mobile apps for preadolescents and adolescents: a systematic review. J. Med. Internet Res. 19, e176 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 97.Grist R, Croker A, Denne M & Stallard P Technology delivered interventions for depression and anxiety in children and adolescents: a systematic review and meta-analysis. Clin. Child. Fam. Psychol. Rev. 22, 147–171 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 98.Stiles-Shields C, Ho J & Mohr DC A review of design characteristics of cognitive behavioral therapy-informed behavioral intervention technologies for youth with depression and anxiety. Digital Health 2, 1–13 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 99.Fedele DA, Cushing CC, Fritz A, Amaro CM & Ortega A Mobile health interventions for improving health outcomes in youth: a meta-analysis. JAMA Pediatr. 171, 461–469 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 100.Punukollu M & Marques M Use of mobile apps and technologies in child and adolescent mental health: a systematic review. Evid. Based Ment. Health 22, 161–166 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 101.Shen N et al. Finding a depression app: a review and content analysis of the depression app marketplace. JMIR MHealth UHealth 3, e16 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 102.Torous J, Levin ME, Ahern DK & Oser ML Cognitive behavioral mobile applications: clinical studies, marketplace overview, and research agenda. Cogn. Behav. Pract. 24, 215–225 (2017). [Google Scholar]
- 103.Larsen ME, Nicholas J & Christensen H Quantifying app store dynamics: longitudinal tracking of mental health apps. JMIR MHealth UHealth 4, e96 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 104.Bry LJ, Chou T, Miguel E & Comer JS Consumer smartphone apps marketed for child and adolescent anxiety: a systematic review and content analysis. Behav. Ther. 49, 249–261 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 105.Psihogios AM, Stiles-Shields C & Neary M The needle in the haystack: identifying credible mobile health apps for pediatric populations during a pandemic and beyond. J. Pediatric Psychol. 10.1093/jpepsy/jsaa094 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 106.Aguirre Velasco A, Cruz ISS, Billings J, Jimenez M & Rowe S What are the barriers, facilitators and interventions targeting help-seeking behaviours for common mental health problems in adolescents? A systematic review. BMC Psychiatry 20, 293 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 107.Whiteside SPH Mobile device-based applications for childhood anxiety disorders. J. Child. Adolesc. Psychopharmacol. 26, 246–251 (2016). [DOI] [PubMed] [Google Scholar]
- 108.Cavazos-Rehg P et al. Parental consent: a potential barrier for underage teens’ participation in an mHealth mental health intervention. Internet Interv. 21, 100328 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 109.Schleider JL & Weisz JR Parent expectancies and preferences for mental health treatment: the roles of emotion mind-sets and views of failure. J. Clin. Child. Adolesc. Psychol. 47, S480–S496 (2018). [DOI] [PubMed] [Google Scholar]
- 110.Lyon AR, Dopp AR, Brewer SK, Kientz JA & Munson SA Designing the future of children’s mental health services. Adm. Policy Ment. Health 47, 735–751 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 111.Scholten H & Granic I Use of the principles of design thinking to address limitations of digital mental health interventions for youth: viewpoint. J. Med. Internet Res. 21, e11528 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 112.Fedele DA, McConville A, Moon J & Thomas JG Topical review: Design considerations when creating pediatric mobile health interventions: applying the IDEAS Framework. J. Pediatr. Psychol. 44, 343–348 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 113.Jones K, Williams J, Sipsma H & Patil C Adolescent and emerging adults’ evaluation of a Facebook site providing sexual health education. Public. Health Nurs. 36, 11–17 (2019). [DOI] [PubMed] [Google Scholar]
- 114.Davis DW, Jones VF, Logsdon MC, Ryan L & Wilkerson-McMahon M Health promotion in pediatric primary care: importance of health literacy and communication practices. Clin. Pediatr. 52, 1127–1134 (2013). [DOI] [PubMed] [Google Scholar]
- 115.Compas BE, Jaser SS, Reeslund K, Patel N & Yarboi J Neurcognitive deficits in children with chronic health conditions. Am. Psychol. 72, 326 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 116.Sheoran B et al. YTH StreetConnect: development and usability of a mobile app for homeless and unstably housed youth. JMIR mHealth uHealth 4, e82 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 117.Scheuerman MK, Branham SM & Hamidi F Safe spaces and safe places: unpacking technology-mediated experiences of safety and harm with transgender people. ACM 10.1145/3274424 (2018). [DOI] [Google Scholar]
- 118.Tomasino KN et al. Harnessing peer support in an online intervention for older adults with depression. Am. J. Geriatr. Psychiatry 25, 1109–1119 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 119.Titov N et al. Clinical and cost-effectiveness of therapist-guided Internet-delivered cognitive behavior therapy for older adults with symptoms of depression: a randomized controlled trial. Behav. Ther. 46, 193–205 (2015). [DOI] [PubMed] [Google Scholar]
- 120.Dear BF et al. Clinical and cost-effectiveness of therapist-guided Internet-delivered cognitive behavior therapy for older adults with symptoms of anxiety: a randomized controlled trial. Behav. Ther. 46, 206–217 (2015). [DOI] [PubMed] [Google Scholar]
- 121.Witlox M et al. Blended acceptance and commitment therapy versus face- to-face cognitive behavioral therapy for older adults with anxiety symptoms in primary care: pragmatic single-blind cluster randomized trial. J. Med. Internet Res. 23, e24366 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 122.Seifert A, Reinwand DA & Schlomann A Designing and using digital mental health interventions for older adults: being aware of digital inequality. Front. Psychiatry 10, 568 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 123.Seifert A, Hofer M & Rössel J Older adults’ perceived sense of social exclusion from the digital world. Educ. Gerontol. 44, 1–11 (2019). [Google Scholar]
- 124.Elena-Bucea A, Cruz-Jesus F, Oliveira T & Coelho PS Assessing the role of age, education, gender and income on the digital divide: evidence for the European Union. Inf. Syst. Front. 23, 1007–1021 (2021). [Google Scholar]
- 125.Pew Research Center. 7% of Americans don’t use the Internet. Who are they? Pew Research Center; https://www.pewresearch.org/fact-tank/2021/04/02/7-of-americans-dont-use-the-internet-who-are-they/ (2021). [Google Scholar]
- 126.Czaja SJ et al. Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychol. Aging 21, 333–352 (2006). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 127.Singh K et al. Patient-facing mobile apps to treat high-need, high-cost populations: a scoping review. JMIR Mhealth Uhealth 4, e136 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 128.Rasche P et al. Prevalence of health app use among older adults in Germany: national survey. JMIR Mhealth Uhealth 6, e26 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 129.Nouri S, Khoong EC, Lyles CR & Karliner L Addressing equity in telemedicine for chronic disease management during the COVID-19 pandemic. NEJM Catalyst https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0123 (2020). [Google Scholar]
- 130.Warren JC & Smalley KB Using telehealth to meet mental health needs during the COVID-19 crisis. The Commonwealth Fund; https://www.commonwealthfund.org/blog/2020/using-telehealth-meet-mental-health-needs-during-covid-19-crisis (2020). [Google Scholar]
- 131.Mohr DC et al. Banbury Forum consensus statement on the path forward for digital mental health treatment. Psychiatr. Serv. 72, 677–683 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 132.Sabesan S, Simcox K & Marr I Medical oncology clinics through videoconferencing: an acceptable telehealth model for rural patients and health workers. Intern. Med. J. 42, 780–785 (2012). [DOI] [PubMed] [Google Scholar]
- 133.Orlando JF, Beard M & Kumar S Systematic review of patient and caregivers’ satisfaction with telehealth videoconferencing as a mode of service delivery in managing patients’ health. PLoS ONE 14, e0221848 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 134.Foster MV & Sethares KA Facilitators and barriers to the adoption of telehealth in older adults: an integrative review. Comput. Inform. Nurs. 32, 523–533; quiz 534–535 (2014). [DOI] [PubMed] [Google Scholar]
- 135.Schueller SM, Hunter JF, Figueroa C & Aguilera A Use of digital mental health for marginalized and underserved populations. Curr. Treat. Options Psychiatry 6, 243–255 (2019). [Google Scholar]
- 136.Pew Research Center. Internet/broadband fact sheet. Pew Research Center; https://www.pewinternet.org/fact-sheet/internet-broadband/ (2019). [Google Scholar]
- 137.Pew Research Center. Mobile fact sheet. Pew Research Center; https://www.pewinternet.org/fact-sheet/mobile/(2019). [Google Scholar]
- 138.Anderson M & Jiang J Teens, social media & technology 2018. Pew Research Center; https://www.pewinternet.org/2018/05/31/teens-social-media-technology-2018/ (2018). [Google Scholar]
- 139.Smith A, McGeeney K & Duggan MUS Smartphone Use in 2015 https://www.pewresearch.org/internet/2015/04/01/us-smartphone-use-in-2015/ (Pew Research Center, 2015). [Google Scholar]
- 140.Rathbone AL & Prescott J The use of mobile apps and sms messaging as physical and mental health interventions: systematic review. J. Med. Internet Res. 19, e295 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 141.Mohr DC et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J. Med. Internet Res. 17, e166 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 142.Atske S & Perrin A Home Broadband Adoption, Computer Ownership Vary by Race, Ethnicity in the U.S https://www.pewresearch.org/fact-tank/2021/07/16/home-broadband-adoption-computer-ownership-vary-by-race-ethnicity-in-the-u-s/ (Pew Research Center, 2017). [Google Scholar]
- 143.Anderson M & Kumar M Digital Divide Persists as Lower-income Americans Make Gains in Tech Adoption https://www.pewresearch.org/fact-tank/2021/06/22/digital-divide-persists-even-as-americans-with-lower-incomes-make-gains-in-tech-adoption/ (Pew Research Center, 2019). [Google Scholar]
- 144.Stiles-Shields C et al. Harnessing phones to target pediatric populations with socially complex needs: systematic review. JMIR Pediatr. Parent. 3, e19269 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 145.Titov N et al. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv. 13, 108–115 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 146.Alegría M, Alvarez K, Ishikawa RZ, DiMarzio K & McPeck S Removing obstacles to eliminating racial and ethnic disparities in behavioral health care. Health Aff. 35, 991–999 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 147.Meyer IH Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: conceptual issues and research evidence. Psychol. Bull. 129, 674–697 (2003). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 148.DiClemente RJ et al. Efficacy of an HIV prevention intervention for African American adolescent girls: a randomized controlled trial. JAMA 292, 171–179 (2004). [DOI] [PubMed] [Google Scholar]
- 149.DiClemente RJ et al. Efficacy of sexually transmitted disease/human immunodeficiency virus sexual risk-reduction intervention for african american adolescent females seeking sexual health services: a randomized controlled trial. Arch. Pediatr. Adolesc. Med. 163, 1112–1121 (2009). [DOI] [PubMed] [Google Scholar]
- 150.Wingood GM, Scd & DiClemente RJ Application of the theory of gender and power to examine HIV-related exposures, risk factors, and effective interventions for women. Health Educ. Behav. 27, 539–565 (2000). [DOI] [PubMed] [Google Scholar]
- 151.Wingood GM & DiClemente RJ in Emerging Theories in Health Promotion Practice and Research: Strategies for Enhancing Public Health (eds DiClemente RJ, Crosby RA & Kegler M) 313–347 (Jossey-Bass, 2002). [Google Scholar]
- 152.Bicket MC, Iyengar V & States L Addressing inequities in older adults’ health: a role for emerging technologies. JAMA Health Forum 2, e210680 (2021). [DOI] [PubMed] [Google Scholar]
- 153.Salamanca-Sanabria A, Richards D & Timulak L Adapting an Internet-delivered intervention for depression for a Colombian college student population: an illustration of an integrative empirical approach. Internet Interv. 15, 76–86 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 154.Ospina-Pinillos L et al. Using participatory design methodologies to co-design and culturally adapt the Spanish version of the Mental Health eClinic: qualitative study. J. Med. Internet Res. 21, e14127 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 155.Jackson CS & Gracia JN Addressing health and health-care disparities: the role of a diverse workforce and the social determinants of health. Public Health Rep. 129 (Suppl. 2), 57–61 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 156.Galán CA et al. Editorial: A call to action for an antiracist clinical science. J. Clin. Child. Adolesc. Psychol. 50, 12–57 (2021). [DOI] [PubMed] [Google Scholar]
- 157.Cuevas CDL, Arredondo MT, Cabrera MF, Sulzenbacher H & Meise U Randomized clinical trial of telepsychiatry through videoconference versus face- to-face conventional psychiatric treatment. Telemed. e-Health 12, 341–350 (2006). [DOI] [PubMed] [Google Scholar]
- 158.Firth J et al. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J. Affect. Disord. 218, 15–22 (2017). [DOI] [PubMed] [Google Scholar]
- 159.Friis-Healy EA, Nagy GA & Kollins SH It is time to REACT: opportunities for digital mental health apps to reduce mental health disparities in racially and ethnically minoritized groups. JMIR Ment. Health 8, e25456 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 160.O’Loughlin K, Neary M, Adkins EC & Schueller SM Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv. 15, 110–115 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 161.Torous J, Nicholas J, Larsen ME, Firth J & Christensen H Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid. Based. Ment. Health 21, 116–119 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 162.Robillard JM et al. Availability, readability, and content of privacy policies and terms of agreements of mental health apps. Internet Interv. 17, 100243 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 163.Nebeker C, Bartlett Ellis RJ & Torous J Development of a decision-making checklist tool to support technology selection in digital health research. Transl. Behav. Med. 10, 1004–1015 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 164.Nahum-Shani I et al. Just-in-time adaptive interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann. Behav. Med. 52, 446–462 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 165.Klasnja P et al. Microrandomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 34S, 1220–1228 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 166.Guo S et al. The efficacy of Internet-based cognitive behavioural therapy for social anxiety disorder: a systematic review and meta-analysis. Clin. Psychol. Psychother. 10.1002/cpp.2528 (2020). [DOI] [PubMed] [Google Scholar]
- 167.Soh HL, Ho RC, Ho CS & Tam WW Efficacy of digital cognitive behavioural therapy for insomnia: a meta-analysis of randomised controlled trials. Sleep Med. 75, 315–325 (2020). [DOI] [PubMed] [Google Scholar]
- 168.Wootton BM Remote cognitive-behavior therapy for obsessive-compulsive symptoms: a meta-analysis. Clin. Psychol. Rev. 43, 103–113 (2016). [DOI] [PubMed] [Google Scholar]
- 169.Cervin M & Lundgren T Technology-delivered cognitive-behavioral therapy for pediatric anxiety disorders: a meta-analysis of remission, posttreatment anxiety, and functioning. J. Child. Psychol. Psychiatry 10.1111/jcpp.13485 (2021). [DOI] [PubMed] [Google Scholar]
- 170.Linardon J, Shatte A, Messer M, Firth J & Fuller-Tyszkiewicz M e-Mental health interventions for the treatment and prevention of eating disorders: an updated systematic review and meta-analysis. J. Consult. Clin. Psychol. 88, 994–1007 (2020). [DOI] [PubMed] [Google Scholar]
- 171.Volk J, Palanker D, O’Brien M & Goe CL States’ actions to expand telemedicine access during COVID-19 and future policy considerations. 10.26099/R95Z-BS17 (2021). [DOI]
- 172.Shah K et al. Focus on mental health during the coronavirus (COVID-19) pandemic: applying learnings from the past outbreaks. Cureus 12, e7405 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 173.Sharma A, Sasser T, Schoenfelder Gonzalez E, Vander Stoep A & Myers K Implementation of home-based telemental health in a large child psychiatry department during the COVID-19 crisis. J. Child. Adolesc. Psychopharmacol. 30, 404–413 (2020). [DOI] [PubMed] [Google Scholar]
- 174.Mishkind MC et al. Rapid conversion to telemental health services in response to COVID-19: experiences of two outpatient mental health clinics. Telemed. J. E. Health 27, 778–784 (2021). [DOI] [PubMed] [Google Scholar]
- 175.Connolly SL et al. Rapid increase in telemental health within the Department of Veterans Affairs during the COVID-19 pandemic. Telemed. J. E. Health 27, 454–458 (2021). [DOI] [PubMed] [Google Scholar]
- 176.Aminoff V et al. Internet-based cognitive behavioral therapy for psychological distress associated with the COVID-19 pandemic: a pilot randomized controlled trial. Front. Psychol. 12, 684540 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 177.Jaworski BK et al. Exploring usage of COVID Coach, a public mental health app designed for the COVID-19 pandemic: evaluation of analytics data. J. Med. Internet Res. 23, e26559 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 178.Job SA & Williams SL Translating online positive psychology interventions to sexual and gender minorities: a systematic review. Psychol. Sex. Orientat. Gend. Divers. 7, 455–503 (2020). [Google Scholar]
- 179.de Mello MF, de Jesus Mari J, Bacaltchuk J, Verdeli H & Neugebauer R A systematic review of research findings on the efficacy of interpersonal therapy for depressive disorders. Eur. Arch. Psychiatry Clin. Neurosci. 255, 75–82 (2005). [DOI] [PubMed] [Google Scholar]
- 180.Dagöö J et al. Cognitive behavior therapy versus interpersonal psychotherapy for social anxiety disorder delivered via smartphone and computer: a randomized controlled trial. J. Anxiety Disord. 28, 410–417 (2014). [DOI] [PubMed] [Google Scholar]
- 181.Doss BD et al. A randomized controlled trial of the web-based OurRelationship program: effects on relationship and individual functioning. J. Consult. Clin. Psychol. 84, 285–296 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 182.Christensen A et al. Traditional versus integrative behavioral couple therapy for significantly and chronically distressed married couples. J. Consult. Clin. Psychol. 72, 176–191 (2004). [DOI] [PubMed] [Google Scholar]
- 183.Lagan S, Sandler L & Torous J Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open 11, e047001 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 184.Gerke S, Stern AD & Minssen T Germany’s digital health reforms in the COVID-19 era: lessons and opportunities for other countries. NPJ Digit. Med. 3, 94 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 185.Richards D et al. Digital IAPT: the effectiveness & cost-effectiveness of Internet-delivered interventions for depression and anxiety disorders in the Improving Access to Psychological Therapies programme: study protocol for a randomised control trial. BMC Psychiatry 18, 596 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 186.Sturk H, Crowther R & Kavanagh DJ Head to health: Practitioner perceptions of the new digital mental health gateway. Aust. J. Rural. Health 27, 448–453 (2019). [DOI] [PubMed] [Google Scholar]
- 187.Lora A, Hanna F & Chisholm D Mental health service availability and delivery at the global level: an analysis by countries’ income level from WHO’s Mental Health Atlas 2014. Epidemiol. Psychiatr. Sci. 29, 1–12 (2020). [DOI] [PubMed] [Google Scholar]
- 188.Carter H, Araya R, Anjur K, Deng D & Naslund JA The emergence of digital mental health in low-income and middle-income countries: a review of recent advances and implications for the treatment and prevention of mental disorders. J. Psychiatr. Res. 133, 223–246 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 189.Stewart S Minorities are Underrepresented in Clinical Trials (American Academy of Family Physicians, 2018). [Google Scholar]
- 190.Flaskerud JH Cultural bias and Likert-type scales revisited. Issues Ment. Health Nurs. 33, 130–132 (2012). [DOI] [PubMed] [Google Scholar]