Skip to main content
NCBI home page
American Journal of Public Health logoLink to American Journal of Public Health
. 2024 Feb;114(Suppl 2):171–179. doi: 10.2105/AJPH.2023.307505

Pragmatic Approach to the Assessment and Use of Digital Mental Health Interventions for Health Workers

Lyndsay Krisher 1,✉, Debra L Boeldt 1, Chloe A Nicksic Sigmon 1, Sarah E Rimel 1, Lee S Newman 1
PMCID: PMC10916727  PMID: 38354345

Abstract

Symptoms of poor mental health among working people, especially health workers, are on the rise in the United States, contributing to a burgeoning market of thousands of mental health technology products, few of which have undergone rigorous evaluation. Most research on these products focuses on deploying digital mental health interventions as ancillary support in clinical practice and community settings. Little is known of the effectiveness of these tools when employers offer them.

We describe the landscape of digital mental health interventions, providing an overview of mental health conditions that are addressed with technology-based solutions in the workplace and the products and services available. We argue for employers to apply a methodical approach to evaluating and selecting technologies for their organizations, and we review relevant frameworks for evaluation.

Considering the rapidly evolving landscape of digital mental health interventions, we offer evidence-informed recommendations to organizations and decision-makers seeking to support workplace mental health and well-being, and we advocate the development of products that help organizations assess how they can mitigate workplace conditions that may contribute to poorer mental health. (Am J Public Health. 2024;114(S2):S171–S179. https://doi.org/10.2105/AJPH.2023.307505)

Mental distress reporting has accelerated since the onset of the COVID-19 pandemic, with nearly 60% of US adults reporting having been concerned for either their own mental health or that of family and friends, an increase of 9% since April 2020.1 Furthermore, poor employee mental health is associated with a $1 trillion annual global cost owing to lost productivity.2 However, addressing well-being in the workplace can make a difference: every dollar spent on health promotion is linked to multiple dollars saved by reducing health care costs and absenteeism.3

Because of the rising prevalence of poor mental health symptoms and the potential benefits of addressing mental health in the workplace, many organizations offer technology-based solutions to augment the treatment of mental health disorders. As a type of “digital health,” which broadly includes categories such as mobile health, telehealth and telemedicine, health information technology, and wearable devices,4 mental health–supportive technology is being used to expand a suite of tools to connect and empower employees to manage their health and well-being.

It is argued that when technologies are strategically used in the workplace, they can improve worker health outcomes on a broad scale because of their potentially wide accessibility and speed of deployment. Technology holds promise as a means of supplementing care delivery, augmenting telehealth, making treatment more personalized and effective, and aiding prevention efforts.5,6 There is evidence that digital mental health solutions are effective and may be good for use in the aftermath of the pandemic.7 In theory and practice via empirical evidence, there is a strong rationale for offering employees mental health assistance through technology, especially considering its scalability with added convenience and ease of use, ideally resulting in greater adoption and benefit.

Because of the ubiquitous nature of and increased access to personal digital devices, such as smartphones, tablets, and wearable fitness trackers, many wellness services have shifted to digital formats. Applications (“apps”) have become a staple of employer wellness offerings.8,9 Technology that can enhance employee mental health, self-efficacy, and well-being in particular are attracting interest. And as employers come to embrace a more comprehensive approach, along the lines of the National Institute for Occupational Safety and Health’s Total Worker Health (TWH) program, we can expect the use of mental health technologies to increase as an investment in a more resilient and productive workforce.

More than 10 000 technology solutions are currently offered to address mental health.10 The use of digital mental health tools increased during the COVID-19 pandemic,11 and the number of apps available will continue to grow.12 Technology solutions include cell phone–based apps, wearables, chatbots, virtual reality, and augmented reality. In addition to wellness and cognitive behavioral therapy offerings, digital mental health interventions encompass a range of content for well-being promotion and assessment as well as for treatment and recovery. They are intended to help with such conditions as depression, anxiety, posttraumatic stress disorder (PTSD), and suicidality.

The significant lag between the invention, marketing, and eventual research evaluation of digital mental health interventions thwarts timely assessment of products and claims. The sheer volume of available mental health technologies impedes systematic and timely evaluation. Only a small proportion has undergone rigorous evaluation for efficacy and effectiveness.13 In response, investigators have begun to propose taxonomies to classify these products for research and evaluation.14 The technologies can fit into more than 1 category in the taxonomy, adding complexity to organizing, reviewing, and evaluating products.

Consequently, products are being marketed to employers and the public based on very limited or no evidence of effectiveness, and their marketers may make unsubstantiated or even misleading claims. Often, companies conduct internal research but do not complete extensive external studies. Most commonly, published articles and reviews evaluating the use of technological tools focus on their potential value in clinical treatment and, to a lesser extent, prevention.14

Given the current limitations of the intervention science, a common cautionary theme in the literature is that digital mental health solutions should be used as a complement to and not a substitute for more traditional mental health prevention and therapy approaches: an ancillary tool kit. However, some technology companies conduct rigorous research to clinically validate their products, and although caution is warranted, there remains immense potential to use technology for both mental health support and health promotion.

On a companywide level, employers, including health care organizations, and their workers face a challenge in determining which products are best for them. Even if a particular technology has demonstrated efficacy in controlled trials, substantial barriers to feasibility, acceptability, and effectiveness in real-world use must be considered. Furthermore, health workers span a range of professions, including physicians, nurses, public health employees, technicians, home health workers, and pharmacists, each with unique needs. The utilization of mental health interventions is likely to differ between and within these groups.

Through the lens of implementation science,15 when an organization prepares to license or purchase a digital mental health product or service, barriers related to implementation, adoption, engagement, and sustainability are essential to consider, as each can undermine effectiveness in practice.16–18 In addition, for many reasons, including stigma and workplace concerns for psychological safety,19 personal health information related to mental health warrants special protection.20 This requires being aware of potential data vulnerabilities and data privacy policies. Data privacy and health information protections must be considered.

We address the state of the art and science of digital mental health interventions, providing an overview of the types of mental health conditions addressed in the workplace with digital mental health tools. We offer an overview of the categories of products and services currently available. We focus primarily on mobile apps and related resources although we note where frameworks and recommendations also apply to other forms of digital mental health interventions. We argue that employers should apply a methodical approach to evaluating and selecting technologies for their organizations. To assist decision-makers, we identify some of the independent resources available for selecting technology solutions and summarize non–technology-related considerations to address when promoting these solutions in the workplace.

CONDITIONS, AUDIENCES, AND TOOL FEATURES

Nearly all mental health conditions have attracted technology developers—from the government, for-profit companies, nonprofit companies, health care companies, and academic institutions. These conditions include mood disorders, anxiety, insomnia, eating disorders, PTSD, and substance use. As a first step in selecting digital tools to assist workers in a health organization, one must consider which conditions are the highest priority. Most health organizations will choose to address the most commonly recognized mental health conditions among health workers, including depression, anxiety, PTSD, psychological strain, and related burnout.21 They may also seek to provide employees with tools to manage stress, such as the so-called mindfulness apps. Many technology solutions are labeled “self-help” or “well-being” tools. Others serve as reference guides. Employers must consider both the evidence for the effectiveness of programs and the features and design elements to verify that they are appropriate for both the general workforce and for specific subgroups of workers; organizations should be mindful that not all employees will benefit from the same type of intervention.8

Many digital products are designed to be accessed through a Web browser or a smartphone. An essential consideration for improving accessibility for all employees, including those with limited Internet access and cell phone coverage, is whether the application can function both offline and online. Smartphone apps carry some advantages over other digital (e.g., computer- and tablet-based) programs, including allowing more convenient access to activities such as behavior tracking and symptom monitoring in real time as well as before and after significant mental health events; they also offer flexibility in the time and location, and thus privacy, of use.22 Smartphone apps’ intended audiences will be workers who meet certain levels of literacy, who can download and navigate technology, and who have access to the required hardware.

Accessibility is an important consideration. The relative ease of accessibility to mobile apps because of growing smartphone usage means potentially ineffective and even harmful programs have proliferated.22,23 Some digital tools, including apps, are available in more than 1 language, most commonly English and Spanish. However, a recent review found only 14.5% of mental health apps to be operable in Spanish.24

In weighing the value of a particular technological solution, it is important to understand its specific features because they will be closely tied to the expected outcomes. Although some interventions have multiple features, most can be grouped into 1 of several categories (Table 1).25

TABLE 1—

Summary of Digital Mental Health Interventions Commonly Used in the Workplace

Category Description
Prevention and treatment solutions May include chatbots/artificial intelligence tools as well as in-person psychotherapeutic prevention and clinical treatment, using methods such as CBT, behavior/attitude/mood tracking, meditation, and hypnosis, along with general well-being, such as physical activity, nutrition, and sleep
Customizable to size of employer and employee needs
Wearables and digital biomarker apps Assess aspects of well-being through collection of physiological data including heart rate, skin temperature or even electrodermal activity
Often also include smartphone apps that monitor mood and emotional state through techniques such as voice recording or prompts
Analytic tools Typically used to integrate data collected from the wearables or other biomarker apps to assist with a referral if needed
In terms of prevention, can alert employees themselves to take some time for their mental health, or alert leaders when teams may be experiencing higher than normal rates of stress
Open in a new tab

Note. CBT = cognitive behavioral therapy.

Source. Adapted from Brassey et al.25

RESEARCH ON TECHNOLOGIES IN THE WORKPLACE

Although some work has been done to evaluate mental health technologies in clinical practice and community settings, less is known of the effectiveness of these modalities when they are offered in the workplace, typically by employers or health plans. Table A (available as a supplement to the online version of this article at https://www.ajph.org) summarizes the results of several key systematic reviews and meta-analyses to help distill the present state of research in this area.

In general, working people’s utilization of psychological interventions is low, even though the workplace has been proposed as an ideal setting for the delivery of poor mental health prevention and mental well-being programs, with the potential to provide employees with direct access to appropriate treatment and to increase productivity.26–28 Furthermore, digital mental health interventions bring ease and flexibility of access, relative affordability, and multiple modes of delivery.29,30 Potential challenges with implementation of and engagement in digital mental health solutions in the workplace have been identified, including real or perceived lack of anonymity, time barriers, lack of job autonomy or needing to obtain permission to access the intervention during the workday, shared workspaces or computer equipment that defeat anonymity, and the perceived need to appear mentally and emotionally strong at work.27,31

RESEARCH TARGETING HEALTH CARE SETTINGS

There is a paucity of evidence on the use of digital mental health interventions specifically addressing the needs of health workers. Following the onset of the COVID-19 pandemic and the frequently abrupt transition to technological solutions, users and employers initially sought already-existing apps and programs. One application applied to health care workers during the pandemic was the government-developed, cognitive behavioral therapy–based program PTSD Coach (https://mobile.va.gov/app/ptsd-coach), which had been deployed for some time already in the Veterans Affairs health care system, for example.32 Telepsychiatry, teletherapy, and other virtual treatment programs and interventions have shown promise and were available to some extent before the pandemic.33 They have since expanded rapidly, and further evaluation of their effectiveness is still needed.6,34 A recent systematic review summarized digital mental health interventions that were used with health care workers, about half of them in China, during the COVID-19 pandemic35 and found that only about one quarter of the interventions had included an evaluation component and that half presented challenges to user adoption.

Before the pandemic, in a 2018 systematic review, Pospos et al. conducted an evaluation of Web-based tools and mobile apps for potential use among health care professionals and students.36 The authors proposed next steps to tailor existing digital mental health interventions more specifically for health care workers, focusing on the issues of burnout, depression, and suicidality.

FRAMEWORKS FOR EVALUATION

Several independent, nonprofit organizations evaluate digital solutions to enhance individuals’ and organizations’ ability to improve their selection process from the array of mental health technology solutions. Without a usable framework for guidance, providers, employers, and end users alike often rely on user star ratings in the app store, which may be misleading. Meanwhile, the US Food and Drug Administration (FDA) has recognized the difficulty in regulating health apps and other digital health tools, having launched the pilot precertification program in 2017 that places the burden on app developers to meet specific standards of quality and safety.37 To date, only a few apps have achieved FDA’s certification. As that program is still in development, at least 45 other evaluation frameworks have been proposed,38 and concerted efforts have been made to assemble and harmonize these various tools, which are updated frequently, into a single, navigable framework.39

The M-Health Index and Navigation Database (MIND; Figure 1) allows users to view app attributes and compare ratings (Table B, available as a supplement to the online version of this article at http://www.ajph.org)40 and is based on the framework developed in collaboration with the American Psychiatric Association.41,42 The framework is a 5-level pyramid with a medical ethics principle at each level.41 The pyramid shape is designed to encourage users to begin their evaluation of an app at the bottom and work their way up (e.g., if an app is not accessible to the user [level 1], they need not continue up the pyramid).41 The MIND database (www.mindapps.org) is operated by the Beth Israel Deaconess Medical Center’s Division of Digital Psychiatry (Boston, MA). It relies on crowdsourcing to conduct quantitative app evaluations using 105 binary or numeric response questions to inform the database so that it keeps up with this constantly changing landscape and is accessible to, navigable by, and usable by broad audiences.41 End users, providers, and employers can evaluate an app to assess whether it will meet their individual needs and goals.

FIGURE 1—

FIGURE 1—

Open in a new tab

American Psychiatric Association Framework and Basis for the M-Health Index and Navigation Database (MIND)

Source. This figure is adapted from Figure 1 in Lagan et al.41 and used under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0).

The levels of the pyramid are as follows:

  • 1.

    Accessibility is grounded in the principle of justice and addresses factors such as ownership of a new enough smartphone to access all features of the app, cost barriers, and offline functionality for those with unreliable Internet access.

  • 2.

    Privacy & Security is based on nonmaleficence: the use of an app should not cause harm, including, for example, through the release of confidential health information. Individuals with mental illness have been shown to use apps less frequently because of concerns over their data privacy.43 Indeed, although FDA guidelines exist to address privacy issues in mobile medical apps, many apps pass as wellness tools and are thus exempt from the FDA guidelines.41,44

  • 3.

    Clinical Foundation pertains to evidence of the app’s benefit, thus relating to the ethical principle of beneficence. This level assesses whether there are published data derived from well-executed research studies, substantial end user feedback, and other evidence to support claims of effectiveness.41

  • 4.

    Engagement Style rests on the principle of autonomy, that is, the assurance that one decides to use the app free of coercion or other undue influence. Importantly, this level focuses less on ease of use and more on the extent of engagement because studies have shown that most users stop using mental health apps after downloading them in less than a month, often less than a week,45 creating challenges for the evaluation of long-term efficacy.

  • 5.

    Therapeutic Goal forms the top of the pyramid. This level considers the connection between the user and clinical care and treatment. The importance of this level depends on the purpose of the app because some are intended to be standalone self-management tools and not designed for clinical referral or as ancillary clinical tools. However, apps that address depression and anxiety have been proven most effective when linked with clinical treatment.22 We argue that such an app should allow integration with an electronic health record or similar connection directly with a medical provider. Other related evaluation questions at this top level ask about the ability for data sharing with other individuals and social networks if desired and integration with other digital devices and wearables. The goal is that the app permits shared decision-making, the final ethics principle referenced.

Another key framework to highlight as a way of vetting mental health apps is the One Mind Psyberguide Credibility Rating System (http://onemindpsyberguide.org), a free online app-rating platform developed and operated by a nonprofit organization of the same name. App reviews, available on the site, are conducted by mental health care providers and researchers with expertise in digital mental health interventions and include links to relevant research. The platform uses an evaluation framework based on 4 primary dimensions: (1) credibility (quality of the evidence base and development process, consumer ratings, intervention specificity); (2) user experience (functionality and software support, engagement); (3) transparency (privacy and security); and (4) professional reviews (narrative reviews from professionals in a relevant field with recommendations for use).13,46 Users can search for and filter out apps by various metrics, including mental health conditions and treatment categories, clinical dimensions, and app name. Limitations that have been cited regarding Psyberguide are that the professional reviews may become outdated fairly quickly; app information is sometimes inaccurate because of frequent changes, such as software updates and cost; and the user is unable to filter by more specific categories (e.g., grief and bereavement, alcohol use tracking, exercise).46 Due to a loss of funding, the site will remain available as a resource but will no longer be updated.

Finally, the International Organization for Standardization in 2021 released a set of quality requirements for health apps, including a quality and reliability label: ISO/TS 82304–2. Health software—Part 2: Health and wellness apps—Quality and reliability. Supported by the European Commission and developed in collaboration with the European Committee for Standardization, the new standard is expected to be a tool for other organizations that evaluate health and wellness apps.

THE MIND DATABASE

Several additional important considerations should be included in decision-making regarding purchasing and promoting digital tools in health care and other industries. The MIND database, for example, includes the following parameters on its Web site (https://mindapps.org/Apps) that can be selected (as displayed in Table B). There are 12 filters, each with a list of commonly searched-for parameters:

  • •

    Cost refers to whether payment is required and when costs are incurred (e.g., during download or in the app).

  • •

    Developer types include government, nonprofit and for-profit companies, health care, and academia.

  • •

    Engagements include user experiences such as data feedback; messaging and chat; gamification; network, peer, and provider support; and whether responses are in real time or asynchronous.

  • •

    Evidence & Clinical foundations relate to the evidence available for the app, including whether supporting research is available, whether it does what it claims, and whether it contains well-written and relevant content.

  • •

    Features are varied and include whether specific intervention modalities are used, such as cognitive behavioral therapy, mindfulness, and peer support, as well as capabilities and services offered, such as tracking of mood, sleep, medications, and symptoms and whether interaction is available with a chatbot or with a therapist.

  • •

    Functionalities involve language options, accessibility options, and ownership of data.

  • •

    Inputs allow the use of camera and microphone, as well as geolocation.

  • •

    Outputs include notifications, references, data summaries, connection to social networks, and linking to formal care.

  • •

    Privacy considers how data are stored and shared, whether a privacy policy exists and is visible, and whether individuals can opt out of data collection or delete their data.

  • •

    Supported conditions include mood disorders, stress and anxiety, PTSD, and substance use.

  • •

    Treatment Approaches include cognitive behavioral therapy, mindfulness, and physical health exercises.

  • •

    Uses include whether the app is designed for self-help, referral, or a hybrid model.

THE EMPLOYER–EMPLOYEE RELATIONSHIP CONTEXT

One of the key advantages of digital mental health tools is that they can be empowering and give employees an opportunity to address their mental well-being on a voluntary basis. Additionally, they can function as secondary prevention before more serious health consequences occur. Employees may be skeptical about using digital mental health tools, especially if their organization is perceived to be a significant contributor to poor mental health. When organizations adopt apps, it is important for them to communicate clearly on such considerations as privacy, data use, and the organization’s intent.

From the TWH perspective, the isolated use of mental health apps falls short if the organization has not seriously examined and addressed how it contributes—and may mitigate—mental health harms that result from health industry working conditions. It is rare to find a digital mental health solution that addresses primary prevention at the organizational level. And many workers already attribute their mental health challenges to using technology. In the case of health workers, it is important to recognize that the use of technology itself, especially the electronic health record, has been identified as a major contributor to health worker burnout.47

There is an opportunity for the digital technology industry to develop tools for health industry leadership to use for assessing and creating organizational solutions that mitigate the anxiety, stress, depression, and burnout that grueling schedules, changing shifts, electronic health record systems, high job demands, exposure to violence, and other factors in the organization’s control can lead to.

Importantly, employers’ goal should be to move toward a holistic approach in which well-being, as described by Brassey et al., is integrated into a broader mental health promotion strategy and ultimately embedded in the organizational culture and overall business strategy through strong leadership commitment and high employee engagement.25

RECOMMENDATIONS

With the present landscape of digital mental health interventions, its ever-changing nature, and the existing state of research, we offer recommendations to organizations and decision-makers (adapted from Park et al.48). The recommendations focus on selecting and implementing technology solutions, specifically mobile applications, to support workplace mental health and well-being.

Selection

Identify and address workers’ needs

First, technology selections should reflect employees’ specific needs from preventive, therapeutic, and practical standpoints. For example, in addition to the particular support area (e.g., burnout), how long, when, and how they can access the technology resource should be considered.

Thoroughly vet technology

Conduct a thorough review of technology options before making a final choice. Review any existing research and evidence base for content, request demonstrations of the technology, and be familiar with personalization options for your employees. Especially for the workplace, prioritize convenience, ease of access, and anonymity to maximize the likelihood that employees will engage with the tool. Where possible, include a diversity of employee perspectives in this process.

Implementation

Communication

Leadership should use positive language (e.g., highlight that the goal of deployment is to “improve well-being”), remove the stigma associated with use, and place the technology into a broader context of how the organization will do its part to mitigate stressful working conditions. This includes transparency about the helpfulness and limitations of the tool and establishing a clear process for onboarding.

Confidentiality

The specifics of data confidentiality and how the data will be used should be elucidated before rollout, ensuring that employees will be fully informed of whether anyone in the organization will have access to their use or engagement data or this will remain private.

Culture

Cultivate a workplace climate that emphasizes the organizational importance and value of employee wellness. Employees should be aware that wellness matters and that the responsibility for employee well-being is shared between the organization and the employee. Equitable access to the digital mental health tool6 and honest discussion of other measures being taken at an organizational level can be part of the orientation. Leadership should emphasize that employee participation is voluntary. Although the use of incentives in workplace wellness programs has pros and cons, rewarding employee involvement (e.g., financial incentives, friendly workplace competition) as part of a multifaceted, nondiscriminatory well-being program should be considered.

Evaluation

Before adopting and launching any digital mental health technology, a plan and method should be established for determining whether the goals of the program are being achieved. Establishing a program evaluation plan before launch will help leadership understand the outcomes of implementing the resource. Again, involving a distribution of employees at the outset will aid in the evaluation process. Depending on the context, multiple perspectives should be considered, including organizational leadership and supervisors, employees, and even employee family members.

Responsibility

Although the technology is a resource, employers must take a full measure of how the conditions of work themselves contribute to stress, burnout, and poorer mental health. Taking steps to mitigate the impact of work on sleep and fatigue, schedule variability, job demands, and job control is the sine qua non of primary prevention and the goal of TWH for health workers.

ACKNOWLEDGMENTS

This work was supported in part by the Centers for Disease Control and Prevention (CDC), National Institute for Occupational Safety and Health (NIOSH; intergovernmental personnel agreements 19IPA1916740, 20IPA2014129-M02).

We wish to thank Matthew Schilz for his assistance with this work.

“Total Worker Health” is a registered trademark of the US Department of Health and Human Services (HHS).

Note. Participation by AJPH does not imply endorsement by the HHS, the CDC, or the NIOSH. The views expressed in this essay are our own and do not necessarily express the views or opinions of the CDC.

CONFLICTS OF INTEREST

The authors have no conflicts of interest to disclose.

HUMAN PARTICIPANT PROTECTION

This project did not include human participants; therefore, institutional review board approval was not obtained.

REFERENCES

  • 1.CVS Health. CVS Health/Morning Consult survey finds mental health concerns increase substantially among Americans of all backgrounds. May 2, 2022. Available at: https://www.cvshealth.com/news/mental-health/cvs-health-morning-consult-survey-finds-mental-health-concerns.html. Accessed February 8, 2023.
  • 2.Chisholm D , Sweeny K , Sheehan P , et al. Scaling-up treatment of depression and anxiety: a global return on investment analysis. Lancet Psychiatry. 2016;3(5):415–424. 10.1016/S2215-0366(16)30024-4 [DOI] [PubMed] [Google Scholar]
  • 3.Cunningham-Hill M , Dodge-Rice Z , Wilson-Myers C , Sherman C , Neary M , Schueller SM. Digital tools for mental health: employer’s guide. 2020. Available at: https://www.thementalhealthco alition.org/resources/digital-tools-for-mental-health-employers-guide. Accessed May 10, 2022.
  • 4.US Food and Drug Administration. What is digital health? Available at: https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health. Accessed February 8, 2023.
  • 5.Torous J , Firth J. Bridging the dichotomy of actual versus aspirational digital health. World Psychiatry. 2018;17(1):108–109. 10.1002/wps.20464 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Lattie EG , Stiles-Shields C , Graham AK. An overview of and recommendations for more accessible digital mental health services. Nat Rev Psychol. 2022;1:87–100. 10.1038/s44159-021-00003-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Rauschenberg C , Schick A , Hirjak D , et al. Evidence synthesis of digital interventions to mitigate the negative impact of the COVID-19 pandemic on public mental health: rapid meta-review. J Med Internet Res. 2021;23(3):e23365. 10.2196/23365 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Stratton E , Lampit A , Choi I , Calvo RA , Harvey SB , Glozier N. Effectiveness of eHealth interventions for reducing mental health conditions in employees: a systematic review and meta-analysis. PLoS One. 2017;12(12):e0189904. 10.1371/journal.pone.0189904 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Anthes E. Pocket psychiatry: mobile mental-health apps have exploded onto the market, but few have been thoroughly tested. Nature. 2016;532(7):20–23. 10.1038/532020a [DOI] [PubMed] [Google Scholar]
  • 10.Torous J , Roberts LW. Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry. 2017;74(5):437–438. 10.1001/jamapsychiatry.2017.0262 [DOI] [PubMed] [Google Scholar]
  • 11.Sorkin DH , Janio EA , Eikey EV , et al. Rise in use of digital mental health tools and technologies in the United States during the COVID-19 pandemic: survey study. J Med Internet Res. 2021;23(4):e26994. 10.2196/26994 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Torous J , Jän Myrick K , Rauseo-Ricupero N , Firth J. Digital mental health and COVID-19: using technology today to accelerate the curve on access and quality tomorrow. JMIR Ment Health. 2020;7(3):e18848. 10.2196/18848 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Neary M , Bunyi J , Palomares K , et al. A process for reviewing mental health apps: using the One Mind PsyberGuide Credibility Rating System. Digit Health. 2021;7:20552076211053690. 10.1177/20552076211053690 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.De Witte NAJ , Joris S , Van Assche E , Van Daele T. Technological and digital interventions for mental health and wellbeing: an overview of systematic reviews. Front Digit Health. 2021;3:754337. 10.3389/fdgth.2021.754337 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Guerin RJ , Harden SM , Rabin BA , et al. Dissemination and implementation science approaches for occupational safety and health research: implications for advancing total worker health. Int J Environ Res Public Health. 2021;18(21):11050. 10.3390/ijerph182111050 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Stoyanov SR , Hides L , Kavanagh DJ , Zelenko O , Tjondronegoro D , Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3(1):e27. 10.2196/mhealth.3422 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Flett JAM , Fletcher BD , Riordan BC , Patterson T , Hayne H , Conner TS. The peril of self-reported adherence in digital interventions: a brief example. Internet Interv. 2019;18:100267. 10.1016/j.invent.2019.100267 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Mani M , Kavanagh DJ , Hides L , Stoyanov SR. Review and evaluation of mindfulness-based iPhone apps. JMIR Mhealth Uhealth. 2015;3(3):e82. 10.2196/mhealth.4328 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Grailey KE , Murray E , Reader T , Brett SJ. The presence and potential impact of psychological safety in the healthcare setting: an evidence synthesis. BMC Health Serv Res. 2021;21(1):773. 10.1186/s12913-021-06740-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Armontrout J , Torous J , Fisher M , Drogin E , Gutheil T. Mobile mental health: navigating new rules and regulations for digital tools. Curr Psychiatry Rep. 2016;18(10):91. 10.1007/s11920-016-0726-x [DOI] [PubMed] [Google Scholar]
  • 21.Melnyk BM , Kelly SA , Stephens J , et al. Interventions to improve mental health, well-being, physical health, and lifestyle behaviors in physicians and nurses: a systematic review. Am J Health Promot. 2020;34(8):929–941. 10.1177/0890117120920451 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Linardon J , Cuijpers P , Carlbring P , Messer M , Fuller-Tyszkiewicz M. The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry. 2019;18(3):325–336. 10.1002/wps.20673 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Loucas CE , Fairburn CG , Whittington C , Pennant ME , Stockton S , Kendall T. E-therapy in the treatment and prevention of eating disorders: a systematic review and meta-analysis. Behav Res Ther. 2014;63:122–131. 10.1016/j.brat.2014.09.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Muñoz AO , Camacho E , Torous J. Marketplace and literature review of Spanish language mental health apps. Front Digit Health. 2021;3:615366. 10.3389/fdgth.2021.615366 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Brassey J , Guntner A , Isaak K , Silberzahn T. Using digital tech to support employees’ mental health and resilience. July 2021. Available at: http://www.mckinsey.com/∼/media/mckinsey/industries/pharmaceuticals and medical products/our insights/using digital tech to support employees mental health and resilience/using-digital-tech- to-support-employees-mental-health-and-resilience-f.pdf. Accessed April 8, 2022.
  • 26.Lehr D , Geraedts A , Persson Asplund R , et al. Occupational e-mental health: current approaches and promising perspectives for promoting mental health in workers. In: Wiencke M , Cacace M , Fischer S , eds. Healthy at Work. Cham, Switzerland: Springer; 2016:257–281. 10.1007/978-3-319-32331-2_19 [DOI] [Google Scholar]
  • 27.Carolan S , Harris PR , Cavanagh K. Improving employee well-being and effectiveness: systematic review and meta-analysis of web-based psychological interventions delivered in the workplace. J Med Internet Res. 2017;19(7):e271. 10.2196/jmir.7583 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Mykletun A , Harvey SB. Prevention of mental disorders: a new era for workplace mental health. Occup Environ Med. 2012;69(12):868–869. 10.1136/oemed-2012-100846 [DOI] [PubMed] [Google Scholar]
  • 29.Phillips EA , Gordeev VS , Schreyögg J. Effectiveness of occupational e-mental health interventions: a systematic review and meta-analysis of randomized controlled trials. Scand J Work Environ Health. 2019;45(6):560–576. 10.5271/sjweh.3839 [DOI] [PubMed] [Google Scholar]
  • 30.Qu K , Ozersky S. Workplace mental health in Canada: towards a digital future. In: Riba MB , Parikh SV , Greden JF , eds. Mental Health in the Workplace: Strategies and Tools to Optimize Outcomes. Cham, Switzerland: Springer; 2019:27–48. 10.1007/978-3-030-04266-0_3 [DOI] [Google Scholar]
  • 31.Carolan S , de Visser RO. Employees’ perspectives on the facilitators and barriers to engaging with digital mental health interventions in the workplace: qualitative study. JMIR Ment Health. 2018;5(1):e8. 10.2196/mental.9146 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Wong AH , Pacella-LaBarbara ML , Ray JM , Ranney ML , Chang BP. Healing the healer: protecting emergency health care workers’ mental health during COVID-19. Ann Emerg Med. 2020;76(4):379–384. 10.1016/j.annemergmed.2020.04.041 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Barnett P , Goulding L , Casetta C , et al. Implementation of telemental health services before COVID-19: rapid umbrella review of systematic reviews. J Med Internet Res. 2021;23(7):e26492. 10.2196/26492 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Hoffman DA. Increasing access to care: telehealth during COVID-19. J Law Biosci. 2020;7(1):lsaa043. 10.1093/jlb/lsaa043 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Drissi N , Ouhbi S , Marques G , de la Torre Díez I , Ghogho M , Janati Idrissi MA. A systematic literature review on e-mental health solutions to assist health care workers during COVID-19. Telemed J E Health. 2021;27(6):594–602. 10.1089/tmj.2020.0287 [DOI] [PubMed] [Google Scholar]
  • 36.Pospos S , Young IT , Downs N , et al. Web-based tools and mobile applications to mitigate burnout, depression, and suicidality among healthcare students and professionals: a systematic review. Acad Psychiatry. 2018;42(1):109–120. 10.1007/s40596-017-0868-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37. US Food and Drug Administration. Digital Health Software Precertification (Pre-Cert) Program. Available at : https://www.fda.gov/medical-devices/digital-health-center-excellence/digital-health-software-precertification-pre-cert-program . Accessed April 7, 2022. .
  • 38.Moshi MR , Tooher R , Merlin T. Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review. Int J Technol Assess Health Care. 2018;34(5):464–475. 10.1017/S026646231800051X [DOI] [PubMed] [Google Scholar]
  • 39.Henson P , David G , Albright K , Torous J. Deriving a practical framework for the evaluation of health apps. Lancet Digit Health. 2019;1(2):e52–e54. 10.1016/S2589-7500(19)30013-5 [DOI] [PubMed] [Google Scholar]
  • 40.Lagan S , Sandler L , Torous J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open. 2021;11(3):e047001. 10.1136/bmjopen-2020-047001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Lagan S , Aquino P , Emerson MR , Fortuna K , Walker R , Torous J. Actionable health app evaluation: translating expert frameworks into objective metrics. NPJ Digit Med. 2020;3:100. 10.1038/s41746-020-00312-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. American Psychiatric Association. The app evaluation model . Available at: https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/the-app-evaluation-model . Accessed April 6, 2022. .
  • 43.Hendrikoff L , Kambeitz-Ilankovic L , Pryss R , et al. Prospective acceptance of distinct mobile mental health features in psychiatric patients and mental health professionals. J Psychiatr Res. 2019;109:126–132. 10.1016/j.jpsychires.2018.11.025 [DOI] [PubMed] [Google Scholar]
  • 44.Schueller SM , Neary M , O’Loughlin K , Adkins EC. Discovery of and interest in health apps among those with mental health needs: survey and focus group study. J Med Internet Res. 2018;20(6):e10141. 10.2196/10141 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Baumel A , Muench F , Edan S , Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. 2019;21(9):e14567. 10.2196/14567 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Garland AF , Jenveja AK , Patterson JE. Psyberguide: a useful resource for mental health apps in primary care and beyond. Fam Syst Health. 2021;39(1):155–157. 10.1037/fsh0000587 [DOI] [PubMed] [Google Scholar]
  • 47.US Surgeon General. Addressing health worker burnout: the U.S. Surgeon General’s Advisory on building a thriving health workforce. 2022. Available at: https://www.hhs.gov/sites/default/files/health-worker-wellbeing-advisory.pdf. Accessed May 25, 2022. [PubMed]
  • 48.Park SY , Nicksic Sigmon C , Boeldt D. A framework for the implementation of digital mental health interventions: the importance of feasibility and acceptability research. Cureus. 2022;14(9):e29329. 10.7759/cureus.29329 [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from American Journal of Public Health are provided here courtesy of American Public Health Association

RESOURCES