Abstract
The behavioral health toll of the COVID-19 pandemic and systemic racism has directed increased attention to the potential of digital health as a way of improving access to and quality of behavioral health care. However, as the pandemic continues to widen health disparities in racially and ethnically minoritized groups, concerns arise around an increased reliance on digital health technologies exacerbating the digital divide and reinforcing rather than mitigating systemic health inequities in communities of color. As funding for digital mental health continues to surge, we offer five key recommendations on how the field can “REACT” to ensure the development of approaches that increase health equity by increasing real-world evidence, educating consumers and providers, utilizing adaptive interventions to optimize care, creating for diverse populations, and building trust. Recommendations highlight the need to take a strengths-based view when designing for racially and ethnically diverse populations and embracing the potential of digital approaches to address complex challenges.
Keywords: digital health, app, public mental health, health disparities, COVID-19, pandemic, mental health, disparity, behavior
Introduction
The dual public health crises of systemic racism and COVID-19 have significantly increased mental health needs while simultaneously limiting access to traditional models of in-person care [1,2]. The pandemic has critically strained an already overburdened system, magnifying existing shortcomings of the mental health system, including inequitable availability, access, and quality of care for the Black, Indigenous, and people of color (BIPOC) community. BIPOC continue to be disproportionately affected (emotionally and physically) by the virus, experiencing more hospitalizations and greater mortality rates as well as increased emotional distress [3,4]. Concurrently, these losses are occurring against a backdrop of inequities due to systemic racism including long-standing disenfranchisement from the health care system, historic traumas, stigma, high cost, and cultural insensitivity, which may put BIPOC communities at highest risk for not receiving mental health care [5,6]. These barriers have persisted in the wake of prior disasters, preventing BIPOC communities from accessing and utilizing mental health treatment after national tragedies [5]. Given the complex and intersecting clinical and social-contextual vulnerabilities that contribute to behavioral health inequities, an urgent need exists for innovative behavioral health approaches that support effective and accessible care to overcome, rather than perpetuate, existing health disparities.
Digital mental health (DMH) tools like mobile apps could play a key role in expanding access to care for those most impacted by COVID-19 and systemic racism by providing remote assessment, support, or intervention; lowering the cost of mental health care; reducing transportation challenges; and providing care in a private and destigmatizing manner [7,8]. However, challenges remain including a lack of culturally grounded DMH interventions, minimal implementation data, low provider and consumer confidence in DMH quality, variability in provider competency in DMH, and lower digital access and literacy in groups at the highest risk for experiencing health care inequities. Digital inequalities span a multidimensional continuum that includes socioeconomic status and location [9], age [10], level of education [11], quality of social support network [12], immigration status [13], location, and health literacy [14]. These inequalities align with social determinants of health [15,16]; therefore, as reliance on digital health approaches increases, digital inequalities may further exacerbate existing health disparities and reduce health care access for those most likely to be affected by the ongoing crises.
In this viewpoint, we first provide a “pulse check” on the DMH field amidst the ongoing pandemic and provide five key corresponding recommendations to ensure optimal leveraging of DMH apps for increasing health equity and ensuring that innovation does not inadvertently widen the digital divide.
The Current DMH Landscape
Given widespread availability and scalability, DMH apps have been lauded as a potential first line of defense for responding to the increase in behavioral health concerns precipitated by COVID-19 and systemic racism. Recognizing the need to increase access to effective and safe technologies that promote physical distancing, the US Food and Drug Administration (FDA) issued an Enforcement Policy on April 14, 2020, allowing the “…distribution and use of computerized behavioral therapy and other digital health therapeutic devices for psychiatric disorders…” without the requirement that those products comply with traditional regulatory policies. Additionally, the guidance reiterates and clarifies that low-risk wellness and digital health products—including apps that promote mindfulness, meditation, sleep, exercise, or manage mental health symptoms without providing a specific treatment—will continue to be exempt from FDA oversight. This important policy is consistent with the FDA’s previously published emphasis on increasing access to inherently safe digital products for mental health and other medical conditions [17]. FDA guidance regarding mental health apps has the potential to spur innovation, but it can also increase ambiguity in a marketplace already marred with concerns of real-world engagement [18] and a limited evidence base [19,20].
The COVID-19 pandemic has stimulated continued interest and investment in digital health approaches. In the first half of 2020, digital behavioral health startup companies reported record funding of over $588 million, roughly the annual funding for this market in any previous full year, reflecting an industry-wide perception of growing demand for DMH products [21,22]. Continued growth, however, portends continued saturation of the marketplace with apps that make unwarranted claims, offer minimal protection of user data, or are ineffective [23-25]. Health care providers, patients, and their families are at once faced with the promise of leveraging easy-to-access digital tools such as apps, and the daunting challenge of choosing from the 10,000+ DMH apps available [26,27]. The lack of reliable, easily accessible efficacy and safety data around these products continues to impede clinician and consumer ability to make effective health care choices. Reliance on the “wisdom of the crowd” to guide decision making is not advised since, among other reasons, user ratings do not correlate well with clinical utility or quality [27].
Recommendations
We offer five key recommendations on how the mental health care and technology fields can “REACT” to ensure the development of approaches that increase health equity by increasing real-world evidence, educating consumers and providers, utilizing adaptive interventions to optimize care, creating for diverse populations, and building trust (Figure 1). As highlighted in Figure 1, these recommendations will only be effective through increased representation of and collaboration with BIPOC researchers, policymakers, educators, providers, developers, and payers, requiring the removal of structural barriers in academia and industry.
1. Real-World Evidence Is Key
DMH apps developed for both underrepresented and majority populations have been hindered by a lack of real-world effectiveness data and a paucity of implementation trials [28]. The lack of timely effectiveness data prohibits consumers and medical providers from making informed choices, reducing trust and uptake. Implementation science offers frameworks and methodologies applicable to both increasing sustained use of new technologies and more rapidly evaluating digital products [29]. Drawing on implementation science and human-computer interaction, the Accelerated Creation to Sustainment model [30] offers an alternative approach to traditional clinical evaluation methods that can take over 10 years to move an intervention from conceptualization to implementation [31]. This approach proposes an iterative process of design and evaluation across three phases (create, trial, sustain) and may be facilitated by real-time engagement and outcomes data gathered by the device itself. This approach highlights the need to utilize user-centered design [32,33] to support both the development of a product and an associated implementation and sustainment strategy. In the trial phase, products and implementation strategies are concurrently tested in “real-world” settings utilizing Optimization, Effectiveness, and Implementation studies that explore effectiveness and implementation outcomes. Outcomes such as fidelity, adoption, uptake, and cost are particularly important to track to ensure new technologies are acceptable to a diverse population of consumers. Finally, in the sustainment phase, ongoing passive or low-effort data collection informs continued optimization as the research team transfers responsibilities to clinical or organizational staff.
2. Educate Providers and Consumers
As the field moves toward generating a more robust and easily interpretable evidence base for specific apps, providers and consumers must be empowered with alternative strategies to make informed decisions on which apps to utilize in clinical practice. The American Psychiatric Association’s app evaluation framework offers a guiding framework for providers and consumers to self-evaluate apps [34]. In principle, this framework offers a much-needed solution to a complex problem, but it also places a potentially high burden on providers to evaluate digital therapies, which the framework itself notes is “not what psychiatrists and mental health clinicians are classically trained to do [provide]” [35]. Given these challenges, it is unclear how provider evaluation of mental health apps will practically fit into the workflow for clinicians.
Moreover, given that a substantial proportion of mental health services is rendered through primary care [36], additional steps need to be taken to help provide primary care providers with the necessary information to make informed decisions about app selection for their patients. It is therefore critical to train both primary care and mental health professionals in digital technologies to support patient care. Frameworks for digital competencies in mental health have primarily focused on telehealth and do not fully address integration of broader products including the prescription of digital therapeutics or integration of apps to support treatment [37,38]. There must be a synergistic balance between providers being empowered to effectively choose and evaluate digital products on an individual basis for their patients, and more transparent and accessible information on individual products. The balance may be facilitated by the inclusion of a digital specialist or “digital navigator” [39]. Finally, broader provider and stakeholder education is critical as lack of understanding or knowledge of digital approaches is a key barrier to DMH uptake and adoption [40].
3. Adaptive Interventions to Optimize Care
Digital approaches offer the opportunity to develop adaptable interventions that provide tailored content based on individual characteristics, such as race, gender, sexual identities, family structure, and language preferences [41]. The pandemic has further highlighted the potential for external stressors to exacerbate previously managed chronic mental health concerns and the need to flexibly ramp up care. We can leverage novel clinical trial methodologies (eg, the Multiphase Optimization Strategy [42] and sequential, multiple assignment, randomized trials (SMART) [43]) to develop adaptive DMH interventions that adjust the type or dosage of the intervention based on patient characteristics. SMART trials facilitate this development by randomizing patients to different treatment options across time based on predetermined decision points, allowing for an assessment of effectiveness for different interventions at each stage. These novel designs allow for the evaluation of the tailoring variables and intervention components in the same trial. Moreover, they support the development of decision rules for assigning treatments based on patient characteristics and response to interventions, rather than a priori decisions. This can result in shorter and more targeted interventions.
4. Create DMH Apps for a Diverse Population of Users
Considerable work is required to ensure the availability of DMH products that fit a wide range of user needs and preferences. Efforts have been hampered by the underrepresentation of BIPOC researchers and developers, calling for fundamental shifts to address structural barriers in academics and industry. In order for digital technologies to recognize their full potential to mitigate widening behavioral health disparities exacerbated by the COVID-19 pandemic and ongoing cultural-based traumas, they must be designed with those who are both at most risk for mental health concerns and who face the greatest barriers to health care engagement. At risk of oversimplification, this fundamentally calls for us to follow the core principle of designing for the end user [32]. This includes both the type and way we utilize technology to deliver care and the intervention content we deliver. Human- or user-centered design approaches that ask the question, “What are the barriers to receiving effective behavioral health care and how can technology help overcome them?” must be used to develop strategies that mitigate disparities. However, such approaches must engage consumers whose voices are traditionally underrepresented in care settings. Culturally salient recruitment strategies [44], which include addressing community mistrust, participant resource constraints, and potential risks (eg, community stigma), could help researchers and developers more effectively integrate the input of those consumers most likely to benefit from their products [45].
BIPOC underrepresentation in the product development process has potentially reinforced structural inequalities in our health care system by limiting the availability of products that are culturally inclusive and effective [8]. The lack of culturally appropriate tailoring, stakeholder input, and broader community-partnered implementation plans limits the effectiveness of DMH tools, leads to decreased uptake, and may contribute to lower rates of DMH utilization in BIPOC populations despite equal or higher preferences for using DMH apps compared to White peers [46,47]. To increase potential effectiveness and uptake, cultural tailoring must go beyond language translation to incorporate cultural values, norms, and references [48]. This tailoring offers the opportunity to capitalize on community or cultural resiliency factors offering opportunities to increase efficacy of interventions in addition to improved engagement. Finally, digital technologies offer the opportunity to develop intervention approaches that are not as inherently embedded in cultural paradigms or at minimum are language agnostic (eg, attentional bias modification) [49].
5. Trust Building Through Monitoring and Independent Vetting
As we work toward the development and rapid real-world evaluation of DMH approaches for diverse populations, there is a strong need for a sustainable ecosystem to increase consumer and provider confidence, and empower users and providers to effectively choose appropriate and effective DMH interventions based on their specific needs and context. At an aspirational level, there are several approaches that could help increase this confidence.
Increase Industry and Marketplace Self-Monitoring
Across all DMH products, there must be increased industry pressure to self-monitor, conduct rigorous research, and disseminate results in a manner that is interpretable by both consumers and providers. A potential short-term step toward this end is clear reporting of digital products released under the COVID-19–related FDA guidance to ensure company accountability to consumers and facilitate uptake of new low-risk products. On the open app marketplace, the inclusion of clear, easily interpretable statements regarding potential efficacy and risks, including data security, must be encouraged. A review of 73 popular apps found that none of these evaluations or membership in app libraries were noted in their app store descriptions [27], suggesting a potential lack of awareness of these schemes outside academic communities or that accreditation could benefit companies. App stores could therefore provide a more standardized way to include reporting of clinical testing results [50]. It is recognized that these recommendations introduce burden to companies and, without enforcement-related contingencies, are unlikely to be fully effective. Nevertheless, continued dialogue among stakeholders, including patient and advocacy groups, providers, and DMH companies, will help advance the field.
Independent Vetting and Structured Dissemination of DMH Products
The lack of reliable, easily accessible efficacy and safety data impedes clinician and consumer ability to make effective health care choices. Digital formularies that provide a curated list of clinically supported DMH products could be an important mechanism for enabling dissemination of products including apps into clinical care [51]. Formularies could integrate into electronic health records and associated patient portals, allowing both patients and providers to know which products are supported by their health care organization. Indeed, some steps have been taken in this regard, and lessons learned from these efforts will be important to refine the approach [52,53]. Similarly, the development of independently curated app libraries [54,55] takes a clear step toward increasing transparency around the safety and potential efficacy of apps outside of a formalized health system. However, attempts are hindered by the vast quantity and rapidly changing nature of products on the market. Conscious of the challenges of keeping up with this massively expanding and growing field, the FDA’s Digital Health Precertification Pilot Program [56] shifts regulations toward the app developers and manufactures themselves, allowing for products to be developed efficiently through rapid iteration more appropriate for digital products, therefore reducing the need for burdensome hurdles to be cleared with frequent software changes, etc. For this to be successful, there must be an understanding between regulators and developers of a consensus framework for product evaluation. In addition, there must be adequate incentives for engagement in any regulatory programs, as prior voluntary certification attempts have failed [57].
Conclusion
COVID-19 and increased awareness of systemic racism have highlighted the promises and pitfalls of DMH. As interest in DMH grows, we have the opportunity and responsibility to increase the availability of safe, effective, and accessible products that reduce rather than perpetuate health disparities. This will require synergistic collaboration among researchers, policymakers, educators, providers, developers, payers, and patients. As researchers and clinicians, we are well positioned to lead this charge; however, we must embrace the rapid nature of technological innovation and development. Concerns around the increasing digital divide must serve as a “wake up call” for the DMH field to ensure products and mHealth (mobile health) approaches are tackling health care disparities rather than contributing to them. Rather than “designing down” to reduce the complexity of interventions for populations with more limited digital literacy, we must take a strengths-based view of designing for diverse populations [8]. By embracing these strengths, digital health has the opportunity to increase access to care for the most vulnerable populations. Finally, fundamental shifts to address structural inequalities in academics and industry, including the underrepresentation of BIPOC researchers and developers, are needed to revolutionize our field.
Acknowledgments
This work was supported in part by the Multi-Health Systems-Conners Fellowship in Digital Health.
Abbreviations
- BIPOC
Black, Indigenous, and people of color
- DMH
digital mental health
- FDA
Food and Drug Administration
- mHealth
mobile health
- SMART
sequential, multiple assignment, randomized trials
Footnotes
Conflicts of Interest: SHK has received research support and/or consulting fees from a number of digital health companies, including Akili Interactive, BehaVR, Limbix, Luoms Labs, OnDosis, Sana Health, and Tali Health. SHK has stock options in Akili Interactive.
References
- 1.Palsson O, Ballou S. The US National Pandemic Emotional Impact Report. 2020. https://pandemicimpactreport.com/index.html. [DOI] [PMC free article] [PubMed]
- 2.Torales J, O'Higgins Marcelo, Castaldelli-Maia J, Ventriglio A. The outbreak of COVID-19 coronavirus and its impact on global mental health. Int J Soc Psychiatry. 2020 Jun;66(4):317–320. doi: 10.1177/0020764020915212. [DOI] [PubMed] [Google Scholar]
- 3.Hamel L, Kearney A, Kirzinger A, Lopes L, Muñana C, Brodie M. KFF Health Tracking Poll - July 2020. KFF. 2020. Jul 27, [2020-09-02]. https://www.kff.org/coronavirus-covid-19/report/kff-health-tracking-poll-july-2020/
- 4.Coronavirus disease 2019 (COVID-19): Cases, data, and surveillance. Center for Disease Control and Prevention. 2020. [2020-10-3]. https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/index.html.
- 5.Goldmann E, Galea S. Mental health consequences of disasters. Annu Rev Public Health. 2014 Mar 18;35(1):169–83. doi: 10.1146/annurev-publhealth-032013-182435. [DOI] [PubMed] [Google Scholar]
- 6.Novacek D, Hampton-Anderson J, Ebor M, Loeb T, Wyatt G. Mental health ramifications of the COVID-19 pandemic for Black Americans: Clinical and research recommendations. Psychol Trauma. 2020 Jul;12(5):449–451. doi: 10.1037/tra0000796. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Cook BL, Trinh N, Li Z, Hou SS, Progovac AM. Trends in Racial-Ethnic Disparities in Access to Mental Health Care, 2004-2012. Psychiatr Serv. 2017 Jan 01;68(1):9–16. doi: 10.1176/appi.ps.201500453. http://europepmc.org/abstract/MED/27476805. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Schueller SM, Hunter JF, Figueroa C, Aguilera A. Use of Digital Mental Health for Marginalized and Underserved Populations. Curr Treat Options Psych. 2019 Jul 5;6(3):243–255. doi: 10.1007/s40501-019-00181-z. [DOI] [Google Scholar]
- 9.Yates S, Kirby J, Lockley E. Sociological Research Online. SAGE Publications: London, England; 2015. Nov 30, Digital Media Use: Differences and Inequalities in Relation to Class and Age; pp. 71–91. [Google Scholar]
- 10.Hall A, Bernhardt J, Dodd V, Vollrath M. The digital health divide: evaluating online health information access and use among older adults. Health Educ Behav. 2015 Apr;42(2):202–9. doi: 10.1177/1090198114547815. http://europepmc.org/abstract/MED/25156311. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Cruz-Jesus F, Vicente M, Bacao F, Oliveira T. The education-related digital divide: An analysis for the EU-28. Computers in Human Behavior. 2016 Mar;56:72–82. doi: 10.1016/j.chb.2015.11.027. [DOI] [Google Scholar]
- 12.Helsper EJ, van Deursen AJAM. Do the rich get digitally richer? Quantity and quality of support for digital engagement. Information, Communication & Society. 2016 Jun 29;20(5):700–714. doi: 10.1080/1369118x.2016.1203454. [DOI] [Google Scholar]
- 13.Haight M, Quan-Haase A, Corbett BA. Revisiting the digital divide in Canada: the impact of demographic factors on access to the internet, level of online activity, and social networking site usage. Information, Communication & Society. 2014 Mar 07;17(4):503–519. doi: 10.1080/1369118x.2014.891633. [DOI] [Google Scholar]
- 14.Bailey SC, O'Conor R, Bojarski EA, Mullen R, Patzer RE, Vicencio D, Jacobson KL, Parker RM, Wolf MS. Literacy disparities in patient access and health-related use of Internet and mobile technologies. Health Expect. 2015;18(6):3079–87. doi: 10.1111/hex.12294. http://europepmc.org/abstract/MED/25363660. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Marmot M. Social determinants of health inequalities. The Lancet. 2005 Mar 25;365(9464):1099–1104. doi: 10.1016/s0140-6736(05)74234-3. [DOI] [PubMed] [Google Scholar]
- 16.Allen J, Balfour R, Bell R, Marmot M. Social determinants of mental health. Int Rev Psychiatry. 2014 Aug 19;26(4):392–407. doi: 10.3109/09540261.2014.928270. [DOI] [PubMed] [Google Scholar]
- 17.Enforcement Policy for Digital Health Devices For Treating Psychiatric Disorders During the Coronavirus Disease 2019 (COVID-19) Public Health Emergency: Guidance for Industry and Food and Drug Administration Staff. FDA. 2020. Apr, [2020-10-3]. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/enforcement-policy-digital- health-devices-treating-psychiatric-disorders-during-coronavirus-disease.
- 18.Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, Barkham M, Bower P, Cooper C, Gask L, Kessler D, Lester H, Lovell K, Parry G, Richards D, Andersen P, Brabyn S, Knowles S, Shepherd C, Tallon D, White D, REEACT Team Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. BMJ. 2015 Nov 11;351:h5627. doi: 10.1136/bmj.h5627. http://www.bmj.com/lookup/pmidlookup?view=long&pmid=26559241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Torous J, Lipschitz J, Ng M, Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: A systematic review and meta-analysis. J Affect Disord. 2020 Feb 15;263:413–419. doi: 10.1016/j.jad.2019.11.167. [DOI] [PubMed] [Google Scholar]
- 20.Lui JHL, Marcus DK, Barry CT. Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Professional Psychology: Research and Practice. 2017 Jun;48(3):199–210. doi: 10.1037/pro0000122. [DOI] [Google Scholar]
- 21.Landi H. Digital behavioral health startups scored $588M in funding amid COVID-19 pandemic. FierceHealthcare. 2020. Jul 6, [2020-09-23]. https://www.fiercehealthcare.com/tech/funding-for-digital-behavioral-health-startups-surged-amid- covid-19-pandemic.
- 22.Shah RN, Berry OO. The Rise of Venture Capital Investing in Mental Health. JAMA Psychiatry. 2020 Sep 16; doi: 10.1001/jamapsychiatry.2020.2847. [DOI] [PubMed] [Google Scholar]
- 23.Weisel KK, Fuhrmann LM, Berking M, Baumeister H, Cuijpers P, Ebert DD. Standalone smartphone apps for mental health-a systematic review and meta-analysis. NPJ Digit Med. 2019 Dec 2;2(1):118. doi: 10.1038/s41746-019-0188-8. doi: 10.1038/s41746-019-0188-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Baumel A, Muench F, Edan S, Kane JM. Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. J Med Internet Res. 2019 Sep 25;21(9):e14567. doi: 10.2196/14567. https://www.jmir.org/2019/9/e14567/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Huckvale K, Torous J, Larsen ME. Assessment of the Data Sharing and Privacy Practices of Smartphone Apps for Depression and Smoking Cessation. JAMA Netw Open. 2019 Apr 05;2(4):e192542. doi: 10.1001/jamanetworkopen.2019.2542. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/10.1001/jamanetworkopen.2019.2542. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Torous J, Firth J, Huckvale K, Larsen ME, Cosco TD, Carney R, Chan S, Pratap A, Yellowlees P, Wykes T, Keshavan M, Christensen H. The Emerging Imperative for a Consensus Approach Toward the Rating and Clinical Recommendation of Mental Health Apps. J Nerv Ment Dis. 2018 Aug;206(8):662–666. doi: 10.1097/NMD.0000000000000864. [DOI] [PubMed] [Google Scholar]
- 27.Larsen ME, Huckvale K, Nicholas J, Torous J, Birrell L, Li E, Reda B. Using science to sell apps: Evaluation of mental health app store quality claims. NPJ Digit Med. 2019;2:18. doi: 10.1038/s41746-019-0093-1. doi: 10.1038/s41746-019-0093-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Anderson-Lewis C, Darville G, Mercado RE, Howell S, Di Maggio Samantha. mHealth Technology Use and Implications in Historically Underserved and Minority Populations in the United States: Systematic Literature Review. JMIR Mhealth Uhealth. 2018 Jun 18;6(6):e128. doi: 10.2196/mhealth.8383. https://mhealth.jmir.org/2018/6/e128/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Buis L. Implementation: The Next Giant Hurdle to Clinical Transformation With Digital Health. J Med Internet Res. 2019 Nov 20;21(11):e16259. doi: 10.2196/16259. https://www.jmir.org/2019/11/e16259/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Mohr DC, Lyon AR, Lattie EG, Reddy M, Schueller SM. Accelerating Digital Mental Health Research From Early Design and Creation to Successful Implementation and Sustainment. J Med Internet Res. 2017 May 10;19(5):e153. doi: 10.2196/jmir.7725. https://www.jmir.org/2017/5/e153/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Mohr DC, Riper H, Schueller SM. A Solution-Focused Research Approach to Achieve an Implementable Revolution in Digital Mental Health. JAMA Psychiatry. 2018 Feb 01;75(2):113–114. doi: 10.1001/jamapsychiatry.2017.3838. [DOI] [PubMed] [Google Scholar]
- 32.Harte R, Glynn L, Rodríguez-Molinero Alejandro, Baker PM, Scharf T, Quinlan LR, ÓLaighin Gearóid. A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology. JMIR Hum Factors. 2017 Mar 16;4(1):e8. doi: 10.2196/humanfactors.5443. https://humanfactors.jmir.org/2017/1/e8/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, Cafazzo J. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol. 2012 Sep;Suppl:49–56. doi: 10.2345/0899-8205-46.s2.49. https://www.researchgate.net/journal/0899-8205_Biomedical_Instrumentation_Technology. [DOI] [PubMed] [Google Scholar]
- 34.Henson P, David G, Albright K, Torous J. Deriving a practical framework for the evaluation of health apps. The Lancet Digital Health. 2019 Jun;1(2):e52–e54. doi: 10.1016/s2589-7500(19)30013-5. [DOI] [PubMed] [Google Scholar]
- 35.The App Evaluation Model. APA. [2020-08-17]. https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/the-app-evaluation-model.
- 36.Beck AJ, Page C, Buche J, Schoebel V, Wayment C. Behavioral Health Service Provision by Primary Care Physicians. Ann Arbor, MI: University of Michigan Behavioral Health Workforce Research Center; 2019. Oct, [2020-10-3]. http://www.behavioralhealthworkforce.org/wp-content/uploads/2019/12/Y4-P10-BH-Capacityof-PC-Phys_Full.pdf. [Google Scholar]
- 37.Hilty D, Chan S, Torous J, Luo J, Boland R. A Framework for Competencies for the Use of Mobile Technologies in Psychiatry and Medicine: Scoping Review. JMIR Mhealth Uhealth. 2020 Feb 21;8(2):e12229. doi: 10.2196/12229. https://mhealth.jmir.org/2020/2/e12229/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Hilty DM, Maheu MM, Drude KP, Hertlein KM. The Need to Implement and Evaluate Telehealth Competency Frameworks to Ensure Quality Care across Behavioral Health Professions. Acad Psychiatry. 2018 Dec 13;42(6):818–824. doi: 10.1007/s40596-018-0992-5. [DOI] [PubMed] [Google Scholar]
- 39.Wisniewski H, Torous J. Digital navigators to implement smartphone and digital tools in care. Acta Psychiatr Scand. 2020 Apr;141(4):350–355. doi: 10.1111/acps.13149. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Williams MG, Stott R, Bromwich N, Oblak SK, Espie CA, Rose JB. Determinants of and barriers to adoption of digital therapeutics for mental health at scale in the NHS. BMJ Innov. 2020 May 07;6(3):92–98. doi: 10.1136/bmjinnov-2019-000384. [DOI] [Google Scholar]
- 41.Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, Rivera DE, West R, Wyatt JC. Evaluating Digital Health Interventions: Key Questions and Approaches. Am J Prev Med. 2016 Nov;51(5):843–851. doi: 10.1016/j.amepre.2016.06.008. http://europepmc.org/abstract/MED/27745684. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med. 2007 May;32(5 Suppl):S112–8. doi: 10.1016/j.amepre.2007.01.022. http://europepmc.org/abstract/MED/17466815. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Lei H, Nahum-Shani I, Lynch K, Oslin D, Murphy SA. A "SMART" design for building individualized treatment sequences. Annu Rev Clin Psychol. 2012;8:21–48. doi: 10.1146/annurev-clinpsy-032511-143152. http://europepmc.org/abstract/MED/22224838. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.UyBico SJ, Pavel S, Gross CP. Recruiting vulnerable populations into research: a systematic review of recruitment interventions. J Gen Intern Med. 2007 Jun;22(6):852–63. doi: 10.1007/s11606-007-0126-3. http://europepmc.org/abstract/MED/17375358. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Ellard-Gray A, Jeffrey NK, Choubak M, Crann SE. Finding the Hidden Participant. International Journal of Qualitative Methods. 2015 Dec 17;14(5):160940691562142. doi: 10.1177/1609406915621420. [DOI] [Google Scholar]
- 46.Pratap A, Renn BN, Volponi J, Mooney SD, Gazzaley A, Arean PA, Anguera JA. Using Mobile Apps to Assess and Treat Depression in Hispanic and Latino Populations: Fully Remote Randomized Clinical Trial. J Med Internet Res. 2018 Aug 09;20(8):e10130. doi: 10.2196/10130. http://www.jmir.org/2018/8/e10130/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Kern A, Hong V, Song J, Lipson SK, Eisenberg D. Mental health apps in a college setting: openness, usage, and attitudes. Mhealth. 2018 Jun;4:20–20. doi: 10.21037/mhealth.2018.06.01. doi: 10.21037/mhealth.2018.06.01. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Bernal G, Jiménez-Chafey MI, Domenech Rodríguez MM. Cultural adaptation of treatments: A resource for considering culture in evidence-based practice. Professional Psychology: Research and Practice. 2009;40(4):361–368. doi: 10.1037/a0016401. [DOI] [Google Scholar]
- 49.Zhang M, Ying J, Song G, Fung DS, Smith H. Attention and Cognitive Bias Modification Apps: Review of the Literature and of Commercially Available Apps. JMIR Mhealth Uhealth. 2018 May 24;6(5):e10034. doi: 10.2196/10034. https://mhealth.jmir.org/2018/5/e10034/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Larsen ME, Nicholas J, Christensen H. Quantifying App Store Dynamics: Longitudinal Tracking of Mental Health Apps. JMIR Mhealth Uhealth. 2016 Aug 09;4(3):e96. doi: 10.2196/mhealth.6020. https://mhealth.jmir.org/2016/3/e96/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med. 2020;3:14. doi: 10.1038/s41746-019-0212-z. doi: 10.1038/s41746-019-0212-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Express Scripts Makes Digital Health Click. Express Scripts. [2020-09-23]. https://www.express-scripts.com/corporate/articles/digital-health-formulary-announced.
- 53.Muoio D. CVS Health kicks off digital health-friendly service for PBM clients with Big Health’s Sleepio. MobiHealthNews. 2019. Jun 12, [2020-09-23]. https://www.mobihealthnews.com/content/north-america/cvs-health-kicks-digital-health- friendly-service-pbm-clients-big-health-s.
- 54.Apps and digital health resources reviewed by experts. One Mind PsyberGuide. [2020-09-23]. https://onemindpsyberguide.org/
- 55.BIDMC - Division of Digital Psychiatry. Beth Israel Deaconess Medical Center. [2020-09-23]. https://apps.digitalpsych.org/
- 56.Digital Health Software Precertification (Pre-Cert) Program. FDA. 2020. Sep 14, [2020-09-23]. https://www.fda.gov/medical-devices/digital-health-center-excellence/digital-health-software-precertification-pre-cert-program.
- 57.Palmer S. Swipe Right for Health Care: How the State May Decide the Future of the mHealth App Industry in the Wake of FDA Uncertainty. J Leg Med. 2017;37(1-2):249–263. doi: 10.1080/01947648.2017.1303289. [DOI] [PubMed] [Google Scholar]