Abstract
Digital mental health technologies such as mobile health (mHealth) tools can offer innovative ways to help develop and facilitate mental health care provision, with the COVID-19 pandemic acting as a pivot point for digital health implementation. This viewpoint offers an overview of the opportunities and challenges mHealth innovators must navigate to create an integrated digital ecosystem for mental health care moving forward. Opportunities exist for innovators to develop tools that can collect a vast range of active and passive patient and transdiagnostic symptom data. Moving away from a symptom-count approach to a transdiagnostic view of psychopathology has the potential to facilitate early and accurate diagnosis, and can further enable personalized treatment strategies. However, the uptake of these technologies critically depends on the perceived relevance and engagement of end users. To this end, behavior theories and codesigning approaches offer opportunities to identify behavioral drivers and address barriers to uptake, while ensuring that products meet users’ needs and preferences. The agenda for innovators should also include building strong evidence-based cases for digital mental health, moving away from a one-size-fits-all well-being approach to embrace the development of comprehensive digital diagnostics and validated digital tools. In particular, innovators have the opportunity to make their clinical evaluations more insightful by assessing effectiveness and feasibility in the intended context of use. Finally, innovators should adhere to standardized evaluation frameworks introduced by regulators and health care providers, as this can facilitate transparency and guide health care professionals toward clinically safe and effective technologies. By laying these foundations, digital services can become integrated into clinical practice, thus facilitating deeper technology-enabled changes.
Keywords: digital implementation, digital mental health, digital psychiatry, digital technology, viewpoint
Introduction
Mental health disorders represent the leading cause of disability worldwide, with over one third of the world’s population being affected by a mental health condition in their lifetime [1]. Given the increasing pressures on mental health care budgets globally, as well as shortages of health care professionals who are facing an overwhelming growing burden of chronic and recurring mental health conditions, prevention strategies and improvements in early identification and treatment are essential. To this end, mobile health (mHealth) tools such as apps can offer innovative ways to help develop and facilitate mental health care provision. Indeed, the unmet need for psychiatric services has sparked significant interest in developing apps to meet patient demand. As a result, more than 10,000 apps addressing mental health symptoms such as anxiety, low mood, and insomnia are now available on the Apple App Store and Google Play Store [2]. Furthermore, despite its catastrophic effects, the COVID-19 pandemic has been a pivot point for digital health implementation, with some describing it as years’ worth of digital transformation in just a few months [3-5]. This transformation has been made possible by the recognition from governments and regulatory bodies of the need for speed, flexibility, and action in response to the pandemic [6-8]. However, many current mHealth technologies rely heavily on patients’ ability to self-diagnose and use self-help rather than being fully integrated into the clinic. In building a holistic and comprehensive digital mental health ecosystem that augments rather than replaces traditional mental health care approaches, strategies for the design, validation, and implementation of mHealth technologies should be carefully considered by innovators moving forward.
mHealth: Promise of Dimensional Psychiatry and Digital Phenotyping
Psychiatric nosology, as described in gold-standard diagnostic manuals, has often been criticized for being overly narrow, failing to capture the wide range of clinical symptoms that are endorsed by individuals with mental health conditions [9,10]. In this regard, moving away from a categorical, symptom-count approach to a transdiagnostic view of psychopathology may facilitate early and accurate diagnosis, and allow for personalized treatment strategies. Given that time is a premium in clinical settings, where relying on brief symptom-count checklists such as the Patient Health Questionnaire-9 [11] and the seven-item Generalized Anxiety Disorder questionnaire [12] is common practice, digital technologies may offer an innovative way to improve diagnostic accuracy and advance mental health care provisions. Indeed, digital technologies such as mHealth apps have the capacity for the collection of a vast range of key transdiagnostic data via active reporting (ie, self-administered symptom reporting or monitoring via an app), providing valuable information on relevant social and demographic factors as well as current and past symptom profiles [13]. Digital technologies, including adaptive/nonlinear questionnaires where patients are required to answer questions based on previous answers, can further personalize and streamline the collection of active cross-disorder symptom data.
Moving beyond active symptom data, passive data, which measure aspects of daily living and can be collected continuously in the background, such as activity rhythms, text and call logs, sleep quality, keyboard reaction time [14], speech phonation, coherence, sentiment, and language patterns [15], can be applied to better inform mental health assessments in a process called digital phenotyping [16]. Data platforms that gather information from various digital sources have started emerging with the goal of collecting large-scale epidemiological and longitudinal data to identify risk factors and invest in mental health resources effectively. An example of this approach is offered by the recently launched Mental Health America (MHA) dashboard supported by MHA and Lundbeck. The dashboard collects publicly available data and anonymized data from web screening tools offered by MHA to highlight mental health hotspots across the United States. The objective is to implement advanced targeted interventions, address disparities in a timely manner, and design policies for at-risk populations [17,18]. Furthermore, advances in mHealth platforms (eg, Apple HealthKit or Google Fit) also allow for the bundling of data from different sensors such as accelerometers, microphones, GPS sensors, and gyroscopes, enabling the collection of physical and mental health information [19].
Critically, digital phenotyping for mental health is still in its infancy, with further research into how these data fit into or build upon the current diagnostic guidelines and criteria being a necessity moving forward. Both active as well as passive data collection also raise potential serious ethical concerns, which will need to be legislated and regulated. In the future, combining data from active and passive monitoring sources may feed new data into mental state examinations, allowing for more comprehensive data collection outside the time-pressured environment of clinical consultations. In turn, mHealth technologies have the potential to streamline mental health care delivery and management by reducing the number of in-person appointments, thereby freeing up clinician time, resulting in cost savings for both the provider and patient. However, these technologies usually require patients and clinicians to engage in new behaviors, which may be accompanied by reluctance to change. As a result, uptake of mHealth tools critically depends on perceived relevance, effectiveness, confidentiality, and engagement of the end users. A way to address this is by utilizing behavior theories and engaging in codesigning approaches when designing digital mental health care solutions.
Designing mHealth Tools With Patients and Clinicians in Mind
Understanding the target patient population’s need is pivotal to the design of effective mHealth tools. To this end, behavior theories such as the Health Belief Model (HBM) [20], Behavioral Intervention Technology (BIT) model [21], and behavior model for Just-in-Time Adaptive Interventions (JITAI) [22] offer frameworks that help identify key drivers and barriers to the uptake of new technology, starting from app subscription to adherence to recommendations delivered by mHealth tools [23-25]. For instance, the HBM can help mHealth developers better understand their target patient population by investigating their inclination to seeking mental health support and what drives their adherence to treatment (eg, trust in clinicians, severity of illness, social stigma). The HBM is based on the theory that a person’s willingness to engage in treatment is primarily due to their health perceptions. Indeed, mental health studies have shown that the perceived severity of mental illnesses impacts treatment outcomes (eg, adherence and beliefs about treatments) [26]. Thus, using the HBM can inform developers on how to best engage patients in adhering to the use of a novel digital tool.
Similarly, the BIT model combines the understanding of technological features of mHealth tools with behavioral principles to guide innovators on how to design and implement technology that is useful and usable. For instance, motivational enhancement strategies (behavior principle) can be used in the form of push notifications (technological feature) to prompt patients to track their mood regularly. The BIT model approach can be especially valuable for researchers looking to develop multifunctional tools for mental health, including tools for mood monitoring and treatment such as electronic cognitive behavioral therapy, as well as tracking adherence to medication. In these cases, the stepwise, structured approach of the BIT model can best inform developers on how to use different behavioral principles and technologies to achieve their clinical aims.
The JITAI model can be employed by researchers developing adaptive sampling and ecological momentary assessments aimed at collecting information at sufficient frequencies and quantities to be useful without overburdening mental health patients. Critically, a delicate balance between utility and tolerability must be sought when developing mHealth monitoring tools in mental health, especially in conditions with specific triggers (eg, social anxiety triggered by meeting strangers, harmful alcohol use triggered by stress). The behavioral model for JITAI offers a framework to systematically identify when patients are most vulnerable to experiencing symptoms and pinpoints circumstances in which patients may be most receptive to the digital intervention.
Transitioning into the digital mental health care space will also involve cultivating a collaborative approach that includes academics, software developers, computer and data scientists, investors, and health care professionals. Indeed, moving forward, the concept of codesigning will need to be at the forefront of development, involving all potential users and stakeholders as active collaborators in the creation of digital diagnostics and interventions. This is especially critical for mental health technologies, which require clinical support from health care professionals to encourage engagement. Moving from the view of designing a digital product to designing a technology-enabled service means that the aims, role of the practitioner, and technology are designed and evaluated simultaneously in the relevant context [27]. Shifting from the practice of designing for to designing with users can help humanize digital health care, and augment rather than replace traditional health care delivery and management systems. By laying these foundations, technology-enabled services can become a part of everyday clinical practice and set the scene for deeper technology-enabled changes. Critically, as well as leveraging on behavior theories and codesigning approaches to increase engagement and relevance, mHealth developers should also pursue implementation studies to validate and assess the efficacy of their interventions.
Clinical Trials to Build a Strong Evidence-Based Case for Psychiatric mHealth Tools
Recent studies show that the most commonly downloaded mental health apps mainly fall under the umbrella of well-being apps offering relaxation, meditation, or mindfulness skills rather than validated treatments, and that the scientific language used to support claims often lacks corresponding evidence in the literature [28,29]. Although these apps can be incredibly useful and helpful for some patients, they may not be effective for more severe mental health concerns. Furthermore, many of these mHealth tools rely heavily on patients’ ability to self-diagnose and use self-help rather than being fully integrated into clinical service delivery. In building a holistic and comprehensive digital mental health ecosystem, innovators are encouraged to design technology in a more sophisticated manner, moving away from a one-size-fits-all well-being approach to embrace the development of clinically validated and integrated treatments.
Clinical trials and observational studies can help mHealth developers build a strong clinical case for their technologies and differentiate them from digital wellness apps. Moving forward, it should be noted that randomized controlled clinical trials typically bear little resemblance to clinical settings. Often, recruitment favors clinicians and patients who are interested in using digital health technologies and are therefore more likely to adhere to them. However, these participants may represent only a small proportion of potential users. It should then come as no surprise if digital technologies fail in general health care settings. Thus, innovators should carefully consider the opportunity to conduct the evaluation of digital mental health technologies within the intended context, and to assess both its effectiveness and implementation requirements [27]. This so-called hybrid pragmatic trial design can assess both treatment effects and implementation strategies [30] in real-world patient populations with outcome measurements routinely used in clinical practice [31]. Thus, implementation and sustainment strategies become integral to the clinical evaluation process and part of the intervention design from the very beginning. Importantly, performing clinical studies and citing published literature in the descriptions of digital tools can only partially address the difficulties encountered by providers, payors, and patients in assessing and choosing effective digital interventions. Standardized evaluation frameworks introduced by regulators and health care providers can create transparency and guide health care professionals toward clinically meaningful technologies, with efforts being made in this direction.
Defining Appraisal Frameworks for mHealth Solutions
Several academic groups have contributed to the design of evaluation frameworks. For instance, the American Psychiatric Association (APA) has developed an app evaluation framework [32] that has been used by the New York Department of Health in the construction of an app library. Following the APA model, Lagan et al [33] designed a health app evaluation model, which harmonized 45 preexisting frameworks and where answers can be either binary or numeric to permit an objective evaluation of (1) background and access, (2) data safety and privacy, (3) app effectiveness and clinical foundation, (4) user engagement, and (5) data integration. The app evaluation model can be used by clinicians and patients to inform decisions on the suitability of candidate apps for the intended clinical use. Mental health apps often require patients to disclose sensitive personal information; thus, data safety and privacy are fundamental. To this end, the framework assesses apps on the possibility of deleting data, the format in which data are shared (nonanonymized, deidentified, anonymized), and whether data are shared with third parties. Thus, data handling procedures should be clearly stated in an app’s description or privacy policy. Ultimately, compliance with data security statutes and regulations (eg, Health Insurance Portability and Accountability Act and General Data Protection Regulation) is paramount for mental health apps collecting sensitive patient data. Clinicians will only recommend apps that fulfill data security requirements and that can easily be integrated into existing clinical workflows.
Regulators and health care providers are introducing additional ways of evaluating and certifying digital health technologies that do not fall under hardware medical device regulations. Innovators should work with regulatory bodies and be aware of the innovation plans for appraisal frameworks. The Food and Drug Administration (FDA) developed the Digital Health Innovation Action Plan to provide a more streamlined and efficient regulatory oversight of digital health technologies [34]. The Plan includes the Software Precertification Pilot Program, which precertifies organizations, not individual products [35]. An “excellence appraisal” is conducted to give the FDA the assurance that the organization is capable of producing high-quality, safe, and effective medical software products so that individual product reviews are carried out more rapidly and swiftly. An international approach to one-off regulatory appraisals also emerged with the Medical Device Single Audit Program (MDSAP), piloted in the United States, Australia, Brazil, Canada, and Japan. The objective of the MDSAP is to develop a single regulatory audit process that can satisfy the needs of multiple regulatory jurisdictions [36]. Europe stands as an observer in the MDSAP, and the new Medical Device Regulation mainly focuses on hardware devices; however, steps have been taken toward a more software-specific approach [37]. In the United Kingdom, the National Health Service (NHS) X developed the beta version of Digital Technology Assessment Criteria for health and social care, which outlines baseline assessment criteria to validate the suitability and function of digital health technologies for use by the NHS, social care staff, or directly by citizens [38]. Trade associations such as the Digital Therapeutics Alliance (DTA) are promoting industry standards and evaluation frameworks to enhance the understanding, adoption, integration, and reimbursement of clinically evaluated digital therapeutics into mainstream health care. The DTA’s member companies include large pharmaceutical companies such as Novartis and Otsuka, as well as digital therapeutics developers such as Pear Therapeutics and SilverCloud, along with technology companies such as Philips, demonstrating the interdisciplinary efforts in pushing forward digital health technologies.
Compliance with data security regulations, assessment against commonly used frameworks, and certifications issued by regulatory bodies can help innovators build transparency and trust with their stakeholders. Importantly, presenting efficacy data and privacy matters in clear terms is essential to engage providers in the uptake of novel digital tools. In other areas of medicine, a clear evidence base, ease of use, and swift integration into clinical workflows have proven to facilitate the uptake of digital health technologies [39,40]. Experts have suggested that the introduction of novel technologies in health care could be facilitated by a dedicated health care profession called a “digital navigator,” whose task in hospitals and clinics would be to select evidence-based apps, troubleshoot, and interpret digital data outputs in a clinically meaningful way [41]. However, mHealth developers should not heavily rely on the presence of specialized personnel; rather, they should focus on communicating evidence transparently, designing tools that fit into current care pathways and information technology system provisions, and providing actionable patient data. Consequently, the use of digital tools will not be limited to settings that have specialized staff members, and as uptake of these apps increases, it will become cost-effective to train and employ digital navigators to support further, more complex implementation of digital health tools.
Conclusion
Digital mental health technologies can enable the early identification of conditions, and facilitate disease management and treatment while empowering patients to make better informed decisions about their own health. Critically, there has been a movement toward technology that replaces the human element, rather than complementing and enhancing the invaluable involvement of health professionals in patient care. However, it is important not to throw decades or even centuries of clinical knowledge overboard. The way forward to building a digital mental health ecosystem must be more nuanced and sensitive. By developing patient-centric solutions, we can enable earlier and more accurate diagnoses, and in turn connect people with the most appropriate and effective treatment and care. As we move toward a postpandemic world, building the digital mental health ecosystem will require optimizing and adapting traditional mental health care services with considered step-wise innovation approaches, allowing health care professionals and patients to adapt to change. Innovative solutions will not come from leaps forward in technology but rather from new models of codesign, careful use of behavior theories, new pragmatic approaches to clinical evaluation, unprecedented collaboration in defining industry standards, and an open mind to trial-and-error implementation approaches.
The long-term goal for digital mental health technology innovators should be to design tools and services that can easily become so engrained in the health care ecosystem that what is now referred to as digital health will simply be called health.
Acknowledgments
This Viewpoint was funded by the Stanley Medical Research Institute (grant number 07R-1888).
Abbreviations
- APA
American Psychiatric Association
- BIT
Behavioral Intervention Technology model
- DTA
Digital Therapeutics Alliance
- FDA
Food and Drug Administration
- HBM
Health Belief Model
- JITAI
Just-in-Time Adaptive Interventions
- MDSAP
Medical Device Single Audit Program
- MHA
Mental Health America
- mHealth
mobile health
- NHS
National Health Service
Footnotes
Conflicts of Interest: SB is a director of Psynova Neurotech Ltd and Psyomics Ltd. SB and NAMK have financial interests in Psyomics Ltd. BS reports no conflicts of interest.
References
- 1.Steel Z, Marnane C, Iranpour C, Chey T, Jackson JW, Patel V, Silove D. The global prevalence of common mental disorders: a systematic review and meta-analysis 1980-2013. Int J Epidemiol. 2014 Apr;43(2):476–493. doi: 10.1093/ije/dyu038. http://europepmc.org/abstract/MED/24648481 .dyu038 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Torous J, Roberts LW. Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry. 2017 May 01;74(5):437–438. doi: 10.1001/jamapsychiatry.2017.0262.2616170 [DOI] [PubMed] [Google Scholar]
- 3.Nadella S. Empowering every developer, with Satya Nadella | Microsoft Build 2020 #MSBuild. Samir Dadoui’s Technical Blog. [2020-12-23]. https://daoudisamir.com/empowering-every-developer-with-satya-nadella-microsoft-build-2020/
- 4.Moltu C, Stefansen J, Svisdahl M, Veseth M. Negotiating the coresearcher mandate - service users' experiences of doing collaborative research on mental health. Disabil Rehabil. 2012;34(19):1608–1616. doi: 10.3109/09638288.2012.656792. [DOI] [PubMed] [Google Scholar]
- 5.Torous J, Jän Myrick K, Rauseo-Ricupero N, Firth J. Digital mental health and COVID-19: using technology today to accelerate the curve on access and quality tomorrow. JMIR Ment Health. 2020 Mar 26;7(3):e18848. doi: 10.2196/18848. https://mental.jmir.org/2020/3/e18848/ v7i3e18848 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Oliver N, Lepri B, Sterly H, Lambiotte R, Deletaille S, De Nadai M, Letouzé E, Salah AA, Benjamins R, Cattuto C, Colizza V, de Cordes N, Fraiberger SP, Koebe T, Lehmann S, Murillo J, Pentland A, Pham PN, Pivetta F, Saramäki J, Scarpino SV, Tizzoni M, Verhulst S, Vinck P. Mobile phone data for informing public health actions across the COVID-19 pandemic life cycle. Sci Adv. 2020 Jun;6(23):eabc0764. doi: 10.1126/sciadv.abc0764. http://europepmc.org/abstract/MED/32548274 .abc0764 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.COVID-19 information governance advice for staff working in health and care organisations. NHSX. [2020-08-04]. https://perma.cc/HA25-ZRSY .
- 8.Notification of enforcement discretion for telehealth remote communications during the COVID-19 nationwide public health emergency. HHS.gov. [2021-05-17]. https://www.hhs.gov/hipaa/for-professionals/special-topics/emergency-preparedness/notification-enforcement-discretion-telehealth/index.html .
- 9.Dalgleish T, Black M, Johnston D, Bevan A. Transdiagnostic approaches to mental health problems: Current status and future directions. J Consult Clin Psychol. 2020 Mar;88(3):179–195. doi: 10.1037/ccp0000482. http://europepmc.org/abstract/MED/32068421 .2020-10232-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kendler KS. The nature of psychiatric disorders. World Psychiatry. 2016 Feb;15(1):5–12. doi: 10.1002/wps.20292. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. https://onlinelibrary.wiley.com/resolve/openurl?genre=article&sid=nlm:pubmed&issn=0884-8734&date=2001&volume=16&issue=9&spage=606 .jgi01114 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006 May 22;166(10):1092–1097. doi: 10.1001/archinte.166.10.1092.166/10/1092 [DOI] [PubMed] [Google Scholar]
- 13.Clarke DE, Kuhl EA. DSM-5 cross-cutting symptom measures: a step towards the future of psychiatric care? World Psychiatry. 2014 Oct;13(3):314–316. doi: 10.1002/wps.20154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Mastoras R, Iakovakis D, Hadjidimitriou S, Charisis V, Kassie S, Alsaadi T, Khandoker A, Hadjileontiadis LJ. Touchscreen typing pattern analysis for remote detection of the depressive tendency. Sci Rep. 2019 Sep 16;9(1):13414. doi: 10.1038/s41598-019-50002-9.10.1038/s41598-019-50002-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Scherer S, Lucas GM, Gratch J, Rizzo A, Morency LP. Self-reported symptoms of depression and PTSD are associated with reduced vowel space in screening interviews. IEEE Trans Affective Comput. 2016 Jan 1;7(1):59–73. doi: 10.1109/taffc.2015.2440264. [DOI] [Google Scholar]
- 16.Torous J, Kiang MV, Lorme J, Onnela J. New tools for new research in psychiatry: a scalable and customizable platform to empower data driven smartphone research. JMIR Ment Health. 2016 May 05;3(2):e16. doi: 10.2196/mental.5165. https://mental.jmir.org/2016/2/e16/ v3i2e16 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Snyder BB. Lundbeck taps Mental Health America—and its big data—to create a national crisis dashboard. Fierce Pharma. [2021-05-14]. https://www.fiercepharma.com/marketing/lundbeck-taps-big-data-partnering-mental-health-america-to-create-a-national-mental .
- 18.Identifying and addressing mental illness hot spots caused by the Covid-19 pandemic. Newsroom. [2021-05-14]. https://newsroom.lundbeckus.com/news-release/2020/identifying-and-addressing-mental-illness-hot-spots-caused-by-covid19-pandemic .
- 19.Hermes S, Riasanow T, Clemons EK, Böhm M, Krcmar H. The digital transformation of the healthcare industry: exploring the rise of emerging platform ecosystems and their influence on the role of patients. Bus Res. 2020 Sep 11;13(3):1033–1069. doi: 10.1007/s40685-020-00125-x. [DOI] [Google Scholar]
- 20.Baranowski T, Cullen KW, Nicklas T, Thompson D, Baranowski J. Are current health behavioral change models helpful in guiding prevention of weight gain efforts? Obes Res. 2003 Oct;11(Suppl):23S–43S. doi: 10.1038/oby.2003.222. [DOI] [PubMed] [Google Scholar]
- 21.Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res. 2014 Jun 05;16(6):e146. doi: 10.2196/jmir.3077. https://www.jmir.org/2014/6/e146/ v16i6e146 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: A pragmatic framework. Health Psychol. 2015 Dec;34S:1209–1219. doi: 10.1037/hea0000306. http://europepmc.org/abstract/MED/26651462 .2015-56045-002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Naslund JA, Aschbrenner KA, Kim SJ, McHugo GJ, Unützer J, Bartels SJ, Marsch LA. Health behavior models for informing digital technology interventions for individuals with mental illness. Psychiatr Rehabil J. 2017 Sep;40(3):325–335. doi: 10.1037/prj0000246. http://europepmc.org/abstract/MED/28182469 .2017-05877-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.An S, Lee H. Adoption of mobile apps for mental health: socio-psychological and technological factors. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering; 7th International Conference, MobiHealth 2017; November 14-15, 2017; Vienna, Austria. 2018. pp. 29–37. [DOI] [Google Scholar]
- 25.Taj F, Klein MCA, van Halteren A. Digital health behavior change technology: bibliometric and scoping review of two decades of research. JMIR Mhealth Uhealth. 2019 Dec 13;7(12):e13311. doi: 10.2196/13311. https://mhealth.jmir.org/2019/12/e13311/ v7i12e13311 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Averous P, Charbonnier E, Dany L. Relationship between illness representations, psychosocial adjustment, and treatment outcomes in mental disorders: a mini review. Front Psychol. 2020;11:1167. doi: 10.3389/fpsyg.2020.01167. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Mohr DC, Weingardt KR, Reddy M, Schueller SM. Three problems with current digital mental health research . . . and three things we can do about them. Psychiatr Serv. 2017 May 01;68(5):427–429. doi: 10.1176/appi.ps.201600541. http://europepmc.org/abstract/MED/28412890 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Larsen ME, Huckvale K, Nicholas J, Torous J, Birrell L, Li E, Reda B. Using science to sell apps: Evaluation of mental health app store quality claims. NPJ Digit Med. 2019;2:18. doi: 10.1038/s41746-019-0093-1.93 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Carlo AD, Hosseini Ghomi R, Renn BN, Areán PA. By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digit Med. 2019;2:54. doi: 10.1038/s41746-019-0129-6.129 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012 Mar;50(3):217–226. doi: 10.1097/MLR.0b013e3182408812. http://europepmc.org/abstract/MED/22310560 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013 Jun;40(3):257–265. doi: 10.1177/1090198113486805.40/3/257 [DOI] [PubMed] [Google Scholar]
- 32.The App Evaluation Model. American Psychiatric Association. 2018. [2020-12-23]. https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/the-app-evaluation-model .
- 33.Lagan S, Aquino P, Emerson MR, Fortuna K, Walker R, Torous J. Actionable health app evaluation: translating expert frameworks into objective metrics. NPJ Digit Med. 2020;3:100. doi: 10.1038/s41746-020-00312-4.312 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Digital Health Innovation Action Plan. US Food and Drug Administration Center for Devices and Radiological Health (CDRH) 2017. [2020-12-23]. https://www.fda.gov/media/106331/download .
- 35.Developing a Software Precertification Program: A Working Model. US Food and Drug Association. 2019. [2020-12-23]. https://www.fda.gov/media/119722/download .
- 36.Medical Device Single Audit Program (MDSAP) TÜV SUD. [2020-12-23]. https://tinyurl.com/27uhv6d7 .
- 37.Reflection Paper on regulatory frameworks for digital health technologies in Europe. Digital Europe. 2019. May 04, [2020-12-23]. https://www.digitaleurope.org/wp/wp-content/uploads/2019/04/DIGITALEUROPE-Reflection-on-regulatory-frameworks-for-digital-health-technologies-in-Europe-.pdf .
- 38.Digital Technology Assessment Criteria – beta. NHSX. [2020-12-23]. https://www.nhsx.nhs.uk/key-tools-and-info/designing-and-building-products-and-services/
- 39.Palacholla RS, Fischer N, Coleman A, Agboola S, Kirley K, Felsted J, Katz C, Lloyd S, Jethwani K. Provider- and patient-related barriers to and facilitators of digital health technology adoption for hypertension management: scoping review. JMIR Cardio. 2019 Mar 26;3(1):e11951. doi: 10.2196/11951. https://cardio.jmir.org/2019/1/e11951/ v3i1e11951 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Slevin P, Kessie T, Cullen J, Butler MW, Donnelly SC, Caulfield B. Exploring the barriers and facilitators for the use of digital health technologies for the management of COPD: a qualitative study of clinician perceptions. QJM. 2020 Mar 01;113(3):163–172. doi: 10.1093/qjmed/hcz241.5572669 [DOI] [PubMed] [Google Scholar]
- 41.Wisniewski H, Torous J. Digital navigators to implement smartphone and digital tools in care. Acta Psychiatr Scand. 2020 Apr;141(4):350–355. doi: 10.1111/acps.13149. http://europepmc.org/abstract/MED/31930477 . [DOI] [PMC free article] [PubMed] [Google Scholar]