Abstract
Background
The mass availability and use of mobile health (mHealth) technologies offers the potential for these technologies to support or substitute medical advice. However, it is worrisome that most assessment initiatives are still not able to successfully evaluate all aspects of mHealth solutions. As a result, multiple strategies to assess mHealth solutions are being proposed by medical regulatory bodies and similar organizations.
Objective
We aim to offer a collective description of a universally applicable description of mHealth assessment initiatives, given their current and, as we see it, potential impact. In doing so, we recommend a common foundation for the development or update of assessment initiatives by addressing the multistakeholder issues that mHealth technology adds to the traditional medical environment.
Methods
Organized by the Mobile World Capital Barcelona Foundation, we represent a workgroup consisting of patient associations, developers, and health authority representatives, including medical practitioners, within Europe. Contributions from each group’s diverse competencies has allowed us to create an overview of the complex yet similar approaches to mHealth evaluation that are being developed today, including common gaps in concepts and perspectives. In response, we summarize commonalities of existing initiatives and exemplify additional characteristics that we believe will strengthen and unify these efforts.
Results
As opposed to a universal standard or protocol in evaluating mHealth solutions, assessment frameworks should respect the needs and capacity of each medical system or country. Therefore, we expect that the medical system will specify the content, resources, and workflow of assessment protocols in order to ensure a sustainable plan for mHealth solutions within their respective countries.
Conclusions
A common framework for all mHealth initiatives around the world will be useful in order to assess whatever mHealth solution is desirable in different areas, adapting it to the specifics of each context, to bridge the gap between health authorities, patients, and mHealth developers. We aim to foster a more trusting and collaborative environment to safeguard the well-being of patients and citizens while encouraging innovation of technology and policy.
Keywords: mhealth, evaluation, assessment, checklist, framework
Introduction
Mobile health (mHealth) is defined by the World Health Organization’s (WHO) Global Observatory for eHealth as “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices” [1]. Despite the lack of safety and quality validation of these technologies by medical regulatory bodies, individuals have adopted mHealth devices as self-management aids, while medical professionals are at a loss for how to relate to them [1]. Due to this consumer-based and rapid introduction within the world of patient health aids, mHealth solutions present unique and stakeholder-specific challenges to the medical environment. Patients, health care providers, administrators, authorities, and mHealth developers alike are operating without clear direction—from potentially improper use of mHealth apps by individuals, to medical systems’ inability to react due to lack of technological and organizational support. Therefore, several questions arise: How should health authorities approach mHealth evaluation and certification? How can this be accomplished without stifling innovation? Should efforts to determine risks, benefits, and appropriate use be held on a global, country, or regional basis? How will policies and strategies be introduced to medical professionals and their practices? Which methods should be used to ensure that patients are properly informed on how to select and use mHealth solutions?
Today, there are many efforts underway to address these challenges [2-5]. However, these are happening in silos and are often specific to a single country or medical system. In this paper, we explore the progress and setbacks of the mHealth assessment environment and propose a collaborative and global approach to assessment, with the aim of applying the competencies of all stakeholder groups. We believe that by structuring our work in this field from a common foundation, assessment initiatives can and will foster a more trusting and collaborative environment to safeguard the well-being of citizens while encouraging innovation of technology and policy.
The purpose of this viewpoint paper is to be informative and provocative, focusing on the changes in mindset and actions that must occur—from the education and perspective of consumers, health assessment evaluations teams, and medical authorities—in the assessment of mHealth solutions. As such, we hope to elicit discussion on the content and methodologies of our suggested common framework for mHealth assessment.
The Potential of Mobile Technologies in the Medical Field
The potential of mHealth solutions lies in their ability to enable chronic diseases management and general wellness motivation [6-8]. By tracking an individual’s health and lifestyle data, and by providing actionable feedback, these tools encourage self-management. With the subsequent promise of equipping health care providers with such detailed information about a patient’s health status in relation to their daily habits, the medical system could not only better understand elusive compliance issues but also propose tailored solutions [9] for individual patients. If integrated and supported appropriately, these tools could improve treatment, empower patients, and foreseeably lower medical costs and streamline use of health care resources [10,11].
However, because these innovations are diverse and unstandardized, mHealth requires that regulatory bodies take a fresh look at evaluation methodologies for health aids. Today, medical actors are expected to adapt to the rapid technology turnover, ubiquity, and connectivity of mHealth by collecting, incorporating, and analyzing unprecedented amounts of personal health information seamlessly within the clinical environment. However, medical research and validation protocols, which providers rely on for guidance, are currently unable to adapt as quickly as such mHealth tools are being released. As a result, health care providers are left in the dark regarding how to relate to mHealth tools, as well as the impacts to their own professional responsibilities. If this goes un-checked, the medical system will continue to lag behind the needs of its patients.
The Current Situation
Just as in the proverbial scenario of the Band-Aid on the leaking dam, the mHealth environment has been met with a series of incomplete or issue-specific solutions to the evaluation of health apps, wearables and sensors. In fact, neither the Health Technology Assessment (HTA) agency [12] nor its collaborative networks, European Network for Health Technology Assessment (EUnetHTA) and HTA Network, have tackled mHealth solutions within their range of technology assessments. As patients continue to use these apps regardless of clinical support or guidance, health authorities and providers fall further behind, stuck in the traditional and hypercontrolled operations of the medical sector. With good intentions, organizations continuously attempt to address the distinct issues facing patients and clinicians. Promising attempts included the National Health Service’s Health Apps Library [13], which aimed to involve clinicians in the review process, yet it is still under maintenance; the Organization for the Review of Care and Health Applications [14], which describes the purpose of their value and risk scores but not which features of an app lead to its value or risk; and PatientView’s “The myhealthapps directory 2015-2016” [15], which presented a summary of consumer-generated app reviews. While these initiatives attempt to provide the public with digestible and relevant information, they simply add to the slew of disjointed and static solutions that are not able to address all stakeholder issues.
A Step in the Right Direction
Recently, governmental organizations and those representing end-user interests have acknowledged that flexible solutions and continuous communication are more appropriate for the mHealth environment than customary static reports. A preparatory review of these evaluation initiatives illustrates the range of developed mHealth evaluation methodologies and adapted health assessment frameworks for use in mHealth. These initiatives focus mainly on the usability and clinical application of mHealth technologies, which are most commonly provided by individual user's commentaries, the organization’s own evaluation team or, very rarely, a representative group of medical practitioners [16-19]. Some, including the United States Food and Drug Administration (FDA), choose to regulate only devices that fall under the definition of “medical device,” enabling them to rely on existing frameworks for evaluation and implementation [20]. Others attempt to address a broader range of tools, with some focusing only on risk assessment [21,22] or even utilizing nontraditional resources (eg, novel technologies and social media outlets [23]) to evaluate mHealth solutions. Finally, most initiatives address only mHealth solutions that are fully operational within the market [24], with no solution for those under development [25].
Too Many Initiatives, Too Few Answers
Despite the diversity of approaches, recent reviews reveal worrisome results—a lack of conclusive or actionable evidence to suggest the ability of commercially available mHealth tools to effect behavioral changes or to manage chronic diseases, inpatient care, or health care delivery [26]. As a result, a primary concern and unfortunate reality is that this environment has allowed for the existence of misinformation regarding appropriate uses for these mHealth solutions. Common labels may even clearly categorize an app as “medical” yet include the warning in fine print that the app is “intended for entertainment only” [27]. Moreover, these evaluation efforts do not involve or inform all stakeholders [28] of relevant results, including potential risks associated with misleading health recommendations or software failures.
To compare the scope of structure and insights offered by today’s mHealth evaluation initiatives to the desired coverage and impact that we believe is possible, we have completed a comparative summary of representative evaluation efforts (see Multimedia Appendix 1 [29-33]). These characteristics are in addition to the assumed basic coverage of usability and security. Such a lack of flexible and inclusive evaluation options has resulted in a lack of empirically demonstrated understanding of the benefits and risks that mHealth solutions provide to care delivery and health management. There is clearly a need to create an environment that explores additional options for mHealth evaluation. For example, individuals outside the medical realm have not traditionally had a majority voice in the assessment of health aids. Yet today, individual app users are becoming more knowledgeable about the daily benefits of mHealth and technology developers are exploring the potential of ever-present self-management systems. Therefore, we should consider these groups as underutilized resources—we have an opportunity to recognize and enthusiastically encourage the insight that these individuals can now contribute to mHealth evaluation efforts.
The Unsung Role of Developers in Assessment
Developers are the key force driving the need for certification due to the astounding rate at which they are producing mHealth solutions [34]. Therefore, those who are developing mHealth solutions should be encouraged to seek assessment for their own technologies through engagement in the evaluation process. Additionally, they should be provided with guidance and educational resources, including explanation of the concepts and benefits of a quality assured and reliable product go hand-in-hand with growth in the market and success, as the result of consumer engagement, loyalty, and trust [35,36]. By creating a mutually beneficial situation for developers and the medical system, evaluation frameworks can facilitate an environment that is transparent, trustworthy, and safe for users. It must also be noted that we must achieve a balance between development oversight and creative freedom. Too much regulation, or an unnecessarily lengthy process, would paralyze certification and inhibit adoption of mHealth solutions, which is evident from responses to the US FDA’s complex and unclear rulings [37]. Therefore, involving developers in the improvement and operation of any mHealth assessment initiatives will ensure the safety of their creativity and the competitive, open health-app market.
The Need to Work From a Common Foundation
While the resources, evidence, and financial support for clinical implementation undoubtedly vary between countries and medical systems, there are shared challenges and themes for assessment. By acknowledging and addressing these common needs, we as a health care community can work in parallel as opposed to reinventing the wheel. Furthermore, these needs cannot hope to be addressed by a single organization or single perspective (eg, safety and privacy vs usability and consumer cost). The needs presented by the mHealth assessment arena can be met only through participation of all stakeholders, hand-in-hand with regulation and legislation, constantly adapting to updates in standards and the capacity of technology.
To facilitate the success of this collaborative approach, we must simultaneously change the paradigm of evaluation to more appropriately relate to the particularities of mHealth. As such, it must be continuous and iterative, while at the same time provide timely conclusions and actionable recommendations for improvement and implementation.
Acknowledging and Merging Universal Needs: Approaching the Solution Together
Organized by the Mobile World Capital Barcelona Foundation, the co-authors of this paper represent a workgroup of several international partners with distinct yet complementary backgrounds and competencies related to mHealth, including patient groups, government, and health authorities, as well as representatives from health research, care providers, and technology developer groups. Exploration of the common needs and themes of mHealth has resulted in a summative list of concepts that we believe are universal to all assessment initiatives. We welcome interested parties and future partners to contribute their competencies and applicable solutions toward the future development of common and foundational guidelines for building mHealth assessment frameworks. Not only are newly validated tools needed in order to improve the quality of assessment steps, but also the perspectives of a diverse group of representatives in all affected fields. The following sections outline the major universal concepts and approaches that can inform proposed and existing evaluation initiatives.
Assigning the Evaluation Team
The concept of specialization is a key component in any effort to impact a multistakeholder system. A single organization cannot be expected to accomplish the diversity of tasks and successfully address the challenges of mHealth evaluation (eg, results of safety, usability, and health change assessments). Therefore, representative organizations should be involved in tasks associated with their competencies, thus providing even distribution of responsibilities as well as relevant input.
In order to achieve a comprehensive process, the evaluation team should possess a broad scope of perspectives that are representative of the following stakeholder groups: patients or patient organizations; commercial and research-based mHealth developers; health care providers, medical professionals, and system administrators; insurance or other reimbursement bodies; and authorities within governmental health and medical system organizations.
It is assumed that the composition of this team, and specific roles of stakeholder representatives, will vary based on the decision of each country or region that chooses to adopt and adapt this approach to their own respective medical system.
Pre-Assessment of mHealth Solutions
The pre-assessment phase is meant to classify any mHealth app and can be based on, for example, the following criteria: (1) Risk classification matrix: combining intervention type and patient type (see Figure 1 [38] for an example); (2) Users: patients, medical professionals, and/or informal caregivers; and (3) Integration: stand-alone, partially integrated, or fully integrated.
Figure 1.
Risk assessment matrix.
In order to address mHealth solutions within any stage of development, pre-assessment guides should be distributed to developers who are still in the process of designing health solutions as well as for members of evaluation teams within any medical system or country.
Checklists
Together with the pre-assessment phase, checklists can act as preparative resources for the evaluation teams to ensure that all relevant information is provided before resources and time are spent assessing an mHealth solution, as well as subsequent public and medical integration. Once pre-assessment has been established, checklists can be directed toward each contextual category of health apps. The checklist will vary from country to country and be provided to the evaluation team. To illustrate this point, a checklist should ensure that documentation related to the following categories is provided for each mHealth solution:
designation of mHealth solutions by intended use, for example, reference guides (eg, for nutrition or weight control), monitoring devices (eg, for blood sugars or blood pressure), or other types of solutions within a matrix
level of development
security and privacy
interoperability standards
usability
functionalities and content
To the best of our knowledge, there are few functional and representative checklists (eg, Catalonia, Andalucía, and WHO’s mHealth Technical Evidence Review Group [29,39,40]). Most others are not fully available, are under development (eg, European Commission mHealth assessment working group [5]), or are focused only on mobile medical devices (eg, Future Internet-STAR checklist model [41]). These checklists propose categories related to functionality including usability, technology, security, content design and pertinence, and services. They also propose more domains related to infrastructure, including intervention delivery, accessibility toward individuals (ie, barriers or facilitators to the adoption of the intervention among study participants), cost assessment, adoption inputs/program entry, limitations for delivery at scale, contextual adaptability, and replicability. Many of these approaches involve classification of information into levels, from mandatory to not applicable, based on the app’s level of risk, thereby providing flexibility to assess not only medical but also nonmedical devices.
Again, each stakeholder representative within evaluation teams should be assigned to the topics they are most capable of assessing. However, not all of these available checklists are accessible or usable by all stakeholders. As previously mentioned, developers are a key stakeholder in the success of assessment initiatives. Therefore, all developers should be able to provide the checklist information themselves. An alternative and more time-consuming option is that facilitators be put in place to gather this information from multiple sources in order to answer evaluation questions. This will affect the level of time and funded investment needed, which supports the involvement and support of developer groups in the design of checklists.
mHealth Evaluation Aspects and Methods
Assessment initiatives must be equally focused on summative (ie, during or post implementation) as well as formative (throughout the development life cycle) evaluation. This will ensure that not only common software development life-cycle stages but also end-user needs and input are incorporated, thus giving any initiative practical dimension. We expect that in doing so, this will promote adoption by the app development community as well as those representing the interests of the medical community.
The categories of information provided by checklists can be evaluated with the common evaluation perspectives of technical readiness and maturity, risks, benefits, and resources needed. The limitations and specifics of what is involved in these evaluation domains should correspond to the purpose of the evaluation (ie, assessing a device as an educational or medical tool) and should vary depending on the level of interoperability and intended use (eg, disease self-management vs activity tracking) of the mHealth solution (eg, stand-alone app vs integrated medical device).
Each of these four domains and their subdomains should be defined to streamline evaluation efforts and organize stakeholder participation based on their respective competencies (see Multimedia Appendix 2 for an example). Ideally, the chosen set of domains and subdomains should address the following needs:
Determine the appropriate use of each mHealth solution (ie, as a medical device or a health and wellness tool, based on the target and breadth of functionalities as well as status of interoperability and safety standards)
Develop expedited and conclusive methods to evaluate the effect(s) that an mHealth solution has on respective clinical outcomes and/or patient lifestyle habits, based on its appropriate and intended use
Assess risk related to (1) patients and their caregivers in relation to personal data security, self-management decision making, and disease understanding; (2) clinicians, including liability to their practice and a greater trust of and reliance on patient-gathered data; and (3) overall health care organizations and systems, including financial impact and liabilities
Inform stakeholders of relevant results through respective and accessible platforms.
Quantitative or qualitative methodologies may be employed through formal evaluation studies (eg, clinical trials) to assess each of these. The evaluation studies should be conducted based on the regulations, practices, and implementation strategies of individual countries or medical systems in which such studies are completed. However, to our knowledge, most of the subdomains still lack validated, standardized, or descriptive approaches and methods for evaluation. Therefore, alternative or unconventional inquiry methods should be considered including involvement of medical education programs as well as clinical and commercial research organizations, all of which use complementary methods such as online platforms, market analysis, and user-involved workshops. Partnership with organizations that are able to develop such novel methods should be sought both within the international realm of health sciences and then adapted to the needs and socioeconomic factors present within each unique medical system.
Conclusions
There is a clear need for defining a standard assessment framework for mHealth technologies that will help separate the wheat from the chaff and identify those solutions that may provide added value to patients and the health care system. By accomplishing this set of methodologies and approaches to evaluation and also the methods, perspectives, and resources used to accomplish these tasks, evaluation would facilitate more educated and informed decision making regarding the choice and use of mHealth solutions—from patients to medical practitioners to the health authorities that are charged with maintaining the foundation of each medical service.
The insights and suggestions provided in this paper are intended for the groups that are completing evaluations, developing apps, and creating health policy and infrastructural support for mHealth implementation. We propose that all organizations and individuals who share a similar passion for a coordinated effort towards a more rigorous and comprehensive evaluation of mHealth technologies join forces to form a virtual community of practice [42]. Discussing the merits and shortcomings of the proposed approach in this paper and its utility in real-world scenarios can be a starting point for such community. We hope that a free-flowing format will foster creative ways to solve the problems put forward in this position paper. A social media‒based discussion group is a feasible starting point for this virtual community of practice. The authors invite the experts in evaluation of health information systems to provide their opinion as to how such a virtual community should form and collaborate.
Abbreviations
- EUnetHTA
European Network for Health Technology Assessment
- FDA
Food and Drug Administration
- FI-STAR
Future Internet Social and Technological Alignment Research
- HTA
Health Technology Assessment Agency
- mHealth
mobile health
- WHO
World Health Organization
Presence of desirable characteristics within existing mHealth assessment initiatives.
Example set of domains and subdomains for an mHealth assessment framework.
Footnotes
Authors' Contributions: This paper was prepared by all the authors in their capacity. Authors EP and CC can be contacted by those interested in this discussion.
Conflicts of Interest: None declared.
References
- 1.World Health Organization Frequently asked questions on Global Task Force on digital health for TB and its work. [2017-02-27]. http://www.who.int/tb/areas-of-work/digital-health/faq/en/
- 2.Chant L, Hoogewerf L, Lewis T, Sathanandam S, Thimbleby H, Wyatt J. Rapid literature review for the National Information Board work stream 1.2. Royal College of Physicians; 2015. Aug 07, [2016-10-19]. http://harold.thimbleby.net/NICE/NIB%20literature%20review%20final%20report%20v096-harold%20edits.pdf . [Google Scholar]
- 3.World Health Organization. UN Foundation. Johns Hopkins University Global mHealth Initiative mHealth MAPS toolkit: mHealth Assessment and Planning for Scale. 2015. [2016-10-19]. http://who.int/life-course/publications/mhealth-toolkit/en/
- 4.Haugli H. Høring: Forslag til selvdeklareringsordning for mobile helseapplikasjoner. Oslo: Direktoratet for ehelse; 2016. Aug 30, [2016-10-19]. https://ehelse.no/Documents/E-helsekunnskap/H%c3%b8ringsnotat-%20Forslag%20selvdeklareringsordning%20mobile%20helseapplikasjoner.pdf . [Google Scholar]
- 5.European Commission. 2016. Jan 15, [2016-10-19]. Working group on mHealth assessment guidelines http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=3390 .
- 6.Talboom-Kamp E, Verdijk N, Harmans L, Numans M, Chavannes N. An eHealth Platform to Manage Chronic Disease in Primary Care: An Innovative Approach. Interact J Med Res. 2016 Feb 09;5(1) doi: 10.2196/ijmr.4217. http://www.i-jmr.org/2016/1/e5/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hamine S, Gerth-Guyette E, Faulx D, Green BB, Ginsburg AS. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res. 2015 Feb 24;17(2) doi: 10.2196/jmir.3951. http://www.jmir.org/2015/2/e52/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Garg S, Garg D, Turin TC, Chowdhury MF. Web-Based Interventions for Chronic Back Pain: A Systematic Review. J Med Internet Res. 2016 Jul 26;18(7):e139. doi: 10.2196/jmir.4932. http://www.jmir.org/2016/7/e139/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Azar KM, Koliwad S, Poon T, Xiao L, Lv N, Griggs R, Ma J. The Electronic CardioMetabolic Program (eCMP) for Patients With Cardiometabolic Risk: A Randomized Controlled Trial. J Med Internet Res. 2016 May 27;18(5):e134. doi: 10.2196/jmir.5143. http://www.jmir.org/2016/5/e134/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Hung CS, Yu JY, Lin YH, Chen YH, Huang CC, Lee JK, Chuang PY, Ho YL, Chen MF. Mortality Benefit of a Fourth-Generation Synchronous Telehealth Program for the Management of Chronic Cardiovascular Disease: A Longitudinal Study. J Med Internet Res. 2016 May 13;18(5) doi: 10.2196/jmir.5718. http://www.jmir.org/2016/5/e102/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Badawy SM, Kuhns LM. Economic Evaluation of Text-Messaging and Smartphone-Based Interventions to Improve Medication Adherence in Adolescents with Chronic Health Conditions: A Systematic Review. JMIR Mhealth Uhealth. 2016 Oct 25;4(4) doi: 10.2196/mhealth.6425. http://mhealth.jmir.org/2016/4/e121/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.EUnetHTA EUnetHTA Joint Action 2 on HTA 2012-2015: 3-year Work Plan. 2013. May 01, [2017-03-14]. http://www.eunethta.eu/sites/5026.fedimbo.belgium.be/files/EUnetHTA%20JA2%203-year%20Work%20Plan%20%282013%29.pdf .
- 13.Hall K. The Register. Situation Publishing; 2015. Jul 09, [2017-03-14]. Privacy campaigners question credibility of NHS ‘endorsed’ apps https://www.theregister.co.uk/2015/07/09/credibility_of_nhs_endorsed_apps_in_question/ [Google Scholar]
- 14.Organisation for the Review of Care and Health Applications . WHo are ORCHA? ORCHA; 2017. [2017-03-14]. https://www.orcha.co.uk/ [Google Scholar]
- 15.My Health Apps: tried and tested by people like you. Patient View; 2013. [2017-04-26]. http://www.patient-view.com/bull-myhealthappsnet.html . [Google Scholar]
- 16.Misra S, Husain I. iMedicalApps. iMedicalApps; 2015. Dec 21, [2016-10-20]. The Best Medical Apps Released in 2015 for iPhone & Android http://www.imedicalapps.com/2015/12/best-medical-apps-iphone-android/ [Google Scholar]
- 17.UMass Center for mHealth and Social Media. Massachusetts: University of Massachusetts Medical School Prevention and Behavioral Medicine; [2016-10-20]. App Evaluation Service http://www.umassmed.edu/mhealth/app-evaluation-service/ [Google Scholar]
- 18.Singh K, Drouin K, Newmark LP, Rozenblum R, Lee J, Landman A, Pabo E, Klinger EV, Bates DW. Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications. Issue Brief (Commonwealth Fund) 2016 Feb;5:1–11. http://www.commonwealthfund.org/publications/issue-briefs/2016/feb/evaluating-mobile-health-apps . [PubMed] [Google Scholar]
- 19.Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto S. Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med. 2014 Dec;4(4):363–371. doi: 10.1007/s13142-014-0293-9. http://europepmc.org/abstract/MED/25584085 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.US Food and Drug Administration Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff. 2015. Feb 09, [2017-02-27]. https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm263366.pdf .
- 21.Wyatt JC, Thimbleby H, Rastall P, Hoogewerf J, Wooldridge D, Williams J. What makes a good clinical app? Introducing the RCP Health Informatics Unit checklist. Clin Med (Lond) 2015 Dec;15(6):519–521. doi: 10.7861/clinmedicine.15-6-519. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer use. J Med Internet Res. 2014 Sep 15;16(9):e210. doi: 10.2196/jmir.3133. http://www.jmir.org/2014/9/e210/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Facebook. London: PatientView; 2013. Nov 01, [2016-10-20]. myhealthapps.net https://www.facebook.com/myhealthapps . [Google Scholar]
- 24.Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3(1):e27. doi: 10.2196/mhealth.3422. http://mhealth.jmir.org/2015/1/e27/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Royal Academy of Engineering . Health apps: regulation and quality control. Academy of Medical Sciences; 2015. [2016-10-20]. http://www.raeng.org.uk/publications/reports/health-apps-regulation-and-quality-control . [Google Scholar]
- 26.Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med. 2015 Apr 15;7(283) doi: 10.1126/scitranslmed.aaa3487. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Krisch J. Questioning the Value of Health Apps. The New York Times; 2015. Mar 16, [2016-10-20]. http://well.blogs.nytimes.com/2015/03/16/health-apps-provide-pictures-if-not-proof-of-health/?_r=1 . [Google Scholar]
- 28.MyHealthApps . Health apps: where do they make sense? Patient View; 2014. Mar, [2016-10-20]. http://www.ehfg.org/blog/wp-content/uploads/2014/03/Health-apps-patients-standards-and-policies-white-paper-March-2014.pdf . [Google Scholar]
- 29.Agencia de Calidad Sanitaria de Andalucía Safety and Quality Strategy in Mobile Health Apps. 2012. [2016-10-20]. AppSaludable Quality Seal http://www.calidadappsalud.com/distintivo-appsaludable/
- 30.Connect DG. Digital Single Market: Digital Economy & Society. European Commission; 2016. May 04, [2016-10-20]. Open stakeholder meeting on mHealth assessment guidelines: Presentations and survey https://ec.europa.eu/digital-single-market/en/news/open-stakeholder-meeting-mhealth-assessment-guidelines-presentations-and-survey . [Google Scholar]
- 31.TicSalut: Technologia, innovacio i salut. TicSalut - Generalitat de Catalunya; 2013. [2016-10-20]. Standards and interoperability http://www.ticsalut.cat/estandards/en_index/ [Google Scholar]
- 32.Future Internet: Social Technological Alignment in Healthcare. 2014. [2016-10-20]. Legal compliance for developers using FI-STAR eHealth platform: Training Materials https://www.fi-star.eu/fileadmin/images/use_cases/Microsoft_PowerPoint_-_Introduction_into_applicable_legal_requirements.pdf .
- 33.Future Internet: Social and Technological Alignment in Healthcare. 2013. [2016-10-19]. Basic validation methodology and quality control indicators https://bscw.fi-star.eu/pub/bscw.cgi/d44362/D6.1-Basic%20validation%20methodology%20and%20Quality%20control%20indicators.pdf .
- 34.Krohn R, Metcalf D, editors. mHealth Innovation: Best Practices From The Mobile Frontier. Chicago: Healthcare Information and Management Systems Society; 2014. [Google Scholar]
- 35.Batchelor C. United Kingdom Regulators Network. London: UK Regulators Network; 2014. Dec 17, [2016-10-20]. Consumer engagement and switching http://www.ukrn.org.uk/wp-content/uploads/2016/07/20141217ConsumerEngSwitch.pdf . [Google Scholar]
- 36.Halim P, Swasto B, Hamid D, Firdaus MR. European Journal of Business and Management. 2014. [2017-02-27]. The Influence of Product Quality, Brand Image, and Quality of Service to Customer Trust and Implication on Customer Loyalty (Survey on Customer Brand Sharp Electronics Product at the South Kalimantan Province) http://iiste.org/Journals/index.php/EJBM/article/viewFile/16089/16236 .
- 37.Cortez NG, Cohen IG, Kesselheim AS. FDA regulation of mobile health technologies. N Engl J Med. 2014 Jul 24;371(4):372–379. doi: 10.1056/NEJMhle1403384. http://www.nejm.org/doi/full/10.1056/NEJMhle1403384 . [DOI] [PubMed] [Google Scholar]
- 38.Vallespin B, Cornet J, Kotzeva A. Ensuring evidence-based safe and effective mHealth applications. Evidence-Based Health Informatics. 2016;222:248–261. doi: 10.3233/978-1-61499-635-4-248. http://ebooks.iospress.nl/publication/42831 . [DOI] [PubMed] [Google Scholar]
- 39.TicSalut: Technologia, innovacio i salut. TicSalut - Generalitat de Catalunya; 2013. [2016-10-20]. Observatory http://www.ticsalut.cat/observatori/en_index/ [Google Scholar]
- 40.Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, Labrique A, WHO mHealth Technical Evidence Review Group Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ. 2016 Mar;352 doi: 10.1136/bmj.i1174. http://www.bmj.com/cgi/pmidlookup?view=long&pmid=26988021 . [DOI] [PubMed] [Google Scholar]
- 41.FI-STAR: Future Internet Social and Technological Alignment in Healthcare. FI-STAR Consortium; 2014. [2016-10-20]. Information for application developers using FI-STAR eHealth platform https://www.fi-star.eu/publications/checklist-for-app-developers.html . [Google Scholar]
- 42.Ford J, Korjonen H, Keswani A, Hughes E. Virtual communities of practice: can they support the prevention agenda in public health? Online J Public Health Inform. 2015 Jul;7(2) doi: 10.5210/ojphi.v7i2.6031. http://europepmc.org/abstract/MED/26284150 . [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Presence of desirable characteristics within existing mHealth assessment initiatives.
Example set of domains and subdomains for an mHealth assessment framework.