Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations

Abstract

Mobile applications (apps) to improve health are proliferating, but before healthcare providers or organizations can recommend an app to the patients they serve, they need to be confident the app will be user-friendly and helpful for the target disease or behavior. This paper summarizes seven strategies for evaluating and selecting health-related apps: (1) Review the scientific literature, (2) Search app clearinghouse websites, (3) Search app stores, (4) Review app descriptions, user ratings, and reviews, (5) Conduct a social media query within professional and, if available, patient networks, (6) Pilot the apps, and (7) Elicit feedback from patients. The paper concludes with an illustrative case example. Because of the enormous range of quality among apps, strategies for evaluating them will be necessary for adoption to occur in a way that aligns with core values in healthcare, such as the Hippocratic principles of nonmaleficence and beneficence.

INTRODUCTION

The use of mobile technologies to track and improve health outcomes, referred to as mHealth, is a rapidly expanding practice [1]. The advent of smartphone technologies that enable quick and easy access, transfer, and tracking of information, as well as interactive displays and interventions that can be highly engaging, have promoted adoption. In 2012, 85 % of adults in the USA owned a mobile phone, of which 53 % were smartphones [2]. Additionally, nearly one-fifth (19 %) of US adults who owned a smartphone had at least one application (app) designed to promote health behavior or health maintenance (health app) on their phone, with the majority being related to diet and physical activity. By 2017, 50 % of mobile phone users are projected to have downloaded at least one health app [3].

As of May 2012, more than 40,000 health apps existed in the USA, and the rate of proliferation is nearly exponential [4]. This incredible volume of apps, combined with the rapid evolution of technology and features, makes it difficult for consumers to keep abreast of which apps are available for use by patients. While the first priority in the field should be creating a primary evidence base by studying apps in well-controlled studies, the pace of science is incongruent with that of the business sector and consumer demand. Healthcare providers may want to stay informed due to interest in recommending apps as part of their treatment planning [5], and healthcare systems, health promotion organizations, and insurance providers (hereafter referred to as healthcare organizations collectively) may want to provide recommendations regarding health apps as part of their health promotion services. However, guidance on how to judge the validity or worth of commercially available apps is lacking [6]. The decision to recommend an app to a patient can have serious consequences if its content is inaccurate or if the app is ineffective or even harmful. For example, a recent study found that a mobile app claiming to provide diagnostic recommendations for suspected melanoma had very low sensitivity and was therefore likely to miss many melanomas [7]. Use of this app had the potential to delay diagnosis and treatment for a condition in which early detection has a significant impact on survival rates. Another recent study showed that only 37 % of physicians recruited from an online social network prescribed health apps to their patients, 42 % said they would not prescribe apps without regulatory oversight, and 37 % said they have no idea what apps are available [8]. Thus, healthcare providers and healthcare organizations are in a quandary: increasingly, patients are using the countless existing health apps, but providers and organizations are hesitant to act because these apps are shrouded in quality and validity concerns, and they do not know which ones to recommend.

This paper outlines a practical guide for healthcare providers and healthcare organizations on the evaluation and selection of publically available apps targeting health behavior or health maintenance. It will focus only on apps designed for use by patients to promote patient health and not on those that provide information directly to healthcare providers, such as apps designed to avoid medication errors or for medical education. In addition, it will not provide information on evaluating medical apps regulated by the Food and Drug Administration (FDA) [9]. Apps regulated by the FDA include only those that are intended as an accessory to a regulated medical device or that transform a mobile platform into a regulated medical device, such as an app intended to diagnose cardiac arrhythmias. Most health-related apps commonly available in apps stores do not fit this definition and are therefore not under the purview of the FDA. The paper is divided into two sections. The first section proposes practical search and selection strategies, while the second provides an illustrative example using a case study of a patient needing dietary assistance to manage celiac disease.

PRACTICAL SEARCH AND SELECTION STRATEGIES

Currently, little or no quality control or regulations exist to ensure health apps are user-friendly, accurate in content, evidence-based, or efficacious. For example, a search using the term “weight loss” on iTunes (January 15, 2014) revealed more than 3800 iPhone apps alone. Reviews of commercially available weight loss or weight management apps have found that most reflect only a narrow range of evidence-based behavioral strategies (e.g., tracking and goal setting) and some include strategies not rooted in any evidence base [10–12]. The following seven strategies balance the ease and efficiency of the search process against the need to understand the accuracy, evidence base, and efficacy of the app. While they could be viewed as sequential steps, they are not designed as such to promote flexibility and to acknowledge that search strategies might depend upon the purpose and baseline knowledge of the consumer. These strategies are: (1) Review the scientific literature, (2) Search app clearinghouse websites, (3) Search app stores, (4) Review app descriptions, user ratings, and reviews, (5) Conduct a social media query in professional and, if available, patient networks, (6) Pilot the apps, and (7) Elicit feedback from patients. These proposed strategies are summarized in Table 1.

Table 1.

Summary of strategies

1. Review the scientific literature: Search the scientific literature for papers reviewing apps in a content domain or strong clinical trials

2. Search app clearinghouse websites: Clearinghouses that review apps can help with identifying strengths and weaknesses

3. Search app stores: App stores are challenging to navigate, so it is important to fine-tune and filter app searches with the most relevant and targeted key words, including words keyed to the pathological state or target behavior

4. Review app descriptions, user ratings, and reviews: Publicized ratings and user reviews can offer evidence of app usability, functionality, and efficacy, which can help to narrow the pool of candidate apps

5. Conduct a social media query within professional and, if available, patient networks: Social networks may reveal new app trends, likability by certain user groups, and other substantive data

6. Pilot test the app: Apps may be piloted by the healthcare provider or a designee, including examinations of functionality, accuracy of content, and usability

7. Elicit feedback from patients: Patients may be able to provide valuable insights after they have used the app a provider recommends

Review the scientific literature

Systematic reviews of health apps are increasingly appearing in the literature. Most reviews evaluate whether apps for a particular condition or health behavior are rooted in evidence-based strategies or theoretical models of behavior change. Examples include reviews of apps for weight loss [10, 11], cancer prevention and management [13], mental health [14], medication self-management [15], chronic medical conditions [16, 17], HIV prevention [18], sports injury prevention [19], and smoking cessation [20]. Reviews are likely to reveal the capabilities and limitations of apps and this information can be incorporated into clinical decisions and patient recommendations. The same criteria used to judge other systematic reviews should be used to evaluate the worth of app review papers, including the comprehensiveness of the search criteria, the rigor of the evaluation methods, and the journal’s peer-review process. In the absence of reviews, providers can search for randomized trials, which are also increasingly common. Search terms “randomized trial,” “mobile app*,” and the name of the condition or health behavior (e.g., tobacco use) should produce the available literature for that condition.

The main weaknesses with reviewing the scientific literature are: (1) despite the increasing attention, there remains only a very small published literature on health apps, (2) it can be time consuming to sift through and digest the literature, (3) evaluations are often focused on one operating system (iPhone or Android), and (4) the literature becomes outdated quickly because of the constantly evolving market. In addition, some reviews do not list the names of the specific apps, and when moving from development to market, apps often change names, making it difficult to know exactly what app to recommend. Contacting the corresponding authors of review papers may be one way to find out the apps included in the review and the ones they recommend based on their review, but this is time consuming and requests are not always acknowledged.

Search app clearinghouse websites

Because of the difficulties inherent in searching the evidence, notable efforts by public and private organizations have begun to help organize, review, and “certify” health apps. The premise of “certifying” apps is similar to existing web certification, such as the Health on the Net Foundation Code of Conduct (http://www.healthonnet.org) and the Information Standard (http://www.theinformationstadard.org), where the credibility and reliability of health information is evaluated against predetermined standards. Ultimately, the goal of app clearinghouses is to help consumers, healthcare providers, and healthcare organizations feel confident about their app selections by evaluating the usability, functionality, accuracy of the content, or evidence base supporting the app. Because of the lack of FDA regulation of most apps, and the burden associated with some of the other strategies we outline, such as reviewing the medical literature, clearinghouses hold much promise in helping providers and organizations select apps to recommend to their patients.

Table 2 describes several existing health app clearinghouses, including the website, a general description, and, importantly, their review standards and who conducts the evaluation. The information in Table 2 was abstracted from each clearinghouse’s website as of July 2014. We have also included points to consider when using clearinghouses to select health apps to recommend to their patients. The clearinghouses and websites listed were those known to us authors and/or discovered by searching Google and PUBMED (e.g. “mhealth app”; “mhealth applications”; “mhealth”; “mhealth clearinghouses”). We also reviewed mobihealthnews (http://mobihealthnews.com/). Given that these searches were conducted in 2014, our list of clearinghouses may not be exhaustive at the time of this publication as mhealth and clearinghouse development is constantly growing. Nevertheless, we encourage providers and organizations to consider the elements listed in Table 2 when searching any app clearinghouse not described here.

Table 2.

Available clearinghouses for selection of health mobile apps: description and review standards

Clearinghouse (website)	Target audience	Description	Review standards	Who reviews	Considerations
National Health Service (NHS) Health Apps Library (http://apps.nhs.uk)	UK (citizens, providers, organizations)	Offers a library of health apps for patients and aims to provide quality assurance to ensure that apps are clinically safe. Apps are organized into listings by Conditions (e.g., diabetes, insomnia), Healthy Living (e.g., alcohol, fitness, weight loss), Health Information (e.g., symptom checker), and Social Care (e.g. service finder). Each app’s information is presented by screenshots, a brief description from the developer, and consumer reviews. Consumers can also search for apps by phone platform and costs.	All submitted apps are evaluated to make sure they are: (1) relevant to those living in England. (2) comply with data protection laws, and (3) comply with trusted information sources. If these minimum requirements are met, NHS evaluates if a person’s health or condition could be harmed. A clinical team from NHS works with the app developer to make sure the app adheres to safety standards. Apps are also reviewed on an ongoing basis and/or if users flag concerns.	Reviewed by a NHS Clinical team that consists of doctors, nurses, and safety specialists. Consumers are able to rate and review apps on website.	The Health Apps library was launched in March 2013 and is still in development, therefore, there are relatively few apps that have been reviewed. Apps are submitted by app developers and may not include all relevant apps. The review process focuses on the clinical safety of app; evidence base for promoting or managing health is not applied.
Happtique (www.happtique.com)	USA—patients, physicians, and medical organizations	A mobile health application store and app management solution that allows organizations to create individually branded, secure, multi-platform application stores for staff and patients. Allows physicians to send their patients a “digital health prescription” with the apps, documents, and videos of their choice. Initially offered an app certification program based on a 19-page document specifying specific certification standards, but this was discontinued.	Includes apps that have been developed by a source Happtique considers reputable, including professional associations, universities/colleges, nonprofit associations, insurance companies, federal government, state government, or pharmaceutical industry. Provides an Engagement Score reflective of content validity, persuasion techniques, and psychosocial factors.	The content of apps are evaluated by internal partners and in partnership with medical professional societies. Technical aspects of evaluation are managed by a 3rd party developer. The company does not evaluate effectiveness or safety of the app for its intended use.	Provides an Engagement Score indicating an app’s potential to compel consistent use. However, it is unclear how the Engagement Score is calculated, as the process and algorithm is proprietary.
iMedicalApps (www.imedicalapps.com)	USA—physician, patients, and mHealth analysts	This is an independent online medical publication. Its goal is to provide reviews, research, and commentary of mobile medical technology. Consumers can read app reviews categorized by medical specialty and phone platform.	Reviews are based on the reviewers’ own hospital and clinical experiences. It is unknown whether each reviewer has similar specific standards when reviewing, though many reviews will include an overall summary that may consist of the following: app price, likes and dislikes, user interface, multimedia usage, and overall score.	Reviewers are 4 editors and 5 writers all of whom are physicians, physicians-in-training, allied health professionals, or mHealth analysts.	Reviews appear to be based on the opinions of the reviewer only. App reviews are categorized by medical specialty, and are primarily focused on medical education. Still, it is difficult to find apps for health behavior change (e.g., weight loss, smoking cessation, fitness/physical activity). May be beneficial to know the name of the app before using their search tool. Website is hard to navigate and has undergone several changes since its launch.
Eat Right (www.eatright.org/appreviews)	Worldwide	Sponsored by the world’s largest organization of food and nutrition professionals, Academy of Nutrition and Dietetics Reviewed apps are described as top-rated, free iPhone apps for those looking to lose weight, manage their diabetes, or eat gluten-free.	Reviewers write a brief evaluation based on (1) overall synopsis of the app, (2) pros and cons, and the (3) bottom line. Reviewers also rate apps from 1 to 5 stars.	The website specifically names 3 reviewers all registered dieticians.	Apps are limited to those regarding weight loss, diabetes management, or gluten-free diets only. Apps are limited to iPhone or iPhone compatible platforms; not useful for patients with Android phones or tablets.
IMS Health’s AppScript (http://www.imshealth.com/portal/site/imshealth/menuitem.3e17c48750a3d98f53c753c71ad8c22a/?vgnextoid=123ee61fec4b2410VgnVCM10000076192ca2RCRD&vgnextfmt=default)	Customers include pharmaceutical, medical device, and consumer health manufacturers and distributors, providers, payers, government agencies, policymakers, researchers, and the financial community.	IMS Health is an information, services, and technology company. Offers IMS Health AppScript, an electronic prescribing service for apps. Physicians are able to securely prescribe, reconcile, and track patient app use from a mobile device.	More than 40,000 healthcare apps from iOS and Android platforms, are assessed using their proprietary IMS Health App Score. The institute published a 65 page report that provides the results of an extensive content analysis of all apps as of June 2013. The overall IMS Health App Score is based on a combination of (1) functionality, (2) peer and patient reviews, (3) certifications, and (4) their potential to improve outcomes and lower the cost of care. Scoring criteria include, but are not limited to, the type and quantity of information provided by the app, how the app tracks or captures user data, the communication processes utilized by the app, and the quantity of device capabilities included in the app. In addition, weightings are applied specific to the category of the app. For example, in a condition management app there will be a higher value placed on results that focus on doctor communications than there would be for a reference/education app.	IMS	With regard to wellness, prevention, and treatment regiments, physicians can organize apps into formularies based on their patient’s needs. IMS Health App Scores are not readily available to the consumer and do not consider physician evaluation or efficacy evidence.
HealthTap’s AppRx (https://www.healthtap.com/)	USA—primarily for patients/consumers	This is a specific function of the larger HealthTap app. The AppRx function allows consumers to read health and medical app evaluation/recommendations from a network of 60,000+ physicians.	Doctors in the network review the apps based on 3 questions: (1) Is the app medically sound?, (2) Is the app useful?, and (3) Is the app easy to use and understand? Doctors have the option to write a review, but reviews do not enter the system until 30 reviews have been made and the HealthTap medical review board has approved them. Patients are also able to recommend the app using an “applaud an app” function.	Physicians from a network of >60,000 provide reviews, but it is unclear who are the physicians and if they all provide a similar review structure.	Have to download free HealthTap app and then create an account and log in in order to access app reviews.

Advantages of using clearinghouses include efficiency, with numerous apps summarized in one place, access to systematic evaluations, and, in some cases, continual updating as new apps become available. However, the use of clearinghouses also has limitations. The quality of the app reviews is dependent on the evaluation methods used by each clearinghouse. Reviewing apps can be resource intensive and time consuming, so the app listing may not reflect the range of available apps on the market at a given time. There is an inherent tradeoff between the thoroughness of review, timeliness of updates, and number of apps reviewed, and different clearinghouses may yield more useful recommendations when searching for different types of apps. For example, the UK’s National Health Service (NHS) Apps Library (http://apps.nhs.uk), launched in March 2013, aims to provide a library of apps endorsed by the NHS. The library started with 70 apps and 1 year later only had about 100 apps, which is few compared to the thousands of apps that are commercially available. The evaluation process can also be subject to error. Happtique (www.happtique.com), a mobile health application store, recently suspended their app certification program when independent groups found security flaws in apps that Happtique had “certified” as secure [21]. Finally, other app clearinghouses primarily focus on a particular health topic, like EatRight (Table 2). This clearinghouse may be quite useful for identifying an app to help patients lose weight but does not provide suggestions for apps to manage stress or help patients quit smoking. Still others are targeted primarily to the healthcare provider or organization only (e.g., IMS www.imshealth.com; Table 2) and are not readily available to patients. Recently, academic institutions or other nonprofit organizations have begun to offer listings of mhealth resources (e.g., http://www.zurinstitute.com/mentalhealthapps_resources.html) or apps for particular behaviors or health conditions (e.g., http://diabetes.ufl.edu/my-diabetes/diabetes-resources/diabetes-apps/). However, many of these listings may lack systematic independent reviews; a description of their search and vetting processes are unclear, as well as their process for updating their lists. Nonetheless, these listings may be helpful for providers when generating an initial pool of candidate apps to consider for further evaluation. In summary, providers and organizations should become familiar with clearinghouses, including their strengths and limitations. Given the differing purposes, review standards, reviewers, and strengths and weaknesses of each clearinghouse, providers and organizations are encouraged to search multiple clearinghouses for candidate apps.

Search app stores

If the scientific literature or clearinghouses do not list good-fit apps for a specific need, app stores can be searched directly. App stores are specific to the mobile device platform, with Apple apps available from iTunes and Android apps available from Google Play. Unfortunately, no method exists for searching all app stores together. The primary advantage of this strategy is its ease and relevance. It is the only way to identify the apps that are currently available to patients. As with other strategies, there are limitations. An important limitation is the lack of systematic evaluation of the apps, so the accuracy and evidence base are difficult to evaluate. Complicating the evaluation process, app stores rely on ranking algorithms such that apps that appear first are often those that are more popular. Thus, apps designed for smaller audiences, such as for specific diseases or behaviors that may be less common in the population, or those that are new to the market will be harder to find. Lower ranks in the app lists should not be viewed as synonymous with poor quality.

Search terms can be important when searching app stores. Entering a term that is too broad likely will result in much irrelevant information. For example, entering “alcohol” as a search term will lead to drinking games along with apps designed to promote abstinence [22]. Search terms aligned with the pathological state or disorder, or with the goal state, such as “alcoholism” or “alcohol abstinence,” are likely to reduce this outcome. However, even highly specific search terms can yield apps that are not a good fit because of the way apps are indexed, which is often determined by the app developer, and will include a broad list of terms to catch a broader consumer base. The term “alcohol abstinence,” for example, yields apps targeting smoking cessation as a result of the vagaries of indexing. Consequently, the initial resulting pool from these searches may need to be refined through additional search terms.

Review app descriptions, user ratings, and reviews

Any search strategy is likely to reveal more than one app that targets the behavior of interest. To narrow apps to one or two “best of breed,” the content of the resulting apps can be reviewed by reading its brief description in the app store, and, in some cases, one can test drive the app, such as many of those found through Amazon.com. In addition, while user ratings are generally not focused on the accuracy of the information or the evidence base supporting the app, they can be helpful in determining usability and functionality. Rating information generally includes the total number and average rating, typically as a number of stars ranging from 0 to 5. The number of ratings provides a relative measure of popularity and/or longevity of the app. This may be more important when a patient is considering paying for an app, as users may feel more comfortable purchasing a highly rated app that has been rated numerous times. In addition to the average rating, the distribution of ratings, which is typically presented, may also be useful. For example, an average rating of three stars might indicate a mediocre app (mostly three-star ratings), or it might represent an app with highly polarized ratings (many one-star and five-star ratings). In the latter case, information contained in reviews may indicate whether these low ratings were due to a glitch-prone update, which can be fixed, or truly polarized opinions of the app, which cannot be fixed. Changes in the distribution of ratings over time may indicate changes in functionality or interface over time, such as solving glitches or, in contrast, an unpopular update. One can start with the most popular apps in the health behavior domain, as these apps are likely to be already used by some of patients, and represent apps with longevity.

Healthcare providers and healthcare organizations may need to consider price; patients may be more willing to try a free app than an app that costs money. A recent review of weight loss mobile apps showed that paid apps did not include more evidence-based strategies than free apps [10]. While reviewing app descriptions, user ratings, and reviews has the benefit of convenience and ease, because the information is readily obtained and easily understood, the primary disadvantage associated with this strategy is that it does not consider the evidence base, validity, or accuracy of the app. In addition, even though the information is easily obtained, reviewing this information can be time consuming.

Conduct a social media query within professional and, if available, patient, networks

Although, as mentioned above, app clearinghouses provide structured evaluations and recommendations from providers and other regarded experts, healthcare providers and healthcare organizations may be able to leverage their professional social media networks to learn what their peers are recommending. Social media is increasingly used by healthcare providers to connect with colleagues. About 25 % of physicians use social media daily as part of their clinical practice [23]. A large medical community is represented on Twitter and hashtags used to identify these communities are archived at Symplur.com [24]. Physician-specific social networks like Sermo (www.sermo.com) might also be used to query colleagues about which apps they recommend to patients or elicit feedback about a specific app. Also, providers can search online social networks that patients use (e.g., Twitter) with the app hashtag as a keyword (e.g., #myfitnesspal) to observe conversations about the app. This will reveal if users are reporting generally positive or negative experiences and give a sense of whether the app is used widely. They can tweet any users they discover and ask for experiences using the app. This approach, may, however, oversample patients with positive experiences with a particular app, and may fail to elicit experiences that resulted in the patient no longer using the app for health promotion.

Pilot the apps

Pilot testing, or using an app to assess its features, functionality, usability, and content accuracy [25], can be completed by a healthcare provider or other designee, such as health informatics personnel. While piloting an app may seem laborious, many apps can be evaluated with only a few minutes of navigation, and it rarely takes more than 1 day of using an app before the major pros and cons are observed [26]. Providers should evaluate the candidate health app for the accuracy of the information provided, as well as the usability and inclusion of desired functionality. Desired functionality depends not only on the health problem or behavior (e.g., weight loss, smoking cessation) but also how the provider intends the patient to use the app. For example, an app for electronically tracking dietary intake as a replacement for paper records requires different functionality than an app that provides more comprehensive assistance with weight loss, which also generally entails increasing caloric expenditure.

Elicit feedback from patients

Once a provider has recommended an app to a patient or discovered that a patient is using an app to manage their health, follow-up is an important step. The provider should review the app with the patient and elicit feedback on its usefulness. Because individuals vary in their ability to evaluate an app, and different patients may find a particular app more or less useful depending on the exact nature of their health condition, behavior, and personal factors, we recommend that providers elicit feedback evaluations from multiple patients to increase confidence of an app’s potential benefit. Because the time spent with an individual patient is limited, questions about the app’s utility will need to be maximally informative, such as asking whether the patient found the app to be useful, whether he or she is continuing to use it, and whether the patient would recommend the app to others. Another more direct indicator of success is whether the patient has successfully changed his or her behavior or exhibits clinical improvement while using the app. Providers or perhaps larger organizations may even want to develop their own standard clearinghouses of apps, and then accumulate feedback from providers and patients as they use the apps to guide updates of their clearinghouses.

CASE STUDY

Dr. Benson treats patients with celiac disease and many of them express difficulty adhering to a gluten-free diet. Dr. Benson heard about diet mobile apps for self-management of a gluten-free diet but had no idea if these were safe and evidence-based. Table 1 summarizes our proposed framework for recommending apps for Dr. Benson’s patient. Her first step was to contact a medical librarian to help her search for systematic reviews and, if none, randomized clinical trials for dietary apps targeting a gluten-free diet. Unfortunately, no studies had evaluated such apps. She moved to another strategy of searching clearinghouses (Table 2). She began with EatRight.org, a website from the Academy of Nutrition and Dietetics, since this is a reputable nutrition professional organization. She identified the highest rated app on this website. Because EatRight.org has individual dietitians rate apps based on a scoring system with no stated criteria, she decided to cross check with iMedicalapps (www.imedicalapps.com), but this app is not in the iMedicalapps database. Her next strategy was to examine the user ratings in iTunes. Of 212 reviews, the average user rating of the app identified at EatRight.org is 1.5 out of 5 stars. She is concerned about such a low score and is now reticent to recommend this to her patients. The next highly rated app on EatRight.org is not in iTunes and the third highly rated app has no user ratings on iTunes. She has two choices, to select the fourth highly rated app on Eatright.org or proceed to investigate the most highly rated app further. She chose the latter.

Dr. Benson decided to tap her online social network. With hundreds of followers on Twitter, many of whom are physicians with specialty treating celiac disease, patients with celiac disease, and professional organizations dealing with celiac disease, she sent a tweet asking for recommendations and for experiences using this app. She performed a search on Twitter of the app name to see the conversations about this app. Her social media search revealed positive experiences and recommendations of the app, which was reassuring.

Dr. Benson decided to use the app for a day herself to evaluate its usability. She discovered no significant usability issues and the content was factually accurate so she recommended her patient pilot test it herself. She showed the patient how to find the app and suggested that it might be helpful in managing her gluten-free diet. Dr. Benson agreed to follow-up with the patient in a week to hear the pros and cons of using the app. The patient reported a positive experience with the app and her symptoms related to celiac decreased over time. Dr. Benson then felt comfortable recommending the app to other patients.

LIMITATIONS

The strategies outlined are not without limitations. The most important limitation is that some of the strategies may be impractical for individual providers, because they can be time consuming, especially if one approaches the seven strategies as a step-by-step plan. We encourage readers to view the strategies as a menu, rather than a stepwise approach, with providers or organizations selecting one or more strategies that are feasible in the context of their needs and constraints. Moreover, these strategies can be used by a group of individuals to spread the burden. For example, a task force or performance improvement team can be commissioned by a healthcare system to complete reviews of various clearinghouses and pilot the most highly ranked apps for a particular health behavior or disease. Another notable limitation includes costs. While many apps are free, some are not, which may hinder piloting the apps, or use by patients if such apps were recommended.

CONCLUSION

With thousands of health apps in the marketplace, healthcare providers and healthcare organizations need guidance on identifying apps that are effective, provide accurate information, and are user-friendly [6]. In addition, more primary research is needed to establish evidence for health apps efficacy. This research can begin with the very strategies outlined herein to help the researcher become better acquainted with the strengths and weaknesses of existing apps and to help guide the design of the next generation of health apps. The guidance in the current paper is similar to steps a provider or organization would use to identify evidence-based interventions of any kind, in that the research evidence is the first place to look, and then in the absence of evidence, defaulting to other reliable sources, case studies, and recommendations coming from professional networks. As the evidence base for health mobile apps continues to grow, guidelines and app clearinghouses will likely be put forth by professional organizations and identifying high-quality health apps will become much easier for providers. Building better app clearinghouses may represent the most important direction of the field, because it relieves some of the burden from the provider. Some of the strategies we describe, such as reviewing the empirical literature to identify evaluations of apps, may apply to building better clearinghouses that review apps not only for usability, privacy, security, and functionality but also on how strongly aligned the app is with the available evidence base. This will be particularly important for providers who are leery of prescribing apps to their patients because of concerns about validity and lack of sufficient regulation. The FDA does not regulate most of these types of apps, and the FDA’s website suggests that it does not intend to expand regulation to nonmedical device apps [27]. Consequently, barring a significant change in FDA policy, providers and healthcare organizations will have to use alternative methods of finding apps for use with their patients, such as clearinghouses. Finally, these strategies may also be appropriate for individual patients and caregivers. Patients and caregivers are likely to have unique insights into the behavior or disease area, and slight modifications to using these strategies may be necessary. For example, a patient/caregiver is likely to rely less on reviewing the published literature and rely more on searching for a credible source for the information and/or piloting the app to determine its utility and effectiveness.