Quality Assessment of Medical Apps that Target Medication-Related Problems

Abstract

BACKGROUND:

The advent of smartphones has enabled a plethora of medical apps for disease management. As of 2012, there are 40,000 health care-related mobile apps available in the market. Since most of these medical apps do not go through any stringent quality assessment, there is a risk of consumers being misinformed or misled by unreliable information. In this regard, apps that target medication-related problems (MRPs) are not an exception. There is little information on what constitutes quality in apps that target MRPs and how good the existing apps are.

OBJECTIVE:

To develop a quality assessment tool for evaluating apps that target MRPs and assess the quality of such apps available in the major mobile app stores (iTunes and Google Play).

METHODS:

The top 100 free and paid apps in the medical categories of iTunes and Google Play stores (total of 400 apps) were screened for inclusion in the final analysis. English language apps that targeted MRPs were downloaded on test devices to evaluate their quality. Apps intended for clinicians, patients, or both were eligible for evaluation. The quality assessment tool consisted of 4 sections (appropriateness, reliability, usability, privacy), which determined the overall quality of the apps. Apps that fulfilled the inclusion criteria were classified based on the presence of any 1 or more of the 5 features considered important for apps targeting MRPs (monitoring, interaction checker, dose calculator, medication information, medication record). Descriptive statistics and Mann-Whitney tests were used for analysis.

RESULTS:

Final analysis was based on 59 apps that fulfilled the study inclusion criteria. Apps with interaction checker (66.9%) and monitoring features (54.8%) had the highest and lowest overall qualities. Paid apps generally scored higher for usability than free apps (P = 0.006) but lower for privacy (P = 0.003). Half of the interaction checker apps were unable to detect interactions with herbal medications. Blood pressure and heart rate monitoring apps had the highest overall quality scores (67.7%), while apps that monitored visual, hearing, and temperature changes scored the lowest (35.5%).

CONCLUSIONS:

A quality assessment tool for evaluating medical apps targeting MRPs has been developed. Clinicians can use this tool to guide their assessments of medical apps that are appropriate for use in the health care setting. Although potentially useful apps were identified, many apps were found to have deficiencies in quality, among which was poor reliability scores for most of the apps. Continued assessments of the quality of apps targeting MRPs are recommended to ensure their usefulness for clinicians and patients.

What is already known about this subject

• Medication-related problems (MRPs) are events or circumstances involving medication therapy that can potentially or actually interfere with desired health outcomes, which, in turn, can lead to morbidity and mortality, as well as increased health care costs.

• As of 2012, there are approximately 40,000 health care apps available, 30% of which specifically target clinicians with little focus on app quality or ability to address MRPs.

What this study adds

• A quality assessment tool has been developed to help clinicians and medical app reviewers assess the quality of medical apps that target MRPs.

• This study showed that apps with medication interaction checkers and monitoring features had the highest and lowest overall qualities, respectively; reliability of apps was generally poor; and paid apps had better usability but poorer privacy scores.

• A recommended list of ranked apps is provided as a guide for clinicians to select or recommend an appropriate app in their clinical practices.

Smartphones and rapid development of mobile health care apps, together with advanced features such as Bluetooth, accelerometers, and 3G/4G networks, are currently supporting patient care activities in many health systems.^1-3 The use of such mobile technologies in health care provision has paved the way for a new domain known as mobile health, or mHealth.^1,2 Eighty-one percent of U.S. physicians own a smartphone and are incorporating their use in practice.^4-6 Moreover, 56% of the U.S. adult population was reported to own smartphone in 2013.⁷ Mobile subscriptions within the Asia-Pacific region account for more than half of the global subscription rate with 3.5 billion subscriptions.⁸ In Singapore, 75% of smartphone users utilize mobile apps, ranking Singapore fifth among 43 countries in terms of app usage.^9,10

As of 2012, there are approximately 40,000 health care apps available.⁴ Thirty percent of these apps specifically target clinicians and the remaining 70% target the general consumer.^4,11 Up to 63.5% of clinicians use apps in clinical practice and 1 in 5 people with smartphones download an app with a healthrelated purpose.^12,13 Little is known about the quality of mobile apps used by clinicians and patients in health care settings. Recently, medical app reviews have discussed the use of apps for medication information.^14,15 However, there is little focus on how these apps can address medication-related problems (MRPs).

MRPs are events or circumstances involving medication therapy that can potentially or actually interfere with desired health outcomes.^16,17 The Pharmaceutical Care Network Europe (PCNE) classifies MRPs as problems with treatment effectiveness, adverse drug events, increased treatment costs, and others. MRPs are associated with increased morbidity and mortality, as well as increased health care costs.^18-20 For example, it is estimated that MRPs cost the U.S. health system over USD$170 billion annually, with hospital admissions accounting for 69% of total costs.²¹ MRPs can be caused by prescribing or medicationuse errors and can be prevented with proper measures in place.¹⁷

While there are numerous healthrelated apps in the different app stores, it is hard to find scientific studies evaluating apps that target MRPs. In the context of patient care, such apps can serve as potentially useful tools for MRP prevention interventions. As such, the objective of this study was to develop a quality assessment tool to evaluate health care apps that target MRPs and provide a non-exhaustive recommended list of apps for clinicians in practice settings. The study focused on the 2 main operating systems, Android and Apple, since they account for over 90% of the market share.²²

Methods

Creation of Quality Assessment Tool

Articles from PubMed and the iMedicalApps website were analyzed to generate the evaluation criteria for the quality assessment tool.²³ The PubMed search included a list of medication-related and mobile technology-related MeSH terms and keywords. Articles were limited to the past 5 years, from July 2008 to October 2013, and filtered based on title and abstract. Articles were included if they were available in English and the mobile apps could be used to target MRPs based on the PCNE classification.¹⁷ Exclusion criteria were non-English articles, abstracts or full-texts that were unavailable, no discussion on mobile app usage, mobile technologies that utilized only short message services (SMSs) or voice calls, articles that focused only on the design and development of apps, and studies on older devices such as personal digital assistants. Additionally, app review articles published on iMedicalApps between January 2012 and November 2013 were analyzed for features of health care apps that could potentially target MRPs. A total of 70 articles were used to develop the quality assessment tool (Figure 1).

FIGURE 1.

Flowchart Showing the Process of Developing the Quality Assessment Tool

The full texts of the 70 articles were reviewed to extract potentially relevant quality assessment criteria for apps targeting MRPs. The criteria extracted from the articles were classified into 2 major categories of quality as “appropriateness of the app to serve its intended function” and “general quality.” The first category was based on the features of the apps that were discussed in the reviewed articles. The features of the apps were broadly categorized into monitoring, medication interaction checker, dose calculator, medication information, and medication records. Each app in the reviewed articles had 1 or more of these features. Criteria for evaluation of the quality of such features were listed under each section. Depending on the number of criteria for each feature, the scores for the quality of each feature varied. For example, the maximum score for the monitoring feature was 24, while for the medication record feature, it was 8. The general quality category included sections on the reliability, usability, and privacy of apps. The corresponding criteria were listed under each of these sections (Appendix A, available in online article). The quality assessment tool was developed in such a way that all apps were rated for their general quality out of a total of 38 points, whereas each app was assessed for the presence of features listed under the appropriateness category and corresponding criteria were used to evaluate the app (the maximum possible score in this section varied with the evaluated feature).

Selection of Apps and Process of Quality Assessment

The top paid and free medical app categories on iTunes and Google Play were searched to identify the first 100 apps in each category as ranked by the app stores themselves. The top 100 apps in each category were screened based on their title and description. Apps were included for assessment if they were available in English and contained 1 or more of the 5 features targeting MRPs (Figure 2). All eligible apps were downloaded to test devices for final evaluation. Android apps were evaluated on a Samsung Galaxy S3 smartphone (version 4.1.2) and Apple apps were evaluated on an iPhone 4s (version 7.0.3).

FIGURE 2.

Number of Apps Evaluated Using the Quality Assessment Tool

Upon download to the test devices, an app was first classified based on the presence of features for the purposes of monitoring treatment effectiveness, checking medication interactions, calculating doses, providing medication information, and keeping medication records. Once the app was identified to have any 1 or more of these features, the corresponding assessment criteria under the section of “appropriateness of feature to serve its intended function” were employed to evaluate its quality. Accordingly, the maximum possible score for an app having any of the features could vary from 46 for the medication record feature to 62 for the monitoring feature. The maximum possible scores for apps with a dose calculator, medication interaction checker, and medication information features were 49, 56, and 60, respectively. App comparisons were made at the level of each feature. To facilitate comparison among apps with different features, percentages of the raw scores of the apps on the assessment tool were calculated. The overall quality of the apps was then calculated as a percentage based on the composite scores for each section (appropriateness, reliability, usability, and privacy).

Statistical Analysis

Apps were grouped according to which MRPs they could target based on their descriptions in the app stores. Descriptive statistics for each feature were computed based on the apps' feature scores on the quality assessment tool and reported as medians and interquartile ranges (IQRs). A Mann-Whitney U test was used to evaluate the differences in scores between free and paid apps and Apple and Android apps. All tests were 2-tailed and P-values below 0.05 were considered statistically significant. Statistical analyses were conducted using Statistical Package for the Social Sciences (SPSS) version 21 (IBM Corp, Armonk, NY).

Results

A total of 400 apps were screened for inclusion in the final evaluation based on review of their app store description pages. Only 59 (14.8%) apps fulfilled the inclusion criteria, of which 5 were available on both platforms. There were 27 apps with monitoring features, 17 apps with medication information features, 14 dose calculators, 8 apps with medication record features, and 6 medication interaction checkers. Six apps contained multiple features and were evaluated according to their respective feature sections (Figure 2).

Apps with medication interaction checkers had the highest overall quality (66.9%; IQR = 18.3%), while apps with monitoring features had the lowest overall quality (54.8%; IQR = 19.4%). Paid Apple apps had higher overall quality compared to free apps (58.1% vs. 55.8%; P = 0.962). Similarly, paid Android apps also had higher overall quality (59.4% vs. 59.1%; P = 0.937). Free and paid apps across both platforms had equally poor reliability (25% each; IQR = 29.2% vs. 18.8%; P = 0.480). Paid apps generally scored higher for usability (79.6% vs. 68.2%; P = 0.006) but lower for privacy (25.0% vs. 50.0%; P = 0.003) compared to free apps. This was a significant trend for Android apps, but not Apple apps (Table 1).

TABLE 1.

Median General Quality Scores of Apple and Android Apps: Free and Paid Versions

Quality Scores (Total Points)	Apple			Android
	Median Score (%, IQR)		P Value	Median Score (%, IQR)		P Value
	Free (n = 16)	Paid (n = 14)		Free (n = 13)	Free (n = 16)
Reliability (12 points)	3/12 (25.0, 37.5)	3/12 (25.0, 25.0)	0.782	3/12 (25.0, 29.2)	3/12 (25.0, 16.7)	0.516
Usability (22 points)	15/22 (70.5, 12.6)	17.5/22 (79.6, 15.9)	0.167	14/22 (63.6, 18.2)	18/22 (81.9, 21.6)	0.012b
Privacy (4 points)	2/4 (50.0, 75.0)	0.5/4 (12.5, 50.0)	0.072	2/4 (50.0, 0.0)	1/4 (25.0, 50.0)	0.012b
General qualitya (38 points)	20/38 (52.6, 16.5)	21/38 (55.3, 17.1)	0.786	21/38 (55.3, 18.5)	23/38 (60.5, 14.5)	0.243

^a General quality scores are the sum of the reliability, usability, and privacy scores.

^b Statistical significance of P < 0.05 for usability and privacy scores between Android free and paid apps.

IQR = interquartile range.

Assessment of Feature Scores

Monitoring.

Paid apps with monitoring features had higher overall quality scores than free apps across both platforms, but the score difference for Android apps was statistically significant (65.3% vs. 53.2%, respectively; P = 0.032; Table 2). The Cancer.Net Mobile app ranked highest (79.0%), followed by Blood Pressure Diary Pro, HeartWise Blood Pressure Tracker, and iBP Blood Pressure (67.7% each; Table 3). Other than the Cancer.Net Mobile app, the top-scoring apps based on their overall quality scores were those that monitored blood pressure and heart rate (66.1%-67.7%), while apps that monitored visual, hearing, and temperature changes scored the lowest (30.6%-38.7%). The appropriateness and usability scores for the top 4 apps were similar, but Cancer.Net Mobile performed better for reliability and privacy (Appendix B, available in online article). A majority of the apps did not allow users to set target goals for physiological parameters such as blood glucose, body temperature, and heart rate (n = 20) nor provide a management plan when an abnormal measurement was recorded (n = 23), thus affecting their appropriateness scores. Only 3 apps (Blood Pressure Diary, Blood Pressure Diary Pro, and iBP Blood Pressure) allowed syncing of data from the blood pressure monitoring device to the app via Bluetooth.

TABLE 2.

Median Overall Quality and Appropriateness Scores of Apple and Android Apps: Free and Paid Versions

Feature	Overall Quality Score			Appropriateness Score
	Median Score (%, IQR)[n]		P Value	Median Score (%, IQR)[n]		P Value
	Free	Paid		Free	Free
Apple platform
Monitoring	31/62 (50.0, 19.4) [n = 9]	34/62 (54.8, 21.0) [n = 9]	0.605	12/24 (50.0, 33.3) [n = 9]	15/24 (62.5, 33.3) [n = 9]	0.423
Medication interaction checker	39/56 (70.5, NA) [n = 2]	NA (NA, NA) [n = 0]	NA	12/18 (66.7, NA) [n = 2]	NA (NA, NA) [n = 0]	NA
Dose calculator	29/49 (59.2, 16.3) [n = 4]	28/49 (56.1, NA) [n = 2]	0.800	5/11 (40.9, 47.7) [n = 4]	6/11 (50.0, NA) [n = 2]	0.812
Medication information	38/60 (63.3, 22.7) [n = 5]	35/60 (58.3, NA) [n = 2]	0.857	14/22 (63.6, 22.7) [n = 5]	12/22 (54.6, NA) [n = 2]	0.693
Medication record	20/46 (42.4, NA) [n = 2]	NA (NA, NA) [n = 1]	0.667	4/8 (43.8, NA) [n = 2]	NA (NA, NA) [n = 1]	0.221
Android platform
Monitoring	33/62 (53.2, 25.0) [n = 5]	40/62 (65.3, 6.5) [n = 4]	0.032a	13/24 (54.2, 37.5) [n = 5]	15/24 (62.5, 20.8) [n = 4]	0.217
Medication interaction checker	36/56 (64.3, 17.4) [n = 4]	NA (NA, NA) [n = 0]	NA	14/18 (77.8, 31.9) [n = 4]	NA (NA, NA) [n = 0]	NA
Dose calculator	29/49 (59.2, NA) [n = 3]	25/49 (51.0, 12.2) [n = 5]	0.071	6/11 (54.6, NA) [n = 3]	4/11 (36.4, 40.9) [n = 5]	0.365
Medication information	36/60 (60.0, 31.7) [n = 7]	31/60 (51.7, NA) [n = 3]	0.667	15/22 (68.2, 59.1) [n = 7]	10/22 (45.5, NA) [n = 3]	0.563
Medication record	NA (NA, NA) [n = 1]	29/46 (64.1, 8.7) [n = 4]	0.400	NA (NA, NA) [n = 1]	5/8 (56.3, 21.9) [n = 4]	0.709

^a Statistical significance of P < 0.05 in overall quality score between Android free and paid apps.

IQR = interquartile range; NA = not applicable.

TABLE 3.

Top 3 Apps Within Each Feature Category Based on Overall Quality Scores

Feature	App Name	Rank	Overall Quality Score (%)	Platform (Version)
Monitoring	Cancer.Net Mobile	1	79.0	Apple (free)
	Blood Pressure Diary Pro	2	67.7	Android (paid)
	HeartWise Blood Pressure Tracker			Apple (paid)
	iBP Blood Pressure			Apple (paid)
	iBP Blood Pressure	3	66.1	Android (paid)
	Blood Pressure Monitor – Family Lite			Apple (free)
Medication interaction checker	Medscape	1	76.8	Android (free)
				Apple (free)
	Skyscape Medical Resources (RxDrugs)	2	69.6	Android (free)
	Epocrates	3	64.3	Apple (free)
Dose calculator	Medscape	1	75.5	Android (free)
				Apple (free)
	MedCalc	2	67.3	Apple (paid)
	Epocrates	3	61.2	Apple (free)
Medication information	Medscape	1	71.7	Apple (free)
				Android (free)
	Epocrates			Apple (free)
	Epocrates	2	66.7	Android (free)
	Lippincott Nursing Drug Handbooks	3	63.3	Apple (free)
	MIMS			Apple (free)
Medication record	Med Helper Pro Pill Reminder	1	67.4	Android (paid)
	Pillboxie			Apple (paid)
	My Medical Info	2	65.2	Android (paid)
	My Medical	3	63.0	Android (paid)

Medication Interaction Checker.

All the evaluated apps in this category were free apps (Table 2). The top app based on overall quality was Medscape (76.8%), while the Drugs.com Medication Guide scored the lowest (57.1%). The Apple version of Epocrates had a higher score than its Android counterpart due to better usability (score: 17/22 vs. 14/22, respectively; Appendix B). A majority of the apps (n = 4) were unable to provide users with a list of all possible interactions associated with a single medication, and half of the apps (n = 3) were also unable to detect interactions with herbal medications.

Dose Calculator.

In general, free apps scored higher for overall quality compared to paid apps (Table 2). The highest-scoring apps based on overall quality were Medscape (75.5%), MedCalc (67.3%), and Epocrates (Apple version: 61.2%; Table 3). Epocrates performed better for overall quality on Apple compared to Android platforms (61.2% vs. 57.1%) due to better usability (score: 17/22 vs. 14/22, respectively). Similarly, the Android version of Calculate by QxMD also fared better than its Apple counterpart due to slightly better usability (score: 16/22 vs. 15/22, respectively; Appendix B). Most apps in this category were unable to calculate dose requirements for antibiotics (n = 10) and did not provide any administration instructions after calculation of doses (n = 13).

Medication Information.

Free apps in this category had higher overall quality and appropriateness scores compared to paid apps (Table 2). The top-ranking apps were Medscape and Epocrates (Table 3). The Apple version of Epocrates scored better for usability than the Android version (score: 17/22 vs. 14/22, respectively), thus affecting its overall quality scores and ranking (Appendix B). Although apps that were among the top 3 ranks had similar appropriateness scores, most of them did not allow for identification of medications based on their physical features (e.g., shape, color, imprint; n = 14). Furthermore, most apps did not provide supporting references for the content provided (n = 13).

Medication Record.

Paid apps in this category were ranked higher in terms of overall quality. The top-ranked apps were Med Helper Pro Pill Reminder and Pillboxie (67.4% each), both of which were paid apps (Table 3). The appropriateness and usability scores were similar for both apps, but Pillboxie had a higher reliability score of 6/12 in contrast to Med Helper Pro Pill Reminder, which had a higher privacy score of 4/4 (Appendix B). More than half of the apps were unable to provide medication reminders or keep track of users' medication schedules (n = 5). Most apps were also unable to alert users if an allergy-causing medication was among their active list of medications (n = 7).

Discussion

According to the recently released U.S. Food and Drug Administration (FDA) guidelines, regulatory oversight is applied to mobile apps that satisfy the definition of a medical device. Depending on the level of risk the use of a mobile medical app poses to a patient's safety, the FDA may require app developers to apply for premarket notification (510k) or premarket approval.^24,25 However, a majority of the healthrelated apps in the app stores do not fall under the FDA's regulatory oversight. This can be a concern for the increasing number of consumers who download these medical apps. The quality assessment tool developed in this study is to serve as a guide for clinicians to assess the quality of the medical apps in the market before using them in their practice settings or recommending them to patients. Although non-exhaustive, this study identified mobile apps that can be potentially useful for both patients and clinicians in addressing MRPs. The study also identified important deficiencies of currently available apps targeting MRPs, which can serve as an input for future improvements.

Paid apps with the monitoring feature are of higher overall quality than their free versions, in contrast to the other evaluated app features. We postulate that the higher price of these apps may be linked to technical complexity of this feature (e.g., Bluetooth syncing with external devices, data storage, and trending capabilities) and the greater effort put in by developers. Clinicians may choose to use or recommend to patients paid versions of monitoring apps but can also consider the free versions of apps with other features.

Free apps scored higher for privacy than paid apps in this study. Our results were consistent with a recent survey that showed that free apps were more likely than paid apps to include privacy policies.²⁶ A reason could be that most paid apps were usually developed by individuals rather than organizations, hence they are less likely to take user's privacy into account during app development.²⁷ However, a clearly stated privacy policy on how user information is handled is important to safeguard the privacy of users, which is critical for handling confidential information in certain apps, such as medical record apps.

In general, the reliability scores of apps assessed in this study were low regardless of platform or free/paid versions. In order for apps to be current and reliable, they should be validated regularly either by quality assessment tools, such as the one in this study, or by third-party certifications. For example, Cancer.Net Mobile was the only app that was certified by the Health On the Net Foundation (http://www.hon. ch/HONcode/), thus scoring the highest (10/12, 83.3%) for its reliability. Such evaluations can help filter out useful and reliable apps from app stores.

It was interesting to note that the overall quality of the Apple and Android versions of some apps were different. For example, the Android version of Epocrates contained an additional intravenous-drip rate calculator that was not available on its Apple version. Similarly, iBP Blood Pressure on the Apple platform was able to sync with an external monitoring device for automatic transfer of readings, but not on Android. Therefore, clinicians should be aware that there might be slight differences in the functionalities of the apps when used on different platforms.

A list of recommended apps within the top 3 ranks for each feature category based on their overall quality scores is provided in Table 3. The recommended apps in this list are consistent with those provided by reviewers from the iMedicalApps website (e.g., Medscape and MedCalc).²⁸ However, the difference is that our study classifies the apps into feature categories that target MRPs. In addition, this study assesses the quality of apps with monitoring and medication record features, which, from our knowledge, are not evaluated by iMedicalApps. Furthermore, this study assesses the quality of the apps objectively through a scoring system, which can complement the recommendations provided by the iMedicalApps reviewers based on their experiences with the apps.

Limitations

Owing to the ever-changing nature of information, some of the questions in the quality assessment tool may be time-sensitive. For example, content evaluation of apps with the medication information feature was based on the recent black box warning for carbamazepine to conduct genetic screening in Asian patients who use this medication due to the possibility of Stevens-Johnson syndrome and toxic epidermal necrolysis.²⁹ Clinicians who intend to use this tool for evaluation of their apps will need to update the content based on their area of specialization.

The apps evaluated were only on the Apple and Android platforms, so the external validity of using this assessment tool across other platforms, such as Windows and Blackberry, is uncertain. This study's results show that the same apps on different mobile platforms are likely to achieve similar scores for appropriateness, reliability, usability, and privacy, suggesting that the quality of apps across other platforms is likely to be similar. It is likely possible to apply this assessment tool across other platforms with minor adjustments.

Finally, the apps evaluated in this study represent a snapshot of the current medical apps that were available during the period of the study. Apps may be removed or updated with time, so the list of recommended apps may change in the future. There is a need to frequently assess the quality of apps to ensure that they remain within certain standards. It is also important to note that user preferences and experiences with the apps will determine whether they are going to continue using recommended apps. Despite the inclusion of criteria to evaluate issues of usability in the quality assessment tool, actual user experiences may have significant influence on the adoption of an app. While this study did not assess if the use of such apps could really reduce or prevent MRPs in clinical practices, future studies can possibly be conducted to determine if there is indeed a role for using medical apps that target MRPs in the health care setting.

Conclusions

A quality assessment tool has been developed to assess the quality of medical apps that target MRPs. This study showed that apps with medication interaction checkers and monitoring features had the highest and lowest overall qualities, respectively. Reliability of apps was generally poor. Paid apps had better usability but poorer privacy scores. Apart from Android apps with monitoring features, the free/paid status of an app generally did not reflect its overall quality. We hope that this tool can assist clinician users and medical app reviewers assessing the quality of apps from a more objective approach. Potential developers of health care apps can also use this tool as a guide to create apps of a better quality standard for appropriate use in the health care setting. The recommended list of ranked apps provided can be utilized as a guide for clinicians to select or recommend an appropriate app in their clinical practices.

APPENDIX A. Quality Assessment Tool for Evaluating Medical Apps

Section A (Assesses the appropriateness of app to serve its intended function for targeting medication-related problems) i. Monitoring Feature (This section assesses an app’s feature that allows users to monitor or record the efficacy/toxicity of medications in terms of improvement in symptoms or adverse effects)
Maximum Possible Score: 24
1. Can app monitor more than 1 physiological parameter (e.g., blood glucose, body temperature, heart rate)?     (0) No, only 1 physiological parameter can be monitored     (1) Yes, more than 1 physiological parameter can be monitored
2. Does app allow users to set target goals for physiological parameters (e.g., blood glucose, blood pressure)?     (0) No     (1) Yes
3. Does app allow recording of symptoms?     (0) No symptoms can be recorded     (1) Only subjective symptoms can be recorded     (2) Objective symptoms can be recorded
4. Can an external monitoring device (e.g., blood pressure monitor, glucometer) sync with the app to record physiological readings?     (0) Yes (Proceed to Q5)     (1) No (Skip Q5)
5. Can the monitoring device automatically transfer readings to the app in real time?     (0) No, manual entry of data by user is required     (1) Yes, data are automatically transferred but at a later time (when user syncs)     (2) Yes, data are automatically transferred to app in real time (by wireless networks/Bluetooth)
6. Does the app need to be used in a controlled environment (e.g., quiet place with no interference)?     (0) Yes     (1) No
7. Does the app require a health care professional to perform the monitoring for patients?     (0) Yes, a health care professional is required     (1) No, monitoring can be performed by patients themselves
8. Is there a specific section that allow recording of medication(s) taken?     (0) No, medication cannot be recorded     (1) Yes, but only medication name can be recorded     (2) Yes, able to record medication name + dose or timing/frequency     (3) Yes, able to record medication name + dose + timing/frequency
9. Does app organize patient’s health monitoring input (e.g., vital signs) in chronological order?     (0) No     (1) Yes
10. Is there a feature to display trends of monitored parameter(s) (e.g., graph)?     (0) No     (1) Yes
11. Does app allow users to display a saved history of measurement results?     (0) No, there is no display of results/user cannot select a time frame for display of results     (1) Yes, but display is based only on a limited number of recordings (e.g., past 10 readings)     (2) Yes, display can be based on a selected time frame (e.g., weeks/months/years)
12. Does app classify severity of medical condition(s) based on pre-set cutoff measurements?     (0) No, measurements are not classified into severity     (1) Yes, but it is not clearly stated whether classification is supported by evidence     (2) Yes, it is clearly stated that classification is based on evidence/clinical guidelines
13. Does app alert users on abnormal readings that are out of the normal range?     (0) No, users are not alerted     (1) Yes, abnormality is indicated qualitatively (e.g., highlighted using color code)     (2) Yes, abnormality is indicated quantitatively (e.g., high/low or normal range is provided to user)
14. Does app provide a management plan for an abnormal measurement?     (0) No, there is no management plan provided     (1) Yes, only a general management plan is provided     (2) Yes, a patient-specific management plan is provided
15. Can a password/pin be set to access the recorded data/parameters?     (0) No     (1) Yes
16. Does the app allow monitored data to be backed up/stored?     (0) No, data cannot be stored     (1) Yes, but data are non-transferable (e.g., stored on device itself)     (2) Yes and data are transferable (e.g., to an external storage device, cloud)
ii. Medication Interaction Checker Feature (This section assesses an app’s feature that has the ability to perform interaction checks between 2 or more medications)
Maximum Possible Score: 18
1. Does app allow medication to be searched by different names?     (0) No search function is provided     (1) Yes, either generic or brand name can be searched     (2) Yes, both generic and brand name can be searched
2. Does app display a list of possible interactions associated with a single medication?     (0) No     (1) Yes
3. Does app detect multiple drug-drug interactions concurrently?     (0) No, only 1 pair of drug-drug interaction can be detected at any 1 time     (1) Yes, multiple drug-drug interactions can be detected at any 1 time
4. Can the app detect interactions with complementary and alternative/herbal medicines?     (0) No     (1) Yes
5. Which of the following details other than the interaction effect(s) are provided: severity of interaction(s), mechanism(s) of interaction, and management plan(s)?     (0) Only 1 of those above is provided     (1) 2 of the above are provided     (2) All 3 details are provided
6. Does app categorize interactions according to severity?     (0) No, app does not classify severity/classification rubric is not available     (1) Yes, classification rubric is provided but information is not supported by evidence/references     (2) Yes, classification rubric is provided and information is supported by evidence/references
7. Does app provide details on the comprehensiveness of the interaction database (e.g., number of medications/interactions detected by the database)?     (0) No details are provided     (1) App is a standalone with no information on the comprehensiveness of the database provided     (2) App is a standalone and information on the comprehensiveness of the database is provided     (3) App follows a parent database but no information on the comprehensiveness of the database is provided     (4) Yes, app follows a parent database and comprehensiveness of database is provided
8. Does app allow interaction checks against a previously saved medication list?     (0) No     (1) Yes
9. Are there spelling or grammatical errors in the interaction content?     (0) Major errors that compromise the level of understanding     (1) Some errors but they do not compromise the level of understanding     (2) No errors present
10. Is the content of the app reviewed?     (0) Not reviewed/not stated     (1) Yes, reviewed internally     (2) Yes, reviewed externally
iii. Dose Calculator Feature (This section assesses an app’s feature that is used explicitly for calculating medication doses)
Maximum Possible Score: 11
1. Can app perform dose calculation based on weight of patient?     (0) No     (1) Yes
2. Does app include an antibiotic dose calculator?     (0) No     (1) Yes, but app does not adjust for renal/liver dysfunction in the calculation     (2) Yes, and app takes into account renal/liver dysfunction in the calculation
3. Does app include an IV drip rate calculator?     (0) No, app is unable to calculate drip rate for IV medications     (1) Yes, app can calculate drip rate for IV medications
4. Can app calculate dose equivalent conversion(s) of medications within the same pharmacologic class where applicable (e.g., statins)?     (0) No, dose conversion(s) cannot be performed     (1) Yes, dose equivalents can be calculated
5. Does app provide a formula and/or explain how calculation is derived?     (0) No information is provided     (1) Yes, either formula or explanation for calculation is derived
6. Once calculation has been performed, is administration instruction provided?     (0) No, administration instruction is not provided     (1) Yes, administration instruction is provided
7. Are the units of medication(s) consistent throughout the calculation     (e.g., if unit for input are in mg, does final calculation show strength in mg/mL)?     (0) No     (1) Yes
8. Is the content of the app reviewed?     (0) Not reviewed/not stated     (1) Yes, reviewed internally     (2) Yes, reviewed externally
9. Is the formula used supported with reference citations/evidence?     (0) No     (1) Yes
iv. Medication Information Feature (This section assesses an app’s medication database on whether it is able to provide clinically relevant information about medications)
Maximum Possible Score: 22
1. Does the app have a search box to search for information on specific drugs/medications?     (0) No     (1) Yes
2. Can the drug/medication be searched based on multiple categories (generic name, brand name, disease/medical condition)? (For example, generic name: carbamazepine, brand name: Tegretol, medical condition: neurologic disorder)     (0) No, the drug can only be searched by 1 category     (1) Yes, the drug can be searched by at least 2 out of 3 categories
3. From the search results, are the core information (medication, indication, dosing instruction, contraindication, and side effects) and supplementary information (precaution, drug interaction) provided?     (0) Not all the core information (listed above) is provided     (1) All the core information (listed above) is provided, without any supplementary information     (2) Yes, all the core information and any supplementary information (listed above) are provided
4. Does medication reference provide information if the medication can be purchased over the counter or requires a prescription?     (0) No     (1) Yes
5. Does app include information for special populations (e.g., pediatrics, pregnancy, renal/liver dysfunction)?     (0) No     (1) Yes
For questions 6-8, refer to the following black box warning about the pharmacogenomic effects of carbamazepine (Tegretol):
Black box warning: Serious dermatologic reactions and HLA-B*1502 allele
• Serious and sometimes fatal dermatologic reactions, including toxic epidermal necrolysis (TEN) and Stevens-Johnson syndrome (SJS), have been reported during treatment with Tegretol.
• These reactions are estimated to occur in 1 to 6 per 10,000 new users in countries with mainly Caucasian populations, but the risk in some Asian countries is estimated to be about 10 times higher.
• Studies in patients of Chinese ancestry have found a strong association between the risk of developing SJS/TEN and the presence of HLA-B*1502, an inherited allelic variant of the HLA-B gene. HLA-B*1502 is found almost exclusively in patients with ancestry across broad areas of Asia.
• Patients with ancestry in genetically at-risk populations should be screened for the presence of HLA-B*1502 prior to initiating treatment with Tegretol.
• Patients testing positive for the allele should not be treated with Tegretol unless the benefit clearly outweighs the risk.
6. Does medication reference classify the above information for carbamazepine (Tegretol) as a black box warning?     (0) No     (1) Yes
7. How comprehensive is the information provided by the app on pharmacogenomic and serious dermatologic effects of carbamazepine?     (0) None of the information is provided     (1) Some of the information is provided     (2) All of the information is provided
8. Does app provide relevant medication-related resources for specific drug classes, where applicable (e.g., antibiotic susceptibility charts; dose-conversion charts for cholesterol-lowering statins, steroids, or opioids)?     (0) No     (1) Yes
9. Does app contain a pill identifier feature?     (0) No, there is no such feature     (1) Yes, but feature does not include pictures of medications     (2) Yes, feature includes pictures of medications
10. Does app provide details on the comprehensiveness of the database (e.g., number of medications in the database)?     (0) No details are provided     (1) App is a standalone with no information on the comprehensiveness of the database provided     (2) App is a standalone and information on the comprehensiveness of the database is provided     (3) App follows a parent database but no information on the comprehensiveness of the database is provided     (4) Yes, app follows a parent database and comprehensiveness of the database is provided
11. Is there a summary of the evidence on a medication’s therapeutic use with supporting reference citations?     (0) No evidence summary or supporting references are provided     (1) Yes, either the evidence summary or supporting references are provided     (2) Yes, both the evidence summary and supporting references are provided
12. Are there any spelling or grammatical errors?     (0) Yes, there are major errors that compromise the level of understanding     (1) Yes, there are some errors but they do not compromise the level of understanding     (2) No, there are no errors present
13. Is the content of the app reviewed?     (0) Not reviewed/not stated     (1) Yes, reviewed internally     (2) Yes, reviewed externally
v. Medication Records Feature (This section assesses an app’s feature that allows users to record their current medication therapy)
Maximum Possible Score: 8
1. Does app allow all of the following patient’s biodata (age, weight, allergy, medical history, medical condition) and current medications to be input?     (0) Not all of the above mentioned data can be input     (1) Both patient’s biodata and current medications can be input
2. Does app allow users to edit their medication list?     (0) No, not at all     (1) Yes, users can add/remove medication according to their prescription
3. Is there an adherence and reminder feature to allow users to record when their medication was taken?     (0) No, there is no such feature available     (1) Yes, there is a recording function but no medication reminder is available     (2) Yes, both the recording and medication reminder functions are available
4. Is the app able to provide/allow users to add basic therapeutic information regarding the medication?     (0) No     (1) Yes
5. Can app alert user on medication allergy(s)/interaction(s) from the user’s active medication list?     (0) No     (1) Yes
6. Can a password/pin be set to access patient’s medication records?     (0) No     (1) Yes
7. Does app allow data to be stored externally?     (0) No, data are non-transferable (e.g., stored on device itself)     (1) Yes, data are transferable (e.g., external storage device, cloud)
Section B (Reliability) (This section assesses if an app is reliable and unbiased for users to utilize for medication-related purposes. Most of the requirements to determine reliability can be found in the “About” section within the app)
Maximum Possible Score: 12
1. Is the target audience for app use clearly defined/stated?     (0) No     (1) Yes
2. Does section icon/heading/title accurately reflect content within section?     (0) No     (1) Yes
3. Are the names and credentials of the app creators/authors stated?     (0) Both not mentioned     (1) Either name(s) or credential(s) indicated     (2) Both name(s) and credential(s) indicated
4. Does app contain a disclaimer (or a statement of similar implication) that the information provided/content will not replace a health care professional’s judgment?     (0) No     (1) Yes
5. Are the sources of funding clearly stated?     (0) Not clearly stated     (1) Yes, clearly stated
6. Are there any conflicts of interests declared?     (0) No declaration stated     (1) Yes, stated clearly     (2) No conflicts of interest exist and this is stated clearly
7. Is there any evidence of bias (e.g., advertisements) present?     (0) Yes     (1) No
8. Is there a third-party certification by a medical or governmental organization?     (0) No certification present     (1) Certification present
9. Is the date of last update stated?     (0) Not stated     (1) Only a states how often app is updated without providing actual date of last update/only provides month and year of update     (2) Yes, actual date of last update is stated
Section C (Usability) (This section assesses if an app is easy to use, provides user support, and is able to cater to the needs of different users)
Maximum Possible Score: 22
1. Is the app intuitive to use?     (0) No     (1) Yes
2. Are there any user interface problems that compromise the ease of navigation within the app?     (0) Yes, to a large extent that it compromises the ease of navigation     (1) Yes, but only to a small extent that does not compromise the ease of navigation     (2) No problems present
3. Is the screen layout well organized/not cluttered?     (0) No     (1) Yes
4. Is the font of a readable size by default?     (0) Not readable by default (can only be viewed on a larger screen)     (1) No, but user can zoom in and panning is required     (2) Yes
5. Are the terminologies understandable by a layperson?     (0) Not at all     (1) Sometimes     (2) Most of the time     (3) All the time
6. Is there a help feature present?     (0) No     (1) Yes
7. Can features of the app be accessed from any page?     (0) No, features can only be accessed from main home page     (1) Yes
8. Does the app require an internet connection to get started?     (0) Yes, the pages will only load with an internet connection     (1) No, I can still retrieve information that I downloaded earlier
9. Are there any broken links?     (0) Many broken links     (1) Few broken links     (2) No broken links
10. Is the graphical user interface adaptable to both portrait/landscape view when the phone is rotated?     (0) No     (1) Yes
11. Is app customizable based on user preference?     (0) Not customizable at all     (1) Yes, but to a limited extent     (2) Yes, mostly customizable
12. Can users bookmark pages for quick access?     (0) No/bookmark feature is unavailable     (1) Yes
13. Are the contact details of developer/feedback section provided?     (0) No     (1) Yes
14. How often does the app crash/exit unexpectedly?     (0) Frequently     (1) Occasionally     (2) Rarely     (3) Never
Section D (Privacy) (This section assesses the privacy protection of user input data; i.e., data security)
Maximum Possible Score: 4
1. How is the privacy policy stated on the app?     (0) No privacy policy present at all     (1) Privacy policy mentions the collection of user data only     (2) Privacy policy mentions the collection of user data and how it is being used
2. Can users choose what content to share?     (0) Users cannot choose what content to share/content cannot be shared at all     (1) Users can choose what content to share
3. Can users choose who to share content with through electronic means?     (0) No feature that allows sharing of content exists/users cannot choose who they want to share content with     (1) Yes, there is a feature that allows users to select a person whom they want to share content with

APPENDIX B. Ranking of Apps Across All Platforms and Versions According to Overall Quality Scores

App Name	Rank	Overall Quality Score (%)a	Appropriateness Score	Reliability Score	Usability Score	Privacy Score	Platform	Version
Apps with Monitoring Feature		Maximum score: 62	Maximum score: 24	Maximum score: 12	Maximum score: 22	Maximum score: 4
Cancer.Net Mobile	1	49 (79.0)	15	10	20	4	Apple	Free
Blood Pressure Diary Pro	2	42 (67.7)	18	3	19	2	Android	Paid
HeartWise Blood Pressure Tracker		42 (67.7)	15	5	21	1	Apple	Paid
iBP Blood Pressure		42 (67.7)	18	3	19	2	Apple	Paid
iBP Blood Pressure	3	41 (66.1)	16	3	20	2	Android	Paid
Blood Pressure Monitor - Family Lite		41 (66.1)	18	2	19	2	Apple	Free
Manage My Pain Pro	4	40 (64.5)	12	5	19	4	Android	Paid
Blood Pressure Diary	5	39 (62.9)	15	3	19	2	Android	Free
Blood Pressure Companion	6	38 (61.3)	16	2	18	2	Apple	Paid
iHeart - Pulse Reader		38 (61.3)	18	3	17	0	Apple	Paid
Blood Pressure (My Heart)	7	37 (59.7)	15	2	18	2	Android	Free
Headache Diary Pro		37 (59.7)	14	2	20	1	Android	Paid
Blood Pressure Companion Free	8	35 (56.5)	16	1	16	2	Apple	Free
BP Tracker - Blood Pressure Tracker	9	34 (54.8)	15	2	17	0	Apple	Paid
Blood Pressure	10	33 (53.2)	13	2	16	2	Android	Free
Blood Pressure - Smart Blood Pressure (SmartBP) BP Tracker		33 (53.2)	13	2	15	3	Apple	Free
Heart Pal - Blood Pressure Tracker	11	32 (51.6)	15	2	15	0	Apple	Paid
Diabetes In Check	12	31 (50.0)	12	3	13	3	Apple	Free
HeartBeat Counter Free	13	30 (48.4)	10	1	19	0	Apple	Free
Vision Test		30 (48.4)	9	2	17	2	Apple	Free
Blood Pressure Tracker by Tapcalc	14	28 (45.2)	10	3	13	2	Apple	Paid
BP Buddy	15	26 (41.9)	8	3	14	1	Apple	Paid
Test Your Hearing	16	24 (38.7)	6	2	13	3	Android	Free
Eye Test	17	23 (37.1)	5	2	16	0	Apple	Paid
Color Blind Test	18	22 (35.5)	6	2	14	0	Apple	Free
Color Blindness Test	19	21 (33.9)	6	1	12	2	Android	Free
DetectsFever	20	19 (30.6)	6	3	10	0	Apple	Free
Apps with Medication Interaction Checker Feature		Maximum score: 56	Maximum score: 18	Maximum score: 12	Maximum score: 22	Maximum score: 4
Medscape	1	43 (76.8)	15	8	16	4	Android	Free
		43 (76.8)	15	8	16	4	Apple	Free
Skyscape Medical Resources (RxDrugs)	2	39 (69.6)	13	6	18	2	Android	Free
Epocrates	3	36 (64.3)	9	8	17	2	Apple	Free
Epocrates	4	33 (58.9)	9	8	14	2	Android	Free
Drugs.com Medication Guide	5	32 (57.1)	16	3	11	2	Android	Free
Apps with Dose Calculator Feature		Maximum score: 49	Maximum score: 11	Maximum score: 12	Maximum score: 22	Maximum score: 4
Medscape	1	37 (75.5)	9	8	16	4	Android	Free
		37 (75.5)	9	8	16	4	Apple	Free
MedCalc	2	33 (67.3)	6	7	18	2	Apple	Paid
Epocrates	3	30 (61.2)	3	8	17	2	Apple	Free
Mediquation Medical Calculator	4	29 (59.2)	7	5	17	0	Android	Paid
Calculate by QxMD		29 (59.2)	6	5	16	2	Android	Free
Calculate by QxMD	5	28 (57.1)	6	5	15	2	Apple	Free
Epocrates		28 (57.1)	4	8	14	2	Android	Free
Lippincott Nursing Drug Handbooks	6	27 (55.1)	3	7	17	0	Apple	Free
Pedi STAT		27 (55.1)	3	6	16	2	Android	Paid
MedCalc 3000 Complete	7	25 (51.0)	8	3	14	0	Android	Paid
IV Drips	8	22 (44.9)	3	3	16	0	Android	Paid
Medical Calculator		22 (44.9)	5	3	14	0	Apple	Paid
Paramedic Meds		22 (44.9)	4	3	15	0	Android	Paid
Apps with Medication Information Feature		Maximum score: 60	Maximum score: 22	Maximum score: 12	Maximum score:22	Maximum score: 4
Medscape	1	43 (71.7)	15	8	16	4	Apple	Free
		43 (71.7)	15	8	16	4	Android	Free
Epocrates		43 (71.7)	16	8	17	2	Apple	Free
Epocrates	2	40 (66.7)	16	8	14	2	Android	Free
Lippincott Nursing Drug Handbooks	3	38 (63.3)	14	7	17	0	Apple	Free
MIMS		38 (63.3)	15	5	18	0	Apple	Paid
MIMS	4	37 (61.7)	15	5	17	0	Android	Paid
Skyscape Medical Resources (RxDrugs)		37 (61.7)	11	6	18	2	Android	Free
Micromedex Drug Information	5	36 (60.0)	15	5	14	2	Android	Free
Drugs.com Medication Guide	6	35 (58.3)	19	3	11	2	Android	Free
Drugs & Medications	7	32 (53.3)	9	3	18	2	Apple	Paid
Johns Hopkins ABX Guide	8	31 (51.7)	10	7	14	0	Android	Paid
MIMS Singapore		31 (51.7)	12	4	15	0	Apple	Free
iPharmacy	9	24 (40.0)	9	3	9	3	Apple	Free
Antibiotics pocket	10	22 (36.7)	5	4	13	0	Android	Paid
Medical Drugs Dictionary: FREE	11	21 (35.0)	3	2	14	2	Android	Free
Drugs Dictionary	12	19 (31.7)	3	1	13	2	Android	Free
Apps with Medication Record Feature		Maximum score: 46	Maximum score: 8	Maximum score: 12	Maximum score: 22	Maximum score: 4
Med Helper Pro Pill Reminder	1	31 (67.4)	5	2	20	4	Android	Paid
Pillboxie		31 (67.4)	5	6	20	0	Apple	Paid
My Medical Info	2	30 (65.2)	5	3	20	2	Android	Paid
My Medical	3	29 (63.0)	4	4	20	1	Android	Paid
Family Medical Info	4	26 (56.5)	3	2	19	2	Android	Paid
Pill Monitor Free–Medication Reminders and Logs	5	21 (45.7)	4	1	15	1	Apple	Free
Drugs.com Medication Guide	6	20 (43.5)	4	3	11	2	Android	Free
iPharmacy	7	18 (39.1)	3	3	9	3	Apple	Free

^a Overall quality score is a composite of appropriateness, reliability, usability, and privacy scores, reflected as a percentage of the total score.