Section A (Assesses the appropriateness of app to serve its intended function for targeting medication-related problems) i. Monitoring Feature (This section assesses an app’s feature that allows users to monitor or record the efficacy/toxicity of medications in terms of improvement in symptoms or adverse effects)
Maximum Possible Score: 24
1. Can app monitor more than 1 physiological parameter (e.g., blood glucose, body temperature, heart rate)?     (0) No, only 1 physiological parameter can be monitored     (1) Yes, more than 1 physiological parameter can be monitored
2. Does app allow users to set target goals for physiological parameters (e.g., blood glucose, blood pressure)?     (0) No     (1) Yes
3. Does app allow recording of symptoms?     (0) No symptoms can be recorded     (1) Only subjective symptoms can be recorded     (2) Objective symptoms can be recorded
4. Can an external monitoring device (e.g., blood pressure monitor, glucometer) sync with the app to record physiological readings?     (0) Yes (Proceed to Q5)     (1) No (Skip Q5)
5. Can the monitoring device automatically transfer readings to the app in real time?     (0) No, manual entry of data by user is required     (1) Yes, data are automatically transferred but at a later time (when user syncs)     (2) Yes, data are automatically transferred to app in real time (by wireless networks/Bluetooth)
6. Does the app need to be used in a controlled environment (e.g., quiet place with no interference)?     (0) Yes     (1) No
7. Does the app require a health care professional to perform the monitoring for patients?     (0) Yes, a health care professional is required     (1) No, monitoring can be performed by patients themselves
8. Is there a specific section that allow recording of medication(s) taken?     (0) No, medication cannot be recorded     (1) Yes, but only medication name can be recorded     (2) Yes, able to record medication name + dose or timing/frequency     (3) Yes, able to record medication name + dose + timing/frequency
9. Does app organize patient’s health monitoring input (e.g., vital signs) in chronological order?     (0) No     (1) Yes
10. Is there a feature to display trends of monitored parameter(s) (e.g., graph)?     (0) No     (1) Yes
11. Does app allow users to display a saved history of measurement results?     (0) No, there is no display of results/user cannot select a time frame for display of results     (1) Yes, but display is based only on a limited number of recordings (e.g., past 10 readings)     (2) Yes, display can be based on a selected time frame (e.g., weeks/months/years)
12. Does app classify severity of medical condition(s) based on pre-set cutoff measurements?     (0) No, measurements are not classified into severity     (1) Yes, but it is not clearly stated whether classification is supported by evidence     (2) Yes, it is clearly stated that classification is based on evidence/clinical guidelines
13. Does app alert users on abnormal readings that are out of the normal range?     (0) No, users are not alerted     (1) Yes, abnormality is indicated qualitatively (e.g., highlighted using color code)     (2) Yes, abnormality is indicated quantitatively (e.g., high/low or normal range is provided to user)
14. Does app provide a management plan for an abnormal measurement?     (0) No, there is no management plan provided     (1) Yes, only a general management plan is provided     (2) Yes, a patient-specific management plan is provided
15. Can a password/pin be set to access the recorded data/parameters?     (0) No     (1) Yes
16. Does the app allow monitored data to be backed up/stored?     (0) No, data cannot be stored     (1) Yes, but data are non-transferable (e.g., stored on device itself)     (2) Yes and data are transferable (e.g., to an external storage device, cloud)
ii. Medication Interaction Checker Feature (This section assesses an app’s feature that has the ability to perform interaction checks between 2 or more medications)
Maximum Possible Score: 18
1. Does app allow medication to be searched by different names?     (0) No search function is provided     (1) Yes, either generic or brand name can be searched     (2) Yes, both generic and brand name can be searched
2. Does app display a list of possible interactions associated with a single medication?     (0) No     (1) Yes
3. Does app detect multiple drug-drug interactions concurrently?     (0) No, only 1 pair of drug-drug interaction can be detected at any 1 time     (1) Yes, multiple drug-drug interactions can be detected at any 1 time
4. Can the app detect interactions with complementary and alternative/herbal medicines?     (0) No     (1) Yes
5. Which of the following details other than the interaction effect(s) are provided: severity of interaction(s), mechanism(s) of interaction, and management plan(s)?     (0) Only 1 of those above is provided     (1) 2 of the above are provided     (2) All 3 details are provided
6. Does app categorize interactions according to severity?     (0) No, app does not classify severity/classification rubric is not available     (1) Yes, classification rubric is provided but information is not supported by evidence/references     (2) Yes, classification rubric is provided and information is supported by evidence/references
7. Does app provide details on the comprehensiveness of the interaction database (e.g., number of medications/interactions detected by the database)?     (0) No details are provided     (1) App is a standalone with no information on the comprehensiveness of the database provided     (2) App is a standalone and information on the comprehensiveness of the database is provided     (3) App follows a parent database but no information on the comprehensiveness of the database is provided     (4) Yes, app follows a parent database and comprehensiveness of database is provided
8. Does app allow interaction checks against a previously saved medication list?     (0) No     (1) Yes
9. Are there spelling or grammatical errors in the interaction content?     (0) Major errors that compromise the level of understanding     (1) Some errors but they do not compromise the level of understanding     (2) No errors present
10. Is the content of the app reviewed?     (0) Not reviewed/not stated     (1) Yes, reviewed internally     (2) Yes, reviewed externally
iii. Dose Calculator Feature (This section assesses an app’s feature that is used explicitly for calculating medication doses)
Maximum Possible Score: 11
1. Can app perform dose calculation based on weight of patient?     (0) No     (1) Yes
2. Does app include an antibiotic dose calculator?     (0) No     (1) Yes, but app does not adjust for renal/liver dysfunction in the calculation     (2) Yes, and app takes into account renal/liver dysfunction in the calculation
3. Does app include an IV drip rate calculator?     (0) No, app is unable to calculate drip rate for IV medications     (1) Yes, app can calculate drip rate for IV medications
4. Can app calculate dose equivalent conversion(s) of medications within the same pharmacologic class where applicable (e.g., statins)?     (0) No, dose conversion(s) cannot be performed     (1) Yes, dose equivalents can be calculated
5. Does app provide a formula and/or explain how calculation is derived?     (0) No information is provided     (1) Yes, either formula or explanation for calculation is derived
6. Once calculation has been performed, is administration instruction provided?     (0) No, administration instruction is not provided     (1) Yes, administration instruction is provided
7. Are the units of medication(s) consistent throughout the calculation     (e.g., if unit for input are in mg, does final calculation show strength in mg/mL)?     (0) No     (1) Yes
8. Is the content of the app reviewed?     (0) Not reviewed/not stated     (1) Yes, reviewed internally     (2) Yes, reviewed externally
9. Is the formula used supported with reference citations/evidence?     (0) No     (1) Yes
iv. Medication Information Feature (This section assesses an app’s medication database on whether it is able to provide clinically relevant information about medications)
Maximum Possible Score: 22
1. Does the app have a search box to search for information on specific drugs/medications?     (0) No     (1) Yes
2. Can the drug/medication be searched based on multiple categories (generic name, brand name, disease/medical condition)? (For example, generic name: carbamazepine, brand name: Tegretol, medical condition: neurologic disorder)     (0) No, the drug can only be searched by 1 category     (1) Yes, the drug can be searched by at least 2 out of 3 categories
3. From the search results, are the core information (medication, indication, dosing instruction, contraindication, and side effects) and supplementary information (precaution, drug interaction) provided?     (0) Not all the core information (listed above) is provided     (1) All the core information (listed above) is provided, without any supplementary information     (2) Yes, all the core information and any supplementary information (listed above) are provided
4. Does medication reference provide information if the medication can be purchased over the counter or requires a prescription?     (0) No     (1) Yes
5. Does app include information for special populations (e.g., pediatrics, pregnancy, renal/liver dysfunction)?     (0) No     (1) Yes
For questions 6-8, refer to the following black box warning about the pharmacogenomic effects of carbamazepine (Tegretol):
Black box warning: Serious dermatologic reactions and HLA-B*1502 allele
• Serious and sometimes fatal dermatologic reactions, including toxic epidermal necrolysis (TEN) and Stevens-Johnson syndrome (SJS), have been reported during treatment with Tegretol.
• These reactions are estimated to occur in 1 to 6 per 10,000 new users in countries with mainly Caucasian populations, but the risk in some Asian countries is estimated to be about 10 times higher.
• Studies in patients of Chinese ancestry have found a strong association between the risk of developing SJS/TEN and the presence of HLA-B*1502, an inherited allelic variant of the HLA-B gene. HLA-B*1502 is found almost exclusively in patients with ancestry across broad areas of Asia.
• Patients with ancestry in genetically at-risk populations should be screened for the presence of HLA-B*1502 prior to initiating treatment with Tegretol.
• Patients testing positive for the allele should not be treated with Tegretol unless the benefit clearly outweighs the risk.
6. Does medication reference classify the above information for carbamazepine (Tegretol) as a black box warning?     (0) No     (1) Yes
7. How comprehensive is the information provided by the app on pharmacogenomic and serious dermatologic effects of carbamazepine?     (0) None of the information is provided     (1) Some of the information is provided     (2) All of the information is provided
8. Does app provide relevant medication-related resources for specific drug classes, where applicable (e.g., antibiotic susceptibility charts; dose-conversion charts for cholesterol-lowering statins, steroids, or opioids)?     (0) No     (1) Yes
9. Does app contain a pill identifier feature?     (0) No, there is no such feature     (1) Yes, but feature does not include pictures of medications     (2) Yes, feature includes pictures of medications
10. Does app provide details on the comprehensiveness of the database (e.g., number of medications in the database)?     (0) No details are provided     (1) App is a standalone with no information on the comprehensiveness of the database provided     (2) App is a standalone and information on the comprehensiveness of the database is provided     (3) App follows a parent database but no information on the comprehensiveness of the database is provided     (4) Yes, app follows a parent database and comprehensiveness of the database is provided
11. Is there a summary of the evidence on a medication’s therapeutic use with supporting reference citations?     (0) No evidence summary or supporting references are provided     (1) Yes, either the evidence summary or supporting references are provided     (2) Yes, both the evidence summary and supporting references are provided
12. Are there any spelling or grammatical errors?     (0) Yes, there are major errors that compromise the level of understanding     (1) Yes, there are some errors but they do not compromise the level of understanding     (2) No, there are no errors present
13. Is the content of the app reviewed?     (0) Not reviewed/not stated     (1) Yes, reviewed internally     (2) Yes, reviewed externally
v. Medication Records Feature (This section assesses an app’s feature that allows users to record their current medication therapy)
Maximum Possible Score: 8
1. Does app allow all of the following patient’s biodata (age, weight, allergy, medical history, medical condition) and current medications to be input?     (0) Not all of the above mentioned data can be input     (1) Both patient’s biodata and current medications can be input
2. Does app allow users to edit their medication list?     (0) No, not at all     (1) Yes, users can add/remove medication according to their prescription
3. Is there an adherence and reminder feature to allow users to record when their medication was taken?     (0) No, there is no such feature available     (1) Yes, there is a recording function but no medication reminder is available     (2) Yes, both the recording and medication reminder functions are available
4. Is the app able to provide/allow users to add basic therapeutic information regarding the medication?     (0) No     (1) Yes
5. Can app alert user on medication allergy(s)/interaction(s) from the user’s active medication list?     (0) No     (1) Yes
6. Can a password/pin be set to access patient’s medication records?     (0) No     (1) Yes
7. Does app allow data to be stored externally?     (0) No, data are non-transferable (e.g., stored on device itself)     (1) Yes, data are transferable (e.g., external storage device, cloud)
Section B (Reliability) (This section assesses if an app is reliable and unbiased for users to utilize for medication-related purposes. Most of the requirements to determine reliability can be found in the “About” section within the app)
Maximum Possible Score: 12
1. Is the target audience for app use clearly defined/stated?     (0) No     (1) Yes
2. Does section icon/heading/title accurately reflect content within section?     (0) No     (1) Yes
3. Are the names and credentials of the app creators/authors stated?     (0) Both not mentioned     (1) Either name(s) or credential(s) indicated     (2) Both name(s) and credential(s) indicated
4. Does app contain a disclaimer (or a statement of similar implication) that the information provided/content will not replace a health care professional’s judgment?     (0) No     (1) Yes
5. Are the sources of funding clearly stated?     (0) Not clearly stated     (1) Yes, clearly stated
6. Are there any conflicts of interests declared?     (0) No declaration stated     (1) Yes, stated clearly     (2) No conflicts of interest exist and this is stated clearly
7. Is there any evidence of bias (e.g., advertisements) present?     (0) Yes     (1) No
8. Is there a third-party certification by a medical or governmental organization?     (0) No certification present     (1) Certification present
9. Is the date of last update stated?     (0) Not stated     (1) Only a states how often app is updated without providing actual date of last update/only provides month and year of update     (2) Yes, actual date of last update is stated
Section C (Usability) (This section assesses if an app is easy to use, provides user support, and is able to cater to the needs of different users)
Maximum Possible Score: 22
1. Is the app intuitive to use?     (0) No     (1) Yes
2. Are there any user interface problems that compromise the ease of navigation within the app?     (0) Yes, to a large extent that it compromises the ease of navigation     (1) Yes, but only to a small extent that does not compromise the ease of navigation     (2) No problems present
3. Is the screen layout well organized/not cluttered?     (0) No     (1) Yes
4. Is the font of a readable size by default?     (0) Not readable by default (can only be viewed on a larger screen)     (1) No, but user can zoom in and panning is required     (2) Yes
5. Are the terminologies understandable by a layperson?     (0) Not at all     (1) Sometimes     (2) Most of the time     (3) All the time
6. Is there a help feature present?     (0) No     (1) Yes
7. Can features of the app be accessed from any page?     (0) No, features can only be accessed from main home page     (1) Yes
8. Does the app require an internet connection to get started?     (0) Yes, the pages will only load with an internet connection     (1) No, I can still retrieve information that I downloaded earlier
9. Are there any broken links?     (0) Many broken links     (1) Few broken links     (2) No broken links
10. Is the graphical user interface adaptable to both portrait/landscape view when the phone is rotated?     (0) No     (1) Yes
11. Is app customizable based on user preference?     (0) Not customizable at all     (1) Yes, but to a limited extent     (2) Yes, mostly customizable
12. Can users bookmark pages for quick access?     (0) No/bookmark feature is unavailable     (1) Yes
13. Are the contact details of developer/feedback section provided?     (0) No     (1) Yes
14. How often does the app crash/exit unexpectedly?     (0) Frequently     (1) Occasionally     (2) Rarely     (3) Never
Section D (Privacy) (This section assesses the privacy protection of user input data; i.e., data security)
Maximum Possible Score: 4
1. How is the privacy policy stated on the app?     (0) No privacy policy present at all     (1) Privacy policy mentions the collection of user data only     (2) Privacy policy mentions the collection of user data and how it is being used
2. Can users choose what content to share?     (0) Users cannot choose what content to share/content cannot be shared at all     (1) Users can choose what content to share
3. Can users choose who to share content with through electronic means?     (0) No feature that allows sharing of content exists/users cannot choose who they want to share content with     (1) Yes, there is a feature that allows users to select a person whom they want to share content with