Tasks	Descriptions	Stakeholders	Estimated Time
Stage 1. TARGETING
1. Targeting	Determine the [target disease and/or symptom]	Researcher who participated in the current study
Stage 2. CHECK LIST CREATION
2. Brainstorming and Mind Mapping	Conducting group brainstorming sessions on the [target disease and/or symptom] and designing mind map about brainstorming	Individual (healthcare provider or researcher) who had experience/majored in informatics (eg, medical informatics, nursing informatics) (≥2 groups, group ≥2 individuals)	1 hour
3. Persona and Scenario	Design a potential user (persona) who uses an app to manage the [target disease and/or symptom] (≥2 personas) Create a scenario for each persona who uses an app to manage the [target disease and/or symptom] (≥2 scenarios)	* The same groups as the previous stage	3 hours
4. User Needs	List of user's needs (requirements) derived from each scenario of each persona who uses an app to manage the [target disease and/or symptom], Write the list followed by order of priority	* The same groups as the previous stage	1 hour
5. Developing Checklist	Draft version of checklist by table format: Compile all of the user’s needs (requirements) included in the scenario into a checklist	* Same individual can do this stage with stage 1 Researcher who participated in the current study	1 hour
	Modifying checklist: Select items (user’s needs, requirements) based on a content validity index of >0.75.	† Different individual with all previous stages Individual (Healthcare provider or researcher) who had experience/majored in informatics (eg, medical informatics, nursing informatics) (≥3 individuals) [55]	1 day
	Final version of checklist: Allocate partial score for each item. If the user's needs are “recording the pain in the app,” allocate 0 points to the item in the checklist if there is no pain recording menu, 1 point if there is a pain recording menu with a dichotomous (Yes/No) scale, and 2 points if there is a pain recording menu with Likert or score recording system.	* Same individual can do this stage with previous stage (second version of checklist) Individual who majored in medical informatics or nursing informatics (≥3)	1 day
Stage 3. APP SELECTION
6. App Searching	Search terms: Found synonyms for [target disease and/or symptom]. If a researcher wants to search for menstrual apps, he/she needs to search using terms such as menstruation, period, and menstrual cycle.	Same individual can do this stage with stage 1 or 6 Researcher who participated in the current study	2–3 hours
	App search: Searched for apps in 1 or more app stores using the derived search terms.	* Same individual can do this stage with stage 1 or 6 or 8 Researcher (≥2)	1 day for 1 search term
7. App Screening	Deduplication: Compared the lists of the searched apps and compiled them into a single list. If the app name and developer were the same, they are considered the same app.	* Same individual as the previous stage Researcher (≥2)	1-2 days
	Remove all apps unrelated to the [target disease and/or symptom] based on the app name or description.	* Same individual as the previous stage Researcher (≥2)	1-2 days
8. App Scoring	Score the apps using the checklist after using app at least 10 mins (Independently).	*Same individual as the previous stage Researcher (≥2)	1 day for 30 apps
	Crosscheck: Compare the scores of the top 100 apps given by each researcher. If the total scores of the app or the scores given for a specific item in the checklist differ between the researchers, the researchers should review the app and discuss it to reach a consensus on the score.	* Same individual as the previous stage Researcher (≥2)	1-2 days
9. App Selecting	Check a review user’s needs (the app rating and the number of reviews) for each app in app store with the top 5 checklist scores. A given checklist score might include multiple apps.	* Same individual as the previous stage Researcher	1 hour
	When multiple apps receive the same score, the app with the highest rating should be selected.	*The same person as the previous stage Researcher	1 hour
	When multiple apps receive the same score and app rating, the app with the highest number of reviews should be selected.	*The same person as the previous stage Researcher	1 hour
Stage 4. APP EVALUATION
10. App Evaluation	Evaluate the candidate apps using an assessment tool like MARS (Mobile Application Rating Scale) after using each app for at least 10 mins.	† App expert with experience in developing and/or designing a health-related app (≥5 individual)	1 day
	Evaluate the candidate apps using an assessment tool like the user-MARS after using each app for at least 10 mins.	† Clinical expert with experience in the treatment of and/or research on [target disease and/or symptom] patients/consumers (≥5 individual)	1 day
	Evaluate the candidate app through focus group interviews (real users). The interview questions should be structured around an assessment tool like the user-MARS. An evaluation (preferred app or not) should be conducted after using each app for at least 10 mins.	† Person who has experienced the [target disease and/or symptom] (≥3 groups, group ≥2 individual)	1 day for 1 group
11. Determining the Best App	Among the preferred apps, the app with the highest score given by the app and clinical experts is selected as the best app.	Same individual can do this stage with stage 1 or 6 or 8 Researcher who participated in the current study	1 day
Stage 5. APP VERIFICATION
12. Randomized Controlled Trial	Randomized controlled trial is conducted. An intervention group use the best app selected in the previous stage.	† Person who has experienced the [target disease and/or symptom] (≥2 groups)

^*Person who participated in previous stage can participate in this step.

^†Person who participated in the previous stage cannot participate in this step. In other words, a new participant/researcher should perform this step.