Abstract
Aims and background
The field of mobile healthcare (mHealth) has attracted attention, and the quality of mHealth applications is also being addressed. Therefore, usability evaluation should be conducted to verify the quality of mHealth applications. The aim of this study was to conduct an expert evaluation to verify the systematic aspects and usability of a mobile application (“CAMBRA-students”) developed to evaluate caries risk in children and adolescents and to provide systematic caries management.
Materials and methods
The expert reviewers consisted of five professionals, including two dental hygienists, one professor of dental hygiene, and two IT specialists. These experts evaluated the technical issues of the mobile application using the heuristic evaluation and the Mobile App Rating Scale (MARS). Frequency analysis was used to determine the heuristic score and MARS.
Results
As a result of the heuristic evaluation, a total of 45 comments were received, of which 23 problems were extracted. The usability evaluation for professionals resulted in an overall score of 3.64. In the subscales (engagement, functionality, esthetics, information, and application subjective quality) of the MARS, the domain “application subjective quality” received the lowest score of 2.95 and the domain “information” received the highest score of 4.18. Through expert evaluation, the “CAMBRA-students” application was improved, and the usability was modified by correcting systemic and usability issues.
Conclusion
The “CAMBRA-students” application is expected to be used in public oral health services for students. It will contribute to improving the oral health status of adolescents using the caries risk management system.
Clinical significance
The “CAMBRA-students” application effectively manages caries risk in adolescents and has enhanced usability and systematic quality through expert evaluation, enabling personalized oral health management. It can significantly improve adolescent oral health and be integrated into school-based programs.
How to cite this article
Kang Y, Yeo A, Lee S. Expert Usability Evaluation of a Mobile Application for Systematic Caries Management in Children and Adolescents: A Cross-sectional Study. Int J Clin Pediatr Dent 2024;17(12):1370–1376.
Keywords: Dental caries, Heuristics, mHealth, Mobile applications
Introduction
Recently, the field of mobile healthcare (mHealth), which combines digital technology and healthcare, has attracted attention1,2 and the quality of mHealth applications is also being addressed.3 In addition, studies are being conducted to verify the effectiveness of mHealth applications.4 The reason for conducting systematic reviews and evaluations of mHealth applications is that the rapid increase in the development of mHealth applications has raised an important issue about reliability, information, and functionality. It has also been noted that consumers have limited reference indicators for selecting high-quality applications.5 Developers need to develop useful, accurate, and adaptable apps. Usability evaluation should be conducted to ensure that users can effectively and subjectively perform specific tasks in any environment.
The Mobile App Rating Scale (MARS) developed by Stoyanov et al.5 is a validated tool designed to assess the quality of healthcare-related applications. This scale has the advantage of being scored from an expert perspective, which compensates for the reliability issue of subjective user ratings.6 The MARS tool has been actively used to evaluate many types of healthcare applications,7,8 including self-management related to the healthcare system.9,14
Heuristic evaluation is a method of identifying usability issues based on usability principles by an expert.15,16 Heuristic evaluation is known to be able to identify many important usability issues with a relatively small number of experts.17 CAMBRA-kids for caries management in infants and young children, has also been subjected to expert heuristic evaluation18 and evaluated for effectiveness.19
As of 2018, there are approximately 1,075 mobile applications focused on oral care.20 The majority of oral health-related mobile application research has focused on adolescents. Mobile applications are mainly used for oral health promotion and behavior management, with limited use for diagnosis and remote consultation.21 Smartphone use is a part of adolescents’ daily lives and has a high value to them. Given the accessibility and convenience of smartphone use, mobile applications could effectively manage dental caries in adolescents.
Dental caries affects approximately 60–90% of adolescents worldwide. The high frequency of snacking and preference for cariogenic foods during this period can negatively affect dental health, making caries management critical.22 Mobile oral health applications accessible to adolescents should not only provide simple information but also be evidence-based and facilitate systematic caries management in collaboration with patient-centered clinicians. Therefore, in this study, an expert evaluation was conducted to verify the systematic aspects and usability of the “CAMBRA-students” application for caries risk assessment and systematic caries management in children and adolescents, to ensure the validity of the application, and ultimately to activate the caries management program for children and adolescents using “CAMBRA-students.”
Materials and Methods
Participants
The research participants were computer scientists and clinicians with sufficient knowledge and experience with the CAMBRA program. They were selected by purposive sampling. They had no limitations in installing and using mobile applications. The minimum number of experts for the evaluation was set at three.16 A total of five participants conducted the evaluation, including two IT specialists, two clinical dental hygienists with a master's degree and >3 years of clinical experience who had completed CAMBRA-related courses, and a professor of dental hygiene.
“CAMBRA-students” Mobile Application Evaluated for Usability
The mobile application “CAMBRA-students” was developed as an Android, iOS, and web-based application with the aim of identifying caries risk in children and adolescents, preventing dental caries, and ultimately promoting oral health. The implemented application was subjected to a usability evaluation method called “Cognitive Walkthrough” based on work scenarios to assess cognitive ease and potential for misuse, followed by modifications.23 After initial adjustments were made, the application was selected for evaluation. The “CAMBRA-students” application consisted of the following sections in Table 1.
Table 1:
Contents of the “CAMBRA-students” application
| Menu | Contents | Purpose |
|---|---|---|
| Introduction of the CAMBRA program | Provide information about the purpose, content, and process of the CAMBRA program | Provide oral health knowledge and information |
| Oral health care information | Provide information and knowledge about oral diseases (periodontal disease, dental caries) and oral health issues in children and adolescents | |
| To assist in the selection of appropriate oral care products | ||
| Caries risk assessment | Assesses the user's level of caries risk using both a questionnaire and clinical assessment for systematic caries risk assessment using assessment items such as “risk factors”, “protective factors”, and “disease indicators” | Increase self-awareness for oral health promotion |
| Allows the user to self-assess the status of their oral health using intraoral photographs taken with the Q-Ray | ||
| Assessment outcome | After risk assessment, the application's algorithm automatically determines individual caries risk and provides users with test results | |
| Oral health management | Provides guidance on personalized treatment regimens that are appropriate for an individual's risk level | Prevent oral disease through self-management |
| Q&A | Provide individualized guidance and information throughout the program | Facilitate communication between users and clinicians |
Research Tool
MARS Evaluation of the “CAMBRA-students” Application
The usability evaluation of the “CAMBRA-students” application was conducted using the MARS6 adapted by Byun.24 MARS was designed to identify and correct problems during the development process. In addition, it was also used to evaluate the usability of applications. It consisted of a total of 23 items in five subcategories: engagement (five items), functionality (four items), esthetics (three items), information (seven items), and subjective quality (four items), all rated on a five-point Likert scale. The usability of the application was evaluated by calculating the average score for each of the five subcategories with a score of three or more indicating fair to good.6,24
Heuristics Evaluation of the “CAMBRA-students” Application
Nielsen's heuristic evaluation is the most widely recognized usability evaluation tool in the field of user interfaces to date, consisting of 10 principles for evaluating technical errors or problems in an application.16,25 For this study, we used a modified tool of Nielsen's heuristic evaluation with eight principles26: visibility of the application, reflection of the real world, consistency and connectivity, ergonomic design, readability of the screen and ease of input, usability and efficiency of user settings, emotional aspects, and error management. The tool scored each heuristic on a 5-point scale (0: no usability problem at all; 1: cosmetic problem only, does not need to be fixed unless extra time is available on the project; 2: minor usability problem, fixing it should be given a low priority, 3: major usability problem, important to fix, should be given a high priority; and 4: usability catastrophe, imperative to fix).16
The experts freely described each heuristic principle. Any problem identified by more than two experts, or any problem rated with a severity level of four was considered for modification in the application.16,25,26 In addition, modifications were made based on the researchers’ judgment to improve usability by addressing problems that could improve the user experience.
Research Procedure
This study was approved by the Institutional Review Board of NAMSEOUL University (NAMSEOUL-202212-001) and conducted from June to July 2023. This is a two-part article due to the nature of a single IRB-reviewed study. In this study, five experts were selected, informed consent was obtained, and a survey was conducted to identify systematic errors or graphical problems in the “CAMBRA-students” application developed to manage dental caries in children and adolescents. Subjects were given instructions on how to download and use the application (Android, IOS) along with the questionnaire. Each of the five experts downloaded and used the application for 1 week before evaluating it. When the experts evaluated the app, they logged in directly from the user's perspective, entered risk and protective factors in the caries risk assessment section, and then used the preprovided administrator account to enter dental records and disease indicators arbitrarily from the expert's perspective. In addition, the application was revisited from the user's perspective to explore “assessment outcome”, “management”, and “oral health care information”. Any usability issues identified during this process were noted in the questionnaire and described freely. Based on the collected questionnaires, the researchers analyzed and discussed the findings according to heuristic standards and finally made modifications to the application according to the identified issues.
Statistical Analysis
The collected data were analyzed using SPSS version 23.0 (IBM Co., Armonk, New York, United States). General characteristics and heuristic scores of the study subjects were subjected to frequency analysis. MARS was analyzed by calculating the mean and standard deviation for each domain using descriptive statistics.
Results
General Characteristics of Study Subjects
Study participants included two IT specialists, two clinical dental hygienists, and one professor of dental hygiene. Of these, four (80%) had a master's degree or higher. All of them had >10 years of experience in their respective fields (Table 2).
Table 2:
General characteristics of study subjects
| Value | Classification | N (%) |
|---|---|---|
| Gender | Male | 2 (40) |
| Female | 3 (60) | |
| Age | 30–39 | 3 (60) |
| 40–49 | 2 (40) | |
| Education level | Bachelor's degree | 1 (20) |
| Master's degree | 2 (40) | |
| Doctoral degree | 2 (40) | |
| Job | Clinical dental hygienist | 2 (40) |
| Professor of dental hygiene | 1 (20) | |
| IT specialist | 2 (40) | |
| Experience (year) | 11–15 | 2 (40) |
| 16–20 | 1 (20) | |
| >20 | 2 (40) |
MARS Evaluation of the “CAMBRA-students” Application
Mobile App Rating Scale had an average objective score of 3.76 and the average score including subjective application quality was 3.64. The highest-scoring question was “credibility” in the “information” category with a score of 4.80. The lowest scoring question was “customization” in the “engagement” category with a score of 2.20. Other scores ranged from 2.00 to 2.90 for the “interactivity” question in the “engagement” category and the “would you pay for this application?” question in the “application subjective quality” category, with 20 questions scoring above the average of 3.00.
When considering the average score for each of the subscales of the tool, the subscale for commitment was the lowest, with an average score of 3.00. The subscale with the highest score was the “information” subscale with a mean score of 4.18, or 83.53% out of a maximum score of 5 (Table 3).
Table 3:
MARS evaluation results
| Subcategory | Item No. | Contents of each item | Mean ± SD | Actual score (A) | A/5 (%)a |
|---|---|---|---|---|---|
| Objective evaluation | 3.76 ± 0.56 | 75.22 | |||
| Engagement | 1 | Entertainment | 3.00 ± 0.71 | 3.00 ± 0.82 | 60.00 |
| 2 | Interest | 3.20 ± 0.45 | |||
| 3 | Customization | 2.20 ± 0.84 | |||
| 4 | Interactivity | 2.80 ± 0.45 | |||
| 5 | Target group | 3.80 ± 0.84 | |||
| Functionality | 6 | Performance | 4.40 ± 0.90 | 4.12 ± 0.83 | 84.00 |
| 7 | Ease of use | 4.00 ± 1.00 | |||
| 8 | Navigation | 4.00 ± 1.00 | |||
| 9 | Gestural design | 4.40 ± 0.55 | |||
| Esthetics | 10 | Layout | 3.80 ± 0.84 | 3.67 ± 0.72 | 73.33 |
| 11 | Graphics | 3.40 ± 0.55 | |||
| 12 | Visual appeal | 3.80 ± 0.84 | |||
| Information | 13 | Accuracy of application description | 4.00 ± 1.00 | 4.18 ± 1.03 | 83.53 |
| 14 | Goals | 3.40 ± 1.82 | |||
| 15 | Quality of information | 4.20 ± 0.84 | |||
| 16 | Quantity of information | 4.20 ± 0.84 | |||
| 17 | Visual information | 4.60 ± 0.55 | |||
| 18 | Credibility | 4.80 ± 0.45 | |||
| 19 | Evidence base | 4.00 ± 1.16 | |||
| Subjective evaluation | 2.95 ± 1.12 | 59.00 | |||
| Application subjective quality | 20 | Would you recommend this application to people who might benefit from it? | 3.20 ± 0.84 | ||
| 21 | How many times do you think you would use this application in the next months if it was relevant to you? | 3.00 ± 0.71 | |||
| 22 | Would you pay for this application? | 2.40 ± 0.89 | |||
| 23 | What is your overall star rating of the application? | 3.40 ± 0.90 | |||
| Total | 3.64 ± 0.70 | 70.00 |
aThe maximum score is a percentage of 5
Heuristic Evaluation of the “CAMBRA-students” Application
A total of 45 issues were raised in the heuristic evaluation, with heuristic #5 receiving the most feedback from the experts. The highest severity level, four points, was given to the heuristic #3.
Among the identified heuristic issues, those raised by two or more evaluators or those with a severity score of four or higher were considered for application modification. This criterion resulted in prioritized modifications for nine issues: including those related to heuristics #1 and #2, which received a severity score of three, and issues under heuristic #3, which received the highest severity score of four. In addition to these nine issues, 14 usability comments were also discussed by the researchers and included in the application modifications. As a result, a total of 23 issues were identified and addressed for revision (Table 4).
Table 4:
Heuristics problems with the severity score of the “CAMBRA-students” application
| Heuristics | No. of comments | Grade of heuristics | |||||
|---|---|---|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 4 | |||
| 1 | Visibility of system status and losability/findability of the mobile device | 6 | 1 | 1 | 4 | ||
| 2 | Match between system and the real-world | 6 | 1 | 1 | 1 | 3 | |
| 3 | Consistency and mapping | 6 | 2 | 1 | 1 | 2 | |
| 4 | Good ergonomics and minimalist design | 5 | 3 | 2 | |||
| 5 | Ease of input, screen readability and glanceability | 7 | 1 | 3 | 3 | ||
| 6 | Flexibility, efficiency of use and personalization | 5 | 3 | 1 | 1 | ||
| 7 | Esthetics, privacy, and social conventions | 5 | 4 | 1 | |||
| 8 | Realistic error management | 5 | 2 | 1 | 1 | 1 | |
| Total | 45 | 17 | 9 | 8 | 9 | 2 | |
Expert Comments Based on the Eight Usability Principles of Heuristic Evaluation
The expert evaluation of the “CAMBRA-students” application resulted in a total of 23 problems (Table 5). The IT specialist described 16 issues. The clinical dental hygienist described six issues and the professor of dental hygiene described one issue.
Table 5:
Expert comments on the principles of heuristic evaluation
| Heuristics | Evaluator | Comments | |
|---|---|---|---|
| 1 | Visibility of system status and losability/findability of the mobile device | IT specialist | Need automatic logout on program exit |
| IT specialist | When attaching a photo image, the “year date” shows unknown text such as NaN year, NaN month, NaN day, etc. | ||
| Dental hygienist | Risks are grayed out on the scan results screen, making them difficult to see, and the descriptions are not separated by risk, making them difficult to read | ||
| 2 | Match between system and the real-world | IT specialist | Unclear messages while waiting for expert response after saving the questionnaire (ex. “no results registered yet”) |
| IT specialist | Lack of guidance text after clicking the “save as PDF” button click in the caries management section for different risk levels, making it unclear whether the saving successful | ||
| IT specialist | Processes appear to be in a “pending inspection” state after the questionnaire is completed, preventing them from being reviewed | ||
| IT specialist | When entering a chart number, the “save membership information” message pops up with inappropriate wording | ||
| Dental hygienists | Need to simplify words in the caries risk assessment to make it easier for elementary school students to understand | ||
| 3 | Consistency and mapping | IT specialists | Inconsistent button display, such as “yes and ok”, and different button displays on Android and IOS Inconsistent display of buttons such as “x, close” |
| 4 | Good ergonomics and minimalist design | IT specialist | Lack of a logout button on the main screen, which reduces usability |
| IT specialist | Recommendation to add a “home button” | ||
| 5 | Ease of input, screen readability, and glanceability | IT specialist | The search bar on the oral health information screen is located at the bottom, making it difficult to access |
| IT specialist | Fixed spacing of “CAMBRA students” in the caries risk screen | ||
| Dental hygienist | Typos in the oral management content need to be corrected | ||
| Dental hygienist | The sentence regarding the amount of toothpaste used in the caries management tab needs to be revised | ||
| Dental hygienist | Many sentences throughout the site seem to need line breaks for better readability | ||
| Professor | The term CAMBRA is used inconsistently in both Korean and English | ||
| 6 | Flexibility, efficiency of use and personalization | IT specialist | Improvements needed to allow users to edit survey results on my page |
| 8 | Realistic error management | IT specialist | After filling in the first questionnaire and pressing the “Save” button without selecting anything, the questionnaire will be saved. A function is needed to verify that the questionnaire has been filled out correctly |
| IT specialist | After saving a clinical chart, the chart number is not assigned in the patient list | ||
| IT specialist | English letters are also entered in the numeric input field | ||
A total of three issues were identified related to heuristic #1, the IT specialist raised the lack of an automatic logout function related to user security and the mislabeling of data related to visibility. The clinical dental hygienist suggested color-coding caries risk to increase visibility.
A total of six issues were raised for heuristic #2. The main issues raised by the experts were the lack of instructions for use and the inappropriate wording and location of the notification button. Dental hygienists suggested the same issue and recommended that it be revised to use words that are easily understood by elementary school students.
The issues related to heuristic #3 were assigned a severity level of four. An inconsistent wording of notifications was reported, noting that the names of the “accept” and “decline” buttons were different in different notification windows, which could be confusing to users.
The issues that fell under heuristic #4 were that the lack of a logout button on the main screen and the lack of an icon to navigate to the home screen hindered user usability. There was a total of six issues that fell under heuristic #5; the IT specialist recommended moving the search bar to the top because it was located at the bottom of the screen and was not easy to use, while the clinical dental hygienist suggested improving the readability by correcting typos and line breaks in the oral health management. The professor of dental hygiene also recommended unifying the English and Korean languages as it could be confusing to users.
The IT specialist recommended several changes related to heuristic #6 and heuristic #8. The expert recommended fixing the usability issues related to user settings and changes to the caries risk assessment that could hinder usability.
Discussion
This study was to evaluate technical errors or issues encountered during the use of the mobile application “CAMBRA-students,” developed for the systematic management of dental caries in children and adolescents. The usability evaluation was performed using the heuristic evaluation16,26 and the MARS6,24 on five experts. The heuristic evaluation followed eight principles and identified a total of 45 issues, with 23 significant issues emerging based on the study's evaluation criteria. Of these, IT specialists identified 16 issues, clinical dental hygienists identified six issues, and a professor of dental hygiene identified one issue.
Based on numerous precedents,18,27,29 this study ranked the issues raised by experts according to their severity and incorporated them into application modifications. Content suggested by IT specialists and clinical dental hygienists under heuristic #1 included automatic logout upon program closure and corrections to date display errors and improved risk level visibility. The most critically received heuristic was #3, which was related to #2. These related issues were addressed by clarifying messaging and improving screen layout and button placement. Changes also included clarifying the postquestionnaire process and improving error management for a more intuitive user experience.
The overall usability evaluation of the “CAMBRA-students” application was conducted using the MARS tool, which could identify problems encountered during the process of using the application for ease of learning and users’ subjective views. The overall average MARS score of this study was 3.64, which was higher than the average MARS score of 3.40 of 37 applications in a previous study30 that systematically evaluated oral health care applications. In addition, Zhang et al.31 have stated that the usability of an application is satisfactory when it scores 3.0 (moderate) or higher. Overall, the usability of the application in this study was rated as satisfactory.
Among the MARS items, the “customization” item in the “engagement” domain scored the lowest with a score of 2.20. Previous studies32,33 have also shown that “engagement” received low scores overall and is an important area for overall improvement. Siddiqui et al.32 have suggested that “engagement” is a measure of how engaged users are with the application and that it is important to focus on the application design, user interest, and targeting the right audience and improving interactive features to enable interaction with clinicians. Patients should be provided with oral health education videos and other information that encourages user engagement. Applications should be updated regularly to provide the latest technology and information, rather than being neglected, which is a problem with all applications.
The expert evaluation of the “CAMBRA-students” application in this study indicates that the application's systematic quality and usability of the application are generally satisfactory, making it suitable for its intended users. The use of the “CAMBRA-students” application in school-based oral health management programs for children and adolescents is expected to facilitate individualized oral care and ultimately contribute to the improvement of oral health among these groups. This study highlights the potential of using the “CAMBRA-students” application for pediatric dental caries management while acknowledging the limitations of expert-only evaluations. Future research should include user-based usability evaluations and qualitative expert ratings for more in-depth analysis. It is anticipated that the application, continuously updated with evidence-based information, will become a widely used tool for pediatric oral health management.
Conclusion
In this study, the expert evaluation of the “CAMBRA-students” application showed that the overall systematic quality and usability of the application were suitable for application users. In the future, when the “CAMBRA-students” application is applied to school-based oral health care for systematic caries management in children and adolescents, it is expected to provide personalized oral health care for each student in the long term, ultimately contributing to improving the oral health of children and adolescents.
Clinical Significance
The “CAMBRA-students” application can be effectively used in public oral health services to manage caries risk in adolescents. Expert evaluation improved the application's usability and systematic quality. This enables personalized oral health management for adolescents. The application has the potential to enhance adolescent oral health and can be integrated into school-based programs.
Orcid
Yu-Min Kang https://orcid.org/0000-0001-8020-4326
An-Na Yeo https://orcid.org/0000-0003-2500-1408
Su-Young Lee https://orcid.org/0000-0002-5502-3037
Footnotes
Source of support: Nil
Conflict of interest: None
References
- 1.Topp CW, Østergaard SD, Søndergaard S, et al. The WHO-5 Well-Being Index: a systematic review of the literature. Psychother Psychosom. 2015;84:167–176. doi: 10.1159/000376585. [DOI] [PubMed] [Google Scholar]
- 2.Gallup . Seoul, Korea: Gallup Report; 2022. 2012–2022 Smartphone Usage Rate and Brand, Smart Watch, Wireless Earphone Survey. [Google Scholar]
- 3.World Health Organization . Geneva, Switzerland: World Health Organization; 2011. mHealth: New Horizons for Health Through Mobile Technologies: Second Global Survey on eHealth. [Google Scholar]
- 4.Cai X, Zhang F, Lin C, et al. Achieving effective and efficient basal insulin optimal management by using mobile health application (APP) for type 2 diabetes patients in China. Diabetes Metab Syndr Obes. 2020;13:1327–1338. doi: 10.2147/DMSO.S244826. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Stoyanov SR, Hides L, Kavanagh DJ, et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3:e27. doi: 10.2196/mhealth.3422. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Park JH, Park HY, Park JH. Qualitative evaluation of lifestyle management apps using the Korean Mobile Application Rating Scale (MARS) J RWEAT. 2020;14:134–141. doi: 10.21288/resko.2020.14.2.134. [DOI] [Google Scholar]
- 7.Weekly T, Walker N, Beck J, et al. A review of apps for calming, relaxation, and mindfulness interventions for pediatric palliative care patients. Children. 2018;5:16. doi: 10.3390/children5020016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Santo K, Richtering SS, Chalmers J, et al. Mobile phone apps to improve medication adherence: a systematic stepwise process to identify high-quality apps. JMIR Mhealth Uhealth. 2016;4:e132. doi: 10.2196/mhealth.6742. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Escoffery C, McGee R, Bidwell J, et al. A review of mobile apps for epilepsy self-management. Epilepsy Behav. 2018;81:62–69. doi: 10.1016/j.yebeh.2017.12.010. [DOI] [PubMed] [Google Scholar]
- 10.Bardus M, van Beurden SB, Smith JR, et al. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. Int J Behav Nutr Phys Act. 2016;13:35. doi: 10.1186/s12966-016-0359-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Masterson Creber RM, Maurer MS, Reading M, et al. Review and analysis of existing mobile phone apps to support heart failure symptom monitoring and self-care management using the Mobile Application Rating Scale (MARS) JMIR Mhealth Uhealth. 2016;4:e74. doi: 10.2196/mhealth.5882. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Tinschert P, Jakob R, Barata F, et al. The potential of mobile apps for improving asthma self-management: a review of publicly available and well-adopted asthma apps. JMIR Mhealth Uhealth. 2017;5:e113. doi: 10.2196/mhealth.7177. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Grainger R, Townsley H, White B, et al. Apps for people with rheumatoid arthritis to monitor their disease activity: a review of apps for best practice and quality. JMIR Mhealth Uhealth. 2017;5:e7. doi: 10.2196/mhealth.6956. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Thornton L, Quinn C, Birrell L, et al. Free smoking cessation mobile apps available in Australia: a quality review and content analysis. Aust N Z J Public Health. 2017;41:625–630. doi: 10.1111/1753-6405.12688. [DOI] [PubMed] [Google Scholar]
- 15.Bowman DA, Gabbard JL, Hix D. A survey of usability evaluation in virtual environments: classification and comparison of methods. Presence. 2002;11:404–424. doi: 10.1162/105474602760204309. [DOI] [Google Scholar]
- 16.Nielsen J. Usability Inspection Methods. New York: John Wiley & Sons; 1994. Heuristic evaluation; pp. 25–62. [Google Scholar]
- 17.Yen PY, Bakken S. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol—an example from a web-based communication tool for nurse scheduling. AMIA Annu Symp Proc. 2009;2009:714–718. [PMC free article] [PubMed] [Google Scholar]
- 18.Kang YM, Lee SY. Heuristics evaluation and development of the caries management by risk assessment (CAMBRA)-kids application for caries management of preschoolers. J Korean Soc Dent Hyg. 2019;19:479–492. doi: 10.13065/jksdh.20190043. [DOI] [Google Scholar]
- 19.Yeo AN, Lee SY. Effect of dental caries management using ‘CAMBRA-kids’ mobile application for children under 5 years old. Int J Dent Hyg. 2022;20:443–452. doi: 10.1111/idh.12565. [DOI] [PubMed] [Google Scholar]
- 20.Parker K, Bharmal RV, Sharif MO. The availability and characteristics of patient-focused oral hygiene apps. Br Dent J. 2019;226:600–604. doi: 10.1038/s41415-019-0197-7. [DOI] [PubMed] [Google Scholar]
- 21.Väyrynen E, Hakola S, Keski-Salmi A, et al. The use of patient-oriented mobile phone apps in oral health: scoping review. JMIR Mhealth Uhealth. 2023;11:e46143. doi: 10.2196/46143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Petersen PE. The World Oral Health Report 2003: continuous improvement of oral health in the 21st century—the approach of the WHO Global Oral Health Programme. Community Dent Oral Epidemiol. 2003;31:3–23. doi: 10.1046/j..2003.com122.x. [DOI] [PubMed] [Google Scholar]
- 23.Lee JH, Park Y, Jung JE, et al. Usability evaluation of a mobile app for sexual violence survivors. J KoCon. 2022;22:700–711. [Google Scholar]
- 24.Byun AJ. Seoul: Seoul National University; 2018. Development and Evaluation of Smartphone App “Healthy kids” for Children's Weight Control with their Parents [Master Thesis] [Google Scholar]
- 25.Nielsen J. Enhancing the Explanatory Power of Usability Heuristics. In: Adelson B, Dumais ST, Olson JS, editors. CHI: ACM; 1994. pp. 152–158. [Google Scholar]
- 26.Bertini E, Gabrielli S, Kimani S. Appropriating and assessing heuristics for mobile computing. In: Proceedings of the Working Conference on Advanced Visual Interfaces. Assoc Comput Mach. 2006;1:119–126. [Google Scholar]
- 27.Jeon E, Park HA. Development of the IMB model and an evidence-based diabetes self-management mobile application. Healthc Inform Res. 2018;24:125–138. doi: 10.4258/hir.2018.24.2.125. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Lee WH. Heuristic evaluation of mobile augmented reality applications: focused on location-based service type apps. Korea Society Design Trend. 2017;5:131–140. [Google Scholar]
- 29.Fung RHY, Chiu DKW, Ko EHT, et al. Heuristic usability evaluation of university of Hong Kong libraries’ mobile website. J Acad Libr. 2016;42:581–594. doi: 10.1016/j.acalib.2016.06.004. [DOI] [Google Scholar]
- 30.Roy A, Nayak PP, Shenoy PR, et al. Availability and content analysis of smartphone applications on conservative dentistry and endodontics using mobile application rating scale (MARS) Curr Oral Health Rep. 2022;9:215–225. doi: 10.1007/s40496-022-00325-w. [DOI] [Google Scholar]
- 31.Zhang H, Gil TS, Kang EJ. A feasibility study of Korean mobile application rating scale (MARS) for evaluating the quality of blood pressure management apps. Korean J Health Educ Promot. 2018;35:41–51. doi: 10.14367/kjhep.2018.35.3.41. [DOI] [Google Scholar]
- 32.Siddiqui NR, Hodges SJ, Sharif MO. Orthodontic apps: an assessment of quality (using the Mobile App Rating Scale (MARS)) and behaviour change techniques (BCTs) Prog Orthod. 2021;22:25. doi: 10.1186/s40510-021-00373-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Reyes A, Qin P, Brown CA. A standardized review of smartphone applications to promote balance for older adults. Disabil Rehabil. 2018;40:690–696. doi: 10.1080/09638288.2016.1250124. [DOI] [PubMed] [Google Scholar]