Abstract
Background
Women’s menstrual cycle is an important component of their overall health. Physiological cycles and associated symptoms can be monitored continuously and used as indicators in various fields. Menstrual apps are accessible and can be used to promote overall female health. However, no study has evaluated these apps’ functionality from both consumers’ and health care providers’ perspectives. As such, the evidence indicating whether the menstrual apps available on the market provide user satisfaction is insufficient.
Objective
This study was performed to investigate the key content and quality of menstrual apps from the perspectives of health care providers and consumers. We also analyzed the correlations between health care provider and consumer evaluation scores. On the basis of this analysis, we offer technical and policy recommendations that could increase the usability and convenience of future app.
Methods
We searched the Google Play Store and iOS App Store using the keywords “period” and “menstrual cycle” in English and Korean and identified relevant apps. An app that met the following inclusion criteria was selected as a research app: nonduplicate; with >10,000 reviews; last updated ≤180 days ago; relevant to this topic; written in Korean or English; available free of charge; and currently operational. App quality was evaluated by 6 consumers and 4 health care providers using Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS). We then analyzed the correlations among MARS scores, uMARS scores, star ratings, and the number of reviews.
Results
Of the 34 apps, 31 (91%) apps could be used to predict the menstrual cycle, and 2 (6%) apps provided information pertinent to health screening. All apps that scored highly in the MARS evaluation offer a symptom logging function and provide the user with personalized notifications. The “Bom Calendar” app had the highest MARS (4.51) and uMARS (4.23) scores. The MARS (2.22) and uMARS (4.15) scores for the “Menstrual calendar—ovulation & pregnancy calendar” app were different. In addition, there was no relationship between MARS and uMARS scores (r=0.32; P=.06).
Conclusions
We compared consumer and health care provider ratings for menstrual apps. Continuous monitoring of app quality from consumer and health care provider perspectives is necessary to guide their development and update content.
Keywords: mobile app, period, menstrual cycle, mHealth, mobile health, evaluation, women’s health, health care provider, consumer, menstrual app, digital health app, health screening, consumer satisfaction
Introduction
Background
Women’s menstrual cycles are important to their overall health [1,2] and are characterized by predictable and recurring symptoms. Continuous tracking of the menstrual cycle can aid in health management. Monitoring systems are needed to optimize menstrual health and provide easily accessible health information for women [3].
Mobile health (mHealth) apps facilitate personalized health monitoring and management [4], and menstrual apps are the most important mHealth apps for women. Such apps are typically highly accessible for most women and provide indicators relevant to various health domains [5].
Typically, menstrual apps also allow the user to log their symptoms, mood changes, and body temperature, and they visually represent statistical data via graphs and tables. Some apps offer professional-level information through communities and links, that is, they promote women’s health care by facilitating smooth communication with medical staff through information-sharing services [6]. The apps’ menstrual cycle tracking functions facilitate health care planning and management in various domains, including contraception, fertility, preparation with respect to pregnancy and ensuring adequate “menstrual supplies,” leisure activities, and travel [7].
Women use these functions for various purposes, such as tracking pregnancy, preventing pregnancy, and managing menstruation periods [8]. The functions desired by consumers depend on the intended purpose of the app. Currently, indicators to help consumers identify and select menstrual apps according to the desired functions are lacking [9]. Recently developed systems recommend apps based on consumer requirements [10,11]. Menstrual apps with various functions have been developed, and related research is being actively conducted. However, most of the existing studies are content reviews or expert evaluations [12-15]. Consumer-centered studies have also started to appear [5,16-18], but quality evaluations of menstrual apps remain scarce.
Menstrual apps are directly relevant to women’s health, so it is necessary for experts and health care providers to evaluate these apps [8]. Health care provider quality evaluations can contribute to the app development, which is important to ensure that consumers have access to high-quality apps [19]. Consumer quality evaluations provide feedback on apps, such as consumer preferences (eg, for easy-to-use content) [20,21]. Consistent use of an app is critical given the recurring nature of the menstrual cycle, but research on this topic is lacking from the developer’s standpoint. To promote sustained app use, evaluations from consumers and health care providers’ perspective are required. However, it is unclear whether current commercially available apps satisfy the quality standards of consumers and health care providers. If we find the differences between consumers and health care providers, they need to be discussed importantly to develop apps that can be used to promote women’s health that satisfy both consumers’ wants and health care providers’ needs.
Purpose
This study examines currently available menstrual apps in term of their main contents and quality based on evaluations by health care providers and consumers. We also correlated the two sets of evaluation scores. The study’s findings could improve the utility and convenience of future mHealth apps.
Methods
App Selection
We searched for keywords related to the development and evaluation of menstrual apps commonly included in previous studies, such as “period” and “menstrual cycle” in both English and Korean. The Google Play Store and the iOS App Store were searched from April 8 to April 15, 2021. Up to 150 apps were screened for each keyword. Since then, to secure the appropriated number of apps that can be statistically analyzed, the following app inclusion criteria have been set, based on previous studies [14,22,23]:
The app is nonduplicate.
It has more than 10,000 reviews.
It has been last updated ≤180 days ago.
It is relevant to this topic.
The app language is Korean or English.
The app is free to use.
It is currently in operation.
Analysis of App Contents
To examine the apps’ main contents, we performed a pilot study of 14 representative apps. The apps’ main contents were classified as follows: menstrual cycle management, education and knowledge, sharing information, and notifications. Frequency analysis was performed to determine the number of apps providing these functions.
Evaluation of App Quality
The quality of 34 menstrual apps were evaluated using Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS). The MARS was developed to evaluate mobile apps; it is a reliable tool with a high internal consistency and interrater reliability [24]. The uMARS was subsequently developed, based on the MARS, to allow consumers to evaluate the quality of mHealth apps. uMARS has excellent internal consistency [25]. Both scales comprise the following five categories: engagement, aesthetics, functionality, information, and subjective quality (Table 1). The MARS has been used to evaluate various health care–related apps, such as apps related to chronic disease, COVID-19, and physical activity, as well as apps related to allergy, hepatitis treatment support, and breast cancer [14,15,19,22,23,26,27]. The uMARS has recently been used to evaluate various types of mHealth apps, including apps pertaining to weight loss and nutrition, rheumatic diseases, and the management of ankylosing spondylitis [28-31].
Table 1.
Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS) evaluation items.
| Measures | Characteristics | |
| Objective measures | ||
|
|
Engagement |
|
|
|
Functionality |
|
|
|
Information |
|
|
|
Aesthetics |
|
| Subjective measures | ||
|
|
Subjective quality |
|
aMARS evaluation items.
buMARS evaluation items.
A total of 6 consumers completed the uMARS between July 9 and July 22, 2021, and 4 nurses (ie, health care providers) majoring in health care and working in medical centers completed the MARS between July 21 and July 30, 2021. Each evaluator was asked to use the app for more than 10 minutes every day and the evaluation was conducted in a blind test. The apps were randomly assigned to evaluators to prevent bias related to subjectivity. Each app was crossevaluated by at least two evaluators.
Comparative Analysis of Consumer and Health Care Provider Data
The MARS results represent health care providers’ perspectives, as stated above. The uMARS results, star ratings, and number of reviews were analyzed from the consumers’ perspective. After normalization, Pearson correlation was used to correlate app content, MARS and uMARS scores, star ratings, and reviews and to correlate perspectives of health care providers and consumers. We identified the top and bottom five apps based on the MARS and uMARS scores and compared app preferences between consumers and health care providers. P values <.05 were considered significant. R software (version 4.1.2; R Core Team,) was used for the analysis.
Results
App Selection
A total of 1127 menstrual apps were initially identified via the keyword search, and 34 apps (Android: n=28; iPhone: n=6) met all of the study criteria and were included in the final analysis (Figure 1).
Figure 1.
Flow diagram of the app review and selection process. C1-7 denotes Criterion 1-7.
The operating system (OS) of apps is linked with the app store, so that the app is updated simultaneously with updates of the OS [32]. Therefore, the app version and function may vary depending on the timing of the OS update. The number of reviews and star ratings differed among the apps in this study. For example, Bom Calendar had star ratings of 4.8 and 4.4 for the Android and iOS versions, respectively (23,437 and 76,258 reviews, respectively). In instances where the same apps were available for different OSs, each version was considered to be a unique app in the analysis.
Analysis of App Contents
Most apps (n=31, 91%) offered a menstrual cycle prediction function. Some apps (n=14, 41%) offered menstruation and fertility period notifications, while others had no specific functions (n=3, 9%). Most apps were confidential (n=29, 85%), allowed data export (n=28, 82%), and had a log-in function (n=25, 74%). However, few apps provided education or knowledge (n=10, 29%), screening-related information (n=2, 6%), or advice (n=4, 12%; Table 2).
Table 2.
App contents analysis result.
| Contents | App, n (%) | |||||
|
|
Android (n=28) | iPhone (n=6) | Total (N=34) | |||
| Menstrual cycle management | ||||||
|
|
Symptoms (pain) | 4 (14) | 0 (0) | 4 (12) | ||
|
|
Additional symptom | 18 (64) | 6 (100) | 24 (71) | ||
| Ovulation management | ||||||
|
|
Calculate pregnancy probability | 4 (14) | 0 (0) | 4 (12) | ||
|
|
Contraception methods | 3 (11) | 0 (0) | 3 (9) | ||
|
|
Both | 12 (43) | 6 (100) | 18 (53) | ||
| Last update (months) | ||||||
|
|
<3 | 21 (75) | 5 (83) | 26 (76) | ||
|
|
3~6 | 7 (25) | 1 (17) | 8 (24) | ||
| Function s | ||||||
|
|
Graphical chart | 17 (61) | 4 (67) | 21 (62) | ||
|
|
Lock | 23 (82) | 6 (100) | 29 (85) | ||
|
|
Advice provision | 2 (7) | 2 (33) | 4 (12) | ||
|
|
Data export | 23 (82) | 5 (83) | 28 (82) | ||
|
|
Predictions | 25 (89) | 6 (100) | 31 (91) | ||
|
|
Log-in | 20 (71) | 5 (83) | 25 (74) | ||
| Education or knowledge | ||||||
|
|
General health information | 5 (18) | 1 (17) | 6 (18) | ||
|
|
Personalized information | 1 (4) | 2 (33) | 3 (9) | ||
|
|
Both | 1 (4) | 0 (0) | 1 (3) | ||
|
|
Health screening | 1 (4) | 1 (17) | 2 (6) | ||
| Sharing information (with health care professionals) | ||||||
|
|
All information | 4 (14) | 3 (50) | 7 (21) | ||
|
|
Only information specified by the consumer | 1 (4) | 0 (0) | 1 (3) | ||
| Visualization | ||||||
|
|
Menstruation or ovulation | 2 (7) | 0 (0) | 2 (6) | ||
|
|
Menstrual cycle | 9 (32) | 0 (0) | 9 (26) | ||
|
|
All data | 16 (57) | 6 (100) | 22 (65) | ||
| Notifications | ||||||
|
|
Menstruation or fertility | 2 (7) | 0 (0) | 2 (56) | ||
|
|
Both | 14 (50) | 0 (0) | 14 (41) | ||
|
|
Personalized alarms | 9 (32) | 6 (100) | 15 (44) | ||
| Other features | ||||||
|
|
Community | 4 (14) | 2 (33) | 6 (18) | ||
|
|
Shopping | 2 (7) | 1 (17) | 3 (9) | ||
Evaluation of App Quality
The MARS and uMARS scores of all apps were obtained through the average of the evaluator’s evaluation scores. The average MARS and uMARS scores (ie, health care provider and consumer scores, respectively) were 3.06 (SD 0.62) and 3.33 (SD 0.57), respectively. The iPhone “Bom Calendar” app had the highest score for both the MARS (4.51, SD 0.22) and uMARS (4.23, SD 0.27). The Android “Period calendar—Women’s menstrual calendar❤” app had the second lowest score for both the MARS (2.05, SD 0.45) and uMARS (2.09, SD 0.05), despite its high star rating of 4.8. The Android “Menstrual calendar—ovulation & pregnancy calendar” app showed contrasting scores between the MARS (2.22, SD 0.07) and uMARS (4.15, SD 0.46); it had the third highest overall uMARS score, with an engagement score of 3.86 (SD 0.49), functionality score of 4.04 (SD 0.49), aesthetics score of 4.25 (SD 0.47), information score of 4.25 (SD 0.47), and subjective quality score of 4.37 (SD 0.41). However, for the MARS, its overall score was the third lowest, with an engagement score of 2.40 (SD 0.20), functionality score of 2.50 (SD 0.50), aesthetics score of 2.00 (SD 0.00), information score of 2.58 (SD 0.42), and subjective quality score of 1.63 (SD 0.38; Table 3 and Table 4). MARS and uMARS scores of all 34 apps are presented in Multimedia Appendix 1.
Table 3.
The five highest- and lowest-scoring apps (N=34) based on the user version of the Mobile Application Rating Scale (uMARS). All the values are mean (SD).
| App | Engagement | Functionality | Aesthetics | Information | Subjective quality | uMARSa | |||||||
| Top fiveb | |||||||||||||
|
3.87 (0.33) | 4.17 (0.26) | 4.33 (0.32) | 4.27 (0.29) | 4.51 (0.22) | 4.23 (0.27) | |||||||
|
4.03 (0.38) | 4.16 (0.44) | 4.23 (0.35) | 4.31 (0.27) | 4.40 (0.25) | 4.23 (0.33) | |||||||
|
3.86 (0.49) | 4.04 (0.49) | 4.25 (0.47) | 4.25 (0.47) | 4.37 (0.41) | 4.15 (0.46) | |||||||
|
3.80 (1.01) | 3.97 (0.78) | 4.30 (0.51) | 4.30 (0.51) | 4.29 (0.49) | 4.13 (0.66) | |||||||
|
3.83 (0.60) | 4.06 (0.43) | 4.16 (0.30) | 4.11 (0.35) | 4.39 (0.36) | 4.11 (0.38) | |||||||
| Bottom fivec | |||||||||||||
|
1.91 (0.34) | 1.96 (0.39) | 1.91 (0.54) | 1.94 (0.75) | 1.90 (0.72) | 1.93 (0.54) | |||||||
|
1.76 (0.11) | 2.04 (0.13) | 2.17 (0.07) | 2.13 (0.10) | 2.33 (0.10) | 2.09 (0.05) | |||||||
|
1.93 (0.38) | 2.11 (0.53) | 2.46 (0.47) | 2.53 (0.37) | 2.46 (0.51) | 2.30 (0.44) | |||||||
|
2.30 (0.30) | 2.50 (0.50) | 2.70 (0.30) | 2.74 (0.16) | 2.90 (0.10) | 2.63 (0.23) | |||||||
|
2.24 (1.05) | 2.43 (1.24) | 3.06 (1.07) | 3.19 (1.19) | 3.33 (1.33) | 2.85 (1.17) | |||||||
aAverage of uMARS scores evaluated by consumers.
bTop five apps: (1) Bom Calendar (iOS), (2) Pregnancy planning & management (Android), (3) Menstrual calendar—ovulation & pregnancy calendar (Android), (4) Flo (Android), and (5) Flo (iOS).
cBottom five apps: (1) Maya—My Period Tracker (Android); (2) Period calendar—Women’s menstrual calendar❤ (Android); (3) Women’s menstrual calendar, menstrual & ovulation day calculator, childbearing age, pregnancy planning (Android); (4) My Days—Ovulation Calendar & period Tracker (Android); and (5) Clover: Period & Cycle Tracker (Android).
Table 4.
The five highest- and lowest-scoring apps (N=34) based on the Mobile Application Rating Scale (MARS).
| App and operating system | Engagement | Functionality | Aesthetics | Information | Subjective quality | MARSa | |||||||
| Top fiveb | |||||||||||||
|
|
4.18 (0.63) | 4.75 (0.25) | 5.00 (0.00) | 4.29 (0.11) | 4.36 (0.25) | 4.51 (0.14) | |||||||
|
|
4.10 (0.10) | 4.06 (0.21) | 4.67 (0.33) | 4.92 (0.08) | 3.93 (0.24) | 4.34 (0.15) | |||||||
|
|
3.85 (0.46) | 4.13 (0.63) | 4.42 (0.43) | 4.00 (0.33) | 3.58 (0.58) | 3.99 (0.48) | |||||||
|
3.65 (0.75) | 4.19 (0.57) | 4.00 (1.00) | 4.42 (0.58) | 3.68 (0.07) | 3.99 (0.59) | |||||||
|
4.00 (0.00) | 3.56 (0.37) | 4.83 (0.17) | 4.33 (0.35) | 3.02 (0.61) | 3.95 (0.29) | |||||||
| Bottom fivec | |||||||||||||
|
|
2.05 (0.65) | 2.13 (0.63) | 1.50 (0.50) | 2.38 (0.71) | 1.53 (0.53) | 1.92 (0.60) | |||||||
|
|
2.10 (0.71) | 2.81 (0.48) | 2.08 (0.60) | 2.25 (0.25) | 1.01 (0.26) | 2.05 (0.45) | |||||||
|
|
2.40 (0.20) | 2.50 (0.50) | 2.00 (0.00) | 2.58 (0.42) | 1.63 (0.38) | 2.22 (0.07) | |||||||
|
2.00 (0.20) | 2.94 (0.57) | 2.42 (0.43) | 2.63 (0.14) | 1.34 (0.09) | 2.26 (0.15) | |||||||
|
2.00 (0.00) | 2.75 (0.25) | 2.33 (0.33) | 2.75 (0.08) | 1.81 (0.56) | 2.33 (0.01) | |||||||
aAverage of MARS scores evaluated by health care providers.
bTop five apps: (1) Bom Calendar (iOS), (2) Pink Diary (Android), (3) Bom Calendar (Android), (4) Clue Period, Ovulation Tracker (iOS), and (5) Femometer—Fertility Tracker (Android).
cBottom five apps: (1) My Days—Ovulation Calendar & period Tracker (Android), (2) Period calendar—Women’s menstrual calendar❤ (Android), (3) Menstrual calendar—ovulation & pregnancy calendar (Android), (4) My Menstrual Diary (Android), and (5) Period Tracker (Android).
Further Comparative Analysis of Consumer and Health Care Provider Data
Table 5 shows the results of correlation analysis of the MARS and uMARS scores, star ratings, number of reviews, and app content. The number of reviews was not correlated with app content and menstrual cycle management (r=0.53; P=.001), and visualization (r=0.51; P=.002) had the highest correlation with star ratings. Among the evaluation scores, the highest correlation was found between uMARS and notification (r=0.39; P=.02) as well as between MARS and ovulation date management (r=0.49; P=.003).
Table 5.
Results of correlation analysis of the Mobile Application Rating Scale (MARS) and the user version of the Mobile Application Rating Scale (uMARS) scores, star rating, number of reviews, and app content types.
| App contents | Health care provider and consumer perspective | ||||
|
|
MARS | uMARS | Star rating | Number of reviews | |
| Menstrual cycle management | 0.39a | 0.18 | 0.53b | 0.21 | |
| Ovulation date management | 0.49b | 0.33 | 0.41a | 0.16 | |
| Functions |
|
|
|
|
|
|
|
Graphic chart | –0.06 | –0.10 | –0.54b | –0.31 |
|
|
Lock | –0.37a | –0.40a | –0.32 | –0.10 |
|
|
Advice provision | –0.05 | –0.31 | –0.25 | –0.01 |
|
|
Data export | –0.22 | –0.20 | –0.53b | –0.10 |
|
|
Prediction | –0.43a | –0.17 | –0.79b | –0.22 |
|
|
Log-in | –0.12 | –0.01 | –0.47b | –0.13 |
| Educational or knowledge | 0.23 | 0.04 | –0.02 | –0.14 | |
| Health screening | 0.25 | 0.13 | 0.12 | 0.01 | |
| Sharing information | 0.16 | 0.14 | 0.04 | –0.13 | |
| Visualization | 0.32 | 0.22 | 0.51b | 0.19 | |
| Notification | 0.32 | 0.39a | 0.24 | 0.06 | |
| Other features |
|
|
|
|
|
|
|
Community | –0.14 | –0.19 | –0.16 | –0.10 |
|
|
Shopping | –0.35a | –0.05 | –0.14 | 0.12 |
aP<.05.
bP<.01.
Multimedia Appendix 2 shows content comparison between the top and bottom five apps in consumer and health care provider evaluation scores. Personalized alarms could be set in the top five apps. In addition, they provided a function to visualize all information through a calendar or to specify and manage ovulation days. On the contrary, in the bottom five apps did not have functions for managing or predicting the menstrual cycle.
Figure 2 shows the correlations among the MARS and uMARS scores, star ratings, and number of reviews to compare the perspective of health care providers and consumers. Figure 2 shows no correlation between MARS and uMARS scores of the health care providers and consumers (r=0.32; P=.06). uMARS scores and star rating (r=0.11; P=.54) as well as uMRAS scores and number of reviews (r=0.07; P=.67) also showed no significant correlations. The number of reviews and star rating (r=0.39; P=.02) showed a very low correlation.
Figure 2.
Result of the correlation analysis of the Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS) scores, star ratings, and number of reviews.
Discussion
Principal Findings and Comparison With Prior Work
Interest in mHealth apps has recently increased, and new apps are continuously being developed, including menstrual apps [33]. However, the needs of consumers and health care providers are different, and studies evaluating whether the available menstrual apps satisfy both of these groups are difficult to find. This study obtained quality evaluations of relevant apps from consumers and health care providers using the uMARS and MARS, respectively; the consistency of the evaluations of these two groups in terms of key app contents was analyzed. The health care providers valued engagement, functionality, and aesthetics when evaluating apps, while consumers valued aesthetics and information provision the most.
The MARS and uMARS scores were not correlated in this study. For example, the app with the third lowest MARS score had the third highest uMARS score. A significant difference was observed in app aesthetics scores between the consumers and health care providers; the scores for this attribute showed the largest group difference. Previous studies on health services have reported disparities between consumers and health care providers, and these results affect the implementation of consumer-centered services. Data from health care providers can provide a basis for high-quality apps [20], while consumer data serves as feedback on app quality [19]. To ensure high app quality and consumer satisfaction, app quality should be continuously monitored from the perspective of both health care providers and consumers. Monitoring can identify the needs of consumers and health care providers, which can in turn help in app development and update [10]. This study focused on evaluating app quality using MARS and uMARS for consistency, but could be extended to qualitative studies, including interviews, to collect in-depth answers in the future [34].
According to our findings, the uMARS (ie, consumer) scores, star ratings, and number of reviews were unrelated variables. The uMARS allows for direct assessment of mHealth apps and is a reliable measure of app quality [25]. However, reviews and star ratings are subjective indicators [24]. The currently available mHealth apps have been evaluated in a simplistic manner, such as through star ratings and reviews, even though they differ significantly with respect to content; thus, appropriate guidelines to aid app selection are lacking. By using the uMARS to guide app selection, the limitations of reviews and star ratings can be overcome such that consumers will likely select more useful apps to meet their particular needs. However, it is difficult for consumers to evaluate the quality of the apps using uMARS every time they download one. New indicators to guide app selection are needed so that consumers can make decisions based on objective evaluation results.
Most of the five apps that scored highly in the quality evaluation in this study included personalized monitoring functions. Menstrual apps are becoming increasingly popular, and apps that include self-monitoring functions and provide related information are continuously being developed [28]. Personalized monitoring can improve user well-being by encouraging them to check for signs and symptoms of health issues. Health-related information can also promote consumer health. mHealth apps that include personalized content of this nature are particularly useful for consumers [23,35]. However, only a few of the apps evaluated in this study facilitated consultations with specialists or provided information relating to women’s health. To increase the utility of apps, notifications, symptom recording functions, and the provision of knowledge should be prioritized.
mHealth apps should provide customized content for individual consumers [36]. However, the five bottom-scoring apps in this study did not meet the needs of the women who were using them. Consumers use apps to predict menstrual cycles and ovulation dates, and to monitor their general health [37]. However, most of the five bottom-scoring apps did not provide content enhancing consumer convenience, such as functions for menstrual cycle and ovulation day management, and some apps also lacked predictive functions. Such apps must be updated to include content allowing for the prediction and management of menstrual cycles based on accumulated menstrual cycle- and health-related information.
Menstrual apps collects personal information from consumers, such as name, date of birth, menstrual cycle, and medical history [38]. Personal data must be protected because it is sensitive information [12,39], but some apps do not provide locking functions, and few apps provide icon change functions for protecting personal information. Most fitness apps that record the number of steps do not consider privacy issues, and the data protection of mHealth apps related to women’s health is typically poor [39]. Therefore, regulations pertaining to app management of private data are necessary [40,41]. In fact, there are existing regulations protecting personal information, such as the European Unions’s General Data Protection Regulation, but no standard regulations are enforced worldwide. mHealth apps that protect personal information tend to be favored by consumers [21]. App developers should improve data protection–related functions to protect the personal information of consumers.
The MARS and uMARS were developed specifically for evaluating mHealth apps that aim to improve consumers’ health. Meanwhile, menstrual apps were designed to help consumers keep track of their current health rather than improve it. Health apps must provide solutions customized to individual consumers [36]; reliability may be key in this respect [42]. This study’s results indicate that current health apps do not fully meet consumers’ requirements or desires with respect to content. To better identify consumer objectives and take account of them during the development of menstrual apps, a new evaluation scale is needed to evaluate menstrual apps.
Limitations
This study had several limitations. First, the results cannot be generalized to all menstrual apps because a small number of evaluators evaluated only the most popular apps. Second, the apps were selected based on App Store searches with a limited timeframe. Updates to apps may result in differences between the analyzed content and that in the future. Third, the database is not an electronic database but the App Store. The App Store’s app recommendation function may have compromised an inconsistent search accuracy.
Conclusions
In this study, consumer and health care provider ratings of menstrual apps were obtained using validated scales. Consumer preferred app had high scores of aesthetics and information, and evaluation scores differed between consumers and health care providers. The findings highlight the importance of consumer participation in menstrual app development and evaluation. This study is significant in that it is the first to compare health care providers’ and consumers’ menstrual app quality ratings. We expect our results to guide future mHealth app development and provide consumers with information on menstrual app content and quality. To provide high-quality apps for consumers, continuous quality evaluation research needs to be conducted, and the perspectives of both consumers and health care providers should be taken into account.
Acknowledgments
This work was supported by a 2022 research grant from Kongju National University (KNU) and the National Research Foundation of Korea (NRF) funded by the Korean government (NRF-2020R1C1C1009679). This study was presented at the 2021 Postacademic Conference of the Korean Academy of Health Policy and Management.
JL was affiliated with the Department of Nursing Science, College of Life & Health Sciences at Hoseo University at the time of the study and is currently affiliated with the Department of Nursing at Gangneung-Wonju National University.
Abbreviations
- MARS
Mobile Application Rating Scale
- mHealth
mobile health
- OS
operating system
- uMARS
user version of the Mobile Application Rating Scale
Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS) score of all 34 apps.
Content comparison between the top and bottom five apps (ranked according to the consumer and health care provider evaluation scores).
Footnotes
Conflicts of Interest: None declared.
References
- 1.Bae J, Park S, Kwon J. Factors associated with menstrual cycle irregularity and menopause. BMC Womens Health. 2018 Feb 06;18(1):36. doi: 10.1186/s12905-018-0528-x. https://bmcwomenshealth.biomedcentral.com/articles/10.1186/s12905-018-0528-x .10.1186/s12905-018-0528-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Critchley HO, Babayev E, Bulun SE, Clark S, Garcia-Grau I, Gregersen PK, Kilcoyne A, Kim JJ, Lavender M, Marsh EE, Matteson KA, Maybin JA, Metz CN, Moreno I, Silk K, Sommer M, Simon C, Tariyal R, Taylor HS, Wagner GP, Griffith LG. Menstruation: science and society. AJOG. 2020 Nov;223(5):624–664. doi: 10.1016/j.ajog.2020.06.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Park H. Health and Welfare Policy Forum. South Korea: Korea Institute for Health and Social Affairs; 2021. Sep 01, Health behavior and health promotion in women; pp. 2–4. [Google Scholar]
- 4.Derbyshire E, Dancey D. Smartphone medical applications for women's health: what is the evidence-base and feedback? Int J Telemed Appl. 2013;2013:782074. doi: 10.1155/2013/782074. doi: 10.1155/2013/782074. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Moglia ML, Nguyen HV, Chyjek K, Chen KT, Castaño PM. Evaluation of smartphone menstrual cycle tracking applications using an adapted APPLICATIONS scoring system. Obstet Gynecol. 2016 Jun;127(6):1153–1160. doi: 10.1097/AOG.0000000000001444.00006250-201606000-00024 [DOI] [PubMed] [Google Scholar]
- 6.Levy J, Romo-Avilés N. "A good little tool to get to know yourself a bit better": a qualitative study on users' experiences of app-supported menstrual tracking in Europe. BMC Public Health. 2019 Sep 03;19(1):1213. doi: 10.1186/s12889-019-7549-8. https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-019-7549-8 .10.1186/s12889-019-7549-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Epstein DA, Lee NB, Kang JH, Agapie E, Schroeder J, Pina LR, Fogarty J, Kientz JA, Munson SA. Examining menstrual tracking to inform the design of personal informatics tools. Proc SIGCHI Conf Hum Factor Comput Syst. 2017 May 02;2017:6876–6888. doi: 10.1145/3025453.3025635. http://europepmc.org/abstract/MED/28516176 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Earle S, Marston HR, Hadley R, Banks D. Use of menstruation and fertility app trackers: a scoping review of the evidence. BMJ Sex Reprod Health. 2021 Apr 06;47(2):90–101. doi: 10.1136/bmjsrh-2019-200488.bmjsrh-2019-200488 [DOI] [PubMed] [Google Scholar]
- 9.DeNicola N, Marko K. Connected health and mobile apps in obstetrics and gynecology. Obstet Gynecol Clin North Am. 2020 Jun;47(2):317–331. doi: 10.1016/j.ogc.2020.02.008.S0889-8545(20)30022-X [DOI] [PubMed] [Google Scholar]
- 10.Lee J, Kim J. Can menstrual health apps selected based on users' needs change health-related factors? A double-blind randomized controlled trial. J Am Med Inform Assoc. 2019 Jul 01;26(7):655–666. doi: 10.1093/jamia/ocz019. http://europepmc.org/abstract/MED/30946478 .5427757 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Lee J, Kim J. Method of app selection for healthcare providers based on consumer needs. Comput Inform Nurs. 2018 Jan;36(1):45–54. doi: 10.1097/CIN.0000000000000399. [DOI] [PubMed] [Google Scholar]
- 12.Dantas LO, Carvalho C, Pena C, Breda CC, Driusso P, Ferreira CHJ, Bø Kari. Mobile health technologies for the monitoring of menstrual cycle: a systematic review of online stores in Brazil. J Obstet Gynaecol Res. 2022 Jan 27;48(1):5–14. doi: 10.1111/jog.15082. [DOI] [PubMed] [Google Scholar]
- 13.Starič KD, Trajkovik V, Belani H, Vitagliano A, Bukovec P. Smart phone applications for self-monitoring of the menstrual cycle: a review and content analysis. Clin Exp Obstet Gynecol. 2019;46(5):731–735. doi: 10.12891/ceog4830.2019. [DOI] [Google Scholar]
- 14.Davalbhakta S, Advani S, Kumar S, Agarwal V, Bhoyar S, Fedirko E, Misra DP, Goel A, Gupta L, Agarwal V. A systematic review of smartphone applications available for corona virus disease 2019 (COVID19) and the assessment of their quality using the Mobile Application Rating Scale (MARS) J Med Syst. 2020 Aug 10;44(9):164. doi: 10.1007/s10916-020-01633-3. http://europepmc.org/abstract/MED/32779002 .10.1007/s10916-020-01633-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rodrigues AT, Sousa CT, Pereira J, Figueiredo IV, Lima TDM. Mobile applications (Apps) to support the hepatitis c treatment: a systematic search in app stores. Ther Innov Regul Sci. 2021 Jan;55(1):152–162. doi: 10.1007/s43441-020-00201-8.10.1007/s43441-020-00201-8 [DOI] [PubMed] [Google Scholar]
- 16.Gambier-Ross K, McLernon DJ, Morgan HM. A mixed methods exploratory study of women's relationships with and uses of fertility tracking apps. Digit Health. 2018;4:2055207618785077. doi: 10.1177/2055207618785077. https://journals.sagepub.com/doi/abs/10.1177/2055207618785077?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .10.1177_2055207618785077 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Starling MS, Kandel Z, Haile L, Simmons RG. User profile and preferences in fertility apps for preventing pregnancy: an exploratory pilot study. Mhealth. 2018;4:21. doi: 10.21037/mhealth.2018.06.02. doi: 10.21037/mhealth.2018.06.02.mh-04-2018.06.02 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Saparamadu AADNS, Fernando P, Zeng P, Teo H, Goh A, Lee JMY, Lam CWL. User-centered design process of an mhealth app for health professionals: case study. JMIR Mhealth Uhealth. 2021 Mar 26;9(3):e18079. doi: 10.2196/18079. https://mhealth.jmir.org/2021/3/e18079/ v9i3e18079 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Narrillos-Moraza. Gómez-Martínez-Sagrera P, Amor-García. Escudero-Vilaplana V, Collado-Borrell R, Villanueva-Bueno C, Gómez-Centurión I, Herranz-Alonso A, Sanjurjo-Sáez M. Mobile apps for hematological conditions: review and content analysis using the Mobile App Rating Scale. JMIR Mhealth Uhealth. 2022 Feb 16;10(2):e32826. doi: 10.2196/32826. https://mhealth.jmir.org/2022/2/e32826/ v10i2e32826 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Nicholas J, Fogarty AS, Boydell K, Christensen H. The reviews are in: a qualitative content analysis of consumer perspectives on apps for bipolar disorder. J Med Internet Res. 2017 Apr 07;19(4):e105. doi: 10.2196/jmir.7273. http://www.jmir.org/2017/4/e105/ v19i4e105 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.van Haasteren A, Gille F, Fadda M, Vayena E. Development of the mHealth App Trustworthiness checklist. Digit Health. 2019;5:2055207619886463. doi: 10.1177/2055207619886463. http://europepmc.org/abstract/MED/31803490 .10.1177_2055207619886463 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Mandracchia F, Llauradó E, Tarro L, Valls RM, Solà R. Mobile phone apps for food allergies or intolerances in app stores: systematic search and quality assessment using the Mobile App Rating Scale (MARS) JMIR Mhealth Uhealth. 2020 Sep 16;8(9):e18339. doi: 10.2196/18339. https://mhealth.jmir.org/2020/9/e18339/ v8i9e18339 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Karasneh RA, Al-Azzam SI, Alzoubi KH, Muflih SM, Hawamdeh SS. Smartphone applications for period tracking: rating and behavioral change among women users. Obstet Gynecol Int. 2020 Sep 01;2020:2192387–9. doi: 10.1155/2020/2192387. doi: 10.1155/2020/2192387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015 Mar;3(1):e27. doi: 10.2196/mhealth.3422. http://mhealth.jmir.org/2015/1/e27/ v3i1e27 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the Mobile Application Rating Scale (uMARS) JMIR Mhealth Uhealth. 2016;4(2):e72. doi: 10.2196/mhealth.5849. http://mhealth.jmir.org/2016/2/e72/ v4i2e72 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lee JB, Woo H. Quality evaluation of smartphone applications for physical activity promotion. KOSHEP_EN. 2019 Oct 01;36(4):67–76. doi: 10.14367/kjhep.2019.36.4.67. [DOI] [Google Scholar]
- 27.Wright A. Evaluation of two mobile health apps for patients with breast cancer using the Mobile Application Rating Scale. Mhealth. 2021;7:60. doi: 10.21037/mhealth-20-161. doi: 10.21037/mhealth-20-161.mh-07-20-161 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Bardus M, Ali A, Demachkieh F, Hamadeh G. Assessing the quality of mobile phone apps for weight management: user-centered study with employees from a Lebanese university. JMIR Mhealth Uhealth. 2019 Jan 23;7(1):e9836. doi: 10.2196/mhealth.9836. http://mhealth.jmir.org/2019/1/e9836/ v7i1e9836 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Li Y, Ding J, Wang Y, Tang C, Zhang P. Nutrition-related mobile apps in the china app store: assessment of functionality and quality. JMIR Mhealth Uhealth. 2019 Jul 30;7(7):e13261. doi: 10.2196/13261. https://mhealth.jmir.org/2019/7/e13261/ v7i7e13261 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Lambrecht A, Vuillerme N, Raab C, Simon D, Messner E, Hagen M, Bayat S, Kleyer A, Aubourg T, Schett G, Hueber A, Knitza J. Quality of a supporting mobile app for rheumatic patients: patient-based assessment using the user version of the Mobile Application Scale (uMARS) Front Med (Lausanne) 2021;8:715345. doi: 10.3389/fmed.2021.715345. doi: 10.3389/fmed.2021.715345. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Song Y, Chen H. Evaluating chinese mobile health apps for ankylosing spondylitis management: systematic app search. JMIR Mhealth Uhealth. 2021 Jul 14;9(7):e27234. doi: 10.2196/27234. https://mhealth.jmir.org/2021/7/e27234/ v9i7e27234 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Novac O, Novac M, Gordan C, Berczes T, Bujdosó G. Comparative study of Google Android, Apple iOS and Microsoft Windows Phone mobile operating systems. 2017 14th International Conference on Engineering of Modern Electric Systems (EMES); June 1-2; Oradea, Romania. 2017. pp. 154–159. [DOI] [Google Scholar]
- 33.Symul L, Wac K, Hillard P, Salathé M. Assessment of menstrual health status and evolution through mobile apps for fertility awareness. NPJ Digit Med. 2019;2:64. doi: 10.1038/s41746-019-0139-4. doi: 10.1038/s41746-019-0139-4.139 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Namey E, Guest G, McKenna K, Chen M. Evaluating Bang for the Buck. AJE. 2016 Jul 09;37(3):425–440. doi: 10.1177/1098214016630406. [DOI] [Google Scholar]
- 35.Zhao J, Freeman B, Li M. Can mobile phone apps influence people's health behavior change? An evidence review. J Med Internet Res. 2016 Oct 31;18(11):e287. doi: 10.2196/jmir.5692. http://www.jmir.org/2016/11/e287/ v18i11e287 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Anderson K, Emmerton LM. Contribution of mobile health applications to self-management by consumers: review of published evidence. Aust Health Rev. 2016 Nov;40(5):591–597. doi: 10.1071/AH15162.AH15162 [DOI] [PubMed] [Google Scholar]
- 37.Adnan T, Coull BA, Jukic AM, Mahalingaiah S. The real-world applications of the symptom tracking functionality available to menstrual health tracking apps. Curr Opin Endocrinol Diabetes Obes. 2021 Dec 01;28(6):574–586. doi: 10.1097/MED.0000000000000682. http://europepmc.org/abstract/MED/34560714 .01266029-900000000-99164 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Tangari G, Ikram M, Ijaz K, Kaafar MA, Berkovsky S. Mobile health and privacy: cross sectional study. BMJ. 2021 Jun 16;373:n1248. doi: 10.1136/bmj.n1248. http://www.bmj.com/lookup/pmidlookup?view=long&pmid=34135009 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Alfawzan N, Christen M, Spitale G, Biller-Andorno N. Privacy, data sharing, and data security policies of women’s mhealth apps: scoping review and content analysis. JMIR Mhealth Uhealth. 2022 May 6;10(5):e33735. doi: 10.2196/33735. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Eng DS, Lee JM. The promise and peril of mobile health applications for diabetes and endocrinology. Pediatr Diabetes. 2013 Jun;14(4):231–8. doi: 10.1111/pedi.12034. http://europepmc.org/abstract/MED/23627878 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Costa Figueiredo M, Huynh T, Takei A, Epstein DA, Chen Y. Goals, life events, and transitions: examining fertility apps for holistic health tracking. JAMIA Open. 2021 Jan;4(1):ooab013. doi: 10.1093/jamiaopen/ooab013. https://europepmc.org/abstract/MED/33718804 .ooab013 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Pichon A, Jackman K, Winkler I, Bobel C, Elhadad N. The messiness of the menstruator: assessing personas and functionalities of menstrual tracking apps. J Am Med Inform Assoc. 2022 Jan 12;29(2):385–399. doi: 10.1093/jamia/ocab212. https://europepmc.org/abstract/MED/34613388 .6382239 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS) score of all 34 apps.
Content comparison between the top and bottom five apps (ranked according to the consumer and health care provider evaluation scores).