Abstract
The past decade marks a surge in the development of mobile apps used to digitally track and monitor aspects of personal health, including menstruation. Despite a plethora of menstruation-related apps, pain and symptom management content available in apps has not been systematically examined. The objective of this study was to evaluate app characteristics, overall quality (i.e., engagement, functionality, design aesthetics, and information), nature and quality of pain and symptom tracking features, and availability and quality of pain-related intervention content. A scoping review of apps targeting facets of the menstrual experience was conducted by searching the Apple App Store. After removal of duplicates and screening, 119 apps targeting menstrual experiences were retained. Pain and menstrual symptoms tracking were available in 64 % of apps. Checkboxes or dichotomous (present/absent) reporting was the most common method of tracking symptoms and was available in 75 % of apps. Only a small subset (n = 13) of apps allowed for charting/graphing of pain symptoms across cycles. Fourteen percent of apps included healthcare professionals or researchers in their development and one app reported use of end-users. Overall app quality measured through the Mobile App Rating Scale (MARS) was found to be acceptable; however, the apps ability to impact pain and symptom management (e.g., impact on knowledge, awareness, behaviour change, etc.) was rated as low. Only 10 % of apps (n = 12) had interventions designed to manage pain. The findings suggest that despite pain and symptom management content being present in apps, this content is largely not evidence-based in nature. More research is needed to understand how pain and symptom management content can be integrated into apps to improve user experiences.
Keywords: Menstrual pain, Dysmenorrhea, Endometriosis, Women's health, Mobile apps, Digital interventions
Highlights
-
•
Apps for managing menstruation rarely include content for pain management.
-
•
Pain tracking features rarely use validated and reliable pain assessment tools.
-
•
Stakeholders are overlooked in the development of apps for menstruation.
1. Introduction
Dysmenorrhea, or menstrual pain, occurs in 45 to 95 % of individuals who menstruate (Burnett et al., 2005; DeSanctis et al., 2015; Schoep et al., 2019). Primary dysmenorrhea is defined as painful menstruation and accompanying symptoms (e.g., headaches, nausea, vomiting, and diarrhea) that occur in the absence of any pelvic pathology (Osayande and Mehulic, 2014). Secondary dysmenorrhea occurs in the course of pelvic pathology such as endometriosis (Osayande and Mehulic, 2014). Dysmenorrhea can significantly impact daily functioning; for example, up to 66 % of adolescents who menstruate report missing at least one day of school per cycle due to pain, and nearly 15 % miss multiple school days per cycle (Pitangui et al., 2013; Söderman et al., 2019). Menstrual pain and associated interference with functioning also persists into adulthood. Specifically, over 40 % of individuals who menstruate between the ages of 16 and 50 report severe pain with menstruation and associated interference in professional, family, and social domains of their lives (Fernandez et al., 2020). Moreover, individuals affected by dysmenorrhea experience more mental health concerns, such as depression and anxiety (Bajalan et al., 2019; González-Echevarría et al., 2018); diminished quality of life (Gallagher et al., 2018; Hoppenbrouwers et al., 2016; Nur Azurah et al., 2013); and are at greater risk for comorbid pain conditions throughout their lifetime (Gagnon et al., 2022).
Nonsteroidal anti-inflammatory drugs (NSAIDs) and hormonal therapies are the recommended first line treatment for dysmenorrhea (Burnett and Lemyre, 2017). Yet, 25 % of individuals with dysmenorrhea do not experience symptom relief from these interventions and many suffer from side effects (Oladosu et al., 2018). Exercise, along with alternative and complementary treatments (e.g., heat, acupoint stimulation), are recommended by treatment guidelines (Burnett and Lemyre, 2017); however, these interventions are not accessible to everyone and can be difficult to implement outside the home. Evidence-based guidelines for the treatment of chronic and persistent pain recommend multimodal treatment approaches that include psychological components (Gatchel et al., 2007; Jensen and Turk, 2014). Despite this, research examining such interventions in the context of dysmenorrhea is limited. Given the strong evidence for psychologically-based interventions in other areas of pain management (Gatchel et al., 2007; Jensen and Turk, 2014), it is likely that such strategies could successfully be applied to menstrual pain. However, it should also be noted that several barriers to accessing psychological care exist including cost, geographical location, and a lack of trained providers (Ehde et al., 2014; Sareen et al., 2007).
Recent years have seen undeniable progress in the development of digital and mobile health (mHealth) interventions to improve health-related outcomes (Kraft and Yardley, 2009). Smartphone apps have the potential to overcome barriers to accessing care as they are low cost and accessible. Apps may also reduce perceived stigma and normalization of symptoms specifically experienced by individuals who menstruate (Berkley and McAllister, 2011; Wong, 2011).
When considering apps for non-menstrual pain, several apps have been developed (Devan et al., 2019) and apps that integrate evidence-based pain management content have been shown to lead to improvements in pain intensity and activity limitations due to pain (Lalloo et al., 2015). Apps for menstrual tracking provide an opportune platform for the inclusion of pain and symptom management content to help users better manage symptoms associated with menstruation. Moreover, apps for menstrual and fertility tracking are abundant (Zwingerman et al., 2020) and commonly used. For instance, among adolescents, menstrual tracking apps have been ranked as the second most downloaded health-related app (Diaz et al., 2006). While reviews of tracking and fertility apps have been conducted (Earle et al., 2021; Kalampalikis et al., 2022), little is known about available apps designed specifically to target menstrual pain and menstrual-related symptoms, including the specific features, tools, and interventions offered within these apps. Across menstrual app content that has been examined (e.g., tracking features), researchers have found that most apps are not developed using evidence-based approaches that consider the needs of users, the apps are not tested clinically with users to determine efficacy in changing behaviour (Karasneh et al., 2020), and apps are low in quality, and contain inaccuracies (Moglia et al., 2016). Thus, despite smartphone apps having the potential to provide viable self-management tools for individuals who experience primary dysmenorrhea, it is unclear whether such content is available, and if present, the nature and the quality of the content. Understanding current available self-management content in available apps is an important first step in informing how to improve on such content, as it can provide indication for future app development, and can inform researchers and app developers on how to better meet the needs of individuals with menstrual pain.
We undertook a scoping review of the content and quality of freely available mobile apps for menstruation. The goal of this review was to examine the pain-related content integrated within available apps, as well as rate the overall quality of the apps. Specifically, we examined (1) app characteristics; (2) app engagement, functionality, design aesthetics, and information using the Mobile App Rating Scale (MARS; Stoyanov et al., 2015); (3) nature and quality of pain and symptom tracking features; (4) and availability and quality of pain self-management content.
2. Methods
2.1. App identification
Given device availability of the researchers, we focused our review on apps for iOS. We conducted a search of the Apple App Store between January 26, 2021 and February 16, 2021. Search terms related to menstruation (e.g., menstrual, period, menstrual cycle) and pain (e.g., period cramps, dysmenorrhea, endometriosis, period pain) were identified (Table 1). Identified apps were compiled and duplicates were removed. Apple App Store information was extracted from the apps generated in this search (i.e., app name, developer, user star ratings, number of user ratings, recommended age, cost, and date of most recent update).
Table 1.
Search terms | Date | Results (n = 1091) |
---|---|---|
‘Menstrual’ | 2021-01-26 | 208 |
‘Menstruation’ | 2021-01-26 | 204 |
‘Period’ | 2021-02-11 | 197 |
‘Period Pain’ | 2021-02-12 | 210 |
‘Menstrual cramps’ | 2021-02-12 | 5 |
‘Menstrual pain’ | 2021-02-12 | 8 |
‘Period Cramps’ | 2021-02-12 | 19 |
‘Menstrual cycle’ | 2021-02-12 | 191 |
‘Dysmenorrhea’ | 2021-02-16 | 2 |
‘Pelvic Pain’ | 2021-02-16 | 17 |
‘Endometriosis’ | 2021-02-16 | 31 |
2.2. Inclusion and exclusion criteria
Based on the preliminary information extracted, apps were then reviewed to determine if the inclusion and exclusion criteria were met. The following were established as inclusion criteria: 1) focused on menstruation, which includes, but is not limited to menstrual pain, period monitoring/tracking, period symptom tracking; 2) available in English; and 3) free to download. Apps that contained in-app purchases were retained. Apps with a non-English name were retained if the Apple App store information was in English and the app contained an in-app English option after downloading. Apps were excluded if their primary purpose was related to conceiving and focused on tracking of fertility and ovulation; however, this exclusion generally occurred after a review of the content of the app, and not based on the inclusion of fertility-related words in the app name. Apps were also excluded if they provided an initial free trial followed by a paid subscription. Lastly, we excluded apps that were primarily used as a phone widget without a fully developed associated app. Throughout the completion of this review, apps that became unavailable or non-functional were excluded from the final count.
Two reviewers (L.T. and M.G.) reviewed the app names and descriptions in the Apple App Store for each app identified through the initial search in order to evaluate whether the app met inclusion criteria. Disagreements regarding whether to retain or exclude an app were resolved via discussion until consensus was achieved.
2.3. Data extraction and analysis
Apps that met the inclusion criteria were downloaded for further data extraction and evaluation. Information to be extracted was determined based on the MARS (Stoyanov et al., 2015) questions, as well as additional questions developed to meet the aims of this review. Apps were tested between February 2021 and October 2021. Each reviewer first downloaded the app, then created a login and password if necessary, and reviewed each page and feature within the app, including entering test data to ensure that app features were functional.
All apps were tested and reviewed by three reviewers (L.T., É.L., and S.B.) to ensure consistency in data extraction. Ratings from all three reviewers were used to examine rating consensus on the MARS. Given adequate consensus across reviewers, the primary coder's (L.T.) ratings and extracted data were used to generate all other results of this review. Data was compiled in a Microsoft Excel spreadsheet. Frequency and descriptive data were calculated to examine app characteristics (i.e., target age group, app store metrics including rating out of 5-stars and number of ratings provided), tracking features, health-care professional or researcher involvement in the app development, stereotypically feminine esthetic. Overall quality of apps was examined by calculating the MARS subscale mean values (described below). Pain and symptom management features and intervention type and nature were examined using calculations of frequencies and descriptive analyses, as well as narrative review of content.
In the months prior to completion of the review (May–July 2022), apps included in the review were re-evaluated to allow for updating of data and confirmation of accuracy of entries. Apps were divided across four reviewers (L.T., É.L., S.B., and C.M.) and each app was only examined by one reviewer.
2.3.1. Mobile Application Rating Scale
The MARS is a validated tool designed to evaluate the quality of mHealth apps (Stoyanov et al., 2015). It is comprised of four sections, which include: (1) app classification, (2) app quality ratings, (3) app subjective rating, and (4) app-specific quality section. For the purposes of this review, only the app classification, quality ratings, and app-specific quality sections were included. The MARS has been shown to have good psychometric properties (Stoyanov et al., 2015; Terhorst et al., 2020).
The app classification section is used to gather descriptive and technical information about the app, such as what topic the app targets (e.g., depression, physical health), the apps theoretical background (e.g., cognitive-behavioural therapy (CBT), behavioural, relaxation,), the apps affiliation (e.g., commercial, government), the target age groups of the app, and technical components (e.g., allows sharing, has an app community) of the app. The app quality rating section contains 19 items that describe overall app quality on four dimensions: engagement, functionality, aesthetics, and information quality. Each item is rated on a 5-point Likert scale ranging from 1 (inadequate) to 5 (excellent). A mean score in each dimension is calculated by summing item scores and dividing by the number of items in the dimension. A total app quality mean score is calculated by summing the average score of each of the app objective quality subscales and dividing it by 4, with higher scores indicating greater overall quality.
The app-specific quality scale includes 6-item scale assessing the impact of the app on users' awareness, knowledge, attitudes, intentions to change, help-seeking, and behaviour change related to the topic under review. The tool is designed so that the target health behaviour chosen by the reviewers completing the MARS scale can be inserted into the item. For the purposes of this review, the target area was defined as “menstrual pain and symptom management” (e.g., “This app is likely to increase awareness of the importance of addressing menstrual pain and symptom management”). Similar to the app quality scale, a mean score in the app-specific quality scale is assessed by summing the ratings and dividing by the number of items. Higher scores are indicative of better app-specific quality in the area assessed.
2.3.2. Menstrual pain and symptom-specific domains
To meet the goals of our review, additional pain and symptom content was examined. To assess the pain features of the app, we examined whether the app tracked pain, pain locations listed, measures used to rate pain, whether pain assessments could be examined across time, and other symptoms of the menstrual cycle tracked. Additionally, we examined if the app included any intervention that could be used to manage pain. We did not rely specifically on the app to state that the intervention was for pain, but rather made this assessment based on the nature of the intervention.
2.3.3. Assessment of app esthetic
Given that menstruation is typically considered as occurring in girls and women, but can occur in individuals of all genders, we examined the inclusivity of apps through consideration of the app esthetic. The review team met to discuss aspects that might be considered as “stereotypically feminine” and generated a list through examination of a small number of apps. Subsequently, a final list of esthetic characteristics was established through discussion and until consensus was reached. App characteristics that were determined to be stereotypically feminine included colour palettes that relied heavily on pinks and purples, flower graphics, female cartoon characters, and gender-focused language (e.g., “Hey girl, welcome to [app name]”).
2.4. Coding consensus and reliability
In line with recommendations from the MARS creators, the three reviewers watched the official training video (Stoyanov, 2016) to familiarize themselves with the scale prior to coding of the apps. The reviewers then met to establish consensus in coding by rating four apps together. Subsequently, each coder rated each app. To reduce coder drift in scoring, approximately halfway through the coding process, the coders met to conduct further consensus coding on three apps. In the event a reviewer was unable to access/download an app, a fourth reviewer (MG) was consulted to determine if the app should be included and to conduct the review of the app, if needed.
To ensure that apps were consistently evaluated by raters, interrater reliability of the MARS subscales and the total score was calculated using intraclass correlation coefficients (ICC). Using a two-way mixed-effects, average measures models account for absolute agreement, ICC for the Engagement Subscale = 0.91, Functionality Subscale = 0.72, Aesthetic subscale = 0.69, Information subscale = 0.74, and Total scale = 0.86. For the app specific scale, wherein we examine perceived impact on pain and symptom management, the ICC was = 0.85.
3. Results
A total of 1091 apps were identified. After removing 657 duplicates, the remaining 434 apps were screened for inclusion/exclusion criteria. After various stages of screening, 315 apps were excluded leaving a total of 119 apps to be evaluated (see Fig. 1 for PRISMA flow chart; Moher et al., 2009).
3.1. App characteristics
An overview of app characteristics can be found in Supplemental Table 1.
Although only freely available apps were included in the current review, in-app purchases were available in 48 % of apps. Such purchases typically provide access to additional app features or content, such as symptom management content. Apple Store metrics were examined to compare average star rating (out of 5) and number of ratings provided, as this information provides indication of use of the apps. Of the 119 apps, 32 had insufficient ratings to allow for a star rating. The remaining 87 apps had a number of ratings ranging from 1 (n = 16) to over 86,000. The star ratings ranged from 0 to 5 (the maximum number of stars available is 5); however, given that smaller number of ratings would skew the star rating, we examined apps that had more than 1000 ratings and the overall star rating of these apps. Only 12 identified apps had at least 1000 ratings (range = 1372–86,606). The average rating of these apps was 4.68/5 stars (range = 4.5–4.9). The most rated apps were Flo Period Tracker & Calendar (86,606 ratings, 4.7/5 stars); Clue Period Tracker, Ovulation (50,083 ratings, 4.7/5 stars); and Period Tracker Period Calendar (10,101 ratings, 4.9/5 stars).
Apps were reviewed to assess evidence of involvement of stakeholders, including healthcare providers or researchers, in the development or review of information. Of the reviewed apps, 77 % (n = 92) gave no indication of involvement of healthcare providers or researchers in the development or review of information. Fourteen percent of apps (n = 17) made clear reference to health professionals or health researchers. The remaining 8 % of apps (n = 10) provided insufficient information to evaluate whether healthcare providers or researchers were included. Reasons for this uncertainty included reference to “experts” but no specification of what kind of experts were meant; the ability to chat with a professional within the app, but no indication of who the professional was; or insufficient information provided within the app and links to external webpages affiliated with the app that were not in English. We found one app (Endometrix) that described using end-users (i.e., individuals with menstrual pain) in the development of the app.
Access to the internet was not required for 86 % of apps (n = 102). Fourteen apps required internet access for the app to function, while three apps (n = 119) required internet access for some app features. Reminders or notifications were available as a feature in 80 % of apps (n = 95). A login and password were required for 22 % of apps (n = 26), whereas 76 % (n = 90) did not require a login and password, and 3 % (n = 3) required a login and password for certain app features. Sharing features were available in 37 % of apps (n = 44).
Specific tracking features varied widely across apps, but frequently included a menstrual calendar (98 %), predicted ovulation and fertility windows or actual indicators of ovulation and fertility (e.g., cervical mucous, basal body temperature) (79 %), flow (58 %), and intercourse (63 %). Of the apps, 59 % were assessed as meeting the criteria for stereotypically feminine, with the remainder employing more neutral language, colour schemes, and images. Female names, or gendered words (e.g., women, female, lady, sister) were included in the names of 23 out of 119 apps (19 %).
3.2. MARS ratings
Average MARS scores across all subscales and items are summarized in Table 2. Overall, the mean item rating on the App Quality Ratings was 3.16 (out of 5), pointing to the apps being of acceptable quality. The mean rating across items on the Engagement subscale (n items = 5) was 2.59/5. Although no areas within this subscale were exceptionally strong, apps received higher scores on Customization (mean rating = 2.78/5) and scored lowest on Interest (mean rating = 2.32/5). The mean item score on the Functionality subscale was 3.81/5, and this subscale received the highest mean rating across all MARS subscales. The range across the four Functionality items was limited with the highest rated item being Gestural Design (mean rating 4.04/5) and the lowest rated item being Ease of Use (mean rating = 3.62/5). The mean score on the Aesthetic subscale was 3.39/5, with item scores ranging from 2.98/5 (Visual appeal) to 4.00/5 (Graphics). The mean score was lowest on the Information subscale (mean rating = 2.85/5), and items on this subscale ranged most widely, with the Credibility item receiving the lowest mean item rating across apps (mean rating = 1.62/5) and Accuracy of App Description item receiving the highest mean item rating (mean rating = 3.56/5).
Table 2.
MARS subscales and items | Mean | Range (min. - max.) |
---|---|---|
Subscale A: Engagement | 2.59 | 1.00–4.80 |
Item 1 - Entertainment | 2.35 | 1.00–5.00 |
Item 2 - Interest | 2.32 | 1.00–5.00 |
Item 3 - Customisation | 2.78 | 1.00–5.00 |
Item 4 - Interactivity | 2.63 | 1.00–5.00 |
Item 5 - Target group | 2.88 | 1.00–4.00 |
Subscale B: Functionality | 3.81 | 1.25–5.00 |
Item 6 - Performance | 3.76 | 1.00–5.00 |
Item 7 - Ease of use | 3.62 | 1.00–5.00 |
Item 8 - Navigation | 3.80 | 1.00–5.00 |
Item 9 - Gestural design | 4.04 | 1.00–5.00 |
Subscale C: Aesthetics | 3.39 | 1.00–5.00 |
Item 10 - Layout | 3.19 | 1.00–5.00 |
Item 11 - Graphics | 4.00 | 1.00–5.00 |
Item 12 - Visual appeal | 2.98 | 1.00–5.00 |
Subscale D: Information | 2.85 | 1.00–4.58 |
Item 13 - Accuracy of app description (in app store) | 3.56 | 1.00–5.00 |
Item 14 - Goals | 3.14 | 1.00–4.00 |
Item 15 - Quality of information | 3.32 | 1.00–5.00 |
Item 16 - Quantity of information | 3.41 | 1.00–5.00 |
Item 17 - Visual information | 2.97 | 1.00–5.00 |
Item 18 - Credibility | 1.62 | 1.00–4.00 |
Item 19 - Evidence base | 3.00 | 2.00–4.00 |
Subscale E: Target behaviour (pain and symptom management) | 1.39 | 1.00–4.00 |
Item 1 - Awareness | 2.34 | 1.00–5.00 |
Item 2 - Knowledge | 1.48 | 1.00–5.00 |
Item 3 - Attitudes | 1.08 | 1.00–4.00 |
Item 4 - Intention to change | 1.09 | 1.00–4.00 |
Item 5 - Help seeking | 1.27 | 1.00–4.00 |
Item 6 - Behaviour change | 1.09 | 1.00–3.00 |
Subscale total (subscales A, B, C, and D) | 3.16 | 1.48–4.58 |
Note. MARS = Mobile App Rating Scale. Subscale E is not included in the calculation of the Total MARS score. MARS ratings can range from 1 to 5.
Given that apps' goals were not necessarily to change pain behaviour and symptom management, not unexpectedly, scores within our App-specific subscale examining impact on pain and symptom management were low. The mean item score on perceived impact on pain and symptom management was 1.39/5. The Awareness item had the highest mean rating (2.34/5) while the Attitudes item had the lowest mean rating (1.08/5).
3.3. Pain and symptom tracking and management
Tracking of pain and other menstrual symptoms was available in 76 of 119 (64 %) apps. We grouped similar body areas together to identify pain sites represented in the apps. Over 15 specific pain sites or areas were listed. The most common sites listed for tracking included head (n = 67), breast/chest (n = 64), back (n = 55), abdomen/stomach (n = 21), neck (n = 19), pelvic (n = 11), and shoulder (n = 9). Several types of pain could also be tracked including cramps (n = 59), ovulation pain (n = 10), sexual pain (n = 5), and general muscle pain, joint pain, or body ache (n = 35). Across apps approaches to provision of pain ratings varied. The use of present/absent or a checkbox rating was used in 75 % of apps that allowed for pain ratings. Numeric rating scales that varied from 3 to 10 points were used by 26 % of apps that allowed pain rating, 20 % of apps used some form of verbal rating scale, and 7 % of apps used pictorial ratings (i.e., stars, lightning bolts, faces) to represent pain severity. Of note, the app that used faces to illustrate pain severity did not use a validated facial rating scale (e.g., Bieri et al., 1990). Thirteen of the apps tracked pain over time or across cycles using graphs or charts.
With regard to rating of menstrual symptoms more generally, gastrointestinal issues were commonly listed (e.g., bloating, gut issues, heartburn/indigestion, diarrhea, nausea, vomiting). Combined, gastrointestinal issues of some sort could be reported in 63 of 116 apps, with the most common symptoms being bloating (n = 56), nausea/vomiting (n = 54), diarrhea (n = 47), constipation (n = 46), and heartburn/indigestion (n = 17). Other common trackable symptoms included acne (n = 56), dizziness (n = 24), pre-menstrual syndrome (PMS) (n = 20), and fever (n = 20).
3.4. Education and intervention
Across apps, freely available interventions, broadly defined, were present in 13 % of apps (n = 16). Several apps (n = 7) had some free content, but only allowed full access to educational and intervention content through in-app purchase. Fourteen apps specifically had some form of pain-related intervention (see Table 3). Many of the interventions focused on topics not related to pain specifically but that may impact pain and symptoms, such as nutrition and exercise. Mindfulness and meditation content was available in several apps (n = 6), and was incorporated through in app access or links to guided audios or videos. One app encouraged mindfulness practice but did not provide access to any instructional content and rather allowed for tracking of the practice. Eleven apps provided educational information, with several of these specifically focused on pain and symptom management. However, the quality of information varied greatly and, in several instances, did not appear to be reviewed by healthcare professionals in the field. One app purported to improve menstrual pain through the use of the phone's vibrations. Only three apps tracked the implementation of the intervention.
Table 3.
Notes. *Not all content available in free version of the app. Some free content is available but full access requires in-app purchases.
PMS = premenstrual syndrome; PCOS = Polycystic Ovary Syndrome.
4. Discussion
Use of mHealth and eHealth technology to help with health-related outcomes and behaviour is a growing focus across areas of healthcare. Despite apps designed to track and manage menstrual pain and symptoms being available in abundance, little comprehensive information of the content, focus, and potential use of these apps is known. This scoping review aimed to address this gap in the literature. Our findings highlight that content geared towards pain and symptom management is very limited in available apps, and few apps are developed through evidence-based methods.
We identified 119 apps available through the Apple App store that focused on various aspects of menstruation, met our inclusion criteria, and were free for users to download. Despite the number of available apps, very few apps had been rated by a large number of users, which suggests that many apps are developed within this area but are not subsequently downloaded or used. Although our review did not allow for user perspectives on why certain apps are not well-used, this points to a need for understanding user needs in app creation, thus ensuring that users are engaging with a product that will work for them.
The MARS ratings provide some indication of strengths and weaknesses of the apps available for menstrual management. Overall, apps are acceptable, but not exceptional in quality. While the functionality of apps is largely acceptable, MARS ratings suggest that the degree to which apps enhances users' engagement and provide adequate information are areas for improvement among menstrual apps.
Overall, very few apps had freely available intervention content, and available content was frequently low quality. Scores from the app-specific subscale of the MARS, which allowed for examination of the pain and symptom management content and degree to which it would lead to knowledge and behaviour change, provided additional support that this content is lacking across apps. Education, such as tips and recommendations were the most common type of intervention content available to help manage pain. Although several evidence-based approaches, including yoga, exercise/stretching, and nutrition management (Burnett and Lemyre, 2017; Kim, 2019) were among the interventions available in apps, there was no consistent indication that the specific content itself was consistent with the available research evidence or developed/reviewed by health professionals.
Despite the successful implementation of psychologically based pain management approaches (e.g., mindfulness, CBT) for non-menstrual pain, within digital and app-based platforms (Chadi et al., 2019; Mikolasek et al., 2018; Pfeifer et al., 2020), only six apps offered access to mindfulness and meditation content. We did not find any apps that focused specifically on the development of cognitive or behavioural skills for pain. Given the pervasive nature of dysmenorrhea, the need for support for people who suffer from it may exceed the availability of health care providers to provide treatment. Access to freely available, evidence-based self-management tools and interventions via digital mediums could thus provide an important avenue for those struggling with their menstrual symptoms.
Several features were consistently observed across most apps, including the ability to enter period information (e.g., start dates, end dates) in a calendar, prediction of upcoming menstruation, and characteristics of the menstrual period (e.g., flow). Apps most often used checkboxes for users to track pain ratings, pain locations, pain severity, and the presence or absence of additional symptoms. There is considerable guidance on reliable and valid tools for pain assessment within the broader pain literature (Atisook et al., 2021; Jensen et al., 1986; Ruskin et al., 2014). Given that these apps were mainly not developed by researchers or clinicians, it is not surprising that they largely do not integrate these reliable and valid pain assessment measures. This brings into question the utility of the pain ratings provided by users in the apps. While some apps used validated approaches (i.e., 10-point numerical rating scales, verbal rating scales) many apps relied on non-validated approaches to tracking pain such as dichotomies, verbal rating scales with non-validated descriptors, or images.
Although tracking features varied from app to app, there is little guidance in the literature related to which tracking features are most important to app users who menstruate. User-centred design, which integrates the needs and preferences of end-users (Schnall et al., 2016), is typically used to help with determining these features, and can also lead to more effective and sustained use of apps (McCurdie et al., 2012; Wolpin and Stewart, 2011). Our results indicate that only one app included in the current review utilized a user-centered design approach. Although developers often use various approaches to gain feedback from users in the app development phase (Van Velsen et al., 2013; Vermeeren et al., 2010), these processes were not reported within any of the apps, and we were unable to find any published research describing design processes of these apps. Moreover, given the health focus of these apps, consultation with experts in the field is critical to the accuracy of the app features and content.
Tailoring apps to the target population, including consideration of age and phase of life of users, is integral to increasing satisfaction and engagement with apps (Kaveladze et al., 2022; McCurdie et al., 2012). In the context of menstrual apps, this may mean the ability to add or remove certain features, or the inclusion of tailored content and features depending on life stage (post-natal, peri-menopause, newly menstruating). Adolescents in particular have different developmental abilities, knowledge needs, and menstrual cycle characteristics than older menstruators (Diaz et al., 2006). Consistent with Eschler et al.' (2019) finding that menstrual tracking apps underserve new menstruators, our review identified only two apps designed specifically for adolescents.
Consideration of gender stereotypes and end-user preferences is also necessary in menstrual app development. Individuals who menstruate and who do not identify as women or girls often report that menstruating or aspects of reproductive health that remind them of their sex is distressing (Chrisler et al., 2016; Epstein et al., 2017). Recent research indicates that individuals who identify as female often perceive these design elements as condescending and undesirable and instead prefer apps that are more neutral and less gendered (Epstein et al., 2017). The current review similarly found that more than half of the apps used stereotypically feminine colours, languages, and images. Thus, inclusivity in aesthetic should be considered in the design and development of menstrual pain focused apps moving forward.
4.1. Future directions and recommendations
Several recommendations and directions for future research can be identified. First, integration of evidence-based approaches is needed both in the development and the content of apps. This would include user-centred designs, evidence-based interventions and self-management content, and use of validated pain assessment tools. Incorporation of end-users and stakeholders (e.g., clinicians) in the design process would address several of the issues identified in our review, including non-inclusive and gendered designs and language, and poor tailoring to developmental needs of users. To address gaps in health care, menstrual apps developed in collaboration with researchers and research teams could integrate content that is, ideally, freely available to address needs of individuals struggling with menstrual symptoms. Additionally, following development, appropriate testing of app efficacy through randomized-controlled trials is warranted. Moreover, there is greater opportunity among app developers to link tracking features with implementation of self-management content to provide indication of whether tools used by users are effective at managing their symptoms. If apps are developed appropriately, there is opportunity for identification of potential problems with menstruation. Symptoms that may be indicative of health issues, or consistently high pain ratings, could be flagged in the app for the individual to follow up with their health care provider.
4.2. Limitations
There are several limitations to this review. First, given device access in our research team, we only conducted this review in iOS. Thus, there may be apps that are available for non-apple devices that were not reviewed. Second, given the everchanging landscape of apps throughout the completion of this review, app features often changed or updated, or apps became inaccessible within the app store. Although we completed a final review of included apps to ensure the most up-to-date information was provided for the apps included in our review, it is likely that new apps that would have fit our inclusion criteria were added to the app store since our initial search. Third, our aim was to focus on apps that were not geared towards fertility and conception; however, there is substantial overlap among apps that are used for menstrual tracking and symptoms, and apps that are used for fertility and conception. Decisions around when to exclude fertility-focused apps were made based on predetermined criteria; however, several of these apps likely held features that we evaluated in the current review and may be useful to individuals interested in having children who wish to rely on a single app for menstrual tracking pre-and post- pregnancy and for assisting with maximizing chances of conceiving. Fourth, cultural-specific factors that might impact the content of the apps and presence of cultural adaptations of apps were not examined and should be considered in future research. Lastly, privacy related to health apps has been of increasing focus in recent years (Filkins et al., 2016; Martínez-Pérez et al., 2015) and particularly with apps that track menstruation (Fabricius, 2021). Although we examined apps that allowed for data security in the sense of use of usernames and passwords, we did not examine the privacy and security settings of the apps themselves nor their hosting services; however, this has been examined by other researchers (Fowler et al., 2020; Shipp and Blasco, 2020).
4.3. Conclusions
There is a lack of evidence-based pain and symptom management content available in current apps for menstrual management. The present review is the first to systematically examine pain and symptom management content by investigating what is available as well as providing numerical evaluation of the available content using the MARS Scale. Our results provide an important overview of what is available in apps, the characteristics of apps focused on menstrual tracking and management, and the nature of available pain and symptom intervention content. Inclusion of health-care providers and experts in the development of apps only occurs in 14 % of apps, and user involvement in design and development was only found to occur in one app. Future studies and app development should focus on including users in the design and development; tailoring content for the specific needs of using, including adapting content for life stage and reproductive needs (e.g., new menstruator, peri-menopause) of users; and inclusion of evidence-based content that is freely available to provide guidance for pain and symptom management.
The following is the supplementary data related to this article.
Funding
L. C. M. Trépanier was supported by a Canadian Institutes of Health Research (CIHR) Canada Graduate Scholarship at the time of the development of this manuscript. N. M. Alberts holds a Canada Research Chair (Tier 2) in Behavioural Health Intervention, and this research was undertaken, in part, thanks to funding from the Canada Research Chairs Program. M. M. Gagnon was funded by a Canadian Institutes of Health Research (CIHR) Early Career Investigators Grant in Maternal, Reproductive, Child & Youth Health during the development of this manuscript.
Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: L. C. M. Trépanier was supported by a CIHR Canada Graduate Scholarship at the time of the development of this manuscript. N. M. Alberts holds a Canada Research Chair (Tier 2) in Behavioural Health Intervention, and this research was undertaken, in part, thanks to funding from the Canada Research Chairs Program. M. M. Gagnon was funded by a Canadian Institutes of Health Research (CIHR) Early Career Investigators Grant in Maternal, Reproductive, Child & Youth Health during the development of this manuscript.
References
- Atisook R., Euasobhon P., Saengsanon A., Jensen M.P. Validity and utility of four pain intensity measures for use in international research. J. Pain Res. 2021;14:1129–1139. doi: 10.2147/JPR.S303305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bajalan Z., Moafi F., MoradiBaglooei M., Alimoradi Z. Mental health and primary dysmenorrhea: a systematic review. J. Psychosom. Obstet. Gynecol. 2019;40(3):185–194. doi: 10.1080/0167482X.2018.1470619. [DOI] [PubMed] [Google Scholar]
- Berkley K.J., McAllister S.L. Don’t dismiss dysmenorrhea! Pain. 2011;152(9):1940–1941. doi: 10.1016/j.pain.2011.04.013. [DOI] [PubMed] [Google Scholar]
- Bieri D., Reeve R.A., Champion G.D., Addicoat L., Ziegler J.B. The faces pain scale for the self-assessment of the severity of pain experienced by children: development, initial validation, and preliminary investigation for ratio scale properties. Pain. 1990;41(2):139–150. doi: 10.1016/0304-3959(90)90018-9. [DOI] [PubMed] [Google Scholar]
- Burnett M., Lemyre M. No. 345-primary dysmenorrhea consensus guideline. J. Obstet. Gynaecol. Can. 2017;39(7):585–595. doi: 10.1016/j.jogc.2016.12.023. [DOI] [PubMed] [Google Scholar]
- Burnett M., Antao V., Black A., Feldman K., Grenville A., Lea R., Lefebvre G., Pinsonneault O., Robert M. Prevalence of primary dysmenorrhea in Canada. J. Obstet. Gynaecol. Can. 2005;27(8):765–770. doi: 10.1016/S1701-2163(16)30728-9. [DOI] [PubMed] [Google Scholar]
- Chadi N., Weisbaum E., Malboeuf-Hurtubise C., Kohut S.A., Viner C., Palaniyar N., Kaufman M., Locke J., Vo D.X. In-person vs. eHealth mindfulness-based intervention for adolescents with chronic illnesses: a pilot randomized trial. Adolesc. Psychiatry. 2019;9(1):11–23. doi: 10.2174/2210676608666181031102235. [DOI] [Google Scholar]
- Chrisler J.C., Gorman J.A., Manion J., Murgo M., Barney A., Adams-Clark A., Newton J.R., McGrath M. Queer periods: attitudes toward and experiences with menstruation in the masculine of Centre and transgender community. Cult. Health Sex. 2016;18(11):1238–1250. doi: 10.1080/13691058.2016.1182645. [DOI] [PubMed] [Google Scholar]
- DeSanctis V., Soliman A., Bernasconi S., Bianchin L., Bona G., Bozzola M., Buzi F., De Sanctis C., Tonini G., Rigon F., Perissinotto E. Primary dysmenorrhea in adolescents: prevalence, impact and recent knowledge. Pediatr. Endocrinol. Rev. 2015;13(2):512–520. https://www.ncbi.nlm.nih.gov/pubmed/26841639 [PubMed] [Google Scholar]
- Devan H., Farmery D., Peebles L., Grainger R. Evaluation of self-management support functions in apps for people with persistent pain: systematic review. JMIR Mhealth Uhealth. 2019;7(2) doi: 10.2196/13080. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Diaz A., Laufer M.R., Breech L.L. Menstruation in girls and adolescents: using the menstrual cycle as a vital sign. Pediatrics. 2006;118(5):2245–2250. doi: 10.1542/peds.2006-2481. [DOI] [PubMed] [Google Scholar]
- Earle S., Marston H.R., Hadley R., Banks D. Use of menstruation and fertility app trackers: a scoping review of the evidence. BMJ Sex. Reprod. Health. 2021;47(2):90–101. doi: 10.1136/bmjsrh-2019-200488. [DOI] [PubMed] [Google Scholar]
- Ehde D.M., Dillworth T.M., Turner J.A. Cognitive-behavioral therapy for individuals with chronic pain: efficacy, innovations, and directions for research. Am. Psychol. 2014;69(2):153–166. doi: 10.1037/a0035747. [DOI] [PubMed] [Google Scholar]
- Epstein D.A., Lee N.B., Kang J.H., Agapie E., Schroeder J., Pina L.R., Fogarty J., Kientz J.A., Munson S. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017. Examining menstrual tracking to inform the design of personal informatics tools; pp. 6876–6888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Eschler J., Menking A., Fox S., Backonja U. Defining menstrual literacy with the aim of evaluating mobile menstrual tracking applications. Comput. Inform. Nurs. 2019;37(12):638–646. doi: 10.1097/CIN.0000000000000559. [DOI] [PubMed] [Google Scholar]
- Fabricius A. 2021 IEEE International Symposium on Ethics in Engineering, Science and Technology (ETHICS), 1–1. 2021. Privacy is a feminist issue: reconsidering data sharing in menstrual self-tracking apps. [DOI] [Google Scholar]
- Fernandez H., Barea A., Chanavaz-Lacheray I. Prevalence, intensity, impact on quality of life and insights of dysmenorrhea among French women: a cross-sectional web survey. J. Gynecol. Obstet. Hum. Reprod. 2020;49(10) doi: 10.1016/j.jogoh.2020.101889. [DOI] [PubMed] [Google Scholar]
- Filkins B.L., Kim J.Y., Roberts B., Armstrong W., Miller M.A., Hultner M.L., Castillo A.P., Ducom J.-C., Topol E.J., Steinhubl S.R. Privacy and security in the era of digital health: what should translational researchers know and do about it? Am. J. Transl. Res. 2016;8(3):1560–1580. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4859641/ [PMC free article] [PubMed] [Google Scholar]
- Fowler L.R., Gillard C., Morain S.R. Readability and accessibility of terms of service and privacy policies for menstruation-tracking smartphone applications. Health Promot. Pract. 2020;21(5):679–683. doi: 10.1177/1524839919899924. [DOI] [PubMed] [Google Scholar]
- Gagnon M.M., Moussaoui D., Gordon J.L., Alberts N.M., Grover S.R. Dysmenorrhea across the lifespan: a biopsychosocial perspective to understanding the dysmenorrhea trajectory and association with comorbid pain experiences. Pain. 2022 doi: 10.1097/j.pain.0000000000002649. Publish Ah. [DOI] [PubMed] [Google Scholar]
- Gallagher J.S., DiVasta A.D., Vitonis A.F., Sarda V., Laufer M.R., Missmer S.A. The impact of endometriosis on quality of life in adolescents. J. Adolesc. Health. 2018;63(6):766–772. doi: 10.1016/j.jadohealth.2018.06.027. [DOI] [PubMed] [Google Scholar]
- Gatchel R.J., Peng Y.B., Peters M.L., Fuchs P.N., Turk D.C. The biopsychosocial approach to chronic pain: scientific advances and future directions. Psychol. Bull. 2007;133(4):581–624. doi: 10.1037/0033-2909.133.4.581. [DOI] [PubMed] [Google Scholar]
- González-Echevarría A.M., Rosario E., Acevedo S., Flores I. Impact of coping strategies on quality of life of adolescents and young women with endometriosis. J. Psychosom. Obstet. Gynecol. 2018;1–8 doi: 10.1080/0167482X.2018.1450384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hoppenbrouwers K., Roelants M., Meuleman C., Rijkers A., Van Leeuwen K., Desoete A., D’Hooghe T. Characteristics of the menstrual cycle in 13-year-old flemish girls and the impact of menstrual symptoms on social life. Eur. J. Pediatr. 2016;175(5):623–630. doi: 10.1007/s00431-015-2681-7. [DOI] [PubMed] [Google Scholar]
- Jensen M.P., Turk D.C. Contributions of psychology to the understanding and treatment of people with chronic pain: why it matters to ALL psychologists. Am. Psychol. 2014;69(2):105–118. doi: 10.1037/a0035641. [DOI] [PubMed] [Google Scholar]
- Jensen M.P., Karoly P., Braver S. The measurement of clinical pain intensity: a comparison of six methods. Pain. 1986;27(1):117–126. doi: 10.1016/0304-3959(86)90228-9. [DOI] [PubMed] [Google Scholar]
- Kalampalikis A., Chatziioannou S.S., Protopapas A., Gerakini A.M., Michala L. mHealth and its application in menstrual related issues: a systematic review. Eur J Contracept Reprod Health Care. 2022;27(1):53–60. doi: 10.1080/13625187.2021.1980873. [DOI] [PubMed] [Google Scholar]
- Karasneh R.A., Al-Azzam S.I., Alzoubi K.H., Muflih S.M., Hawamdeh S.S. Smartphone applications for period tracking: rating and behavioral change among women users. Obstet. Gynecol. Int. 2020;2020:1–9. doi: 10.1155/2020/2192387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kaveladze B.T., Wasil A.R., Bunyi J.B., Ramirez V., Schueller S.M. User experience, engagement, and popularity in mental health apps: secondary analysis of app analytics and expert app reviews. JMIR Hum. Factors. 2022;9(1) doi: 10.2196/30766. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kim S.-D. Yoga for menstrual pain in primary dysmenorrhea: a meta-analysis of randomized controlled trials. Complement. Ther. Clin. Pract. 2019;36:94–99. doi: 10.1016/j.ctcp.2019.06.006. [DOI] [PubMed] [Google Scholar]
- Kraft P., Yardley L. Current issues and new directions in psychology and health : what is the future of digital interventions for health behaviour change? Psychol. Health. 2009;24(6):615–618. doi: 10.1080/08870440903068581. [DOI] [PubMed] [Google Scholar]
- Lalloo C., Jibb L.A., Rivera J., Agarwal A., Stinson J.N. There’s a pain app for that: review of patient-targeted smartphone applications for pain management. Clin. J. Pain. 2015;31(6):557–563. doi: 10.1097/AJP.0000000000000171. [DOI] [PubMed] [Google Scholar]
- Martínez-Pérez B., de la Torre-Díez I., López-Coronado M. Privacy and security in Mobile health apps: a review and recommendations. J. Med. Syst. 2015;39(1):181. doi: 10.1007/s10916-014-0181-3. [DOI] [PubMed] [Google Scholar]
- McCurdie T., Taneva S., Casselman M., Yeung M., McDaniel C., Ho W., Cafazzo J. mHealth consumer apps: the case for user-centered design. Biomed. Instrum. Technol. 2012;46(s2):49–56. doi: 10.2345/0899-8205-46.s2.49. [DOI] [PubMed] [Google Scholar]
- Mikolasek M., Berg J., Witt C.M., Barth J. Effectiveness of mindfulness- and relaxation-based eHealth interventions for patients with medical conditions: a systematic review and synthesis. Int. J. Behav. Med. 2018;25(1):1–16. doi: 10.1007/s12529-017-9679-7. [DOI] [PubMed] [Google Scholar]
- Moglia M.L., Nguyen H.V., Chyjek K., Chen K.T., Castaño P.M. Evaluation of smartphone menstrual cycle tracking applications using an adapted APPLICATIONS scoring system. Obstet. Gynecol. 2016;127(6):1153–1160. doi: 10.1097/AOG.0000000000001444. [DOI] [PubMed] [Google Scholar]
- Moher D., Liberati A., Tetzlaff J., Altman D.G. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7) doi: 10.1371/journal.pmed.1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nur Azurah A.G., Sanci L., Moore E., Grover S. The quality of life of adolescents with menstrual problems. J. Pediatr. Adolesc. Gynecol. 2013;26(2):102–108. doi: 10.1016/j.jpag.2012.11.004. [DOI] [PubMed] [Google Scholar]
- Oladosu F.A., Tu F.F., Hellman K.M. Nonsteroidal antiinflammatory drug resistance in dysmenorrhea: epidemiology, causes, and treatment. Am. J. Obstet. Gynecol. 2018;218(4):390–400. doi: 10.1016/j.ajog.2017.08.108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Osayande A.S., Mehulic S. Diagnosis and initial management of dysmenorrhea. Am. Fam. Physician. 2014;89(5):341–346. http://www.ncbi.nlm.nih.gov/pubmed/24695505 [PubMed] [Google Scholar]
- Pfeifer A.-C., Uddin R., Schröder-Pfeifer P., Holl F., Swoboda W., Schiltenwolf M. Mobile application-based interventions for chronic pain patients: a systematic review and meta-analysis of effectiveness. J. Clin. Med. 2020;9(11):3557. doi: 10.3390/jcm9113557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pitangui A.C.R., Gomes M.R.de A., Lima A.S., Schwingel P.A., de Araújo R.C., Albuquerque A.P.dos S. Menstruation disturbances: prevalence, characteristics, and effects on the activities of daily living among adolescent girls from Brazil. J. Pediatr. Adolesc. Gynecol. 2013;26(3):148–152. doi: 10.1016/j.jpag.2012.12.001. [DOI] [PubMed] [Google Scholar]
- Ruskin D., Lalloo C., Amaria K., Stinson J.N., Kewley E., Campbell F., Brown S.C., Jeavons M., McGrath P.A. Assessing pain intensity in children with chronic pain: convergent and discriminant validity of the 0 to 10 numerical rating scale in clinical practice. Pain Res. Manag. 2014;19(3):141–148. doi: 10.1155/2014/856513. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sareen J., Jagdeo A., Cox B.J., Clara I., ten Have M., Belik S.-L., de Graaf R., Stein M.B. Perceived barriers to mental health service utilization in the United States, Ontario, and the Netherlands. Psychiatr. Serv. 2007;58(3):357–364. doi: 10.1176/ps.2007.58.3.357. [DOI] [PubMed] [Google Scholar]
- Schnall R., Rojas M., Bakken S., Brown W., Carballo-Dieguez A., Carry M., Gelaude D., Mosley J.P., Travers J. A user-centered model for designing consumer mobile health (mHealth) applications (apps) J. Biomed. Inform. 2016;60:243–251. doi: 10.1016/j.jbi.2016.02.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schoep M.E., Nieboer T.E., van der Zanden M., Braat D.D.M., Nap A.W. The impact of menstrual symptoms on everyday life: a survey among 42,879 women. Am. J. Obstet. Gynecol. 2019;220(6) doi: 10.1016/j.ajog.2019.02.048. 569.e1-569.e7. [DOI] [PubMed] [Google Scholar]
- Shipp L., Blasco J. How private is your period?: A systematic analysis of menstrual app privacy policies. Proc. Priv. Enhancing Technol. 2020;2020(4):491–510. doi: 10.2478/popets-2020-0083. [DOI] [Google Scholar]
- Söderman L., Edlund M., Marions L. Prevalence and impact of dysmenorrhea in swedish adolescents. Acta Obstet. Gynecol. Scand. 2019;98(2):215–221. doi: 10.1111/aogs.13480. [DOI] [PubMed] [Google Scholar]
- Stoyanov S.R. YouTube; 2016. MARS training video.https://www.youtube.com/watch?v=25vBwJQIOcE [Google Scholar]
- Stoyanov S.R., Hides L., Kavanagh D.J., Zelenko O., Tjondronegoro D., Mani M. Mobile app rating scale: a new tool for assessing the quality of health Mobile apps. JMIR Mhealth Uhealth. 2015;3(1) doi: 10.2196/mhealth.3422. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Terhorst Y., Philippi P., Sander L.B., Schultchen D., Paganini S., Bardus M., Santo K., Knitza J., Machado G.C., Schoeppe S., Bauereiß N., Portenhauser A., Domhardt M., Walter B., Krusche M., Baumeister H., Messner E.-M. Validation of the Mobile application rating scale (MARS) PLOS ONE. 2020;15(11) doi: 10.1371/journal.pone.0241480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Van Velsen L., Wentzel J., Van Gemert-Pijnen J.E. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res. Protoc. 2013;2(1) doi: 10.2196/resprot.2547. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vermeeren A.P.O.S., Law E.L.-C., Roto V., Obrist M., Hoonhout J., Väänänen-Vainio-Mattila K. Proceedings of the 6th Nordic Conference on Human-Computer Interaction Extending Boundaries - NordiCHI ’10. 2010. User experience evaluation methods; p. 521. [DOI] [Google Scholar]
- Wolpin S., Stewart M. A deliberate and rigorous approach to development of patient-centered technologies. Semin. Oncol. Nurs. 2011;27(3):183–191. doi: 10.1016/j.soncn.2011.04.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wong L.P. Attitudes towards dysmenorrhoea, impact and treatment seeking among adolescent girls: a rural school-based survey. Aust. J. Rural Health. 2011;19(4):218–223. doi: 10.1111/j.1440-1584.2011.01213.x. [DOI] [PubMed] [Google Scholar]
- Zwingerman R., Chaikof M., Jones C. A critical appraisal of fertility and menstrual tracking apps for the iPhone. J. Obstet. Gynaecol. Can. 2020;42(5):583–590. doi: 10.1016/j.jogc.2019.09.023. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.