The Needle in the Haystack: Identifying Credible Mobile Health Apps for Pediatric Populations during a Pandemic and beyond

Abstract

Background

The COVID-19 pandemic has ignited wider clinical adoption of digital health tools, including mobile health apps (mHealth apps), to address mental and behavioral health concerns at a distance. While mHealth apps offer many compelling benefits, identifying effective apps in the crowded and largely unregulated marketplace is laborious. Consumer demand and industry productivity are increasing, although research is slower, making it challenging for providers to determine the most credible and safe apps for patients in need.

Objectives/Methods

This commentary offers a practical, empirically guided framework and associated resources for selecting appropriate mHealth apps for pediatric populations during the pandemic and beyond.

Results

In the first stage, Narrow the target problem, end user, and contender apps. Beginning the search with continuously updated websites that contain expert app ratings can help expedite this process (e.g., Psyberguide). Second, Explore each contender app’s: (a) scientific and theoretical support (e.g., are app components consistent with health behavior change theories?), (b) privacy policies, and (c) user experience (e.g., through crowdsourcing feedback about app usability and appeal via social media). Third, use clinical expertise and stakeholder feedback to Contextualize whether the selected app is a good fit for a particular patient and/or caregiver (e.g., by considering age, race/ethnicity, ability, gender, sexual orientation, technology access), including conducting a brief self-pilot of the app.

Conclusion

Youth are increasingly turning to technology for support, especially during the pandemic, and pediatric psychologists must be primed to recommend the most credible tools. We offer additional recommendations for rapidly disseminating evidence-based apps to the public.

Introduction

Now more than ever, the current global COVID-19 pandemic is emphasizing the value of leveraging digital approaches to deliver pediatric health care safely at a distance (Stiles-Shields et al., 2020). Preliminary research from China suggests that the prevalence of depression, anxiety, insomnia, and other stress-related symptoms have spiked since the COVID-19 outbreak (Liu et al., 2020). Given the restrictions on face-to-face psychology and medical visits, many are realizing the potential reach of tools such as telehealth, wearable devices, and mobile health applications to help manage these challenges (i.e., mHealth apps; Badawy & Radovic, 2020). This sudden adoption of digital tools referred to as healthcare’s digital revolution (Keesara et al., 2020), is a proposed solution for delivering mental and behavioral health services during the pandemic (Holmes et al., 2020). Children, adolescents, and many caregivers own smartphones, use these devices almost constantly throughout the day, and are native app explorers, making mHealth apps particularly relevant for pediatrics (Anderson & Jiang, 2018). Indeed, nearly two-thirds of adolescents and young adults in the United States report that they have used an app for health behavior support, including fitness, sleep, meditation, and medication reminders (Rideout & Fox, 2018).

There are over 10,000 publicly available mHealth apps in the Apple iTunes and Google Play stores. These apps target diverse health-related outcomes and offer a variety of features (e.g., chat bots and symptom tracking). For example, there are apps that can promote medication adherence through reminders and incentives (e.g., Mango Health), support users in managing distressing thoughts and feelings through cognitive-behavioral principles (e.g., Woebot; Fitzpatrick et al., 2017), or guide caregivers to help their children develop problem-solving and emotion regulation skills (e.g., Breathe, Think, Do with Sesame). mHealth apps offer many compelling benefits, including abilities to deliver interventions and practice therapeutic skills in real-world environments, assess patient-reported outcomes between clinical encounters, and address access-to-care issues by offering free or low-cost interventions at home (Fedele et al., 2017). Moreover, there is a growing evidence base that mHealth apps are potentially efficacious for improving youth health behaviors (e.g., physical activity; Fedele et al., 2017) and mental health (e.g., depressive symptoms; Firth et al., 2017). In one meta-analysis (Linardon et al., 2019), effect sizes were larger for apps that incorporated professional guidance compared to standalone apps, supporting the use of apps to supplement (rather than replace) care from a pediatric psychologist.

Yet, while some mHealth apps are efficacious, the vast majority have not undergone rigorous testing or expert vetting (Leigh & Flatt, 2015; Neary & Schueller, 2018; Powell et al., 2014; Torous & Powell, 2015). For example, in 2013, only 32 of 1536 (2%) publicly available depression apps had published articles exploring their use (Martínez-Pérez et al., 2013). With limited exceptions (e.g., WebMAP—an app to support teenagers with pain management; Palermo et al., 2018), apps developed and tested by behavioral scientists are rarely disseminated to the public. Moreover, the majority of apps are not regulated by the United States Food and Drug Administration’s (FDA) because they are not considered medical devices that pose immediate safety risks if they malfunction. FDA regulation has been relaxed further during the pandemic to expand the availability of digital health therapeutics (FDA, 2020). This problem of “high availability but low evidence base” (Leigh & Flatt, 2015. p. 97) makes the crowded, rapidly changing, and largely unregulated app marketplace challenging to navigate.

While the first priority should be establishing an evidence base for mHealth apps, research is slower and less nimble than industry productivity and cannot keep pace with consumer demand (Boudreaux et al., 2014; Jake-Schoffman et al., 2017). People are increasingly turning to technology for support during the pandemic (Liu et al., 2020; NHS, 2020), and app developers are meeting this need; a number of apps have been rapidly developed or modified to support users during this time (e.g., COVID Coach, Nod, and SuperBetter). Many users cannot afford to wait for research to catch up. Practice guidelines for selecting appropriate mHealth apps in pediatrics are not well-established, potentially increasing the likelihood that providers will avoid apps altogether (despite their potential benefits), and/or prescribe ineffective, inappropriate, or potentially harmful apps. These challenges pose barriers to pediatric psychologists who seek to recommend credible mHealth apps. With a lack of evidence base and formal regulation, app practice guidelines are needed now, in the era of COVID-19, when pediatric behavioral health concerns may be increased and unique barriers to care exist.

In this commentary, we propose a three-step decision-making framework for selecting appropriate mHealth apps for youth presenting to pediatric specialty medical clinics or primary care and their caregivers: (a) Narrow, (b) Explore, and (c) Contextualize (see Figure 1). While the latter stage relies on clinical judgment and collaborative feedback from patients and caregivers, there are resources available to help with the first and second stages. This framework builds upon previously discussed strategies for evaluating and selecting internet-based (Ritterband et al., 2009) and app interventions (Boudreaux et al., 2014; Jake-Schoffman et al., 2017) for adults by focusing specifically on pediatric populations and their relevant social ecological contexts, outlining special considerations during the COVID-19 pandemic, and providing up-to-date resources. Further, the current framework is aimed at psychologists who may be stretched particularly thin during the pandemic (e.g., full case load while managing extra childcare) and therefore requires only three steps. We then discuss future directions for advancing the public health agenda to improve the scientific rigor of pediatric mHealth apps.

Figure 1.

Decision-making framework for selecting credible mHealth apps in pediatrics.

Stage 1: Narrow

In the first stage, Narrow the target problem (e.g., treatment nonadherence), the intended recipient of the intervention (i.e. child, caregiver, or both) and determine the range of treatment options available (including mHealth apps). The target problem may be related to the pandemic itself, such as coping with loneliness, improving sleep, and reducing sedentary behaviors during social isolation measures. In pediatric populations, the desired end user may be the caregiver to provide additional support with managing their child’s disease or their own stress management skills. Pediatric psychologists may want to seek insight from the literature to determine the best apps to recommend. For some target problems (e.g., medication adherence; Carmody et al., 2019), peer-reviewed reviews offer lists of existing apps and expert ratings. However, reviewing the literature may be a challenging starting point for a few reasons: (a) these papers quickly become obsolete as the app marketplace evolves, (b) few reviews have focused on pediatric populations specifically, and (c) it can be time-consuming for individual providers (Boudreaux et al., 2014), especially with competing demands during the pandemic.

For these reasons, beginning the search with app clearinghouses may be timelier. Specifically, there are several continuously updated websites available to help consumers and psychologists identify mHealth apps, which are outlined in Table I (see Neary & Schueller, 2018 for a full review). Many of these clearinghouses apply developed metrics to produce app rating scores for comparisons. Using PsyberGuide as an example, users can search over 200 mHealth tools based on treatment modalities and conditions (e.g., sleep, symptom tracking/self-monitoring, and chronic pain), and filter based on audience, platform, and cost. PsyberGuide contains expert ratings of each app based on three distinct but equally valued metrics (credibility, transparency, and user experience), which also offer considerations for the Explore stage. Presently, app clearinghouses typically offer more options that target mental health and coping in older adolescents and adults (which may be especially suitable for caregiver use). For pediatric health-specific recommendations, other hospital and organizational websites, such as Solutions for Kids in Pain, have distributed additional app recommendations.

Table I.

Overview of available mHealth app clearinghouses

Name	Evaluation metrics	App scores/ranks assigned	Cost	Users can search/filter for apps specifically for children and youth	Link
CredibleMind	App store star ratings Research about people, organizations, and expert reviews	Yes	Free (additional features available in paid version)	Yes	https://crediblemind.com/
Division of Digital Psychiatry, BIDMC	Origin and functionality Privacy and security Inputs and outputs Clinical foundation Engagement style Interoperability and sharing	No	Free	No	https://apps.digitalpsych.org/
MindTools	Program quality Score credibility Privacy explanation	Yes	Free	Yes	https://mindtools.io/
One Mind PsyberGuide	Credibility (indirect and direct research evidence, developmental processes) User experience Transparency of privacy policies Narrative reviews from mental health professionals	Yes	Free	Yes	https://onemindpsyberguide.org/
ORCHA	Data privacy Clinical assurance User experience	Yes	Pay to access reviews	Yes	https://appfinder.orcha.co.uk/

Stage 2: Explore

After narrowing the presenting problem and a list of contender apps, the next proposed step is to Explore: (a) the scientific evidence supporting the app’s use and theoretical basis (through a literature search of peer-reviewed manuscripts), (b) the existence and sufficiency of the app’s privacy policies, and (c) user experience (i.e., level of engagement, interest, entertainment, and ease of use). In addition to evaluating efficacy publications, apps should be reviewed on their congruence with health behavior change theories. For example, the Theoretical Domains Framework synthesized 33 theories and 128 theoretical constructs into 14 health behavior change domains (e.g., knowledge, reinforcement; Cane et al., 2012). Pediatric researchers have characterized mHealth behavior change features in this manner (Brannon & Cushing, 2015; Carmody et al., 2019), and this strategy may be helpful to pediatric psychologists who seek to quickly determine whether components of the app are theory-informed, even if app efficacy data is limited or absent altogether.

The American Psychiatric Association’s App Evaluation framework is a useful exploration tool (https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/app-evaluation-model; American Psychiatric Association, 2020). The framework provides a number of questions for psychologists to consider in a stepwise manner, including questions about privacy (e.g., a clear and accessible privacy policy, terms for data collection/sharing) and usability (e.g., the app’s engagement features, ease of use, and customization). Of note, for children, additional privacy restrictions may apply (e.g., the Children’s & Online Privacy Protection Act requires parental consent before collecting personal information from children under 13). For an expedited assessment of user experience, crowdsource feedback from stakeholders on social media networks (e.g., Twitter; Boudreaux et al., 2014). For psychologists interested in doing further exploration, the Mobile App Rating Scale (MARS; Stoyanov et al., 2015) is an additional tool for evaluating user experience. The MARS can be used to score apps based on their engagement, functionality, esthetics, and information quality. This deeper dive into user experience may be particularly valuable for pediatric populations, as user experience is an important driving factor for app use among youth (Garrido et al., 2019).

Stage 3: Contextualize

In the Contextualize stage, we recommend using clinical expertise and stakeholder feedback to determine whether the app is a good fit for a particular patient and/or caregiver. There is not a one-size-fits-all app, and pediatric psychologists must consider how the app fits into patient and family life and the context of their technology use. Notably, the context of daily life may look very different for certain vulnerable youth and families during the pandemic. Some families may have limited or no access to Wi-Fi at home, and youth may have less supervision (and as a result, lower engagement in self-guided apps) because of their caregiver’s status as an essential employee (Valenzuela et al., 2020). While these factors are always important to consider, they may be taken for granted during the well-intended push toward remote digital support. Other logistical factors to assess include: phone platform (not all apps are available for both Android and Apple/iOS), app cost, and data usage (high data usage could force undue financial burden), and smartphone accessibility—both ownership (some youth may share with a caregiver/sibling) and access (if the phone is not available at night, engaging with a meditation tool to fall asleep is not feasible).

Regarding patient/family factors, consideration of the age appropriateness of the app is essential. There are many apps that were specifically designed with youth in mind (e.g., Woebot, Stop, Breathe, and Think, Positive Penguins). Additionally, pediatric psychologists should assess whether an app was inclusively designed to accommodate physical disabilities (e.g., visual or hearing impairments) and contains diverse/representative depictions (e.g., based on race/ethnicity, ability, gender, sexual orientation). Other patient/caregiver factors to assess include: literacy, preferred language, comfort/trust of technology, the willingness of the caregiver to engage (caregiver involvement in mHealth trials yields greater effect sizes than youth involvement alone; Fedele et al., 2017), and clinical contraindications (e.g., screen time interfering with daily tasks such as sleep, trauma history and/or poor frustration tolerance that might be triggered by gamified apps). During the pandemic, caregivers may be hesitant to adopt an app that encourages even more screen time (Nagata et al., 2020). For this reason, it may be useful to share the American Academy of Pediatrics new recommendations (2020) to flexibly set limits with screens, such as prioritizing media that supports mental and physical health (e.g., exercise, social connection).

At the provider/healthcare system level, pediatric psychologists should pilot the app before recommending it to provide informed instructions about the app’s purpose and features. Many apps can be piloted thoroughly after only a few minutes of use. Ideally, the psychologist should help the patient/caregiver download the app, then provide a brief tutorial (Armstrong et al., 2018). Throughout the tutorial, clearly describe the purpose of the app and rationale for use, state expectations for use, and demonstrate key features. This can be done in conjunction with a telehealth visit, such as sharing a screen to demonstrate key features of the app. At the healthcare-level, shared resources (electronic health record smart phrases, internal app recommendation lists) may reduce unnecessary replication of the above steps and expedite wider and longer-term adoption of credible mHealth apps. Digital health equity also requires that every level of the health care system is able to detect disparities (e.g., screening for phone access among all patients) and work toward improving access of culturally safe digital health tools (Crawford & Serhal, 2020).

Finally, after recommending the app, it is important to re-evaluate actual app use and benefits. This includes following up with the patient/caregiver to ask if the app was accessed, how and when they engaged with the app, what features were liked or disliked, and perceived app benefits. Of note, health apps are uninstalled after an average of 8.8 days (Adjust, 2018), so assessing user engagement is particularly important if the app is intended for longer-term use. Problem-solving barriers to app usage and practicing the skills recommended in the app may also be necessary. This reevaluation process can also yield useful feedback to guide app decision-making with future patients (e.g., additional usability information).

Case Example

Jordan is a 16-year-old patient with leukemia who is struggling with his adjustment to cancer and the COVID-19 pandemic, including depressed mood. He is sedentary, spends most of the day on his phone or playing video games and is gaining weight, which can compromise his oncology treatment efficacy. Jordan is frustrated when his parents prompt him to “be more active” around the house. He is working on behavioral activation with his pediatric psychologist via telehealth, but has limited uptake of these recommendations. Since the onset of the COVID-19 pandemic, Jordan frequently tells his psychologist, “I can’t even go anywhere, so I just give up, and don’t look for the list of ideas we made in our last session.” Given Jordan’s draw to technology, his psychologist wonders if there are apps that could support both his behavioral activation goals and autonomy. The psychologist searches the iTunes store using the term “Behavioral Activation” and the first apps that appear are “CBT Companion,” “Mood Space,” and “DBT Coach” (search conducted in August 2020). She scrolls down to see that this page continues to expand with other app options—but it is unclear which one, if any, to recommend.

Jordan’s pediatric psychologist Narrows a pool of behavioral activation apps by referencing PsyberGuide and other websites (e.g., https://www.div12.org/treatment/behavioral-activation-for-depression/). This leaves her with a significantly smaller list of apps to consider than her first search on the app store. While Exploring each app on this new list, the psychologist considers the user experience for a teenage patient, evidence base, and privacy policies. To answer these questions, she conducts a brief literature search and surveys colleagues on Twitter. From this process, an app called SuperBetter stands out as a strong choice; this is a gamified app in which users conquer real-life quests by completing activities to boost health and well-being (e.g., walking around the block, drinking water). There are also new games relevant to the pandemic, including the “Stay at Home Scavenger Hunt” designed to build social connections while social distancing. In a randomized controlled trial with adults, SuperBetter users achieved greater reductions in depressive symptoms than a waitlist control (Roepke et al., 2015). Moreover, this app has published an accep and transparent privacy policy (e.g., app data are not shared with Google or other social media apps).

The psychologist Contextualizes SuperBetter by piloting the app herself and determining that the app is appealing for teenage patients who enjoy gaming, available on Android and iOS for free, and customizable so that it is appropriate for each user. In the next session with Jordan, she provides a background on the app’s purpose and functionality, downloads the app with Jordan, collaboratively sets a short-term goal (i.e., use the app to complete a task once a day until the next session), and shares these recommendations with Jordan’s caregivers. One week later, the psychologist followed up with Jordan to assess his usage of the app and goal progress. Jordan reported enjoying the app and expressed an intention to use it further, although he forgot to use it after the first two days. They worked together to better incorporate app use and associated behavioral activation goals into other daily routines (e.g., using the app each day after dinner).

Future Directions

This commentary offers pediatric psychologists and other healthcare providers a three-stage decision-making framework for selecting appropriate mHealth apps for youth and their caregivers who need remote behavioral health support. Unique considerations for pediatric populations during and after the COVID-19 pandemic are discussed at each stage, including whether the app was specifically and inclusively designed for children and adolescents, data privacy laws for minors, the importance of user experience for facilitating engagement, the availability and inclusion of caregivers, and concerns about excessive screen time.

Several challenges will need to be overcome to accelerate the pace at which evidence-based apps are made available and accessible to the public. Efforts to expertly rate pediatric-focused health apps should be continued, updated, and disseminated. These ratings should be disseminated collaboratively (e.g., through the American Psychological Association, the American Psychiatric Association, and the American Academy of Pediatrics)—to avoid the currently siloed practice of separate databases—and published on websites caregivers already commonly use (e.g., www.commonsensemedia.org). Apps need to be reviewed not only using a top-down approach (e.g., expert reviews) but a bottom-up approach that elicits stakeholder feedback and provides timely efficacy data (e.g., through N-of-1 trials; Jake-Schoffman et al., 2017). A scientific shift toward rapid implementation, dissemination, and sustainment of efficacious apps is needed to share mHealth research discoveries with the public and determine generalizability (Mohr et al., 2017). Given the increasing relevance of technology in practice, incorporating mHealth into pediatric psychology education and training is also needed. Finally, integrating mHealth apps into clinical care to screen and provide referral options in settings that reach nearly all pediatric patients (e.g., primary care) may help support patients who otherwise receive little to no behavioral health care (Lavigne et al., 2016).

The COVID-19 pandemic sparked an abrupt shift to digital support for many—but this shift was long overdue given the potential benefits and reach of mHealth. Pediatric psychologists should leverage what they are learning during this unprecedented time to strengthen the promise of mHealth for pediatric patients and families going forward.