Skip to main content
PMC home page
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2021 Dec 20;29(3):520–535. doi: 10.1093/jamia/ocab277

Review, Assess, Classify, and Evaluate (RACE): a framework for studying m-health apps and its application for opioid apps

Upkar Varshney 1,✉, Neetu Singh 2, Anu G Bourgeois 3, Shanta R Dube 4
PMCID: PMC8827031  PMID: 34939117

Abstract

Objective

The proliferation of m-health interventions has led to a growing research area of app analysis. We derived RACE (Review, Assess, Classify, and Evaluate) framework through the integration of existing methodologies for the purpose of analyzing m-health apps, and applied it to study opioid apps.

Materials and Methods

The 3-step RACE framework integrates established methods and evidence-based criteria used in a successive manner to identify and analyze m-health apps: the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, inter-rater reliability analysis, and Nickerson-Varshney-Muntermann taxonomy.

Results

Using RACE, 153 opioid apps were identified, assessed, and classified leading to dimensions of Target Audience, Key Function, Operation, Security & Privacy, and Impact, with Cohen’s kappa < 1.0 suggesting subjectivity in app narrative assessments. The most common functions were education (24%), prescription (16%), reminder-monitoring-support (13%), and treatment & recovery (37%). A majority are passive apps (56%). The target audience are patients (49%), healthcare professionals (39%), and others (12%). Security & Privacy is evident in 84% apps.

Discussion

Applying the 3-step RACE framework revealed patterns and gaps in opioid apps leading to systematization of knowledge. Lessons learned can be applied to the study of m-health apps for other health conditions.

Conclusion

With over 350 000 existing and emerging m-health apps, RACE shows promise as a robust and replicable framework for analyzing m-health apps for specific health conditions. Future research can utilize the RACE framework toward understanding the dimensions and characteristics of existing m-health apps to inform best practices for collaborative, connected and continued care.

Keywords: framework, structured taxonomy, m-health apps, evidence-based criteria, opioids

Lay Summary

What is already known?• M-health apps are increasingly utilized to address overall health and well-being.• M-health apps are currently studied using a wide range of methodologies.• There is a need for a framework to guide systematic investigation of m-health apps.What did we find?• Three-step RACE framework is designed for systematic investigation of m-health apps.• By applying RACE to opioid apps, we derived numerous insights for practice and research.• Lessons learned from this implementation can be applied to the study of m-health apps for other health conditions.

OBJECTIVE

In the last decade, the development and use of mobile health (m-health) applications (apps) for health interventions has exponentially proliferated.1–3 M-health apps are developed for multiple audiences to deliver evidence-based interventions for behavioral modification and wellbeing. As a result, systematic studies that analyze m-health apps designed for specific health conditions have led to a rapidly evolving area of research.4–7 Most of the studies have used Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for structured review but lack use of a structured method for assessment and classification. This reduces the ability to systematically analyze m-health apps and compare findings from different studies or replicate findings. Thus, our objective is to develop an integrated stepwise approach that can systematically be used in the investigation of m-health apps for specific conditions. To address this objective, we develop and propose the Review, Assess, Classify, and Evaluate (RACE) as a structured framework to study m-health apps.

After designing RACE by integrating existing methodologies (Grundy et al5 and Nouri et al6), we describe the application of RACE for m-health apps. The first step of RACE utilizes PRISMA to conduct a systematic review of literature and multiple app platforms to identify existing m-health apps for the specific condition of interest. The second step utilizes inter-rater reliability (IRR) and Nickerson-Varshney-Muntermann (NVM) taxonomy as an iterative process to assess and classify apps for specific dimensions and characteristics. The third step utilizes the developed taxonomy from step 2 and evidence-based criteria to evaluate the apps for patterns and functionality gaps that may guide future research and improve design of existing m-health apps. Using the step-wise RACE framework to gather and analyze m-health apps for specific conditions can lead to systematization of information8,9 that reveals patterns and gaps for improving current and developing future m-health apps.

To demonstrate the process and potential of RACE, we apply the proposed framework by identifying and evaluating existing opioid apps. Opioid use disorder (OUD) is a global public health crisis affecting hundreds of millions of people and costing trillions of dollars.10–12 In recent years, emerging technology has led to the development and deployment of numerous apps that can provide a wide range of interventions to address and prevent opioid use. Given that there are a large number of opioid apps currently available, there is an opportunity to apply the RACE framework to identify opioid intervention gaps, using a 3-level prevention (primary, secondary, and tertiary) approach.13,14 To apply RACE with opioid apps, (a) review existing literature and apps from multiple platforms,15,16 (b) build a structured taxonomy,8 (c) develop evidence-based guidelines for possible extensions and design of new m-health apps to generate insights for future opioid research. In using this process for opioid apps, we demonstrate how the RACE framework can be applied for the systematic investigation of other m-health apps.

BACKGROUND AND SIGNIFICANCE

Methodological background

We identified 3 notable and prevailing approaches for assessing m-health apps and their attributes.5,6,17 The first approach, RE-AIM framework,18 has been extensively used in postimplementation analysis of health interventions including m-health apps.19 RE-AIM uses Public Health Impact Score as the indicator of Reach (proportion of the target population that participated in the intervention), Efficacy (success rate if implemented as in guidelines, defined as positive outcomes minus negative outcomes), Adoption (proportion of settings, practices, and plans that will adopt this intervention), Implementation (extent to which the intervention is implemented as intended in the real world), and Maintenance (extent to which a program is sustained over time). For our current study on the exploration of m-health apps, and not on postuse assessment, RE-AIM is beyond the scope of our analytic approach. The second approach, by Nouri et al,6 includes multiple studies to identify 7 classes of m-health app attributes. These are Design, Information/Content, Usability, Functionality, Ethical Issues, Security and Privacy, and User-perceived Value. The third approach, by Grundy et al,5 highlights the use of a systematic literature review and app searches with content analysis to identify key attributes of m-health apps. The important steps of the Grundy et al5 and Nouri et al6 methods are shown in Figure  1. Our scope to “systematically review m-health apps, assess and classify to find themes and patterns, and evaluate implemented apps using evidence-based criteria to generate guidelines for future app design” is considerably wider (Figure  1). We use evidence-based criteria including Purposeful Design, Functionality & Usability, Ethical Considerations, Reach & Impact, and Public Health & Clinical Guidelines. This can map to target audience, domain expert evaluation, key function, operation, security & privacy, cultural competence, and impact. We also construct a more robust framework by integration of existing methodologies, primarily Grundy et al5 and Nouri et al.6 Unlike other app assessments primarily relying on PRISMA, the novelty of RACE framework includes the use of NVM taxonomy8 method and IRR using Cohen’s kappa.20,21

Figure 1.

Figure 1.

Open in a new tab

Development of RACE (Review, Assess, Classify, and Evaluate) framework based on existing methods.

The proposed RACE framework has 3 main steps (Figure  1). For review (step 1), we use a systematic review technique PRISMA, designed to provide clarity, inclusion, comprehensiveness, and replicability.15,16 For assessment and classification (step 2), we use NVM taxonomy8 method to identify meta-characteristic followed by the dimensions and characteristics, perform IRR using Cohen’s kappa, and finalize the taxonomy using NVM method. For evaluation (step 3), we use content analysis to categorize each m-health app to specific characteristics of the taxonomy using evidence-based criteria (Figure  1). RACE includes several novel categories under evaluation and allows addition of both deductively and inductively derived categories in classification as well as in evaluation.

As we will apply the RACE framework to assess and evaluate opioid apps, we now present a brief background of the opioid epidemic and current interventions.

Opioid background

OUD is a major public health crisis due to extremely high healthcare costs and serious harm to patients.22 In the United States alone, at least 2 million patients are suffering from OUD, leading to $100B/year in healthcare costs10 and almost 93 000 deaths in 2020.23 Patients may have certain vulnerability based on their medical history,24 with some needing expensive inpatient treatment (detoxification)25,26 while those untreated have a high chance of opioid overdose and long-term suffering from addiction.10

Figure  2 depicts a 6-state OUD model that a patient may progress through, albeit not always linearly. These states include (1) Pain, emotional and/or physical27; (2) Consumption via prescriptions or illicit opioids28; (3) Dependence, reliance on opioids29; (4) Addiction, a chronic physiological need24; (5) Recovery, abstinence and a new way of living30; and (6) Remission, maintaining recovery.31 The progression is not linear due to possible relapse and setbacks along the way. Also a patient may experience an Overdose event from any state, requiring emergency and expensive treatment.

Figure 2.

Figure 2.

Open in a new tab

Opioid use disorder states and examples of healthcare interventions.

Possible interventions administered by a healthcare professional (HP) include medical, clinical, nursing, counseling, and/or psychiatric care. Figure  2 provides intervention examples to prevent and treat opioid use at different states. In particular, Recovery requires multifaceted solutions, such as outpatient therapy, maintenance opioids, frequent drug urine tests, and considerable support from friends and family. These challenges and variant patient conditions and backgrounds must be included in the design, development, testing, and adoption of interventions, along with randomized control trials (RCTs) for showing the effectiveness of any new intervention.

MATERIALS AND METHODS

In this section, we delineate the general process of applying the RACE framework for the analysis of m-health apps for a particular condition. To better explain the process, we show how to apply the 3 steps of RACE for the analysis of opioid apps, which can also be applied in the same way for other m-health apps.

Step 1: Systematic review of apps using PRISMA

To evaluate the m-health apps for a specific health condition, the first step is to identify the set of corresponding apps. The PRISMA approach has been designed for reporting in which authors can summarize the healthcare interventions accurately and reliably.15,16 This includes Identification, Screening, Eligibility, and Inclusion. We identified opioid apps in existing literature and on the Google Play and Apple Stores, which are likely to contain most of the apps with a small number available on only 1 platform. Our search terms included “opioid” AND “app OR apps” on Google Play, Apple Store, and in the abstract search of Web of Science, PubMed, IEEE Xplore, ACM Digital Library, and AIS eLibrary. This resulted in the identification of 432 opioid apps (Supplementary Table A1).

Step 2: Assessment and classification

After the apps are selected, the next step is to assess and classify the apps. NVM method8 is well-established and widely used for taxonomy development in multiple disciplines.32–37 As shown in Figure  3, it is an iterative process that starts with the identification of the meta-characteristics of m-health apps to define the scope of taxonomy. For example, for opioid apps analysis we identified “m-health apps for opioids” as the meta-characteristic. Then, subjective and objective ending conditions (Figure  3) are identified to stop the iterative process after the classification is completed. Each iteration results in adding one dimension with mutually exclusive characteristics to the taxonomy.

Figure 3.

Figure 3.

Open in a new tab

NVM (Nickerson-Varshney-Muntermann) method8 for taxonomy of m-health apps.

As shown in Figure  3, there are 2 complementary approaches to develop the taxonomy. Objects or apps can be studied using Empirical-to-Conceptual approach (steps 4e, 5e, and 6e of Figure  3) to derive characteristics and then merge these to create a dimension. Under Conceptual-to-Empirical approach (steps 4c, 5c, and 6c of Figure  3), an initial dimension can be identified and/or selected from one of the evidence-based criteria of the RACE framework (Figure  1). Through multiple iterations, Empirical-to-Conceptual (inductively) or Conceptual-to-Empirical approach (deductively) is used to develop the taxonomy until the ending conditions are met.

After identifying the dimensions and characteristics of the taxonomy, at least 2 researchers will independently review and code the app information from various platforms, such as Google Play and Apple Store. A researcher not involved in the coding process will compile and stratify the data by platform to conduct the IRR. Stratification allows for more granular assessment since different platforms request different types of information from developers prior to pushing out the apps. A preferred statistical tool, such as SPSS, will be used to obtain the Cohen’s kappa (1.0 = 100% agreement).20,21 Cohen’s kappa of 100% is required to finalize the taxonomy with mutually exclusive characteristics.8 Any Cohen’s kappa < 1.0 must be resolved by reaching consensus among all raters, data analyst, and domain experts.

Step 3: Evaluate apps

After completing step 2 (taxonomy development), m-health apps are evaluated based on the evidence-based criteria from RACE framework (Figure  1). The first phase of the evaluation step is to use the taxonomy data (Table  2) to identify patterns related to Purposeful Design, Functionality and Usability, Ethical Considerations, and Reach and Impact. Further analysis of the patterns will show how the apps align with the Public Health & Clinical Guidelines. This evaluation will reveal patterns and identify any potential gaps to improve existing apps and provide considerations for new designs.

Table 2.

Cohen’s kappa rater agreement on characteristics for opioid apps

Dimension variable Number of valid cases Cohen’s kappa (Individual rater) Group consensus for taxonomy
Google Play Apps Target Audience 140 0.89 100%
Key Function 141 0.82 100%
Operation 137 0.91 100%
Security & Privacy 138 0.73 100%
Impact 141 1.00 N/A
Apple Store Apps Target Audience 12 0.68 100%
Key Function 12 0.80 100%
Operation 12 0.40 100%
Security & Privacy 12 0.70 100%
Impact 12 Insufficient data
Open in a new tab

Cohen’s kappa: 0.01–0.20 as none to slight; 0.21–0.40 as fair; 0.41– 0.60 as moderate; 0.61–0.80 as substantial; 0.81–1.00 as almost perfect agreement.

RESULTS

In this section, we demonstrate how to apply the 3 steps of the RACE framework to analyze opioid apps. The process revealed patterns of evidence-based criteria (Figure  1) among the apps, along with new insights into guidelines for adapting existing apps and consideration for new designs.

Step 1: Review

Of the 432 identified opioid apps, we found 245 apps on Google Play, 33 on Apple Store, and 154 articles in the published literature that appeared to indicate opioid apps (Figure  4). We closely examined the app description and using team consensus excluded 128 apps, as opioid was not the focus but an ancillary component. Some of these apps included substance use (eg, tobacco and alcohol), pain, and other health conditions. We also reviewed 154 articles and excluded 103 opioid articles that did not present an opioid app. Further, 48 apps from literature not focusing on opioid apps or overlapping with Google Play or Apple Store were excluded. Finally, we included 153 opioid apps for our qualitative analysis and developing a structured taxonomy.

Figure 4.

Figure 4.

Open in a new tab

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) method for identification of opioid apps.

Step 2: Assessment and classification

We identified the meta-characteristic of our taxonomy as “m-health apps for opioids”. After reviewing and analyzing 153 opioid apps, we observed that by using the Empirical-to-Conceptual approach of taxonomy development, these apps focus on 3 different group of users: “Patients,” “Healthcare Professionals,” or “Others.” These can be considered as 3 characteristics of the opioid apps and can be naturally grouped into the “Target Audience” dimension. At the end of the first iteration (iteration 1), the taxonomy can be expressed as Target Audience {Patients, Healthcare Professionals, Others}.

We used a Conceptual-to-Empirical approach8 for iteration 2 since we have identified certain terminologies and contexts in the first iteration to further classify the apps. After considering the Target Audience dimension, we focused on the functionality of opioid apps using 3-level prevention (primary, secondary, or tertiary) that informs Public Health interventions.13,14 Primary prevention interventions, such as education and prescription management, aim to avoid a health condition from occurring. Secondary prevention interventions, such as screening and monitoring, are aimed at early identification to stop the disease process before it clinically manifests. Tertiary prevention focuses on preventing complications of an already established disease state, thereby making it more synonymous with treatment. After further analysis of 153 m-health apps, we observe that the new characteristics represent the key functions of m-health apps. These functionalities represent evidence-based interventions to promote behavior change for patients, healthcare professionals, and other audience. Moreover, we did not find any functionality directly referring to behavioral change in opioid apps. We decided to group these characteristics in the “Key Function” dimension. We found some of the apps have overlapping functionalities and few do not align with any of the categories. Iteration 2 adds Key Function {Education, Prescription, Reminder-Monitoring-Support, Treatment & Recovery, Overlapping, None of the Categories} to the taxonomy. Evidence-based criteria from the RACE framework enabled the derivation of characteristics for Key Function dimension.

After identifying the characteristics of Target Audience and Key Function dimensions, we further evaluated 153 opioid apps using the Empirical-to-Conceptual approach8 for iteration 3. In the context of existing dimensions, passive and interactive characteristics can indicate whether the app is used as 1-way or 2-way interaction tool. We observed that these characteristics can be grouped into the “Operation” dimension. After iteration 3, the revised taxonomy now includes Operation {Passive, Interactive} along with the existing dimensions of Target Audience and Key Function, as shown in Table  1.

Table 1.

Opioid apps dimensions and characteristics using the NVM method

Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
App name Target Audience
 Key Function
 Operation
 Security & Privacy
 Impact
Pat HP O EDU PRES RMS TX OL NOC Pas Int Imp Exp N/A L M H N/A
Open in a new tab

Abbreviations: EDU: education; Exp: explicit; H: high; HP: healthcare professionals; Imp: implicit; Int: interactive; L: low; M: medium; N/A: not available; NOC: none of the categories; NVM: Nickerson-Varshney-Muntermann; O: other; OL: overlapping; Pas: passive; Pat: patients; PRES: prescription; RMS: reminder-monitoring-support; TX: Treatment & Recovery.

For iteration 4, we used Empirical-to-Conceptual approach8 to further analyze the opioid apps. Based on the observed patterns, we identified 3 new characteristics as implicit, explicit, or no indication of m-health app privacy. We observe from the analysis of 153 apps that these characteristics can be grouped into the dimension Security & Privacy {Implicit, Explicit, Not Available} as shown in Table  1. The revised taxonomy now includes 4 dimensions identified in iterations 1 to 4. Since one new dimension is added in the fourth iteration, the taxonomy development will be repeated.

In iteration 5, we continued with the Empirical-to-Conceptual approach8 to further assess the opioid apps. We observed that to understand the awareness and potential use of the apps, the number of downloads can be conceptualized into Low, Medium, and High characteristics. During the evaluation of 153 apps, we observed some apps do not include the number of downloads and this can be characterized as Not Available. This led to the grouping of the 4 characteristics into the “Impact” dimension as follows: Impact {Low, Medium, High, Not Available}.

After iteration 5, we did not identify any new app characteristics using Empirical-to-Conceptual approach or new dimensions using Conceptual-to-Empirical approach. This leads us to an ending condition for the taxonomy development resulting in 5 dimensions and 18 characteristics (Table  1). The dimensions and corresponding characteristics are Target Audience {Patients, Healthcare Professionals, Other}, Key Function {Education, Prescription, Reminder-Monitoring-Support, Treatment & Recovery, Overlapping, None of the Categories}, Operation {Passive, Interactive}, Security & Privacy {Implicit, Explicit, Not Available}, and Impact {Low, Medium, High, Not Available}.

These 5 dimensions align with the majority of the evaluation criteria in step 3 of the RACE framework (Figure  1). NVM method supports extendable taxonomies. Therefore, additional dimensions and characteristics can be identified and added to the taxonomy as new opioid apps become available.

After identifying the characteristics of the taxonomy (Table  1), 2 researchers independently reviewed and coded the characteristics from 153 apps found on Google Play and Apple Store. The third researcher not involved in the coding process compiled and stratified the data by platform to conduct the IRR using SPSS (v25). All IRR kappa values for subjective characteristics from both Google Play and Apple Store apps fell below 1.0 (Table  2). For Google apps, coding of each characteristic resulted in Cohen’s kappa values of 0.89 (Target Audience), 0.82 (Key Function), 0.91 (Operation), and 0.73 (Security & Privacy). Cohen’s kappa for Apple Store apps were 0.68 (Target Audience), 0.80 (Key Function), 0.40 (Operation), and 0.70 (Security & Privacy). We also conducted IRR on assessments of impact, in-app purchases, and user rating based on concrete standard information provided by Google Play and acquired kappa of 1.0, suggesting that these types of information are far more objective resulting in greater agreement. For Apple Store apps, there was insufficient data for raters to code and analyze. Acquiring less than 100% agreement on most characteristics across both platforms speaks to having less concrete information in terms of what an app was developed to do; it is the reason why our methodology integrates the 100% consensus agreement criteria for assessment prior to development of the taxonomy.

The Objective Ending Condition8 is met, as the taxonomy has 5 mutually exclusive dimensions and each dimension has mutually and collectively exhaustive characteristics. The Subjective Ending Condition is met, as these 5 dimensions lead to a meaningful and cognitively appealing taxonomy for opioid apps and the resulting taxonomy is robust, comprehensive, extendible, and explanatory.8 The developed taxonomy of opioid apps (Table  3) gives us the concise, rigorous, and comprehensive classification system extendible to all opioid apps.

Table 3.

Opioid apps taxonomy using the NVM method

Iteration 1
 Iteration 2
 Iteration 3
 Iteration 4
 Iteration 5
App Name Target Audience
 Key Function
 Operation
 Security & Privacy
 Impact
Pat HP O EDU PRES RMS TX OL NOC Pas Int Imp Exp N/A L M H N/A
Opioid Convertera 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 0 X 0
CDC Opioid Guidelinea 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
Opioid Calculatora 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0 0
Pain & Opioid Safetya 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
Reducerea 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
Orthodosea 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
Opioid Medicine Abuse & Misusea X 0 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
Narcotic & Opioid Convertera 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
Narcan Nowa X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
Palliative Care Toolsa 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0 0
COR-12 Appa X 0 0 0 0 0 0 X 0 X 0 X 0 0 X 0 0 0
Get Smarta 0 X 0 X 0 0 0 0 0 0 X X 0 0 X 0 0 0
Opioid RXa 0 X 0 0 X 0 0 0 0 0 X X 0 0 X 0 0 0
Kaden Healtha X 0 0 0 0 0 X 0 0 0 X 0 X 0 X 0 0 0
Opiate Abuse: The Trutha X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
OpiRescuea 0 0 X 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
ODS Opioid Drug Screena 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
PursueCarea X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
Pear reSET-Oa X 0 0 0 0 0 0 X 0 0 X 0 X 0 X 0 0 0
Painkiller Calculatora 0 X 0 0 X 0 0 0 0 X 0 0 0 X 0 X 0 0
Analgesic Convertera 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
NEJM Knowledge +IM Reviewa 0 X 0 X 0 0 0 0 0 0 X X 0 0 0 X 0 0
Opioids Certificationa 0 X 0 X 0 0 0 0 0 0 X 0 0 X X 0 0 0
Anesthesiologist Adfreea 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0 0
MME ODS Opioid Drug Screena 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
FENDa X 0 0 X 0 0 0 0 0 0 X X 0 0 0 X 0 0
Pocketbook of Palliative Medicinea 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
Conversor de Opioidesa 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
Opioids Certificationa 0 X 0 X 0 0 0 0 0 X 0 0 0 X X 0 0 0
OpiSafea X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Drugs & Pain Control Made Easya 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
Talk About Opioidsa 0 X 0 X 0 0 0 0 0 0 X X 0 0 X 0 0 0
OnTrack—Preventing Injectiona X 0 0 0 0 0 0 X 0 0 X X 0 0 X 0 0 0
LUminate Supporta X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
Stop OD NYCa 0 0 X 0 0 0 X 0 0 0 X X 0 0 0 X 0 0
Anesthesiologista 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 0 X 0
NaloxoFinda 0 0 X 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Pocket Rehab: Get Sober & Addiction Recoverya X 0 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0
FentaTIPSa 0 0 X 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
KnowYourMedsa X 0 0 0 0 0 0 X 0 0 X X 0 0 0 0 X 0
Drugs Dictionary Offline—Drug A—Z Lista 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
Drugs Dictionary Offlinea 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 0 X 0
Staying Safea X 0 0 0 0 0 0 X 0 0 X 0 X 0 X 0 0 0
Drug Interactionsa 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
Tarascon Pharmacopoeiaa 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
Easy Drug Dose Calculatora 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 0 X 0
eBroselow SafeDosea 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 0 X 0
NARCANsasa 0 0 X 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
PRMT—MyTherapya X 0 0 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0
PRMT—Medisafea X 0 0 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0
MySafeRxa X 0 0 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0
MATxa 0 X 0 0 0 0 X 0 0 0 X 0 X 0 0 X 0 0
Manage My Paina X 0 0 0 0 X 0 0 0 0 X X 0 0 X 0 0 0
eOpioida 0 X 0 0 X 0 0 0 0 X 0 X 0 0 X 0 0 0
Recovery Patha X 0 0 0 0 0 0 X 0 0 X X 0 0 0 X 0 0
Drugs.com Medication Guidea X 0 0 X 0 0 0 0 0 0 X X 0 0 0 0 X 0
Drugs (Classifications, Dosage & Dictionary)a 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 0 X 0
SoberToola X 0 0 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0
Recovery Todaya X 0 0 0 0 X 0 0 0 X 0 0 0 X 0 X 0 0
Workit Healtha X 0 0 0 0 0 X 0 0 0 X X 0 0 0 X 0 0
iPrescribea 0 X 0 0 X 0 0 0 0 0 X X 0 0 0 X 0 0
Opio -Opioid Craving Trackera X 0 0 0 0 X 0 0 0 0 X X 0 0 X 0 0 0
Drug Addict Advicea X 0 0 0 0 0 X 0 0 X 0 0 0 X 0 X 0 0
MDCalca 0 X 0 0 X 0 0 0 0 0 X X 0 0 0 0 X 0
Recovering(me)a X 0 0 0 0 0 X 0 0 0 X 0 X 0 0 X 0 0
Infinite Dosea 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0 0
Opioid Addiction Preventiona 0 0 X 0 0 0 X 0 0 X 0 0 0 X X 0 0 0
WebMDa X 0 0 0 0 0 0 X 0 X 0 X 0 0 0 0 X 0
I am Sobera X 0 0 0 0 X 0 0 0 0 X X 0 0 0 0 X 0
MyCharliea X 0 0 0 0 0 0 X 0 0 X X 0 0 X 0 0 0
Addiction Recovery Devotionalsa X 0 0 0 0 0 X 0 0 X 0 X 0 0 0 X 0 0
Palliative Care Fast Factsa 0 X 0 X 0 0 0 0 0 X 0 0 X 0 0 X 0 0
PainScalea X 0 0 0 0 X 0 0 0 0 X X 0 0 0 X 0 0
Poisoning & Drug Overdosea 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 0 X 0
Drug Addiction Counselinga X 0 0 0 0 0 X 0 0 0 X X 0 0 0 X 0 0
Mosby’s Drug Referencea 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 0 X 0
Wellness Recovery Action Plan (WRAP)a X 0 0 0 0 X 0 0 0 0 X X 0 0 0 X 0 0
I.V. Drug Handbooka 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 0 X 0
OPITa X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Pain Relief Hypnosisa X 0 0 0 0 X 0 0 0 X 0 X 0 0 0 X 0 0
Celebrate Recoverya X 0 0 0 0 X 0 0 0 X 0 X 0 0 0 X 0 0
Nurse’s Drug Handbooka 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 0 X 0
AA 12 Step Appa X 0 0 0 0 0 X 0 0 0 X 0 X 0 0 0 X 0
Davis’s Drug Guide for Nursesa 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
Psychiatry Pro-Diagnosisa X 0 0 0 0 0 0 X 0 0 X 0 0 X 0 X 0 0
CareZonea X 0 0 0 0 X 0 0 0 0 X 0 X 0 0 0 X 0
Addiction Counselors Handbooka 0 X 0 0 0 0 X 0 0 X 0 0 0 X 0 X 0 0
NATB 12 Steps for Narcotics Addictsa X 0 0 0 0 0 X 0 0 0 X 0 X 0 0 X 0 0
Drug Databasea 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
AHFS Drug Informationa 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
Pink Clouda X 0 0 0 0 0 X 0 0 0 X 0 X 0 0 X 0 0
12 Step Guide Narcotics Addictsa X 0 0 0 0 0 X 0 0 X 0 X 0 0 0 X 0 0
Drug Abuse Symptoms, Facts & Treatmenta X 0 0 0 0 0 0 X 0 X 0 X 0 0 0 X 0 0
Pain Management Pocketcardsa 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
Pear reSETa X 0 0 0 0 0 0 X 0 0 X X 0 0 0 X 0 0
NA Meeting Searcha X 0 0 0 0 0 X 0 0 0 X X 0 0 0 0 X 0
Drug Overdose Coacha 0 0 X 0 0 X 0 0 0 X 0 X 0 0 X 0 0 0
Flappy Junkya X 0 0 0 0 0 X 0 0 0 X 0 0 X 0 X 0 0
Kratom Magazinea X 0 0 0 0 0 X 0 0 X 0 0 0 X X 0 0 0
BUPa X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Current Diagnosis & Treatment Psychiatrya 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
Narcotics Anonymousa X 0 0 0 0 0 X 0 0 X 0 X 0 0 0 X 0 0
Opiate Illusionsa X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
RSquared—Rehab & Recoverya X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
OD-MEa 0 0 X 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
OARSa 0 X 0 0 0 0 X 0 0 0 X 0 X 0 X 0 0 0
Get Ready Emergencya 0 0 X 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
3 Steps to Save a Lifea 0 0 X 0 0 0 X 0 0 X 0 0 0 X X 0 0 0
TryCyclea X 0 0 0 0 X 0 0 0 0 X X 0 0 X 0 0 0
Pain Management Guidelinesa 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
2RescAppa 0 0 X 0 0 0 X 0 0 X 0 0 0 X 0 X 0 0
Livret Douleur AP-HPa 0 X 0 X 0 0 0 0 0 X 0 0 0 X 0 X 0 0
Douleur Hôpital Saint Josepha 0 X 0 X 0 0 0 0 0 X 0 X 0 0 0 X 0 0
BHMEDS-R3a 0 X 0 X 0 0 0 0 0 X 0 X 0 0 X 0 0 0
Recover OR Diea X 0 0 0 0 X 0 0 0 X 0 X 0 0 X 0 0 0
Boulder Carea X 0 0 0 0 0 X 0 0 0 X 0 X 0 X 0 0 0
Ostego NY Healtha X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
Marsi Roada X 0 0 0 0 0 X 0 0 X 0 X 0 0 0 X 0 0
Blue Caresa X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
DSM-5 Diagnostic Criteriaa 0 X 0 X 0 0 0 0 0 X 0 0 0 X 0 X 0 0
UnityPhillya 0 0 X 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Telemed Expressa X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
NeuroX addictiona X 0 0 0 0 0 X 0 0 X 0 0 0 X X 0 0 0
Nopioida X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Hamilton County Coalitiona 0 0 X 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
TDS Centrala 0 X 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
MedManagea X 0 0 0 0 X 0 0 0 0 X X 0 0 X 0 0 0
EpioneGenea 0 X 0 X 0 0 0 0 0 0 X 0 0 X X 0 0 0
PainFocusa 0 X 0 0 X 0 0 0 0 X 0 0 0 X X 0 0 0
OSORD calculatora 0 X 0 0 X 0 0 0 0 0 X 0 0 X X 0 0 0
Dopesicka 0 0 X X 0 0 0 0 0 X 0 0 0 X X 0 0 0
Navjeevan: DeAdictiona 0 0 X 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
FlexDeka X 0 0 0 0 X 0 0 0 0 X X 0 0 X 0 0 0
eClinician: M&B OUDa X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
Pilleve Companion Appa X 0 0 0 0 X 0 0 0 0 X X 0 0 X 0 0 0
ODBustera 0 0 X 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Live4Lali Mobile Outreacha X 0 0 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
YMO Storiesa X 0 0 X 0 0 0 0 0 X 0 0 0 X X 0 0 0
You are Linked to Resourcesa 0 0 X 0 0 0 X 0 0 X 0 X 0 0 X 0 0 0
TONIa 0 0 X 0 0 0 0 0 X 0 X 0 0 X X 0 0 0
WellBrain Goa X 0 0 0 0 0 X 0 0 0 X X 0 0 X 0 0 0
Opioid Toolb 0 X 0 0 X 0 0 0 0 X 0 0 0 X 0 0 0 X
My Opioid Managerb X 0 0 0 0 X 0 0 0 0 X 0 0 X 0 0 0 X
Drug Facts by PillSync.comb 0 X 0 X 0 0 0 0 0 0 X X 0 0 0 0 0 X
pH-Medical Opioid Converterb 0 X 0 0 X 0 0 0 0 X 0 0 0 X 0 0 0 X
emochab X 0 0 0 0 0 0 X 0 0 X X 0 0 0 0 0 X
Opioid Managerb 0 X 0 0 X 0 0 0 0 X 0 X 0 0 0 0 0 X
SleepSmartzb X 0 0 0 0 0 0 0 X X 0 0 X 0 0 0 0 X
Day by Day Meditationsb X 0 0 0 0 0 X 0 0 X 0 X 0 0 0 0 0 X
Connections: from CHESS Healthb X 0 0 0 0 0 X 0 0 0 X X 0 0 0 0 0 X
Manage Addiction Lifelineb X 0 0 0 0 0 X 0 0 0 X 0 X 0 0 0 0 X
GetHelpNow Montgomery Countyb X 0 0 0 0 0 X 0 0 X 0 X 0 0 0 0 0 X
Buprenorphine Home Inductionb X 0 0 0 0 0 X 0 0 0 X X 0 0 0 0 0 X
Open in a new tab

Abbreviations: EDU: education; Exp: explicit; H: high; HP: healthcare professionals; Imp: implicit; Int: interactive; L: low; M: medium; N/A: not available; NOC: none of the categories; NVM: Nickerson-Varshney-Muntermann; O: other; OL: overlapping; Pas: passive; Pat: patients; PRES: prescription; RMS: reminder-monitoring-support; TX: Treatment & Recovery.

Step 3: Evaluate apps

As shown in Figure  1, we provide our findings by applying the evidence-based criteria of RACE framework including (a) Purposeful Design, (b) Functionality & Usability, (c) Ethical Considerations, (d) Reach & Impact, and (e) Public Health & Clinical Guidelines.

Purposeful Design (Target Audience)

Most opioid apps are for patients (75/153), followed by healthcare professionals (60/153) and the remaining for others (bystanders/emergency personnel) intervening in emergencies (18/153). Most apps are designed for one type of user (patients or healthcare professionals). Only a handful of apps are intended for multiple user roles, leading to a disconnected, rather than a collaborative care approach. A few apps were classified to include supporting members for the patient (family/friends).

Functionality & Usability (Key Function)

The key functions are education (37/153), prescription (24/153), reminder-monitoring-support (20/153), treatment & recovery (57/153), overlapping (13/153), and none of the categories (2/153). Most educational or information apps are handy reference guides targeted for healthcare professionals (HPs). Some prescription apps provide HPs with drug conversion tools. We observed that 13/153 opioid apps provide multiple functions and the remaining focusing on one function.

Functionality & Usability (Operation)

There is a reasonable balance between passive (85/153) and interactive apps (68/153). Passive apps provide static content based on users’ queries. Interactive apps include daily interaction, journal keeping, or reminders for patients and monitoring support by healthcare professionals. In terms of interactivity, we found that most apps are not context-aware and do not change their operation based on patients’ context. By incorporating context-aware operation, the app could detect triggers that lead to seeking and using opioids. This would be accomplished by determining correlations between triggers with user location activity patterns, and/or proximity to interpersonal interactions and subsequently generating critical responses and alerts.

Ethical Considerations (Security & Privacy)

Very few opioid apps (19/153) explicitly address security and privacy. Most provide some security and include a privacy policy (110/153), but do not explicitly include it in their description. Further, 24/153 apps do not include any information on security and privacy. There are multiple apps that connect patients with healthcare providers. These apps would be required to follow HIPAA regulations for protection of patient health information collected. However, there are numerous apps that are interactive, in which the patient is tracking their medications, recording a daily journal, or connecting with support groups. While these data do not fall under HIPAA compliance, it is imperative that there be security and privacy provisions to protect sensitive data of the app users. Privacy provisions are important in establishing trust for adherence when using the apps to achieve the intended health benefit.

Reach & Impact (Impact)

Most downloads (10K–1000K) were for pill reminder and medication tracking apps. The opioid specific apps have very few downloads (500–100K). It is likely that some patients do not see much value and/or there are usability challenges with many apps being proof-of-concept only with limited functions. We found support for these observations from both structured data (ratings, downloads, and update frequency) and nonstructured data (patients’ feedback and comments).

Figure  5 visualizes the above observations in terms of target audience (patients vs healthcare professionals), operation (passive vs interactive), key functions (education, prescription, reminder-monitoring-support, treatment & recovery, overlapping, and none of the categories), and security & privacy. The size of a circle represents the number of apps in that subcategory. The ratio in each quadrant (security & privacy) shows the number of apps supporting security & privacy to the apps in that quadrant.

Figure 5.

Figure 5.

Open in a new tab

Visualizing key observations of opioid apps.

Public Health & Clinical Guidelines (Expert Evaluation, Cultural Competence, and Prevention & Treatment)

From the evaluation step of our RACE framework, we identify patterns and critical gaps that lead to proposed guidelines for evidence-based opioid app interventions. Additionally, based on the assessment and evaluation, several key apps include the CDC guidelines app, FDA-certified reSET-O app, SAMHSA app, and Anthem’s endorsed app MATx. Interestingly, 3 apps are from federal agencies with a focus on prevention and treatment of OUD. Except for reSET-O, other apps do not have FDA certification or will not qualify as Digital Therapeutics.38,39 Although some apps may have been evaluated in RCTs or received FDA certification, but we did not find such information in their description.

We observed that the existing apps were not evaluated by experts, operationalized for cultural competence, nor supporting continuity of care. To address these limitations, current and future apps should involve domain expertise from multiple disciplines. Cultural competence (cultural awareness) and continuity of care (personalization and suitability) should be supported by involving users and experts (focus groups) from diverse cultures and backgrounds.

We propose following guidelines based on multidisciplinary evidence-based criteria for collaborative care to leverage the patient-provider relationship and modification of patient’s behavior.

  1. Personalized & Proactive Interventions: To support patients and healthcare professionals, the interventions should be dynamic and context-aware.40,41 The interventions should support prevention, treatment, and recovery24,30 by incorporating: (a) patient risk assessments by healthcare professionals, (b) opioid resources and information, (c) cognitive behavior therapy, and (d) context-aware alerts.

  2. Secure & Safe Interventions: Opioid apps must support privacy, security, and safety42 for patients and/or families for adopting and continuing the use of an intervention.

  3. Testing & Evaluation: After app design, several phases of testing and evaluation are needed.40,43 The evaluation should include usability testing with real patients and domain experts, and an RCT to show clinical efficacy.

  4. Certified & Regulated Interventions: Opioid interventions must address FDA certification or exemption requirements for digital health software.44

  5. Collaborative & Connected Care: Opioid interventions should utilize platforms that can connect the key persons involved in treatment and recovery.45

DISCUSSION

The RACE framework is introduced for m-health app analysis. The viability of RACE framework highlights the importance of (a) identifying app characteristics and ensuring inter-rater agreement since there is subjectivity introduced in coding certain characteristics, and (b) the NVM taxonomy is essential for identifying gaps in m-health apps related to addressing health conditions. We observed that systematic literature review, assessment and classification, evaluation and guidelines for future m-health apps can lead to systematization of the fragmented knowledge and scope for future research (see Figure  6).

Figure 6.

Figure 6.

Open in a new tab

Application of RACE (Review, Assess, Classify, and Evaluate) to opioid apps and insights for validation.

We present our findings from applying RACE to opioid apps and insights learned for the validation of RACE when assessing other m-health apps. As shown in Figure  6 (Using RACE in Opioids), our assessment of opioid apps identified a strong emphasis on physician education, but less so for patients. We also found no evidence of opioid apps that might be able to address the secondary prevention through screening tools. Finally, for tertiary prevention or treatment, there is a significant gap in the much-needed continuity of care posttreatment. Generally, for m-health apps, we highlight in Figure  6 (Lessons for Other Health Conditions) the importance of utilizing evidence-based public health and clinical approaches when defining app characteristics. Furthermore, given that app descriptions are not always clearly outlined it is critical to include options of overlapping and unclear categories. Additionally, we highlight the importance of longitudinal assessment given that technology advancements lead to new and evolving m-health apps.

An important finding of the IRR analysis indicates that rater agreement differed for the app dimension of Key Function, highlighting the importance of variations in app descriptions across platforms. Another finding is that domain experts from multidisciplinary perspectives can provide insights from their respective fields when evaluating m-health apps. For example, a public health perspective was used to define the characteristics of the key function based on primary, secondary, and tertiary interventions. Similarly, evaluating cultural competence of m-health apps will require a domain expert for the targeted audience. Certainly, more work is needed in operationalizing the cultural competence (cultural awareness) and continuity of care (personalization and suitability) by involving users and experts (focus groups) from diverse cultures and backgrounds. This also highlights the need for guidelines to improve current and future m-health apps. We position the RACE framework with respect to other established methods to study m-health apps in Table  4.

Table 4.

Use of RACE framework and other methods

Goal Suggested methods/framework Comments
To design or develop new m-health apps Select methods from RACE and then the guidelines for future apps One of the strengths of RACE framework
To evaluate the quality of existing m-health apps Use Grundy et al, Nouri et al or RACE frameworks Either is suitable with variations (cultural competence in RACE)
To study m-health apps utilized as health interventions Use RE-AIM RE-AIM is designed specifically for this purpose
To classify and or analyze the current m-health apps Use steps 1 and 2 of RACE These steps are sufficient
Open in a new tab

Abbreviations: RACE: Review, Assess, Classify, and Evaluate.

Through our analysis of opioid apps, we identified a number of insights applicable to m-health apps in general. With low IRR, there is a major need to improve the app description to improve interpretation (for both researchers and practitioners). There is a need for new functions, cultural competence, longitudinal studies/postuse evaluation, and continuity of care. We identify the need for an integrated model with multiple stages, interventions, theories, and outcomes (Figure  1 being the first step toward such model).

In addition to these generalized insights, we also discovered some findings that are specific to opioid apps. Opioid apps must address prescription, over the counter (available in some countries), and illicit opioids. In some cases, there is an overlap among use and once addiction develops, the treatment, recovery, and remission for opioids (irrespective of their sources) will be similar. There is a need for holistic intervention for prescription, over the counter, and illicit opioids. However, prescription and monitoring functions, designed for physician prescribed opioids, will not be applicable for illicit or over-the-counter opioids. There are numerous legal, law enforcement and societal challenges with illicit opioid use that should be addressed. A major problem faced in the opioid epidemic is that relatively few patients receive the necessary treatment and support for recovery, resulting in a continuous cycle of dependence and a high mortality rate.46,47 Therefore, the field needs novel and evidence-based interventions to overcome barriers in obtaining treatment46,48–51 and to supplement existing healthcare interventions. M-health apps for opioid interventions are showing promise to lift the burdens of cost, time, convenience and reduce stigma, often cited as the reasons why patients do not to seek care.46,48–52

The RACE framework can be applied to other health conditions such as, but not limited to, type 2 diabetes, hypertension, and anxiety disorder. For example, applying RACE to type 2 diabetes apps may result in similar dimensions, such as Key Function, but widely different characteristics. The Key Function dimension of type 2 diabetes apps possibly includes {Education, Medications, Diet, Exercise, Glucose Monitoring, and Reminders} characteristics. These characteristics can lead to behavioral modification and wellbeing.

While RACE appears to be a promising methodology, there are several limitations that should be noted. First, we limited our opioid app review to the descriptions provided on the 2 app platforms (Google Play and Apple Store), rather than downloading each app for review. This was due to various restrictions with some apps requiring fee or subscription, prescription, or permission of a clinic. Downloading some apps and not others would reduce our ability to provide a fair and consistent review. Second, many apps did not have an informational website, resulting in our need to rely on less detailed descriptions provided on the app store platforms. Third, as part of our inclusion criteria, we decided to only consider apps that were either in English or were translatable to English on the app store platform. There may be many other apps that were not reviewed as result of this exclusion. However, we did not have the ability to review those that are not in English. Finally, we conducted a cross-sectional review. Future research can benefit from a longitudinal analysis.

CONCLUSION

With over 350 000 existing and emerging m-health apps,53 there is a need for systematic and replicable research for the analysis of m-health apps to understand their contributions for specific conditions. This study introduces the RACE framework for m-health app analysis and show its application to examine opioid apps. We observed that systematic literature review, assessing the apps, classification, evaluation, and guidelines for future m-health apps can lead to systematization of the fragmented knowledge and identification of future research. The RACE framework led to the identification and analysis of 153 opioid apps and dimensions of Target Audience, Key Function, Operation, Security & Privacy, and Impact. Cohen’s kappa differed across app platforms for Key Function, indicating the importance to examine app platforms separately because they may vary in how apps are presented and described. By applying the RACE framework to opioid apps, we demonstrate its systematic and interdisciplinary approach to analyze m-health apps.

Using the RACE framework, assessment of opioid apps led to the identification of several gaps that can be addressed through evidence-based interventions. First, the apps lacked collective engagement of patients, healthcare professionals, family members, and other decision-makers. Second, app assessment using the RACE framework identified a need for decision support systems for healthcare professionals, and reminders and monitoring for patients and family members. Most importantly, the RACE framework led to the development of a structured taxonomy and visual analysis. This analysis demonstrates that opioid apps must be dynamic, interactive, predictive, personalized, and context-aware.40 These characteristics can lead to an all-encompassing platform with multiple evidence-based interventions. Future studies can utilize the RACE framework toward understanding the dimensions and characteristics of m-health apps for other conditions to inform best practices for collaborative, connected and continued care.

FUNDING

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

All authors made significant contributions in the conception, design and analysis of the RACE framework and its application to opioid apps. The search procedure (screening the papers, the opioid apps, and conducting assessment for inclusion and exclusion) was carried out by all 4 authors. Author 2 and Author 3 carried out app coding; Author 4 conducted analysis of inter-rater reliability. Author 4 also contributed as a public health subject matter expert for substance use disorders. Author 1 and Author 2 and conducted the taxonomy analysis and visualization and all authors reviewed and approved. All authors worked on developing, finalizing and approving the final version submitted for publication.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

The authors have no competing interests to declare.

DATA AVAILABILITY STATEMENT

The data underlying this article are available in the article and in its online supplementary material.

Supplementary Material

ocab277_Supplementary_Data
Click here for additional data file. (109.4KB, pdf)

REFERENCES

  • 1.Istepanian RS, AlAnzi T.. Mobile health (m-health): evidence-based progress or scientific retrogression. In: Feng DD, ed. Biomedical Information Technology. Cambridge, MA: Academic Press; 2020: 717–33. [Google Scholar]
  • 2.Alshurafa N, Jain J, Alharbi R, Iakovlev G, Spring B, Pfammatter A.. Is more always better? Discovering incentivized mHealth intervention engagement related to health behavior trends. Proc ACM Interact Mob Wearable Ubiquitous Technol 2018; 2 (4): 1–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Jusoh S. A survey on trend, opportunities and challenges of mHealth apps. Int J Interact Mob Technol 2017; 11 (6): 73–85. [Google Scholar]
  • 4.McKay FH, Cheng C, Wright A, Shill J, Stephens H, Uccellini M.. Evaluating mobile phone applications for health behaviour change: a systematic review. J Telemed Telecare 2018; 24 (1): 22–30. [DOI] [PubMed] [Google Scholar]
  • 5.Grundy QH, Wang Z, Bero LA.. Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods. Am J Prev Med 2016; 51 (6): 1051–9. [DOI] [PubMed] [Google Scholar]
  • 6.Nouri R, R Niakan Kalhori S, Ghazisaeedi M, Marchand G, Yasini M.. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018; 25 (8): 1089–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.BinDhim NF, Hawkey A, Trevena L.. A systematic review of quality assessment methods for smartphone health apps. Telemed J E Health 2015; 21 (2): 97–104. [DOI] [PubMed] [Google Scholar]
  • 8.Nickerson RC, Varshney U, Muntermann J.. A method for taxonomy development and its application in information systems. Eur J Inf Syst 2013; 22 (3): 336–59. [Google Scholar]
  • 9.Vakkari P. Searching as learning: a systematization based on literature. J Inf Sci 2016; 42 (1): 7–18. [Google Scholar]
  • 10.NIH. Opioid addiction, National Institute of Health, Genetics Home Reference; 2019. https://ghr.nlm.nih.gov/condition/opioid-addiction.
  • 11.CDC. Annual surveillance report of drug-related risks and outcomes. CDC National Center for Injury Prevention and Control, Centers for Disease Control and Prevention; 2019. https://www.cdc.gov/drugoverdose/pdf/pubs/2019-cdc-drug-surveillancereport.pdf.
  • 12.United Nations Office of Drugs and Crime. World Drug Report 2019: 35 million people worldwide suffer from drug use disorders while only 1 in 7 people receive treatment; 2019. https://www.unodc.org/unodc/en/frontpage/2019/June/world-drug-report-2019_-35-million-people-worldwide-suffer-from-drug-use-disorders-while-only-1-in-7-people-receive-treatment.html.
  • 13.Simeonsson RJ. Primary, secondary, and tertiary prevention in early intervention. J Early Interv 1991; 15 (2): 124–34. [Google Scholar]
  • 14.Boufford JI, Cassel C, Bender K, Berkman L, Bigby J, Burke T.. The Future of the Public's Health in the 21st Century. Washington: Institute of Medicine of the National Academies; 2002. [Google Scholar]
  • 15.Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 2009; 6 (7): e1000100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Moher D, Shamseer L, Clarke M, et al. ; PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015; 4: 1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019; 7: 64. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Yoshida Y, Patil SJ, Brownson RC, et al. Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support. J Am Med Inform Assoc 2020; 27 (6): 946–56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Glasgow RE, Vogt TM, Boles SM.. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89 (9): 1322–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Cicchetti DV, Sparrow SA.. Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 1981; 86 (2): 127–37. [PubMed] [Google Scholar]
  • 21.Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 2012; 8 (1): 23–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Blendon RJ, Benson JM.. The public and the opioid-abuse epidemic. N Engl J Med 2018; 378 (5): 407–11. [DOI] [PubMed] [Google Scholar]
  • 23.Volkow N. Drug overdose deaths in 2020 were horrifying. Radical change is needed to address the drug crisis. National Institute on Drug Abuse Advancing Addiction Science National Institute on Drug Abuse; 2021. https://www.drugabuse.gov/about-nida/noras-blog/2021/08/drug-overdose-deaths-in-2020-were-horrifying-radical-change-needed-to-address-drug-crisis.
  • 24.APA. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). Arlington, VA: American Psychiatric Publishing; 2013. [Google Scholar]
  • 25.Brown TK, Alper K.. Treatment of opioid use disorder with ibogaine: detoxification and drug use outcomes. Am J Drug Alcohol Abuse 2018; 44 (1): 24–36. [DOI] [PubMed] [Google Scholar]
  • 26.Zhu H, Wu L-T.. National trends and characteristics of inpatient detoxification for drug use disorders in the United States. BMC Public Health 2018; 18 (1): 1–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Villwock JA, Villwock MR, New J, Ator G.. The false vital sign: when pain levels are not predictive of discharge opioid prescriptions. Int J Med Inform 2019; 129: 69–74. [DOI] [PubMed] [Google Scholar]
  • 28.Rodgers J, Cunningham K, Fitzgerald K, Finnerty E.. Opioid consumption following outpatient upper extremity surgery. J Hand Surg 2012; 37 (4): 645–50. [DOI] [PubMed] [Google Scholar]
  • 29.Mattick RP, Breen C, Kimber J, Davoli M.. Methadone maintenance therapy versus no opioid replacement therapy for opioid dependence. Cochrane Database Syst Rev 2009; 3 (3): CD002209. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Garland EL, Manusov EG, Froeliger B, Kelly A, Williams JM, Howard MO.. Mindfulness-oriented recovery enhancement for chronic pain and prescription opioid misuse: results from an early-stage randomized controlled trial. J Consult Clin Psychol 2014; 82 (3): 448–59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Calabria B, Degenhardt L, Briegleb C, et al. Systematic review of prospective studies investigating “remission” from amphetamine, cannabis, cocaine or opioid dependence. Addict Behav 2010; 35 (8): 741–9. [DOI] [PubMed] [Google Scholar]
  • 32.Hors-Fraile S, Rivera-Romero O, Schneider F, et al. Analyzing recommender systems for health promotion using a multidisciplinary taxonomy: a scoping review. Int J Med Inform 2018; 114: 143–55. [DOI] [PubMed] [Google Scholar]
  • 33.Singh N, Varshney U.. IT-based reminders for medication adherence: systematic review, taxonomy, framework and research directions. Eur J Inf Syst 2019; 29 (1): 1–25. [Google Scholar]
  • 34.Oberländer AM, Lösser B, Rau D. Taxonomy research in information systems: a systematic assessment. In: Proceedings of the 27th European Conference on Information Systems (ECIS); June 8–14, 2019; Stockholm & Uppsala, Sweden.
  • 35.Ross-Hellauer T. What is open peer review? F1000Res 2017; 6: 588. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Prat N, Comyn-Wattiau I, Akoka J.. A taxonomy of evaluation methods for information systems artifacts. J Manag Inf Syst 2015; 32 (3): 229–267. [Google Scholar]
  • 37.Julianelli V, Caiado RGG, Scavarda LF, de Mesquita Ferreira Cruz SP.. Interplay between reverse logistics and circular economy: critical success factors-based taxonomy and framework. Resour Conserv Recycl 2020; 158: 104784. [Google Scholar]
  • 38.Afra P, Bruggers CS, Sweney M, et al. Mobile software as a medical device (SaMD) for the treatment of epilepsy: development of digital therapeutics comprising behavioral and music-based interventions for neurological disorders. Front Hum Neurosci 2018; 12: 171. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Lougheed T. How “digital therapeutics” differ from traditional health and wellness apps. CMAJ 2019; 191 (43): E1200–1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Singh N, Varshney U.. Medication adherence: a method for designing context-aware reminders. Int J Med Inform 2019; 132: 103980. [DOI] [PubMed] [Google Scholar]
  • 41.van Engelenburg S, Janssen M, Klievink B.. Designing context-aware systems: a method for understanding and analysing context in practice. J Log Algebr Methods Program 2019; 103: 79–104. [Google Scholar]
  • 42.Sinha S, Jensen M, Mullin S, Elkin PL.. Safe opioid prescription: a SMART on FHIR approach to clinical decision support. Online J Public Health Inform 2017; 9 (2): e193. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.CDC, Guideline Resources: CDC Opioid Guideline Mobile App, CDC Gov. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control; 2019. [Google Scholar]
  • 44.Hamel MB, Cortez NG, Cohen IG, Kesselheim AS.. FDA regulation of mobile health technologies. New Engl J Med 2014; 371: 372. [DOI] [PubMed] [Google Scholar]
  • 45.Harris RA, Mandell DS, Kampman KM, et al. Collaborative care in the treatment of opioid use disorder and mental health conditions in primary care: a clinical study protocol. Contemp Clin Trials 2021; 103: 106325. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Lapham G, Boudreau DM, Johnson EA, et al. ; PROUD Collaborative Investigators. Prevalence and treatment of opioid use disorders among primary care patients in six health systems. Drug Alcohol Depend 2020; 207: 107732. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Hser Y-I, Mooney LJ, Saxon AJ, et al. High mortality among patients with opioid use disorder in a large healthcare system. J Addict Med 2017; 11 (4): 315–319. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Ahmed OM, Mao JA, Holt SR, et al. A scalable, automated warm handoff from the emergency department to community sites offering continued medication for opioid use disorder: lessons learned from the EMBED trial stakeholders. J Subst Abuse Treat 2019; 102: 47–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Dube SR, Felitti VJ, Dong M, Chapman DP, Giles WH, Anda RF.. Childhood abuse, neglect, and household dysfunction and the risk of illicit drug use: the adverse childhood experiences study. Pediatrics 2003; 111 (3): 564–572. [DOI] [PubMed] [Google Scholar]
  • 50.Derefinko KJ, García FIS, Talley KM, et al. Adverse childhood experiences predict opioid relapse during treatment among rural adults. Addict Behav 2019; 96: 171–174. [DOI] [PubMed] [Google Scholar]
  • 51.Mirick RG, Steenrod SA.. Opioid use disorder, attachment, and parenting: key concerns for practitioners. Child Adolesc Soc Work J 2016; 33 (6): 547–557. [Google Scholar]
  • 52.Aroke H, Buchanan A, Wen X, Ragosta P, Koziol J, Kogut S.. Estimating the direct costs of outpatient opioid prescriptions: a retrospective analysis of data from the Rhode Island prescription drug monitoring program. J Manag Care Spec Pharm 2018; 24 (3): 214–224. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Byambasuren O, Beller E, Glasziou P.. Current knowledge and adoption of mobile health apps among Australian general practitioners: survey study. JMIR mHealth Uhealth 2019; 7 (6): e13199. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ocab277_Supplementary_Data
Click here for additional data file. (109.4KB, pdf)

Data Availability Statement

The data underlying this article are available in the article and in its online supplementary material.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES