Abstract
Purpose of review:
Opioid misuse, addiction, and related harm is a global crisis that affects public health and social and economic welfare. Many of the strategies being used to combat the opioid crisis could benefit from improved access and dissemination, such as that afforded by smartphone apps. The goal of this study was to characterize the purpose, audience, quality and popularity of opioid-related smartphone apps. Using web scraping, available information from 619 opioid-related apps (e.g., popularity metrics) was downloaded from Google Play, and 59 apps met criteria for review. The apps were additionally coded for quality by two raters using an 8-item screener for the American Psychiatric Association App Evaluation Model.
Findings:
Sixty one percent of apps targeted patients, 29% providers, 8% the general community, and 2% healthcare trainees. Regarding app purpose, 49% addressed treatment, 27% prevention, and 24% overdose. Only one app met all criteria on the screener for quality, and there was no association between a total score calculated for the screener and measures of app popularity (e.g., star ratings; R2=0.10, p=0.19).
Summary:
Opioid-related apps available for consumers addressed key stakeholders (patients, providers, community) and were consistent with strategies to address the opioid crisis (prevention, treatment, overdose). However, there was little evidence that available opioid-related apps meet basic quality standards, and no relationship was found between app quality and popularity. This review was conducted at the level of consumer decision-making (i.e., the app store), where only a handful of opioid-related apps met quality standards enough to warrant a more detailed evaluation of the app before recommendation for use. Because smartphone apps could be a critical tool to increase access to and utilization of opioid prevention, treatment, and recovery services, further development and testing is sorely needed.
Keywords: opiate, opioid use disorder, overdose, smartphone app, mobile health, mHealth
Introduction
Opioid use disorder (OUD) is a global epidemic, affecting over 16 million people and leading to 120,000 deaths per year worldwide. [1] Along with the personal costs to families and communities, OUD carries a substantial economic burden including costs of treatment, loss of productivity, use of prescription opioids, and criminal justice. [e.g., 2] Multiple strategies are being used to reduce opioid misuse, OUD, and related deaths, including strategies focused on restricting supply, influencing prescribing practices, reducing demand, and reducing harm. [3] Mobile technologies, in particular smartphone apps, are a promising tool for carrying out many of these strategies to address the opioid epidemic. Smartphone ownership is high globally, [4] and among a diverse population.[5] Smartphone apps are being used widely for healthcare, [5] with some apps even being covered by health insurance companies. [6] In the healthcare context, smartphone apps have shown initial utility in patient care and monitoring; medical communication, education, and research; and clinical reference, among other uses.[7] Therefore, smartphone apps for opioid misuse and OUD are both in line with current health consumer trends, and provide a novel opportunity to reduce opioid-related harm and death worldwide.
As the use of smartphone apps in healthcare has increased, so has concern about the quality, efficacy, and privacy of these apps, [8] including potential risks to consumers. Risks of apps may impact patient or provider users, health organizations, or other stakeholders, and include risks related to privacy and security, quality of patient data, lifestyle or clinical decision making by the patient, and/or clinical actions made by provider/professionals.[9] Risk variables may be inherent to the app (e.g., app contains inaccurate or out-of-date content), which could be reduced by risk assessment (e.g., peer review) and regulation (e.g., best practice guidelines, US Federal and Drug Administration [FDA] regulation). Risk variables may be external factors dependent on app use (e.g., inappropriate use by someone or for a purpose not intended by the developer) which could be addressed by educating users about potential risks. [9] Another concern is that consumers may be utilizing apps based on popularity (e.g., top returned searches in the app store) [10] rather than any evidence that the app will minimize risk and maximize benefit for a healthcare need. Examples in multiple contexts suggest that app popularity is not an indicator of clinical utility or usability, or even accuracy [10, 11]. For apps addressing opioid use, the risks can be high; for example, poor accuracy in dose calculations has been reported for opioid dose conversion apps, with prescription error a significant risk factor for opioid-related deaths. [12, 13]
In response to concerns about the quality, safety, and potential efficacy of smartphone apps for healthcare, several models for evaluating and recommending healthcare apps have emerged. These include risk frameworks [9], US FDA guidelines [14] and “precertification program” [15], UK National Health Service (NHS) Health Apps Library [16], and even a proposed “self-certification program” for developers that is subsequently validated or challenged by app users (i.e., patients, clinicians) [17], among other emerging models. One useful tool is the American Psychiatric Association (APA) App Evaluation Model [18], described as a hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. [19] The APA evaluation model is a multistep process for patients and providers to consider information that is important when picking an app for healthcare (focused on mental health apps), including: (1) access and background information, (2) privacy and security, (3) clinical foundation, (4) usability, and (5) data integration toward a therapeutic goal. Also offered is an Evaluation Model Screener, a brief version of the model addressing what are considered to be the most fundamental questions to ask when considering an app. [18]
The purpose of this review was to characterize currently available smartphone apps for the prevention, management, and treatment of opioid use, misuse, and related harm from the most-highly used app platform worldwide, Google Play. [20] The review process involved categorizing the apps based on target audience and purpose; screening the apps using the APA Evaluation Model Screener; examining Google Play metrics for app popularity (e.g., number of downloads); and evaluating associations between adherence to APA standards and app popularity. The main goals of this review were to characterize the scope of available opioid-related smartphone apps, and to better understand how measures of app popularity by (presumably) consumers relate to information considered to be important to health professionals in clinical-decision making. In this way, the review aimed to identify gaps in development and testing of opioid-related apps.
Methods
Data from Google Play were obtained via web scraping on April 14th, 2020. Web scraping (i.e., web data extraction) involves using a software program (code) to identify Hypertext Markup Language (HTML) elements on a webpage to extract and organize the data. Web scraping has been used by behavioral researchers to efficiently extract and organize data from publicly available websites for systematic review and analysis. [21] Since Google Play web developers protect their HTML content to prevent large scale automated scraping, a modified open-source Node.Js code was utilized [22] that leverages the Google Play application programming interface (API) to request and pull data. The code was run for the following keywords: opioid use disorder, opioid abuse opioid misuse, opioid addiction, prescription opioid misuse, prescription opioid abuse, opioid abuse treatment, opioid abuse intervention, opioid abuse therapy, opioid abuse management, and opioid addiction recovery.
The same web scraping procedure was applied to the Apple App Store using the same list of keywords. However, the procedure resulted in a very small sample of apps (n = 25), many of which overlapped with those identified in Google Play. Additionally, key to the goals of this review, the App Store only provided one popularity metric, star rating, and star ratings were computed using different algorithms in the App Store and Google Play, making cross-platform comparisons difficult. Therefore, in line with the goals of this review to evaluate app quality and popularity, only apps from Google Play were reviewed.
The following information was extracted for each identified opioid-related app: app description, number of installs, average star rating, number of star ratings, number of reviews, price, and date of last version update. Despite some limitations such as the potential noise in a given review (e.g., number of reviews could be high but negative), these are the popularity metrics in Google Play and therefore comprise the information available to consumers when deciding whether or not to download an app. Note that in Google Play, popularity metrics including average star rating, number of star ratings, and number of reviews are calculated based on the apps’ current ratings, rather than the lifetime average. This is described by Google Play as an effort to give users a better understanding of the apps’ current state, since apps can change over time, and can often add and remove features. [23] Although this is a limit to data transparency in Google Play, it is in line with the conceptual framework of this review by including information that is available to consumers at the level of decision-making (i.e., the app store) in the evaluation of apps. Other fields such as app developer website, privacy policy, and Google Play app URL were kept to facilitate app review and analysis. The results of each query were outputted to a JSON file, which were then inserted into a Python notebook for pre-processing.
All of the search results were merged, and duplicates were removed, resulting in the final dataset for analysis. Two coders independently reviewed the apps and identified those offering specific opioid-related content. Apps were then classified based on target app purpose and target audience. Finally, two coders independently reviewed each app and applied the APA Evaluation Model Screener. [18] The first and last author reviewed the evaluation results, refined codes as needed, and made final decisions about for any discrepancies.
The APA Evaluation Model Screener [18] includes eight questions addressing information considered important by health professionals in evaluating apps for healthcare. Each screener question can be answered as yes/no. This includes: (1) On which platforms/operating systems does the app work? (i.e., both Google Play and the App Store) (2) Has the app been updated in the last 180 days?; (3) Is there a transparent privacy policy that is clear and accessible before use?; (4) Does the app collect, use, and/or transmit sensitive data? If yes, does it claim to do so securely?; (5) Is there evidence of specific benefit from academic institutions, end user feedback, or research studies?; (6) Does the app have a clinical/recovery foundation relevant to your intended use?; (7) Does the app seem easy to use?; and (8) Can data be easily shared and interpreted in a way that’s consistent with the stated purpose of the app? Note that questions (1) and (8) were modified for this review. In the APA screener, question (1) includes, “Does it also work on a desktop computer?” Because this review focused on apps, desktop versions were not evaluated. In addition, question (8) about data sharing was not considered applicable for some apps (i.e., education apps, in which there are no data to be shared), and therefore those apps were not scored on that question, and total scores were adjusted (as indicated in Table 3). Note also that for question (5), evidence of specific benefit from end users was not evaluated using Google Play ratings/reviews because these were tested as popularity metrics and furthermore ratings/reviews were not considered to contribute to an app’s evidence base. All criteria were evaluated at the level of consumer decision-making (i.e., the app store), to best approximate the information available to consumers when choosing whether or not to download and try an app.
Table 3.
Opioid-related apps on Google Play. Raw data for app categories, popularity metrics, and APA screener items, sorted by APA screener “total score.”
| App | Purpose | Audience | # Installs | Star Rating | # Ratings | # Reviews | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Total Score |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pear reSET-O | Treatment | Patients | 1000 | 4.71 | 157 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Kaden Health | Treatment | Patients | 10 | NA | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0.875 |
| OARS | Treatment | Providers | 100 | NA | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0.875 |
| Opioid Rx | Treatment | Patients | 50 | NA | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0.875 |
| Workit Health | Treatment | Patients | 1000 | 4.90 | 39 | 29 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0.875 |
| BUP | Treatment | Patients | 100 | NA | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | NA | 0.857 |
| Stop OD NYC | Overdose | Patients | 5000 | 4.86 | 22 | 5 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | NA | 0.857 |
| LUminate Support | Treatment | Patients | 500 | NA | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0.75 |
| NaloxoFind-Trainer | Overdose | Community | 10 | NA | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0.75 |
| Opio | Treatment | Patients | 50 | NA | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0.75 |
| Telemed Express Video Visits | Treatment | Patients | 10 | NA | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0.75 |
| Reducere | Treatment | Patients | 500 | 2.60 | 5 | 5 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0.75 |
| COR-12 App | Treatment | Patients | 1000 | 3.67 | 12 | 4 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0.75 |
| Opioid Calculator | Prevention | Providers | 10000 | 3.79 | 43 | 25 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0.75 |
| NA Meeting Search 5.0 | Treatment | Patients | 500000 | 4.14 | 2754 | 838 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0.75 |
| Boulder Care | Treatment | Patients | 100 | 4.29 | 7 | 4 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0.75 |
| PursueCare: Addiction Recovery | Treatment | Patients | 500 | 4.75 | 8 | 4 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0.75 |
| UnityPhilly | Overdose | Patients | 100 | 4.90 | 10 | 3 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0.75 |
| Blue Cares | Treatment | Patients | 10 | NA | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | NA | 0.714 |
| FentaTIPS | Overdose | Providers | 500 | NA | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.714 |
| Get SMART | Prevention | Providers | 500 | NA | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.714 |
| OD-ME | Overdose | Patients | 10 | NA | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | NA | 0.714 |
| OpiSafe | Prevention | Providers | 100 | 1.83 | 6 | 3 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.714 |
| CDC Opioid Guideline | Prevention | Providers | 10000 | 2.64 | 28 | 16 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.714 |
| NaloxoFind | Overdose | Patients | 500 | 3.63 | 16 | 10 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | NA | 0.714 |
| OpiRescue | Overdose | Patients | 1000 | 4.43 | 7 | 3 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.714 |
| OPIT | Overdose | Patients | 10 | 5.00 | 9 | 6 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.714 |
| HC-CAN | Prevention | Community | 5 | NA | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | NA | 0.571 |
| Live4Lali | Treatment | Patients | 10 | NA | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | NA | 0.571 |
| Narcan Now | Overdose | Providers | 1000 | NA | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | NA | 0.571 |
| NARCANsas | Overdose | Patients | 1000 | NA | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | NA | 0.571 |
| Talk About Opioids | Treatment | Providers | 50 | NA | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | NA | 0.571 |
| Cordata Go | Overdose | Providers | 10 | NA | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 1 | 0.5 |
| Pilleve Companion App | Prevention | Patients | 10 | NA | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0.5 |
| 3 Steps To Save A Life | Overdose | Community | 50 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.429 |
| Nopioid | Treatment | Patients | 1 | NA | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | NA | 0.429 |
| Opioid Addiction Prevention | Prevention | Community | 1 | NA | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | NA | 0.429 |
| OpiRescue DE | Overdose | Patients | 100 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.429 |
| Pain Management pocketcards | Treatment | Providers | 500 | 4.25 | 8 | 2 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.429 |
| NA Speakers | Treatment | Patients | 100000 | 4.77 | 2447 | 978 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | NA | 0.429 |
| COWS | Treatment | Providers | 10 | NA | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| FlexDek: MAT Edition | Treatment | Patients | 100 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| OnTrack | Treatment | Patients | 10 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| Pain & Opioid Safety | Treatment | Community | 100 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| Staying Safe | Overdose | Patients | 50 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| Tablet Tally | Prevention | Providers | 10 | NA | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| Opioid Converter | Prevention | Providers | 50000 | 3.87 | 173 | 46 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| ORTHODOSE | Prevention | Providers | 10000 | 4.54 | 63 | 20 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.375 |
| Heroin & Tar Heroin | Treatment | Patients | 50 | NA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.286 |
| Opioid Medicine Abuse and Misuse | Prevention | Patients | 500 | NA | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | NA | 0.286 |
| Opioids Certification | Prevention | Providers | 10 | NA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.286 |
| Treatment Hub 365 | Treatment | Patients | 100 | NA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.286 |
| Opiate Abuse: The Truth | Prevention | Patients | 1000 | 4.86 | 44 | 26 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | NA | 0.286 |
| Opiate Illusions | Treatment | Patients | 10 | NA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0.25 |
| Narcotic & Opioid Converter | Prevention | Trainees | 1000 | 2.44 | 9 | 7 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.25 |
| Analgesic Converter | Treatment | Providers | 1000 | 3.57 | 7 | 5 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.25 |
| Painkiller Calculator | Prevention | Providers | 5000 | 4.09 | 22 | 8 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0.25 |
| Recover Or Die | Treatment | Patients | 10 | NA | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0.125 |
| Krokodil | Prevention | Patients | 10 | NA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| PERCENT | 68% | 31% | 68% | 41% | 2% | 90% | 97% | 25% |
American Psychiatric Association (APA) App Evaluation Screener questions: (1) On which platforms/operating systems does the app work? Does it also work on a desktop computer? (2) Has the app been updated in the last 180 days? (3) Is there a transparent privacy policy that is clear and accessible before use? (4) Does the app collect, us, and/or transmit sensitive data? If yes, does it claim to do so securely? (5) Is there evidence of specific benefit from academic institutions, end user feedback, or research studies? (6) Does the app have a clinical/recovery foundation relevant to your intended use? (7) Does the app seem easy to use? (8) Can data be easily shared and interpreted in a way that’s consistent with the stated purpose of the app?
For the current review, a total score was calculated as the average of yes=1 and no=0 for the eight questions, where a score closer to 1 indicated higher adherence to the APA standard. This approach was utilized to provide a summary measure of app quality, and to enable testing relations between app quality and app popularity, however, it is noted that the APA Evaluation Model Screener does not specify a number of items required to consider an app of usable quality.
Finally, correlations were tested between each of the app popularity metrics (number of installs, average star rating, number of star ratings, number of reviews) and the APA Evaluation Model Screener score. Additionally, a linear regression model was fit with APA Evaluation Model Screener as the dependent variable and each of the app popularity metrics as the predictors. The main predictor in this model, number of installs, was highly skewed and kurtotic, and was therefore log transformed. However, model estimates and fitness did not substantially change as a result of this transformation, therefore, the untransformed variable was used for final analysis. All analyses were conducted in R (Version 3.6.1).
Results
Web scraping identified a total of 619 opioid-related apps from Google Play. On a first review, duplicates were removed, as well as apps that were found to be unrelated to health behavior and substance use, apps that were under development (as indicated in Google Play), and apps that were not developed in English, resulting in 399 apps. Subsequently, two coders reviewed the apps and categorized them as: “includes specific opioid-related content,” “addresses substance use but does not contain opioid-related content,” or “other apps.” Fifty-nine apps identified as including opioid-related content were included in the final dataset for analysis.
Target purpose and audience
Opioid-related apps were first categorized by target audience, which was found to be 61% (n=36/59) designed for patients, 29% (n=17/59) for providers, 8% (n=5/59) for the community, and 2% (n=1/59) for healthcare trainees. Next, the same apps were categorized by intended purpose, which indicated that 49% were related to treatment, 27% addressed prevention, and 24% specifically addressed opioid overdose (see Figure 1). Examples of treatment-related apps included those supporting medication assisted treatment, cognitive behavioral therapy, and motivational interviewing. Note that the same apps were categorized by audience and purpose, and therefore categories overlap; for example, apps targeted at patients could address treatment, prevention, or overdose.
Figure 1.
Percentage and number (n) of opioid-related apps (N=59) by target audience (Panel A), and target purpose (Panel B).
App popularity
Regarding Google Play popularity metrics for opioid-related apps, the number of app installations was available for all reviewed apps (n=59) and averaged 11,938 ± 66,235 installations. For other popularity metrics, only some apps had current ratings (n=23), with an average star rating of 1.57 ± 2.06 of out of 5 stars; average number of ratings of 100 ± 474 ratings, and average number of written reviews of 36 ± 166 reviews.
App quality
We found that only 2% of the apps met all 8 evaluation criteria. The same percentage applied to apps that met either 1 criterion (2%), or none of the criteria (2%). Seven percent of the apps met 7 criteria, whereas 22%, 12%, 24%, and 15% met 6, 5, 4, and 3 criteria respectively. Average APA Evaluation Model Screener “total score” across all apps was 0.56 (range = 0 to 1; Table 3). One opioid-related app was found to meet all criteria on the APA Evaluation Model Screener (i.e., “yes” answer to all eight questions), the Pear reSET-O app. The next set of six apps did not pass the screener (score=.875) due to one factor: lack of evidence of specific benefit from academic institutions, end-user feedback, or research studies. Following this, eleven apps failed to pass the screener (score=0.75) due to several additional factors, including that most of these apps (69%; n=41) had not been updated in the past 180 days.
The majority of apps (67%; n=40) were found to have a transparent privacy policy that was available to read before use, by linking to a privacy policy from Google Play. The remaining apps either did not provide a link, or linked to an external website that did not contain a privacy policy. Upon inspection of the privacy policy, however, the language used to describe data security for over half of the apps (59%; n=35) was found to be insufficient to meet APA criteria. This includes privacy policies that did not state whether or not sensitive data were collected, used, or transmitted; did not directly address administrative or technical data security strategies to protect the data; or made vague claims about data security, such as “we are striving to use” appropriate security measures.
The majority of apps (97%, n=57) passed the APA Evaluation Model Screener question for usability, evaluated as whether or not the app seemed easy to use from the available screenshots on Google Play. Only two apps did not meet this criterion due to not providing adequate screenshots on Google Play.
Finally, many opioid-related apps (46%; n=27) were not evaluated on whether data can be easily shared and interpreted in a way that is consistent with the stated purpose of the app, because data sharing was not relevant to the app (e.g., education apps). For other apps (29%; n=17), data sharing and interpretation was not available despite relevance to the intended use of the app; for example, many apps providing opioid dosage calculators did not include functions to track or share these data. However, other apps (25%; n=15) enabled data sharing and interpretation, including several providing telehealth services with links to providers and the clinic (e.g., Kaden Health app).
Association between app popularity and quality
No significant associations were found between the APA Evaluation Model Screener total score and any app popularity metric, including number of installs, average star rating, number of reviews, and number of written reviews (Table 1). A linear regression model examining the association between the APA Evaluation Model Screener total score and number of installs, adjusting for all the other popularity metrics, was not significant (see Table 2).
Table 1.
Correlations between measures of APA Evaluation Model Screener “total score” and measures of app popularity.
| Variable | M | SD | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|---|
| 1. APA Screener Score | 0.56 | 0.23 | ||||
| 2. Number of Installs | 11,938 | 66,235 | .09 | |||
| 3. Average Stars Rating | 1.57 | 2.06 | .23 | .23 | ||
| 4. Number of Ratings | 99.93 | 474.34 | .04 | .86** | .30* | |
| 5. Number of Reviews | 36.39 | 165.88 | .04 | .78** | .31* | .99** |
M- mean; SD - standard deviation.
p < .05.
p < .01.
Table 2.
Results from a linear regression model using APA Screener “total score” as the criterion, and each of the app popularity metrics as predictors (apps N = 59). The model indicates that app popularity did not predict app quality.
| Predictor | b | b 95% CI | beta | beta 95% CI | Fit |
|---|---|---|---|---|---|
| (Intercept) | 0.52** | [0.44, 0.59] | |||
| Number of Installs | 0.00 | [−0.00, 0.00] | 2.18 | [−0.36, 4.71] | |
| Average Stars Rating | 0.02 | [−0.02, 0.05] | 0.14 | [−0.16, 0.44] | |
| Number of Ratings | −0.00 | [−0.01, 0.00] | −9.01 | [−20.15, 2.13] | |
| Number of Reviews | 0.01 | [−0.00, 0.02] | 7.22 | [−1.91, 16.36] | |
| R2 = .106, p = 0.189 |
Note. A significant b-weight indicates the beta-weight and semi-partial correlation are also significant. b represents unstandardized regression weights. beta indicates the standardized regression weights.
p < .05.
p < .01.
App cost
Four opioid-related apps (4/59, 7%) were found to be associated with a cost at the level of Google Play (range $0.99–4.99).
Discussion
This review identified a range of smartphone apps related to opioid use, misuse, and harm in Google Play. The identified apps were targeted at important stakeholders and addressed key efforts to combat the opioid epidemic. However, based on the app descriptions in Google Play, multiple gaps were found in intended app audience and purpose. Additionally, despite a growing number of opioid-related apps (e.g., a 2018 search identified 27 apps across platforms using ‘opiate’ and ‘opioid’ as key words [24] compared with 59 identified in the current review), the reviewed apps were found to have low popularity ratings overall, and low adherence to clinical standards upon evaluation.
With regard to target audience, opioid-related apps were found to be targeted to several key stakeholders –patients, providers, and the community. Additionally, with regard to target purpose, opioid-related apps were found to address key areas of importance to the opioid epidemic –treatment, prevention and overdose. While opioid-related apps were somewhat distributed among categories, only 8% of apps were designed for the general community, and 27% for prevention. This finding is in line with another recent review of opioid apps which identified 72 apps (55 Android or both Android/iOS apps) in 2019, with most designed to support clinicians,[25] with opioid conversion as the purpose of most apps. In addressing the opioid crisis, there have been calls to focus efforts beyond treatment of individuals already addicted to opioids [e.g., 26]. However, apps that address other prevention efforts, for example, substance use prevention among youth, were not included in this study because they were not specifically opioid-related. Such apps have the potential to contribute to prevention efforts and address risk factors for opioid use among youth [27]. Overall, findings suggest that there is considerable potential when it comes to developing apps to address target audience and purpose in the opioid crisis.
Overall, the reviewed opioid-related apps were found to be in line with common strategies to address opioid use, misuse, and harm [as outlined in: 3], including to reduce demand, such as apps to educate patients and the public about opioids and increase access to and utilization of treatment for OUD, and to reduce harm, including apps primarily related to locating and using naloxone to reverse overdose. Other strategies to reduce harm such as reducing disease transmission (via needle exchange, supervised injection facilities, drug checking, and behavioral interventions) were less well-represented among apps. Although, one app using this strategy is Live4Lali, a mobile outreach truck locator, with trucks offering harm reduction services such as drug-checking tools, safer sex supplies, and other support services. Apps were in line with additional common strategies to influence prescribing practices, including apps for provider education and prescribing guidelines, such as opioid dose calculators. One example is the Center for Disease Control and Prevention (CDC) Opioid Guideline app, described as a reference guide for prescribing opioids for chronic pain in clinical practice. Other strategies to influence prescribing practices, such as electronic medical records and decision support, were not found among apps. Finally, very few apps related to the strategy of restricting supply, such as by drug regulation (e.g., abuse-deterrent opioids) and restricting lawful access such as by scheduling, preventing and penalizing diversion, drug take-back programs, and restricting access, [3] although apps have the potential to enable services related to these strategies.
The wide range of opioid-related app purposes identified by this review highlights the utility of smartphone apps to address the opioid epidemic using a number of strategies, although the evidence base for many of these strategies is lacking, [3] and few-to-no studies have evaluated the impact of delivering these tools and services via smartphone app. Nevertheless, smartphone apps put some potentially useful tools, such as prescription calculators and naloxone locators, as well as opioid education, into the hands of patients, providers, and the community. Additionally, opioid apps have the potential to address further recommendations to reduce opioid misuse and harm; for example, by improving access to drug take-back programs. [3]
Only one app met all criteria on the APA Evaluation Model Screener, Pear reSET-O, which is also the only FDA-approved app for treating OUD. re-SET-O provides cognitive behavioral therapy as an adjunct to buprenorphine and contingency management, and was the only opioid-related app found to be associated with a research study. [28] However, one potential limitation of reSET-O is that it must be prescribed by a provider, and therefore individuals who are not able to seek in-person treatment due to lack of accessibility or insurance coverage [29] are not able to access this app. Nevertheless, the approach used to develop re-SET-O is consistent with the FDA Center for Devices and Radiological Health effort to regulate the quality, safety and efficacy of digital devices designed to treat health conditions. Such an approach is also consistent with the Digital Therapeutics Alliance,[30] an academic and industry effort to ensure that digital interventions (1) are tested in clinical trials or developed according to the best science available (i.e., peer review research); (2) follow best standards for data security, privacy, and usability; and (3) are vetted by regulatory agencies. The low quality standards of opioid-related apps in this review supports the need for continued development of such efforts.
No other app besides re-SET-O met criteria for the screener question (5), having evidence of specific benefit from academic institutions, end user feedback, or research studies, because no other app was associated with a research study. This screener question is related to evaluating the clinical foundation of healthcare apps, given that many apps make clinical claims without clinical studies. Lack of evidence from research studies is critical to evaluate because it can provide evidence to users that the app may offer a clinical benefit. However, because very few smartphone apps for healthcare in general are supported by clinical studies, [31] the full APA Evaluation Model provides guidance on evaluating the evidence base of an app using additional criteria. Therefore, apps failing only this screener question would be recommended for further evaluation along this criterion. For example, the Buprenorphine Home Induction Tool (BUP) is a buprenorphine home induction app developed by academic researchers based on RAND Corporation’s Procedures for Medication-Assisted Treatment of Alcohol or Opioid Dependence in Primary Care [32]. In line with this, although only the reSET-O app was associated with a research study, almost all of the apps reviewed met criteria for screener question (6), having a clinical/recovery foundation relevant to the apps’ intended use (all but 6 apps). Another recommendation would be to search an app of interest on a clinical trials database (e.g., https://clinicaltrials.gov/ in the US, https://bepartofresearch.nihr.ac.uk in the UK, and https://apps.who.int/trialsearch/globally ) to identify related clinical studies underway.
Another key failure was that most opioid-related apps had not been updated in the past 180 days. For example, the CDC Opioid Guideline app has not been updated since July 2018. Education apps that have not been updated may contain information that is out-of-date. For example, a recent content analysis of smartphone apps for alcohol and illicit substance use found that several apps (e.g., FlexDek: MAT Edition; updated August 2017) lacked updated information such as outdated contact information and broken links to external websites. [8] Indeed, ‘days since last update’ has been recommended as an easy and useful single-item screening test for health apps more broadly.[10]
Additionally, many apps did not meet APA Evaluation Model Screener criteria for privacy and security. On this issue, the screener includes two questions: whether the app has a transparent privacy policy, and whether the app claims to handle sensitive data securely. Only about a quarter of opioid-related apps were found to meet both criteria. Privacy and security are relatively unique concerns for healthcare delivered by smartphone app, and are critical to evaluate given the potential risks with disclosure of personal health information. Sharing of user data and lack of transparency have been identified as privacy risks in medical apps more generally, [33] and in apps for alcohol and illicit substance use[8]. Other privacy risks identified [8] include inadequate user authentication (e.g., password protection, 2-step verification); lack of encryption for data transmission; and others. The full APA App Evaluation Model considers additional risks including data costs (i.e., with cell provider), social profiling, and loss of insurance benefits or insurability.
The majority of apps passed the screener question for usability, evaluated as whether or not the app seemed easy to use from the available screenshots on Google Play. Because apps were not downloaded for review, this usability criterion was evaluated based on subjective impressions of basic information from app screenshots, such as language, legibility, function, and design. These types of subjective impressions could ultimately drive consumer decisions about downloading an app. The limitation of this approach was that most apps appear usable from screenshots; therefore, almost all of the apps reviewed at the level of Google Play met this criterion. This is not an endorsement of the subjective usability of these apps, because many may be found to be unusable upon testing. However, a further evaluation of apps by download and testing was outside of the scope of the current review. In the comprehensive APA Evaluation Model, usability, considered a subjective measure, is evaluated by testing the app and asking: What are the main engagement styles of the app? Do the app and its features align with your needs and priorities? Is it customizable? Does the app clearly define functional scope? Does the app seem easy to use? Multiple additional methods and measures exist for evaluating the usability of smartphone apps for healthcare, [34] for example the Mobile App Rating Scale [35] is a useful tool for evaluating the quality of mobile health apps on engagement, functionality, aesthetics, information quality and subjective quality. Finally, a more complete evaluation of app usability would require user-centered design research,[36] which would provide more rigorous data to further develop and test these apps.
The final question on the APA Evaluation Model Screener is whether data can be easily shared and interpreted in a way that is consistent with the stated purpose of the app. This criterion relates to the evaluation of data integration toward a therapeutic goal, and is focused on the ability to share data between patient and provider. For many informational apps, this criterion was not relevant. For other apps, about half enabled data sharing and interpretation. Data sharing is considered important for healthcare apps to ensure that data are available to both patient and provider to guide care, make treatment decisions, and avoid fragmented care.[19]
A main purpose of this review was to evaluate opioid-related apps with regard to associations between app quality and app popularity. The current study found no significant associations between the APA Evaluation Model Screener score and Google Play metrics of app popularity including average star rating, or number of ratings, written reviews, or app installs. This finding is in line with previous studies finding a lack of clear associations between measures of app popularity (e.g., top returned searches in the app store, average consumer ratings) and evaluations of app quality. [e.g., 10, 11] Examining this association is critical to identify popular opioid-related apps that may provide misinformation or potentially harmful advice without an evidence base. [9]
Finally, very few apps were associated with a price at the level of Google Play. However, many apps that are free to download require in-app purchases to access full functionality, and this review did not download apps to evaluate in-app purchases. Additionally, there can be costs associated with medical apps, for example, the reSET-O app requires a prescription and other telehealth apps can require insurance coverage.
There were several limitations to this review. First, only opioid-related apps from Google Play were reviewed, thereby omitting apps that may only be available on the Apple App Store or third-party Android stores. However, Google Play is the most popular app store in the world (2.6 million apps),[37] and therefore a useful platform to evaluate associations between app popularity and app quality. Additionally, two-thirds of the apps reviewed were found to be available in the App Store, suggesting significant overlap between the two major platforms.[20] This overlap was also identified by another opioid app review,[25] which reported that in recent years (2018–2019), opioid apps were found to be available on the Android operating system only, or on both the iPhone and Android operating systems, but not iOS only. Next, only the APA Evaluation Model Screener was utilized, rather than the full APA Evaluation Model, and apps were not downloaded for evaluation. The purpose of this study was to provide an ecologically valid evaluation of opioid-related apps at the level of consumer decision-making (i.e., the app store). The screener is considered a first step, and apps that scored reasonably well warrant a more detailed evaluation prior to use. Additionally, a number of other evaluation models and guidelines are emerging that may provide alternative or complementary strategies for informed decision-making around healthcare apps.
Conclusions
OUD is a chronic relapsing disorder, typically requiring medication and psychological support, with challenges in symptom management and treatment adherence, and risk of death from overdose and other causes [38]. Treating OUD presents a complex set of challenges, most prominently difficulties in accessing high quality medication assisted treatment, [29] the gold standard of care for individuals with OUD. [37] Additional complexities exist when considering the wide scope of opioid-related harm. As this review suggests, there are many opportunities for addressing these factors using smartphone apps, and there are numerous efforts to do so, as is evident in the 59 available Google Play apps addressing a range of issues related to opioid use, misuse, and harm. However, only one opioid-related app met all criteria on a basic clinical evaluation screener, and there was no association between adherence to clinical standards and levels of popularity and use by real world users. This is a reason for concern considering the dimensions of the opioid epidemic and the serious consequences of opioid use and misuse. Despite this, many apps were found to have a clinical or recovery foundation relevant to addressing the opioid epidemic. Because smartphone apps have the potential to be a critical tool in increasing access to and utilization of opioid prevention, treatment, and recovery services, further development and testing is sorely needed.
Acknowledgements
KG was supported by the National Center for Complementary and Integrative Health (#1R34AT010365-01) and National Institute on Drug Abuse (#R21DA049583-01). RV was supported by the National Cancer Institute (#R21CA243911) and the National Institute on Drug Abuse (#R01DA047301).
Footnotes
Conflict of Interest
The authors declare no conflicts of interest
Compliance with Ethics Guidelines
Human and Animal rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors
Publisher's Disclaimer: This Author Accepted Manuscript is a PDF file of a an unedited peer-reviewed manuscript that has been accepted for publication but has not been copyedited or corrected. The official version of record that is published in the journal is kept up to date and so may therefore differ from this version.
References
- 1.Dydyk AM, Jain NK, Gupta M. Opioid Use Disorder. Treasure Island (FL): StatPearls Publishing. [Updated 2020 Apr 12]. https://www.ncbi.nlm.nih.gov/books/NBK553166/. [Google Scholar]
- 2.Florence CS, Zhou C, Luo F, Xu L. The Economic Burden of Prescription Opioid Overdose, Abuse, and Dependence in the United States, 2013. Med Care. 2016;54(10):901–6. doi: 10.1097/MLR.0000000000000625. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.National Academies of Sciences Engineering, and Medicine; Health and Medicine Division; Board on Health Sciences Policy; Committee on Pain Management and Regulatory Strategies to Address Prescription Opioid Abuse. Pain Management and the Opioid Epidemic: Balancing Societal and Individual Benefits and Risks of Prescription Opioid Use. Evidence on Strategies for Addressing the Opioid Epidemic. Washington (DC): National Academies Press (US); 2017. [PubMed] [Google Scholar]
- 4.Newzoo. Global mobile market report. 2019. https://newzoo.com/insights/trend-reports/newzoo-global-mobile-market-report-2019-light-version/.
- 5.Krebs P, Duncan DT. Health App Use Among US Mobile Phone Owners: A National Survey. JMIR mHealth uHealth. 2015;3(4):e101. doi: 10.2196/mhealth.4924. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Bini M Express Scripts Makes Digital Health Click 2019. https://www.express-scripts.com/corporate/articles/digital-health-formulary-announced. Accessed June 11, 2020.
- 7.Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res 2012;14(5):e128. doi: 10.2196/jmir.1994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Tofighi B, Chemi C, Ruiz-Valcarcel J, Hein P, Hu L. Smartphone Apps Targeting Alcohol and Illicit Substance Use: Systematic Search in in Commercial App Stores and Critical Content Analysis. JMIR mHealth uHealth. 2019;7(4):e11831. doi: 10.2196/11831. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer use. J Med Internet Res. 2014;16(9):e210. doi: 10.2196/jmir.3133.* Helpful guidance for risk assessment of healthcare apps
- 10.Wisniewski H, Liu G, Henson P, Vaidyam A, Hajratalli NK, Onnela JP et al. Understanding the quality, effectiveness and attributes of top-rated smartphone health apps. Evid Based Ment Health. 2019;22(1):4–9. doi: 10.1136/ebmental-2018-300069. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Singh K, Drouin K, Newmark LP, Lee J, Faxvaag A, Rozenblum R et al. Many Mobile Health Apps Target High-Need, High-Cost Populations, But Gaps Remain. Health Aff 2016;35(12):2310–8. doi: 10.1377/hlthaff.2016.0578. [DOI] [PubMed] [Google Scholar]
- 12.Haffey F, Brady RR, Maxwell S. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf 2013;36(2):111–7. doi: 10.1007/s40264-013-0015-0. [DOI] [PubMed] [Google Scholar]
- 13.Rich BA, Webster LR. A review of forensic implications of opioid prescribing with examples from malpractice cases involving opioid-related overdose. Pain Med 2011;12 Suppl 2:S59–65. doi: 10.1111/j.1526-4637.2011.01129.x. [DOI] [PubMed] [Google Scholar]
- 14.Mobile medical applications: guidance for industry and Food and Drug Administration staff. Published February 9, 2015. https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf. Accessed 23 Aug 2019.
- 15.US Food and Drug Administration. Digital health innovation action plan: FDA Center for Devices and Radiological Health. 2017. https://www.fda.gov/media/106331/download. Accessed 25 June 2020.
- 16.UK National Health Service. NHS Apps Library. 2017. https://www.nhs.uk/apps-library/?page=4. Accessed 11 June 2020.
- 17.Rodriguez-Villa E, Torous J. Regulating digital health technologies with transparency: the case for dynamic and multi-stakeholder evaluation. BMC medicine. 2019;17(1):226. doi: 10.1186/s12916-019-1447-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.American Psychiatric Association. App Evaluation Model. 2018. https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/app-evaluation-model. Accessed 11 June 2020.
- 19.Torous JB, Chan SR, Gipson SYT, Kim JW, Nguyen TQ, Luo J et al. A Hierarchical Framework for Evaluation and Informed Decision Making Regarding Smartphone Apps for Clinical Care. Psychiatr Serv 2018;69(5):498–500. doi: 10.1176/appi.ps.201700423.** Paper outlining the healthcare app evaluation model and screener applied to opioid-related apps in this review.
- 20.Cheney S, Thompson E. The 2017–2022 App Economy Forecast: 6 Billion Devices, $157 Billion in Spend & More. 2018. https://s3.amazonaws.com/files.appannie.com/reports/1805_Report_2022_Forecast_EN.pdf. Accessed 12 June 2020. [Google Scholar]
- 21.Bradley A, James R. Web Scraping Using R. Advances in Methods and Practices in Psychological Science. 2019;2(3):264–70. doi: 10.1177/2515245919859535. [DOI] [Google Scholar]
- 22.Google-play-scraper. https://github.com/facundoolano/google-play-scraper. Accessed 14 June 2020.
- 23.Google Play. Ratings & Review on the Play Store. https://play.google.com/about/comment-posting-policy/. Accessed 16 June 2020.
- 24.Aggarwal M, Borycki E. Review of Mobile Apps for Prevention and Management of Opioid-Related Harm. Improving Usability, Safety and Patient Outcomes with Health Information Technology, vol 257. 2019. doi: 10.3233/978-1-61499-951-5-1. [DOI] [PubMed] [Google Scholar]
- 25.Nuamah J, Mehta R, Sasangohar F. Technologies for Opioid Use Disorder Management: Mobile App Search and Scoping Review. JMIR mHealth uHealth. 2020;8(6):e15752. doi: 10.2196/15752. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.National Institutes of Health. The Helping to End Addiction Long-term Initiative. https://heal.nih.gov/. Accessed 15 June 2020.
- 27.Volkow N The Importance of Prevention in Addressing the Opioid Crisis. National Institute on Drug Abuse, Nora’s Blog. 2019. https://www.drugabuse.gov/about-nida/noras-blog/2019/06/importance-prevention-in-addressing-opioid-crisis. Accessed 15 June 2020.
- 28.Christensen DR, Landes RD, Jackson L, Marsch LA, Mancino MJ, Chopra MP et al. Adding an Internet-delivered treatment to an efficacious treatment package for opioid dependence. J Consult Clin Psychol 2014;82(6):964–72. doi: 10.1037/a0037496.* The only research study of an opioid-related app identified by this review, reporting results from a randomized clinical trial of the re-SET-O app.
- 29.Jones CM, Campopiano M, Baldwin G, McCance-Katz E. National and State Treatment Need and Capacity for Opioid Agonist Medication-Assisted Treatment. Am J Public Health. 2015;105(8):e55–63. doi: 10.2105/AJPH.2015.302664. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Digital Therapeutics Alliance. 2017. https://dtxalliance.org. Accessed 24 June 2020. [Google Scholar]
- 31.Safavi K, Mathews SC, Bates DW, Dorsey ER, Cohen AB. Top-Funded Digital Health Companies And Their Impact On High-Burden, High-Cost Conditions. Health Aff 2019;38(1):115–23. doi: 10.1377/hlthaff.2018.05081. [DOI] [PubMed] [Google Scholar]
- 32.Heinzerling KG, Ober Allison J., Lamp Karen, De Vries David, and Watkins Katherine E.. SUMMIT: Procedures for Medication-Assisted Treatment of Alcohol or Opioid Dependence in Primary Care. Santa Monica, CA: RAND Corporation; 2016. [Google Scholar]
- 33.Grundy Q, Chiu K, Held F, Continella A, Bero L, Holz R. Data sharing practices of medicines related apps and the mobile ecosystem: traffic, content, and network analysis. BMJ. 2019;364:l920. doi: 10.1136/bmj.l920. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Bradway M, Gabarron E, Johansen M, Zanaboni P, Jardim P, Joakimsen R et al. Methods and Measures Used to Evaluate Patient-Operated Mobile Health Interventions: Scoping Literature Review. JMIR mHealth uHealth. 2020;8(4):e16814. doi: 10.2196/16814.* A useful review of methods and measures to evaluate meaningful user engagement and self-efficacy in mobile interventions.
- 35.Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3(1):e27. doi: 10.2196/mhealth.3422. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Rubin J, Chisnell D. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. 2nd ed. Indianapolis, IN: Wiley Publishing, Inc.; 2008. [Google Scholar]
- 37.Connery HS. Medication-assisted treatment of opioid use disorder: review of the evidence and future directions. Harv Rev Psychiatry. 2015;23(2):63–75. doi: 10.1097/HRP.0000000000000075. [DOI] [PubMed] [Google Scholar]
- 38.Schuckit MA. Treatment of Opioid-Use Disorders. N Engl J Med 2016;375(4):357–68. doi: 10.1056/NEJMra1604339. [DOI] [PubMed] [Google Scholar]