Usability and Learnability of RxUniverse, an Enterprise-Wide App Prescribing Platform Used in an Academic Tertiary Care Hospital

Abstract

The objective of this study was to assess the usability of RxUniverse, a novel platform that enables health care providers to directly disseminate proven, evidence-based mobile health apps to patients. Among five pilot clinical sites, 40 physicians and front-line providers were trained on the RxUniverse platform. They were educated on the platform’s functionality and instructed how to prescribe apps to their patients. The well-validated System Usability Score (SUS) was used to assess the usability of the platform. The adoption goal was set as 100 prescriptions of relevant apps within an 8-week pilot period. Within the pilot period, over 2000 apps were prescribed. Nineteen responses were received from the System Usability Score survey, and the platform received a usability score of 84.2, which is in the 96^th percentile across all systems. The pilot study outcomes demonstrate the high adoption and usability of the RxUniverse platform.

Introduction

Digital medicine is the subset of new technologies and mobile applications that demonstrate, or aim to demonstrate, positive impacts in the treatment and management of disease. Several hundred thousand digital medicine tools, such as mobile health (mHealth) applications, have been developed and released. In fact, there are approximately 259,000 mHealth apps available across the major app stores, and these are produced by an estimated 58,000 mHealth publishers¹. These numbers are increased from 2015 and years prior¹, and as the clinical community continues to seek solutions that further engage and empower patients, we can expect continued growth within the mHealth space in the years to come.

Such extensive proliferation of mobile health technologies can in part be explained by the increasing body of research that correlates more engaged and activated patients with improved clinical, behavioral, and utilization outcomes². Mobile health applications can potentially serve as tools to empower patients to become more actively engaged in their care, and thus more effective self-managers of their own health. For these reasons, developers are creating mHealth solutions ranging widely from remote monitoring tools, to chronic disease management platforms, to behavioral modification apps.

The rapid proliferation of mHealth apps has led to two notable unintended challenges. The first is that providers have no standardized way of ascertaining which apps are best suited to their patients. Evaluation of clinical interventions traditionally relies on rigorous study design and peer-review, but these are lacking for the vast majority of mHealth apps. Many developers advertise the potential health benefits of a given mHealth app to would-be patients and prescribers, and consumer reviews for some products do exist, but it is difficult to ascertain how much of these claims are substantiated by rigorous, evidence-based proof. As a result, health care providers find it difficult to objectively assess an app’s efficacy and usability.

Second, providers have no streamlined way of directly disseminating the appropriate mHealth apps to patients. Current practices are bulky and place the burden on patients to locate and download the appropriate app, and on providers to accurately and efficiently direct them to the accurate app. There is a lack of effective filtering and organization of the mHealth apps within any given app store, and this translates to worsened execution in finding and disseminating a specific mHealth tool. As such, the benefits of mobile health technologies can neither be fully felt nor properly evaluated.

To address these concerns, RxUniverse (http://rxuniverse.com) was developed at the Mount Sinai AppLab. It is a digital medicine-centric care delivery platform that features a catalogue of apps that have been curated based off of published, evidence-based reviews regarding their efficacy and usability; additionally, it incorporates user-centered features that enable efficient, direct dissemination to patients. The purpose of RxUniverse is to enable physicians to rapidly adopt digital medicine technologies and strategies within clinical care, to facilitate the effective dissemination of these technologies to patients, and to provide an ecosystem that promotes the evidence-based review of various digital medicine technologies.

A pilot study was conducted at the Mount Sinai Health System to assess the usability and adoption of RxUniverse, the first standardized platform to facilitate both curation and dissemination of mHealth apps.

Methods

Platform Description and Specifications

The platform assessed in this study was the RxUniverse mobile health platform, which was developed by Mount Sinai researchers at the Mount Sinai AppLab. RxUniverse integrates into any web browser, Android, or iOS devicebased workflow. Development environment produced a Node.js web application built using AngularJS 1.4.3 framework hosted on an Azure back-end with a data model persisted in Microsoft SQL Server. Communication on the backend is done through full RESTful API for a data layer running on Azure SQL server. All data was stored on a HIPPA secure database that could only be securely accessed by members of the RxUniverse research team.

The application solves discovery challenges by allowing providers to find healthcare apps using an advanced filter relevant to their specialties, disease, type of app, or functionality, and to prescribe apps directly from their mobile device or within a linked EHR. RxUniverse addresses the evidence challenge by allowing providers to rank apps and share their rating with their peers, to find apps which are FDA approved, and to read a brief peer-reviewed synopsis about the app and links to evidence published in literature.

Clinical Sites and Participants

Five outpatient clinical sites within the Mount Sinai Health System were selected to participate in the eight-week pilot study: two primary care clinics, one pulmonology clinic, one gastroenterology clinic, and one cardiology clinic. Clinic sites were chosen based off the following general criteria: outpatient primary care or specialty clinic, standardized work-flows, clinician interest in integrating use of at least one mHealth app into clinical care. Project procedures were carried out as part of standard of care.

User Training on RxUniverse Platform

Among these sites, 40 physicians and front-line providers, consisting of medical assistants and receptionists, were trained on the use of the RxUniverse platform. They were individually instructed by one of two researchers using a standardized protocol of instruction, lasting approximately 5-7 minutes. Each instruction session consisted of introduction to the platform’s purpose, a demonstration of its functionality in both the web and mobile interfaces, and an observed trial process of prescribing an app. Upon completion of the training, each new user was given a unique username and password.

Users were instructed to direct message via text or email from the platform’s interface, or “prescribe”, a mobile health app to each patient seen. Eight to ten institutionally approved apps were pre-selected for use during the pilot study and were available to the users on the platform to prescribe to patients. Careful consideration was given to the specific operational workflows of each clinical site, and both office managers and user volunteers were consulted in devising the optimal implementation plan for RxUniverse at each pilot site so as to minimize time burden on the volunteer users. Three clinics primarily relied on front desk receptionists to disseminate mHealth apps upon patient check-in. Two clinics utilized medical assistants to prescribe apps, either during patient intake or while patients waited in the waiting area. Two of these five clinical sites used a hybrid method that integrated both medical assistants and front-desk receptionists.

Measures

The System Usability Scale (SUS) is a very reliable tool^4,5 designed to obtain subjective feedback on overall usability and user satisfaction. According to Nielsen, usability is a “quality attribute that assesses how easy user interfaces are to use”³. Learnability is considered to be a component of usability and refers to the ease by which users are able to learn to use the technology³. This is a 10-item questionnaire with a 5-point Likert scale, with response options ranging from 1 (Strongly disagree) to 5 (Strongly agree). Items 1, 3, 5, 7, and 9 are positively worded and items 2, 4, 6, 8, and 10 are negatively worded. The SUS is able to effectively discern both good and bad usability features even with small sample sizes (<10). Ratings for SUS scores are as follows: 0–64 is unacceptable, 65–84 is acceptable, 85–100 is excellent, with a score of 82 representing the likelihood to recommend (LTR) threshold. Users are likely to recommend a product that has an average SUS score of 82, whereas users would not recommend a product that has an average SUS score of 67⁶. Using factor analysis, the SUS is able to provide additional information via two sub-scales: an 8-item “Usability” and 2-item “Learnability” scale^7,8.

Analysis

Success of the RxUniverse platform was measured in terms of number of apps prescribed and system usability. The number and type of apps prescribed by each user was tracked on a daily basis. The prescription of 100 relevant app prescriptions within the 8-week pilot period was set as the adoption goal. Adoption targets were based off of a prior study conducted by Mount Sinai AppLab researchers in determining the impact of a novel mobile health app, HealthPROMISE, in improving health and satisfaction outcomes. Weekly progress updates were sent to volunteer users and their respective clinic managers and physician sponsors.

Following the completion of the pilot study, the well-validated System Usability Score (SUS) was used to assess the usability of the platform. Descriptive analysis of demographic and individual SUS questionnaire items were conducted in excel to examine measures of central tendency and variability as well as level of use of the users. User responses were then converted to 0–100 percentiles as per SUS guidelines⁹ and scored by comparing them to standard rating scales¹⁰. The mean SUS score for all participants was calculated. Likelihood to Recommend (LTR) was determined by comparing overall mean SUS score to industry thresholds.

Results

Adoption

Within the 8-week pilot period, over 2000 apps were prescribed across all users among the five clinical sites. Of the 40 providers trained on the RxUniverse platform, 26 prescribed >5 apps during the trial period. Of these 26 individuals, 18 prescribed >20 apps, 14 prescribed >50 apps, and 5 prescribed >80 apps (Table 1). 58% of users reported frequent use (weekly or daily) of the platform.

Table 1.

RxUniverse Survey Statistics

Total Apps RX	2000
Time Period	8 weeks
People Trained	40
Provider prescribed greater than 5 apps	26
Provider prescribed over 20 apps	18 of 26
Provider prescribed over 50 apps	14 of 26
Provider prescribed over 80 apps	5 of 26

Demographics

A total of 19 care providers / users completed the SUS questionnaire (response rate: 47.5%). Three respondents had more than 2 items with missing data on the SUS portion of their survey, they were still reported (Table 3), however were excluded from the tabulation of the SUS scoring. In all, 63.16% of included respondents (n=12) reported their age as being between 36-45 and between 46-55 respectively. 73.68% of respondents were female (n=14). The sample was also diverse in terms of frequency with which users interacted with RxUniverse. In all, 57.9% (n=11) of the participants reported using RxUniverse “daily” or “weekly”, with the remaining 26.32% (n=5) only using the system a few times per month (Table 2, 3).

Table 3.

Summary of SUS Questionnaire Results for Overall Sample

	SUS Itemsi	Mean	95% CI	SD	Variance	Count	Min	Max
Q1	I think that I would like to use RxUniverse frequently.	3.79	3.27-4.31	1.06	1.11	19	1	5
Q2	I found RxUniverse unnecessarily complex.	2.06	1.32-2.80	1.39	1.94	17	1	5
Q3	I thought RxUniverse was easy to use.	4.61	4.19-5.03	0.83	0.68	18	2	5
Q4	I think I would need the support of a technical person to be able to use RxUniverse.	1.4	0.82-1.98	1.02	1.04	15	1	5
Q5	I found the various functions in RxUniverse were well integrated.	4.12	3.58-4.66	1.02	1.04	17	1	5
Q6	I thought there was too much inconsistency in the RxUniverse platform.	1.67	1.02-2.32	1.14	1.29	15	1	5
Q7	I would imagine that most people would learn to use RxUniverse very quickly.	4.44	4.02-4.87	0.83	0.69	18	3	5
Q8	I found the RxUniverse platform very cumbersome to use.	1.82	1.19-2.46	1.2	1.44	17	1	5
Q9	I felt very confident using RxUniverse.	4.67	4.29-5.05	0.75	0.56	18	2	5
Q10	I needed to learn a lot of things before I could get going with the RxUniverse platform.	1.5	1.06-1.94	0.79	0.63	16	1	5

¹Items 2,4,6,8, and 10 are negatively worded. Lower Means for these items represent higher perceived satisfaction.

SUS: System Usability Scale; CI: confidence interval; SD: standard deviation.

Table 2.

Demographic Variables of SUS Respondents

Characteristic	Number	Percent
Gender
Male	5	26.32%
Female	14	73.68%
Total	19	100%
Age
18-29	2	10.53%
30-35	4	21.05%
36-45	6	31.58%
46-55	6	31.58%
55+	1	5.26%
Total	19	100.00%
Frequency
Never	0	0%
Rarely	3	15.79%
Few times per Month	5	26.32%
Weekly	6	31.58%
Daily	5	26.32%
Total	19	100.00%

System Usability and Satisfaction

Overall, users felt that RxUniverse performed well. The group mean for overall SUS score was 84.2, an “Excellent” rating based on standard SUS⁹. RxU met the industry benchmark SUS score of at least 80 for users to likely promote your product. The mean score for the “Usability” sub-scale was 82.7 and the mean score for the “Learnability” sub-scale was 90. Individual item means are reported in Table 4. The majority of users had a favorable opinion about RxUniverse in terms of how confident they felt using the system (Q9) (4.67 with 95% CI of 4.29-5.05) and ease of use (Q3 and Q7). The “I found RxUniverse unnecessarily complex” question (Q2) indicates the respondents who felt mild concern.

Table 4.

SUS Questionnaire Percentile and Subscales

User	Percentile	SUS	Usability	Learnability
1	81.20%	77.5	81.3	62.5
2	97.20%	85	87.5	75
3	100%	92.5	90.6	100
4	100%	95	93.8	100
5	48.40%	67.5	59.4	100
6	48.40%	67.5	59.4	100
7	99.80%	90	87.5	100
8	64.80%	72.5	68.8	87.5
9	88.10%	80	78.1	87.5
10	100.00%	97.5	96.9	100
11	88.10%	80	75	100
12	100%	95	93.8	100
13	97.20%	85	81.3	100
14	81.20%	77.5	87.5	37.5
15	100.00%	100	100	100
Mean	96.20%	84.2	82.7	90

A raw SUS score of 84.2 has a higher SUS score than 96.19% of all products. We can be 95% confident the population SUS score is between 78.28 and 90.06. With a mean SUS of 84.2 and a standard deviation of 10.6 – compared to global SUS population benchmark of 68 and standard deviation of 12.5 – a t-statistic was calculated. The sample standard deviation was selected as a point of comparison, since it is more specific to this technology. The resulting t-score yields a p-value below 0.005 (0.001), allowing us to confirm the statistical significance of this pilot sample (Figures 1, 2). RxUniverse scored highest in the learnability category of the SUS score, with an average score found to be 90 (Figure 1).

Figure 1.

SUS Questionnaire Results

Figure 2.

SUS Questionnaire: Usability, Learnability Discussion

Discussion

As the pace of innovation continues to accelerate, health care providers will need to quickly integrate new digitally-based tools into their workflows, and patients will need to be able to easily and readily access these tools. It is equally important, however, to ensure that the technologies being disseminated to providers and patients pass performance and quality standards. RxUniverse not only provides the necessary mechanisms, user-friendly interface, and EHR integration functionality to accomplish these tasks, but it also surpasses industry standards in terms of usability and learnability.

The total number of apps prescribed on this platform surpassed 2000, which far exceeded the initial target of 100 apps. The type of mHealth apps prescribed varied widely, but the most common were MyChart, an app to help patients connect to their personal health records, a Mount Sinai Health Systems app, and two internally developed apps targeted Inflammatory Bowel Disease and Heart Failure. Additionally, the platform also scored an 84.2 (p = 0.001) on the System Usability Score, which is a score greater than 96.19% of all products and is considered to fall in the highest usability category of “excellent” (Table 4). According to industry standards, a score of 68 is considered to be average across all systems. By comparison, other health apps considered to be of respectable usability have reported scores of 77.5¹⁰. Additionally, RxUniverse received a high score within the learnability component of the SUS, which demonstrates the ease by which users were able to learn the technology utilized by the platform (Table 3). Research has shown that SUS scores provide reliable measures of usability and user satisfaction, and the high performance of RxUniverse on this scale demonstrates high achievement in these areas⁸. These outcomes demonstrate the high adoption and usability of the RxUniverse platform, an important platform that can be used to prescribe the latest technologies directly to patients.

This study may have some potential limitations. Surveys were offered to all users, but some bias may exist with respect to which users chose to fill out the survey or to participate in the pilot study. Additionally, the patient perspective was not assessed; the primary focus of this study, however, relates to the usability from the prescriber standpoint. Future iterations of this study will more closely involve the patient perspective.

Conclusion

RxUniverse is the first platform that creates an ecosystem that facilitates a standardized process for dissemination and curation of mobile health apps. The goals of this pilot study were to evaluate the RxUniverse platform and to compare to industry standards of usability and learnability. To further validate this platform, it will be important to conduct additional studies both health system-wide, as well as in partnership with other health systems across the United States. The platform is now being expanded for multisite collaborators. Future plans include full integration within the Mount Sinai health system’s EHR, release of additional mHealth apps and content, assessment of usability from the patient perspective, and continued platform development and modification.

It is our belief that RxUniverse can serve as a valuable tool in connecting health technology innovation to the end users of clinicians and patients, as well as enable evidence-based review of mHealth solutions.