Remote Usability Testing and Satisfaction with a Mobile Health Medication Inquiry System in CKD

Abstract

Background and objectives

Inappropriate medication use is common in the care of patients with CKD. The feasibility of a simple mobile health tool designed to advise patients on safe medication usage in CKD was examined.

Design, setting, participants, & measurements

Participants with predialysis CKD (defined as eGFR<60 ml/min per 1.73 m²) in the Safe Kidney Care Cohort Study were recruited for home usability testing of a novel medication inquiry system between January and September of 2013. Testing was through two mobile platforms: (1) short messaging service text or (2) personal digital assistant (e.g., iPod Touch). Twenty participants (one half assigned to one device and one half assigned to the other device) were enrolled and received an in-center tutorial on device usage before the end of the study visit. Participants were subsequently mailed three sample pill bottles with the name of randomly selected medications and asked to input these medications into the medication inquiry system. The medication inquiry system response options were as follows: (1) safe in CKD, (2) not safe in CKD, (3) use with caution/speak with your health care provider, or (4) error message (for an incorrectly inputted medication). Participants were asked to record the response issued by the medication inquiry system for each medication sent for usability testing. A user satisfaction survey was administered after completion of the protocol.

Results

All participants owned a mobile telephone, but few owned a smartphone. Of 60 total medication queries, there were only three recorded errors, two of which occurred in the short messaging service texting group. Overall satisfaction with the application was high, with slightly higher satisfaction noted in the personal digital assistant group compared with the short messaging service group.

Conclusions

The mobile health medication inquiry system application had general ease of use and high acceptance across two platforms among individuals representative of the CKD population. Tailored mobile health technology may improve medication safety in CKD.

Introduction

Improperly dosed or ill-advised medication usage are a significant hazard in the care of individuals with CKD (1), which is likely partially attributable to failed recognition or underappreciation of CKD by both providers and patients (2–5). Despite electronic health record prescription monitoring systems that reduce the likelihood of drug-related problems, the complex care of patients with CKD, often characterized by interactions with diverse health systems and health providers, requires innovative means that place patients in the center of their care for self-management and enable them to safeguard themselves from drug-related problems and other medical errors.

Mobile devices, such as cellular telephones or tablets, may provide new tools with the potential to reduce unsafe practice in CKD management. An explosion of new technologies has provided consumers with access to myriad applications and portals to access health information, but how well the target population can effectively use such applications has not been adequately assessed. Integration of mobile devices into CKD health management requires an iterative user-centered design that incorporates usability testing and user feedback in the final development process to improve the potential effectiveness of the tool. Although in-center usability testing provides valuable insight into the functionality of developing technologies, it does not incorporate home-based processes of care that may influence a user’s ability to navigate the technology under more real-life circumstances.

In this study, we set out to build on prior scripted in-center usability testing, which has been previously reported (6), of a disease-specific application designed as a medication inquiry system (MIS) offering patients information on the safety of drugs in CKD. This study evaluates home-based usability of two mobile health MIS platforms, including short messaging service (SMS) texts or an application embedded in a personal digital assistant (PDA), and reports on the acceptance of the mobile health platform by usability participants.

Materials and Methods

MIS

The MIS application was developed as a patient-centered tool to assist patients with predialysis CKD in identifying the safety of their medications with impaired renal function. Features available for the SMS text platform have been previously described (6) and include the abilities to send the name of a medication by SMS text message and receive a response text informing the patient of the medication’s safety in CKD with three potential responses: not safe in CKD, use with caution/speak with your health care provider, and safe in CKD. The PDA application platform allows users to search by the medication name or class (e.g., ibuprofen or pain medication). PDA responses include traffic light imagery and text to emphasize safety responses: a red light for a medication that is not safe in CKD, a yellow light for use with caution/speak with your health care provider, and a green light for medications deemed safe in CKD (Figure 1). Both platforms had a fourth ability to display an error message if a medication was incorrectly inputted into the device.

Figure 1.

Personal digital assistant (PDA) platform medication inquiry system display. Introduction screen, search screen, and result screen of PDA application.

Study Participants

The Safe Kidney Care (SKC) Cohort Study (ClinicalTrials.gov NCT 01407367) is an ongoing observational cohort of individuals with predialysis CKD intended to assess the frequency of adverse safety events with a protocol that has been previously described (7). To be eligible for the SKC Study, participants required two measures of renal function with an eGFR of <60 ml/min per 1.73 m² at least 90 days apart and ≤18 months before enrollment. Participants were excluded if they were expected to reach ESRD or die within 1 year from enrollment. As an ancillary study, participants in the SKC Study were invited at an annual SKC Study visit to participate in home usability and feasibility testing of the mobile device–based MIS, with a target of 20 participants. The SKC Study and its ancillary study were approved by the University of Maryland School of Medicine Institutional Review Board and the Veterans Affairs Maryland Health Care System Baltimore Research and Development Committee. Informed consent was obtained from all participants in accordance with the Declaration of Helsinki.

Remote Usability Testing

Between January and September of 2013, in total, 20 usability participants were randomly assigned one of two mobile devices to bring home: (1) a traditional cellular telephone (flip telephone) with SMS texting capabilities or (2) a PDA without mobile telephone capabilities (iPod Touch) on which the MIS application had been previously loaded. At enrollment, all participants were given an in-person tutorial on device usage, and PDA participants were given printed instructions on MIS application usage and encouraged to take them home. After the initial visit, usability participants were mailed sample pill bottles with the names of three hypothetical medications that had been randomly selected for each participant from the available MIS database of medications to mimic the method by which many patients receive their home medications. Although the PDA platform included a comprehensive database of medications, the SMS texting platform prototype was only available for use with eight prespecified medications. Therefore, the sample pill bottles consisted of randomly selected medications that were only available from the eight medications programmed into the SMS texting prototype.

Participants were asked to input each of three medications into the respective MIS application and record the device’s responses on a paper diary. At 1 month, participants returned to turn in their devices, review the paper diary, and complete a user satisfaction survey. Participants were compensated $50 for their time at the completion of the protocol.

eHealth Literacy

MIS remote usability testing results were linked with core demographics from the parent SKC Cohort Study as well as information pertaining to participants’ access to and characteristics of computer and mobile telephone usage. As part of this e-literacy assessment, all participants were administered the eHealth Literacy Scale (eHEALS), an eight-item measure of eHealth literacy developed to measure individuals’ combined knowledge and perceived skills at finding, evaluating, and applying electronic health information to health problems (8). Participants were asked to rate their level of agreement with various statements, such as, “I know how to find helpful health resources on the internet,” “I know how to use the health information that I find on the internet to help me,” and “I can tell high-quality from low-quality health resources on the internet.” The questionnaire has been previously validated on patients with HIV and health care trainees (9,10). The eHEALS survey can be scored in its entirety on the basis of the cumulative responses to eight questions (the first two eHEALS questions serve as supplementary questions to understand general interest in eHealth and are not included in overall score), or each question can be evaluated independently. Cumulative scores range from 8 to 40, with higher scores reflective of higher levels of eHealth literacy.

Statistical Analyses

No formal hypothesis testing was conducted in this analysis, and descriptive analyses were consistent with the applied qualitative study methods of usability testing. Sample size was consistent with target samples for usability testing (11). Continuous variables were presented with means and SDs. Binomial and categorical variables were presented as N (%). Frequency of errors was presented per platform type.

Results

Ten participants were assigned to each device, and all completed home usability testing of the MIS. Demographic characteristics grouped by device assignment are shown in Table 1. Usability participant ages ranged from 47 to 76 years old. Participants were predominantly black, with a high proportion reporting a history of diabetes or cardiovascular disease. Although most reported earning more than a high school diploma, annual household income levels were predominantly ≤$50,000. The vast majority reported access to a computer and prior internet use, and although all participants owned a mobile telephone, few owned a smartphone. There was high variability in reported mobile telephone uses, with text messaging and camera use being the two most commonly reported, whereas few reported participating in social networking through their mobile device among all usability participants.

Table 1.

Usability participant demographics by device type

Characteristic	SMS (n=10)	PDA (n=10)	Total N (%)
Age, yr
≤65	7	6	13 (65)
>65	3	4	7 (35)
eGFR, ml/min per 1.73 m2
≥45	6	4	10 (50)
<45	4	6	10 (50)
Sex
Men	5	7	12 (60)
Women	5	3	8 (40)
Race
Black	8	7	15 (75)
Nonblack	2	3	5 (25)
Self-reported comorbidity
Diabetes	6	8	14 (70)
CVD	7	6	13 (65)
Cancer	1	2	3 (15)
Education
Less than or equal to high school diploma	2	4	6 (30)
More than high school diploma	8	6	14 (70)
Annual household income
≤$50,000	1	2	13 (65)
>$50,000	9	8	7 (35)
Computer usage
Access to computer
No	1	1	2 (10)
Yes	9	9	18 (90)
Ever used internet
No	1	4	5 (25)
Yes	9	6	15 (75)
Mobile telephone usage
Mobile telephone type
Cellular telephone	7	9	16 (80)
Smartphone	3	1	4 (20)
Used only for calls
No	4	5	8 (42)
Yes	6	5	11 (58)
Mobile telephone uses
Text messaging	6	4	10 (50)
Mobile web	3	2	5 (25)
Social networking	2	1	3 (15)
Apps	3	1	4 (20)
Email	4	1	5 (25)
Camera	6	4	10 (50)

CVD, cardiovascular disease; SMS, short messaging service; PDA, personal digital assistant.

Mean eHEALS score was 29 of a possible 40 (SD of 9.4). Although on the screening questions, the majority of participants reported that the internet was a useful source of health information and felt that it was important to access health resources on the internet, more detailed responses regarding individual comfort with using the internet for health information showed greater variability (Table 2). For example, 80% of participants agreed that they knew what internet health resources are available, but only 55% of participants agreed that they knew where to find helpful health resources on the internet. Whereas the majority of participants feel that they have the skills that they need to evaluate internet health resources, only a minority of participants agree that they can tell high- from low-quality health resources or that they feel confident using internet information to make health decisions.

Table 2.

eHealth Literacy Scale survey results

Survey Question and Available Responses	n	%
How useful do you feel the internet is in helping you in making decisions about your health?
Not useful at all/not useful	3	15
Unsure	2	10
Useful/very useful	15	75
How important is it for you to be able to access health resources on the internet?
Not useful at all/not useful	3	15
Unsure	0	0
Useful/very useful	17	85
Question 1. I know what health resources are available on the internet.
Strongly disagree/disagree	2	10
Undecided	2	10
Agree/strongly agree	15	80
Question 2. I know where to find helpful health resources on the internet.
Strongly disagree/disagree	4	20
Undecided	5	25
Agree/strongly agree	11	55
Question 3. I know how to find helpful health resources on the internet.
Strongly disagree/disagree	5	25
Undecided	1	5
Agree/strongly agree	14	70
Question 4. I know how to use the internet to answer my questions about health.
Strongly disagree/disagree	5	25
Undecided	2	10
Agree/strongly agree	13	65
Question 5. I know how to use the health information I find on the internet to help me.
Strongly disagree/disagree	4	20
Undecided	3	15
Agree/strongly agree	13	65
Question 6. I have the skills I need to evaluate the health resources that I find on the internet.
Strongly disagree/disagree	4	20
Undecided	2	10
Agree/strongly agree	14	70
Question 7. I can tell high–quality health resources from low–quality health resources on the internet.
Strongly disagree/disagree	4	20
Undecided	7	35
Agree/strongly agree	9	45
Question 8. I feel confident in using information from the internet to make health decisions.
Strongly disagree/disagree	3	15
Undecided	8	40
Agree/strongly agree	9	45

MIS error rates were low using either type of device. With both devices, the small number of errors was with those medications where there are no renal specifications to dosing. Of 60 medication queries inputted into the MIS platforms (30 for each device), there were only three recorded errors. Two of these errors occurred in the SMS group and were recorded by a single participant (age >65 years of age, grade 6 or less education, and no prior internet use) as a malfunction with device output; the error in the PDA group occurred as an inaccurate output response to the reportedly queried medication (participant was >65 years of age and a high school graduate with no prior internet use). Seven of 30 (23%) queries in the PDA group were searched by medication class rather than medication name. Participant satisfaction survey responses for each MIS platform were fairly similar, with slightly higher overall satisfaction in the PDA (41 of a possible 42; range =39–42) than the SMS group (39.5 of a possible 42; range =34–42). Satisfaction survey responses are further detailed in Figure 2. Testimonials after usability testing are shown in Table 3.

Figure 2.

Medication inquiry system satisfaction survey by device type. Percentage of participants in agreement with satisfaction survey questions. PDA, personal digital assistant; SMS, short messaging service.

Table 3.

Medication inquiry system usability participant testimonials

Device and Testimonials
SMS
“Very easy to understand the process and procedures.”
“I would have liked more explanation on how to work the system.”
“System helped me to understand medications that may or may not be good for my health.”
“Would have preferred to use a computer instead of the cell phone. Also, would have liked to see more content, such as vitamins, health foods, supplements, etc.”
“Did not like the cell phone. Letters too small for fingers and difficult to read.”
“As a senior, I felt very uncomfortable using the cell phone. I had to get help with the texting. I would only recommend this device if the cell phone had bigger letters.”
“I became more aware of medicines that might be harmful. Also, I am a low reader, but I found it easy to use the device.”
“Keyboard on the cell phone is too small. Overall process of looking up information was very easy.”
PDA
“No, I wouldn’t like to see anything changed. Study is very informative.”
“No changes needed. Really enjoyed using the iPod Touch. System was very educational.”
“I would like to see more categories, such as lab work, and more variety of medications.”
“I would like only to use the iPod Touch device. It was easy to handle.”
“I would like to have had a bigger key pad. Also an electronic pointer. Would like to see the study on a bigger scale.”
“No changes needed. The letters on the app need to be bigger.”
“Should be easier to get from where to find the app to the typing. Buttons were too small. Provide a class on application usage.”
“Would like to see more information regarding the medicines on the search system. For example, what are the side effects of each medicine.”

SMS, short messaging service; PDA, personal digital assistant.

Discussion

In this study, we set out to evaluate the performance of patients with CKD on a prototype mobile health application when users were tested in their home environment. Our findings show relatively high aptitude for usage of the mobile health system across both platforms. Perceived comfort with searching for and processing digital health information was highly variable in the study population.

Delivery of care for patients with kidney disease has changed from a provider-centered framework to a patient-centered framework, with emphasis on shared decision-making and patient empowerment (12–14). This shift in focus has resulted in a growing emphasis on patient engagement and the incorporation of digital solutions to support patient self-management practices (15,16). Individuals with CKD, often characterized by low disease awareness, limited health literacy, and frequent interactions with the health care system (17,18), are at high risk of adverse safety events related to their care, such as drug-related problems. This opens potential to introduce tailored digital tools to this population as patient-initiated safeguards (7,19–22). Mobile technologies, such as cellular telephones, provide instant access to information that previously was not promptly available to patients. Recent studies examining mobile health tools across various diseases have shown significant improvement in medication adherence, weight management, glycated hemoglobin levels, stress levels, smoking quit rates, and self-efficacy (23–29), although the long-term sustainability of such interventions remains less clear (30). The general or disease-specific health information, such as medication safety, on such devices may increase patients’ interest in self-care, raise their awareness regarding their personal medical conditions, and overcome barriers of the wide range in literacy in the target population.

Despite general proficiency with the mobile health MIS application, this study suggests that individuals with CKD have variable eHealth literacy, which is an important finding with direct relevance to the future development of health information technology applications designed to engage and safeguard individuals with CKD. It has been shown that more eHealth-literate individuals derive greater benefit from digital health information than those who are less eHealth literate, which creates new inequities in digital health education and contributes to the notion of the digital divide (31). The positive results of our remote usability testing suggest that individuals with CKD are capable of correctly using mobile health technologies if provided with accurate tutorials on proper device usage, even if testing is performed in the absence of a trained moderator. In a manner similar to traditional health literacy, tailored and simplified approaches to health education may be more effective in promoting understanding of complex health concepts than a one-size-fits-all approach (32,33), because prior studies support the belief that digital learning ability is not restricted to young healthy users (34). The MIS tool presented here has been developed with the capabilities of the target population in mind: simplified dialogue, universal imagery, and the ability to be used across various platforms. Such an application for general use in CKD would need to incorporate a significantly broader pharmacopeia of medications and require pilot testing in the target population. How use of such an application might be imbedded in the self-management practices of patients with kidney disease and how best to incorporate its relevance in the care provided by other members of a multidisciplinary CKD team (such as pharmacists) remain uncertain. The challenges for future projects are to evaluate whether patients will actually adhere to the technology and whether medication safety can be improved for patients with CKD by such a system.

The home usability testing described has limitations related to the descriptive and qualitative nature of the study. The small sample size limits hypothesis testing but is consistent with the number of participants recommended for usability testing (35,36). Furthermore, the MIS testing described here was specific in its intent to assess usability rather than validity, which may limit the generalizability of the findings to other health information technology applications in kidney disease, particularly given the limited number of medications available within the prototype system. It is unclear if familiarity of the inputted medications by study participants affected the recorded MIS responses given the lack of errors noted in the renal-pertinent medication categories, and it requires additional study. The MIS provides information on the basis of published safety guidelines of medications in CKD but does not have the functionality to incorporate tailored provider judgement into the programmed recommendations, which may differ from guidelines on the basis of individual risk assessment (for example, the use of aspirin in individuals with coronary artery disease). Finally, although mobile telephone and PDA usage is considered commonplace, we anticipate that some individuals will be unfamiliar with the usage of these devices or have limited cellular service availability, which may limit acceptance of the MIS tool to the target population. Additional work needs to go into understanding the human-device interactions that are unique to this patient population and refining the platform with which we expect that patients will use to communicate with their health providers and access relevant health information.

Novel strategies are needed to promote patient engagement and self-directed patient safety among the high-risk population with CKD. Mobile devices may provide a ubiquitous and powerful conduit through which health information can be easily accessed. Additional studies are needed to assess the potential benefit of such mobile health strategies in improving patient safety outcomes in CKD.

Disclosures

J.C.F. has received prior research funding from Amgen, Inc. and honoraria from Sandoz, Inc. and Amgen, Inc.