A Multi-Level Usability Evaluation of Mobile Health Applications: A Case Study

Abstract

Objective:

To report a methodological approach for the development of a usable mHealth application (app).

Materials and Methods:

This work was guided by a 3-level stratified view of health information technology (IT) usability evaluation framework. We first describe a number of methodologies for operationalizing each level of the framework. Following the description of each methodology, we present a case study which illustrates the use of our preferred methodologies for the development of a mHealth app. At level 1 (user-task), we applied a card sorting technique to guide the information architecture of a mobile HIV symptom self-management app, entitled mVIP. At level 2 (user-task-system), we conducted a usability evaluation of the mVIP in a laboratory setting through end-user usability testing and heuristic evaluation with informatics experts. At level 3 (user-task-system-environment), usability of the mVIP was evaluated in a real-world setting following the use of the app during a 3-month trial.

Results:

The 3-level usability evaluation guided our work exploring in-depth interactions between the user, task, system, and environment. Integral to the findings from the 3-level usability evaluation, we iteratively refined the app’s content, functionality, and interface to meet the needs of our intended end-users.

Discussion and Conclusion:

The stratified view of the health IT usability evaluation framework is a useful methodological approach for the design, development, and evaluation of mHealth apps. The methodological recommendations for using the theoretical framework can inform future usability studies of mHealth apps.

1. INTRODUCTION

Approximately two-thirds (66%) of Americans use mobile applications (apps) to manage their health,[1] and nearly 165,000 mobile health (mHealth) apps are now available in the Apple iTunes and Android app stores in the US.[2] More than a few mHealth apps are designed/developed with minimal end-user feedback, and continue to proliferate with little evidence supporting user engagement of the app.[3,4] For example, only 4% of mHealth apps providing breast-feeding support had any evidence ensuring their use.[3] Apps are frequently produced with poor design and inadequate consideration of the needs of end-users.[5] Apps of poor quality that are disseminated can be difficult to use, or misused or underutilized, and will ultimately fail to accomplish their goals.[4,6–9] Thus, it is essential for apps to provide the required functionality and also to ensure quality.[10,11] This highlights the importance of usability evaluation of mHealth apps throughout the development process – before and after prototyping takes place.[12–14] More than 95% of mHealth apps have not been tested,[15] and existing studies have evaluated apps’ usability at a certain single stage (mainly at an early stage) of development, and/or without using a solid theoretical framework.[16–18] A usability assessment which focuses on a single measure (e.g., time efficiency or user acceptance) at a single stage (e.g., prototype) cannot capture the complete usability of the system. The purpose of this paper is to describe a theoretically driven methodological approach for the development of a mHealth app. The multi-level usability evaluation mapped each methodology to a specific level in the conceptual framework. Following our description of the framework and the relevant methodologies, we present a case study to illustrate the operationalization of the framework for the development of a mobile app.

2. METHODS / RESULTS

2–1. Theoretical Framework: A Stratified View of Health Information Technology (IT) Usability Evaluation

The Stratified View of Health Information Technology (IT) Usability Evaluation Framework [19] provides a categorization of study approaches of usability assessments by three levels: user-task, user-task-system, and user-task-system-environment (Figure 1–1).

Figure 1–1.

A stratified view of health information technology (IT) usability evaluation

2–2. Methods for Operationalizing Framework

In the evaluation framework, level 1 targets system specification to understand user-task interaction for system development. Level 2 examines the task performance to assess system validation and human-computer interaction in a laboratory setting. Level 3 aims to incorporate environmental factors to identify system impact in a real-world setting. It is of great importance that mHealth app developers be not only aware of various usability methods, but also able to determine which method is best suited to each level of the development of mobile apps. Table 1 presents potential methods for usability evaluations and the selected methods that are described in a case study.

Table 1.

Usability evaluation methods by three levels

	Potential Methods	Selected Methods
Level 1	• Interview   • Focus group/expert panel meeting   • Questionnaire   • Use-case analysis/modeling   • Design task identification   • Card sorting technique	• Card sorting technique
Level 2	➢ End-user-based:   • Think-aloud protocols   + other techniques/technologies providing objective cues   (e.g., video recording, mouse-clicks, facial expression coding, galvanic skin response, electroencephalography, etc.)   • Field observation   • Time-and-motion study   • Task analysis   • Cognitive Task Analysis   • Interview   • Focus group   • Questionnaire ➢ Expert-based:   • heuristic evaluation   • Cognitive Walkthrough	➢ End-user usability testing:   • Think-aloud protocols + eye-tracking ➢ Heuristic evaluation
Level 3	• Interview   • Focus group   • Questionnaire   • Survey	• Interview   • Survey

Study context:

This project designed, developed, and evaluated a mHealth app to support symptom self-management for persons living with HIV (PLWH). It was conducted as part of an Agency for Healthcare Research and Quality initiative to incorporate the use of patient-centered outcomes research (PCOR) onto a mobile platform.[17] We developed mVIP (mobile Video Information Provider), by incorporating evidence-based self-care strategies which had been previously disseminated through a paper-based manual [18, 19] into a mobile tool.

Level 1 (User-Task):

To develop an effective mHealth app, it is necessary to start by incorporating users’ requirements into the app’s design. The early feedback of potential users can improve the quality of the system.[20] In our case study, a card sorting technique [21] was selected for level 1 usability assessment since we had existing health information (i.e., HIV-related symptoms and self-management strategies from a paper-based HIV/AIDS symptom management manual).[22] Given that the card sorting technique is classified as ‘open’, ‘closed’, and ‘reverse’ card sorting, either online or in person, we chose a reverse card sorting since it is well-suited for testing an existing structure of categories (i.e., HIV-related symptoms) and sub-categories (i.e., self-care strategies).[21,23] Physical index cards were used in person for the reverse card sort since with an online card sorting study, researchers may miss out on the insights and additional comments end-users can provide in person.[24]. A card sorting technique is one of the most effective methods for acquiring categorical and hierarchical data about existing domains.[25]

Level 2 (User-Task-System):

Usability evaluations in a laboratory setting are foundational to the success of achieving systems that meet human-computer interaction principles. In our case study, two usability evaluations: think-aloud protocols by end-users and heuristic evaluation by experts,[26,27] were chosen since they are two methods most frequently used to guide system modification.[28] Moreover, eye-tracking data were integrated and synthesized in the think-aloud protocols [29] since the use of eye-tracking has the potential to improve usability of health IT by providing complementary objective data (i.e., responses provided by end-user during the think-aloud protocols).[30]

Level 3 (User-Task-System-Environment):

System usability is closely linked to the interaction of users performing tasks in the system within a specified environment. Change in any of the components of user, task, system, and environment may change the entire interaction and influence the usability of the system.[31] In our case study, an interview with a survey [32] was selected since an interview is well-suited for exploring and gaining an in-depth understanding of end-users’ experiences, opinions, expectations, wishes, and concerns, especially after they have used the technologies in a real-world setting.

2–3. Case Study: A Multi-Level Usability Evaluation

To illustrate the operationalization of this framework, we present a case study employing a 3-level usability evaluation for the development of the mHealth app. Figure 1–2 provides an outline of the 3-level usability evaluation of mVIP.

Figure 1–2.

An outline of studies: 3-level usability evaluation

2–3-1. Level 1 (User-Task): User-Centered Design

Sample:

We recruited 20 PLWH from an academic medical center and 4 community-based organizations in New York City between December 2015 and May 2016. Inclusion criteria were: adults (>18 years of age) who were diagnosed with HIV; English-speaking; having at least 2 of 13 HIV-related symptoms in the past week; and a cognitive state with acceptable responses for 6 items on a shortened version of the Mini-Mental State Examination (MMSE).[33]

Procedures:

A reverse in-person card sorting exercise [21] guided the information architecture of mVIP. Users were presented with a pile of cards; each card contained a symptom and self-management strategy for ameliorating the symptom. There were a total of 154 self-management strategies for 13 symptoms derived from the paper-based HIV/AIDS symptom management manual.[22] Participants were first ‘provided 13 index cards of symptoms and asked to select the index cards of symptoms they experienced in the past week. Participants were then provided with index cards of strategies for each symptom they chose. The participants were asked to place the index cards of strategies in order of individual priority applicable to the selected symptom. They were allowed to place cards on an ‘irrelevant/unhelpful’ pile for strategies they thought were not relevant to the symptom or were unwilling to try. Finally, participants were asked to add comments on a blank index card if appropriate. At the end, all index cards were photographed (Figure 2).

Figure 2.

Sample picture of card sorting activities

Data Analysis:

A hierarchy analysis established the rank order of symptoms and strategies. Mean scores of the ordinal numbers of the cards for strategies were calculated for each of the symptoms. A lower mean score indicated a higher priority order of strategies for each symptom.

Results:

Rank order of the symptoms and self-management strategies was established and reported elsewhere.[34] 85% (N=17) of participants reported fatigue, and 60% (N=12) difficulty sleeping in the prior 7 days. 3 self-management strategies were excluded in response to participants identifying as ‘irrelevant/unhelpful’. Findings were incorporated into the information architecture of mVIP. The rank order of the 13 symptoms and 151 self-management strategies determined the order of appearance to end-users of the mVIP app, with higher-ranked symptoms and strategies appearing first (Figure 3).

Figure 3.

mVIP Alpha version

2–3-2. Level 2 (User-Task-System): Usability Evaluation in a Laboratory Setting

mVIP was designed and implemented by software developers at Northwestern University based on findings from the card sorting study. We conducted two types of usability evaluations of the mVIP alpha version. The first end-user usability testing examined task performance by PLWH, and the second heuristic evaluation assessed the user interface by usability experts.

2–3-2–1. End-User Usability Testing

We conducted end-user usability testing with an eye-tracking and retrospective think-aloud method.[29]

Sample:

We recruited 20 PLWH (10 Android and 10 iPhone users) from June to July 2016. Participants met the inclusion criteria for the level 1 study and also self-identified as a heavy smartphone user, defined as a smartphone user for more than 1 year who also used mobile apps more than 3 hours/day on average.[35] Heavy smartphone use was necessary to ensure that usability issues identified while using mVIP were not related to lack of technology skills, but from actual shortcomings related to its usability. Participants who wore bifocal/progressive glasses were excluded since these types of glasses affect the precision of the gaze estimation while collecting participants’ eye-tracking data.[36]

Procedures:

End-user usability testing was comprised of the following 2 processes: eye-tracking while using the app, and a retrospective think-aloud protocol.[29] First, participants were provided with a use case scenario designed to determine usability of mVIP and asked to complete 2 app sessions using mVIP (Table 2). While performing the tasks, participant’s eye movements and smartphone screen were video-recorded using Tobii X2–60 with a mobile device stand with embedded camera and microphone (Figure 4).[36] After completing the tasks, participants were asked to watch a recording of their task performance that depicted their eye movements overlaid on the app screen. They were encouraged to retrospectively think-aloud and asked to verbalize their thoughts about the tasks they completed while watching the recording. All verbalizations were audio-recorded.

Table 2.

Task included in a use case scenario

Session_1		Session_2
Task 1	Log-in	Task 1	Log-in
Task 2	Update a password	Task 2	N/A
Task 3	Start a session – Get strategies for the first two symptoms, fatigue and difficulty sleeping	Task 3	Start a session – Get strategies for one symptom, difficulty sleeping
Task 4	Review the recommended strategies	Task 4	Review the recommended strategies

Figure 4.

Sample picture of eye-tracking app testing (taken of a research team member with permission)

Data Analysis:

Data analysis was based on the Tobii audio/video-recordings. Gaze plots were created from screen-recordings synchronized with eye movements. Participants’ vocalizations from the audio-recordings were transcribed verbatim. Notes of critical incidents, characterized by comments, silence, repetitive actions, and error messages were compiled from the recordings. Task performance time analysis: The mean performance time of each task was calculated and compared among participants with/without trouble using a two-sample t-test (α=0.05). Participants with trouble in this study were defined as those who received error messages during the app testing and self-reported difficulties during the think-aloud protocol. Eye movement analysis: Gaze plots depicting participants’ eye movements were reviewed in conjunction with notes of critical incidents. The gaze plots were compared among participants with/without trouble. Content analysis: Free text was excerpted from the transcripts and coded based on 9 concepts of the Health IT Usability Evaluation Model (Health-ITUEM),[31] and the 9 codes were broken into positive, negative, and neutral codes.

Results:

In task 1 at the first session, 50% of the participants (N=10) had trouble and mean time for completion of the task was 241 seconds, whereas the mean time for those without trouble was 49.5 seconds (p=0.002). The mean time to perform each task during the second session was much lower than during the first session, suggesting that the app is highly learnable.

Based on eye-tracking data of a larger red circle or longer red lines for participants with trouble, compared to those shown while a participant was performing a task without trouble (Figure 5–1), we identified several participants who encountered challenges navigating the app. These challenges were specific to ambiguity regarding the ‘Continue’ and ‘Log-in’ button, which showed as an extremely long eye fixation and distractive eye scan path up and down the smartphone screen (Figure 5–2).

Figure 5–1.

Sample gaze plot for a participant without trouble

Figure 5–2.

Sample gaze plots for a participant with trouble

Sample excerpts for each code of the 9 Health-ITUEM concepts are presented in Table 3. Based on end-users’ recommendations and integration of the results, the content, functionality, and interface of mVIP were refined. For example, the error message ‘Please check your credentials’ was changed to ‘The email address or password you entered is not valid’ since the term ‘credentials’ was unfamiliar to several participants. We added instructions for ‘how our app works’ on the home page. Several participants suggested changing the main logo of mVIP to look more informative and professional (Figure 6).

Table 3.

Health-ITUEM concepts, codes, and representative quotes

Concepts	Description and Representative Quotes
Error prevention	mVIP offers error management, such as error messages as feedback, error correction through undo function, or error prevention, such as instructions or reminders, to assist users in performing tasks.
+ Error prevention	“It was very nice that it told me that I made a mistake instead of going right into it even if I did make a mistake. That alerted me that I put in the wrong email. The error message was pretty clear.”
– Error prevention	“I didn’t understand what ‘please check your credentials’ meant. I didn’t understand that at all. What exactly did it mean?” (unclear error messages)
	“It tells me if the password failed, and it tells me it was updated, so that’s wrong. Because it’s giving me two different messages. That’s pretty confusing.” (contradictory information in error messages)
	“It might be beneficial to have a back feature, because you know, you never know and you might be just rushing through, or something like that.” (back button)
	“Give it a back button on certain pages. Maybe give it a menu screen. maybe like three lines right here that you can hit the menu and maybe it will bring up dashboard” (home menu)
	“Well, for those who don’t know how to use apps, they would need more instructions. They would need more instructions, more simple instructions for them to adapt to. For them not to frustrated, but some way of making it fun, to where they will enjoy using the app, or enjoy getting into the app. You know, instead of saying, Okay, alright, I’ve been in this app. What am I supposed to do?” (instruction for how the app works)
Completeness	mVIP is able to assist users to successfully complete tasks.
+ Completeness	Task success rate: 80% (N = 16)
– Completeness	Task failure rate: 20% (N = 4)
Memorability	Users can remember easily how to perform tasks in mVIP.
+ Memorability	“I remember the steps. I logged in and created a new password after I logged in. And then, I had started a session, which asked me about various symptoms that I may have experienced in the past week.”
– Memorability	None
Information needs	The information content offered by the mVIP for basic task performance, or to improve task performance.
+ Information needs	“You can use the app to gain a lot of information. If you’re not feeling too well and you have certain symptoms that match with the ones that are on this app, then it would give you helpful information right then and there.”
– Information needs	“The one thing I had to question is, these 13 strategies are related to what particular? They don’t tell you what category. Is it fatigue or…what?”
Flexibility/Customizability	mVIP provides more than one way to accomplish tasks, which allows users to operate system as preferred.
+Flexibility/Customizability	“There are different kinds of expression on the avatar’s face. So I also look at that as well. It is meaningful. The reason being because if you’ve a man and thinking of yourself, it’s good to remember okay, this is me, so they’re talking about me.”
–Flexibility/Customizability	“I would like to save it there…at least the ID, then the password. I can have my own password, but at least the ID can be saved there so I don’t have to be typing the ID and then the password all the time.” (option of saving ID&PW)
	“I don’t see any here that represent me, but you at least have a choice. I don’t think any of them. You should be able to create your own avatar, make it look like you. Because I have an app and I made it look like me.” (option of choosing an avatar)
	“If I can review this information before I fax it off or send it off to someone else, email it to someone else.” (option of Fax/email)
Learnability	Users are able to easily learn how to operate mVIP.
+ Learnability	“It was very easy to follow after the first use. I found the questions easy. They were precise and they were straight to the point. Once I answered the question, the suggestions they gave were easy. They were easy to follow, so I think I can self-manage myself quite well with this app.”
– Learnability	“I didn’t see the continue button. I thought this system was automatic, when I checked ‘yes,’ I thought it would move on.” (continue button)
	“Well, the way I saw it was that it was checked already, so I guessed it continued, leave it as is because there is no change. Unless they were both blank, then there is ‘no’ and ‘yes,’ and then I’ll check. But since it was checked for ‘yes,’ I left it at that. So then I hit the Continue. But I didn’t think to hit it because it was already checked. If it had been blank, then I would have checked it. But I didn’t see it unchecked to put a check in. I left it as it was and hit Continue.” (checkmark)
Performance speed	Users are able use mVIP efficiently.
+ Performance speed	“It was very short so it was real quick. I know I had to go with ‘yes’ for most of them and ‘no’ with everything else.”
– Performance speed	None
Competency	Users are confident in their ability to perform tasks using mVIP, based on social cognitive theory.
+ Competency	“I would feel very confident, very, very confident, because I mean it gives you pretty much straightforward strategies to try. Like I said, trial and error, so whichever ones do work, then do, if none of them work, then it’s time to see the doctor. But chances are if everything else is going the way it’s supposed to go, as far as you taking care of yourself, then you know, your symptoms should be able to be relieved with these strategies.”
– Competency	“And I think I was done. I wasn’t sure if there was anything else to be done that didn’t get done. So that’s why I didn’t understand that question. Was it complete, or is there something else I have to follow through with? But I guessed I was done. So I was confused as to whether there was something else I needed to do.”
Other outcomes	Other mVIP-specific expected outcomes representing higher level of expectations (uses of non-phone app technology (i.e., phone, books), non-mobile resources (i.e., parents, friends, siblings), other health related entities not directly related to the usability of mHealth (outside of study protocol))
+ Other outcomes	“I mean, not just mine, but I mean everybody that the virus, it could change their quality of life, giving them a better quality of life, you know, it’s kind of like being your own doctor these days, without having to go to the doctor. And, being able to take better care of yourself through the app.”
– Other outcomes	“I don’t think the app can very much change my life.”

Figure 6.

Change of mVIP main logo

2–3-2–2. Heuristic Evaluation

Sample:

5 informatics experts participated in the heuristic evaluation of the web version of mVIP. Inclusion criterion were: a minimum of a master degree in the field of informatics and training in human computer interaction.[37]

Procedures:

Experts were provided with the same use case scenario employed during the end-user usability testing. Each expert was encouraged to explore the user interface of mVIP at least twice, and to think-aloud while they performed the evaluation. The process was recorded using Morae™ software.[38] Following completion of the tasks, evaluators rated the severity of the violations using a paper-based heuristic evaluation form based on Nielsen’s heuristics.[39] Usability problems were rated into 5 categories: no problem (0), cosmetic problem only (1), minor problem (2), major problem (3), and usability catastrophe (4).

Data Analysis:

Content analysis of the transcripts of the experts’ vocalizations during the think-aloud was organized by the usability factors of the 10 heuristics. The mean severity scores of the identified heuristic violations were calculated for each principle.

Results:

Mean scores and sample comments were organized into Nielsen’s 10 usability heuristics.[26,27] Findings are reported elsewhere.[40] The content, functionality, and interface of mVIP were refined based on the evaluators’ recommendations. For example, the term ‘Dashboard’ was changed to ‘VIP Home,’ which was a more familiar term to our end-users. To make the app’s functionality more generalizable, we added a response option of ‘Didn’t try’ in addition to the ‘Yes/No’ options when assessing helpfulness of previously suggested strategies.

The 143 videos created using the GoAnimate™ software [41] to present each of the 143 self-management strategies, were inserted into the refined mVIP with the existing text strategies (i.e., 63 strategies were reworded and 8 were removed). While our mVIP was initially designed as a native app for mobile devices, the refined mVIP (i.e., mVIP Beta version) was developed as a mobile web-app, due to different capabilities between Android and iOS platforms (Figure 7).

Figure 7.

mVIP Beta version

2–3-3. Level 3 (User-Task-System-Environment): Usability Evaluation in a Real-World Setting

After mVIP was refined based on the second level usability evaluation, we implemented a 3-month RCT to test the feasibility of using mVIP for improving symptoms in PLWH. The intervention group was provided with self-management strategies for self-reported symptoms through the app, while the control group self-reported symptoms but was not provided with any strategies. A survey and interview were conducted to evaluate the usability of the app by identifying user-task-system-environment interaction.

Sample:

Eligibility criteria were the same as that in the level 1 study. 80 PLWH who own a smartphone/tablet were randomized (i.e., 40 in the intervention and 40 in the control) and 76 completed the 3-month RCT between December 2016 and June 2017. Of the 76 PLWH, 10 PLWH were recruited for interviews.

Procedures:

All participants in the RCT were asked to complete the Health IT Usability Evaluation Scale (Health-ITUES)[42] survey administered via Qualtrics^® [43] at baseline and 3-month follow-up. At the end of the RCT, a research coordinator facilitated 10 one-on-one interviews using a semi-structured interview guide designed based on the Health-ITUEM.[31] Participants were encouraged to talk about their experiences, perceptions, and satisfaction of their app use. The interviews were audio-recorded. Data collection continued until saturation of themes was reached.

Data Analysis:

The Health-ITUES consists of 20 items rated on a 5-point Likert scale from strongly disagree (1) to strongly agree (5).[42] The overall Health-ITUES scores between baseline and follow-up visit were analyzed using a linear mixed model controlled for age, sex, race, education, and CD4 count (α=0.05). We used thematic analysis to explore patterns and themes that emerged across interviews through NVivo™ software.[44]

Results:

Overall, participants in both intervention and control groups rated the usability of mVIP app as being high. There was no significant difference in the overall Health-ITUES scores across the groups at baseline and follow-up. The mean scores of the overall Health-ITUES were more than 4.19 (SD = .50) of 5.00 (i.e., where the higher the response, the higher the subject’s usability satisfaction with the app), indicating a high user satisfaction of the mVIP app.

A total of 15 themes were identified from interviews. Of the subjective constructs of Health-ITUEM, 9 themes identified in the intervention group related to Perceived usefulness, and 6 themes identified in the control group related to Perceived ease of use (Table 4). Findings from the interviews showed that first, mVIP is useful for HIV-related symptom self-management and has the potential for being used as a communication tool with healthcare providers; and second, mVIP is easy to use to monitor symptom experience over time. At the same time, participants suggested mVIP be more sensitively tailored based on years from initial diagnosis of HIV, an individuals’ age, and conditions.

Table 4.

Themes and quotes of content analysis from the interviews

Intervention Group: Usefulness of mVIP and additional user expectations
Theme I-1. Usefulness for information needs – symptom problem solving
“It was very helpful (for reducing my symptoms), for different situations. It (app) gave me a suggestion and I tried it and got better.”
Theme I-2. Usefulness for interaction needs with healthcare providers
“Sometimes you go to the doctor and forget the symptoms you have been going through. If you bring this app to the doctor and go to the review part (in the app), this app could show the stuff, everything, you’ve been going through.”
Theme I-3. Usefulness of mobile app format as a perceived facilitator
“It’s very convenient because I can use it almost anywhere. While I’m in public transportation, on the buses, at the clinic, at home…everywhere.”
Theme I-4. Additional preference of strategy design (text & video with sounds)
“What the video is saying can give you a little more insight on how to do things and how to go about them and whatever, but if you don’t have the audio it’s like, ‘Just let me read this, click and answer and just go on to the next one.’ So, the audio would help.”
Theme I-5. Additional preference of available language
“Because my community, the Latino community in NYC is very big and increasing in HIV. The Spanish community… It’s very important the Latino community can comprehend the app. You can get two options, in English or in Spanish.”
Theme I-6. Intrinsic motivation of the frequency of app use (for enjoyment)
“I use it (app) 3 times a day sometimes. Just playing around. Just to play a little game. I’m playing a game and I’m tired of the game, I just start to do the app instead. I don’t have anything else to do. I don’t think more (app use) would be helpful. Once a week is right on point.”
Theme I-7. More symptoms and strategies needs according to years of diagnosis
“I think it (strategies) will help people that are newly diagnosed that they are dealing with something new that they’re not familiar with, something that they didn’t expect to have and, as a newly diagnosed person, they go through a lot of confusion, a lot of questions in their mind. We need something geared towards where we are right now because we have much more issues than the app is talking about for us.”
Theme I-8. More individually-sensitive self-tailoring symptom management
“Everyone is different when it comes to their health. It (strategies) was less personal. We may not have the same status. We all should have our own things that we’re dealing with in (personal) life…”
Theme I-9. More communication needs with social groups
“I think you should create a network where we can network among each other within the app. (So) you can respond to someone and you can say, ‘I’m feeling the exactly same way today (like you).’ Or, if someone is not feeling well you can send a message back like, ‘(tell me). Let me see how you’re feeling. We can share the feelings (because we are all HIV+).’”
Control Group:
Ease of mVIP use to track symptoms but also acknowledged its deficits
Theme C-1. Easy app as a regular tool to facilitate self-awareness of symptoms
“I like to listen to my body and it made me more aware of what’s going on with me. I like the app because it’s simple and easy to use and there were something that were listed in the app that I had no idea that were related to my HIV. So, it caused me to listen more closely to what’s going on with me.”
Theme C-2. Lack of action planning of symptom self-management
“It was easy (to use the app) but I thought there could be another portion that would deal with stress (symptoms).”
Theme C-3. Lack of symptom summary to share with healthcare providers
“In regards to answering yes or no, and at the end. A short summary of what our symptoms were… when you see your doctor it just totals the graphs (charts) down and then he can see what’s going on me (symptoms)… We can build that kind of provider relationship (using symptom report summary).”
Theme C-4. Tedious experience of the repeated questions
“The app, it kept repeating itself over and over again. It was like the same thing (questions) over and over, so it got kind of boring for me.”
Theme C-5. Extrinsic motivation of the frequency of app use (for rewards)
“Once a week is good. (But I used the app) At least twice a week because I want to build up my chances for being accepted for the research study next time. For the research study…”
Theme C-6. Appraisal of ease of use vs. security of the password
“9 out of 10 times, I forgot the password. The app had a little button that says: remember me. Therefore, I didn’t have to remember my password. It was very easy to use.”
“For me, I never do that. Like, speaking to the gentleman who had his phone stolen – I have the experience. If you have that ‘remember me’ and somebody accesses your phone…let’s say you have it on your bank account (because we usually use the same passwords). They can immediately see what your bank account level is, they have access to your HIV.”

3. DISCUSSION

In order to ensure usability throughout the system development process, we employed a 3-level usability evaluation of a mHealth app (mVIP), exploring potential interactions between the user, task, system, and environment that was guided by the Stratified View of Health IT Usability Evaluation Framework.[19] Methods included a card sorting technique, eye-tracking and retrospective think-aloud, a heuristic evaluation, a survey and in-depth interview. Integral to the findings from the 3-level usability evaluation, we iteratively refined the app’s content, functionality, and interface, and we found the app was rated as highly usable by our end-users. Findings from this study are unique in that the mVIP app was developed by delivering evidence-based health information through a mobile health platform and assessed its usability at every level throughout the development process.

Suboptimal usability is a major obstacle to technology adoption and poor usability is a primary reason for discontinuing the use of mHealth technology [45]; therefore, usability factors must be considered before and after prototyping takes place to support the quality of the technology and the end-user experience.[12,13] Despite its importance, most mHealth apps are released to the public without sufficient scientific effort devoted to their design, development, and evaluation.[3,46] At the same time, there has been no clear methodological approach to using a theoretical framework in apps’ usability studies.[16,17,19] To overcome this challenge, we used several usability evaluation methodologies at each stage of the development process to operationalize the Stratified View of Health IT Usability Evaluation Framework. It is critical to select the most appropriate evaluation techniques that best meet the study aims at every level of the system development process and ultimately achieve the goals of the system. While the methods that we selected are just a few of many possible methods, this mapping can serve as a methodological guide for the multi-level usability evaluation for development of a user-facing mHealth technology.

Card sorting technique:

This technique is well-suited for incorporating existing health information into a mobile app, particularly for the evidence-based information design phase since it generates an overall information structure and suggestions for navigation, menus, and possible taxonomies of the system.[47] In our case the technique helped us identify the information priorities of our intended end-users and establish the app’s components before prototyping. Despite these benefits, there have been no usability studies using card sorting activities for mHealth app’s development. Good information architecture is key to developing, easy to use, and intuitive technology.[48–50] Application of the card sorting technique to the app design process is user-centered and innovative. We recommend that it be integrated into the mHealth app development process.

End-user usability testing and heuristic evaluation:

In our case study, two usability evaluations were conducted in a laboratory setting to capture different usability perspectives from end-users and experts. First, end-user usability testing showed significant differences in task performance duration between participants who experienced difficulties and those that did not. These usability problems were identified by using both eye-tracking and think-aloud protocols. When a participant had trouble with a task, (e.g., finding a certain button) long eye fixation or distractive eye movements were found upon replay of the screen-recordings. We learned the reason for their unusual eye movements using the retrospective think-aloud protocol. This combination provides valuable information that surpasses usability problems reported without a cue, which are often biased.[51,52] A stand-alone think-aloud protocol may fail to identify additional objective cues that provide insight into participants’ expectations about where information should be located and their level of confidence about information found.[53] The usefulness we have documented leads us to recommend incorporating these methods in combination for future mHealth app development. Second, while our heuristic evaluators identified similar usability issues to those identified by our end-users, they were more likely to focus on ‘making things work’ in a natural and logical order.[28,54] For example, the experts identified the usability factor ‘match between the system and the real world’ regarding the helpfulness assessment question for each of the strategies suggested in the previous session. They suggested that an additional response option for end-users who did not try that particular strategy be included. They suggested that no strategies be offered for symptoms the user marked ‘not bothersome’. To identify usability problems of different sorts and scope, we recommend combining usability evaluations from both end-user and expert perspectives as an effective and thorough approach for evaluating mHealth apps prior to real-world deployment.

Survey and interview in a real-world setting:

Numerous studies have evaluated usability only in a laboratory setting.[55] Conducting usability testing in a laboratory setting presents threats to external validity specifically overlooking usability issues related to the interaction between user, task, system, and environment. In our work, inclusion of interviews following testing of the app during participants’ everyday lives allowed us to measure users’ actual experience with the mVIP app in daily life. The overall usability scores as measured by the Health-ITUES were high at baseline, and the app was perceived as highly usable over time. Eventually, the app was efficacious at improving outcomes in a 3-month feasibility trial.[56] These results are an important strength of our study.

It is vital to take into consideration specific aspects of a mobile device when selecting evaluation methods during app development process. Identification of the usability evaluation methods are unique for mobile devices because they have small screens. For example, a particular usability problem related to the screen size identified by eye movements collected during the participant’s app testing process and verbalizations collected during the retrospective think-aloud protocol was the placement of the ‘Continue’ button. Since the button was placed under the response options because of a small mobile screen, participants were required to scroll down to find the button. The usability problem was identified as a result of using the eye-tracking method with a mobile device stand, which captured the smartphone screen and the user interactions. To resolve the usability issue related to the ‘Continue’ button being hidden, we transitioned the mVIP app from a native app to a web-app optimized for mobile devices.

Concerns specific to the privacy and confidentiality of transmission and storage of health information via a mobile device emerged during our evaluation. Several participants expressed preferences for a function of saving their password if it is stored, whereas a few participants raised privacy concerns about the security of the password. This was specific to our study population where stigma associated with an HIV diagnosis persists.[57] Findings from this study demonstrate the value of rigorous usability evaluations with inclusion of end-users at every stage and support our recommendations for this approach.

Limitations

We faced several challenges when collecting eye-tracking data. To avoid losing participants’ eye movements, we dimmed the lights and asked the participants to maintain the same position. These specifications may be tiring to participants, and time-consuming for researchers. Since there has been no rigorous standard of measuring what is considered a good eye pattern, it was hard to set standards for a given interface.[58] Despite these limitations, eye-tracking proved to be a valuable tool to explore usability issues in conjunction with a think-aloud protocol as opposed to a stand-alone method.

4. CONCLUSIONS

In this paper, we presented methodologies for operationalizing the Stratified View of Health IT Usability Evaluation Framework [19] and a case study to illustrate the use of these methods for the development of a mHealth app, specifically focusing on HIV symptom self-management. The operationalization of the Stratified View of Health IT Usability Evaluation Framework [19] in the context of mHealth technology can provide a methodological approach for future mHealth apps’ development. This scientific and systematic approach to usability evaluations may encourage other researchers to use an evidence-based evaluation framework when planning and designing their usability studies for the development of mHealth apps and to choose the best evaluation methods applicable to the goal of the technology.