A user-centered model for designing consumer mobile health application (apps)

Abstract

Background

Mobile technologies are a useful platform for the delivery of health behavior interventions. Yet little work has been done to create a rigorous and standardized process for the design of mobile health (mHealth) apps. This project sought to explore the use of the Information Systems Research (ISR) framework as guide for the design of mHealth apps.

Methods

Our work was guided by the ISR framework which is comprised of 3 cycles: Relevance, Rigor and Design. In the Relevance cycle, we conducted 5 focus groups with 33 targeted end-users. In the Rigor cycle, we performed a review to identify technology-based interventions for meeting the health prevention needs of our target population. In the ISR Design Cycle, we employed usability evaluation methods to iteratively develop and refine mock-ups for a mHealth app.

Results

Through an iterative process, we identified barriers and facilitators to the use of mHealth technology for HIV prevention for high-risk MSM, developed ‘use cases’ and identified relevant functional content and features for inclusion in a design document to guide future app development. Findings from our work support the use of the ISR framework as a guide for designing future mHealth apps.

Discussion

Results from this work provide detailed descriptions of the user-centered design and system development and have heuristic value for those venturing into the area of technology-based intervention work. Findings from this study support the use of the ISR framework as a guide for future mobile health (mHealth) app development.

Conclusion

Use of the ISR framework is a potentially useful approach for the design of a mobile app that incorporates end-users’ design preferences.

Background and Significance

Mobile health technology (mHealth) is a promising tool for engaging patients in their own health care because most people own and regularly use a mobile phone and mHealth can be an appropriate medium for delivering health information [1]. In specific, mHealth can be used as a powerful health behavior change tool for health prevention and self-management as they are ubiquitous, carried on the person and are capable of advanced computational capacity [2 3]. Given the great potential of mHealth, it is not surprising that current estimates suggest that there are more than 40,000 mHealth applications (apps) [4].

As these mHealth technologies proliferate, the design of effective tools is becoming increasingly important [5]. mHealth apps continue to proliferate with little evidence for their effectiveness and little support for understanding how best to design the apps [6]. Many current mHealth interventions are designed on the basis of existing healthcare system constructs and may not be as effective as those that involve end-users in the design process [7]. Apps need to be produced with adequate consideration of the needs of their intended users so that they are easy to use and perceived as useful [8]. In a healthcare system already burdened with suboptimal outcomes and excessive costs, premature adoption of untested mHealth technologies may limit positive health behavior change. mHealth tools may then ultimately fail to accomplish their objectives and may be ineffective or, at worst, yield adverse outcomes [9].

Our study focused on designing a mHealth app for men who have sex with men (MSM), a population who are heavily affected by HIV and frequent users of mobile technology [10]. In response to this need, we set out to use the ISR framework to design a mobile app for improving prevention behaviors in high-risk MSM.

Objective

With the proliferation of mobile health (mHealth) apps, there is an increased need for design processes that result in apps that are robust, usable, and effectively support healthful behaviors in consumers’ daily lives. The purpose of this paper is to report on a process that can be employed as a guide for the design of mHealth apps. The user-centered design processes that are reported in this paper have been used in previous formative work (eg. focus groups [11] and participatory design sessions [12]), however they have not been widely applied to the design of mHealth apps. More importantly, while user-centered methodologies have been independently applied in technology development, the holistic approach, reported in this manuscript, incorporates a series of different methodologies which are iteratively applied across the design process. Our approach is unique since it incorporates multiple user-centered design methods (focus groups, participatory design sessions, usability evaluation methods) and end-user feedback to inform the design of a mHealth app.

Methods

Theoretical Framework: The Information System Research (ISR) Framework

The Information System Research (ISR) Framework guided the implementation of user-centered human-computer interaction research methods to identify mHealth needs of user, mobile app design preferences, and the barriers and facilitators that prohibit or encourage the uptake and sustained use of mobile apps for HIV prevention, treatment and care [13]. The ISR Framework (Figure 1) employs various design processes in order to build a product or design an artifact such as a mHealth app. As applied to this project, it comprised three research cycles: the Relevance Cycle in which we sought to understand the environment of the end-user by determining requirements through a series of focus group interactions with stakeholders; the Design Cycle in which artifacts were produced and evaluated; and the Rigor Cycle, in which evaluation of theories and artifacts contributed to the design science and application domain knowledge base [13]. The cycles do not need to be conducted linearly. In fact, it may be preferable to conduct them in an iterative process, which was the case in our research activities. We briefly summarize published work related to the Relevance and Rigor Cycles to emphasize the interactions among the cycles and then provide detailed methods and results from the Design Cycle for the mHealth MSM app.

Fig. 1.

The Information System Research (ISR) Framework.

To illustrate the use of the ISR framework for the design of a mobile app, we conducted a series of design processes with high-risk MSM. In this paper, we illustrate the operationalization of the ISR framework employing user-centered design methods for the design of a mHealth app for MSM. Similar procedures for the formative development for an app for persons living with HIV (PLWH) are reported elsewhere [10 14]. The overarching goal of this project was to inform the design of a mHealth app for HIV prevention and to identify technology preferences and features, HIV-related content requirements, design specifications, and issues related to long-term appeal and maintenance of apps in at-risk or affected populations.

The Relevance Cycle

To meet the goals of the Relevance Cycle, we conducted a series of focus group sessions with intended end-users. Eligibility criteria included: Men who reported having unprotected (condomless) anal sex with HIV-positive or unknown-status male partners, age 13-64, had the ability to provide written consent/assent and the ability to read and write in English or Spanish, and self-reported being HIV-negative or of unknown HIV status. Exclusion criteria included: consistent condom use over the past year for both receptive anal intercourse (RAI) and insertive anal intercourse (IAI), or monogamous unprotected anal intercourse (UAI) with partner of certified seronegative status. Flyers detailing the study were distributed at community-based organizations in New York City and at the Harlem Pride Day, an event celebrating the lesbian, gay, bisexual and transgender community in New York’s Harlem neighborhood. Interested persons were screened over the phone.

From July – November 2013, we conducted five focus groups with 33 high-risk MSM ages 18-57 years to identify the desired functions and design of the proposed mobile app. Thematic analysis of the focus group session revealed five categories of functional requirements: My Information Management, Staying Healthy, HIV testing, Chat/ Communication Function and Resources (Table 1). Each of these categories was discussed during later sessions with end-users to identify the desired content and functional requirements of a mobile app [15].

Table 1.

User-Centered Design HIV Prevention App Categories and Sample Content

Category	Content
My Information Management	Log of Past Partners
Staying Healthy	HIV Information, prevention, diet, fitness
HIV Testing	HIV testing site information, testing log
Chat/ Communication	Medical Providers and social peers
Resources	Support group locations, Condom distribution locations, latest HIV news

The Rigor Cycle

To meet the goals of the Rigor Cycle, we performed a review to identify technology-based interventions including mobile apps related to HIV testing and prevention and apps specifically designed for MSM [10]. In our review of the literature, we did not find any studies reporting on the development or evaluation or mHealth apps. There were reports of text messaging interventions. We also searched both grey literature for existing artifacts that could meet the functional specifications and systems qualities drawn from the relevance cycle. These artifacts were included in our Design Cycle activities and sample apps are illustrated in Figure 2. Following our identification of existing mobile apps, we commenced our Design Cycle activities. The goal of this cycle was to improve the design and to increase the likelihood of technology acceptance. We conducted two phases of design sessions for the MSM app (Develop/Build and Evaluate); each successive design session was based on what we learned from the previous session. The design sessions were an iterative process whereby study participants provided feedback regarding interfaces, contributed to the design ideas, and explained which aspects of mobile interfaces, functions, and tasks they found important.

Fig. 2.

Sample apps about HIV testing.

The Design Cycle: Develop/Build Phase

In the Develop/Build phase, the focus is on the creation of a highly usable artifact. To achieve the goal of this cycle, we incorporated findings from the earlier parts of our study, including a list of content, features, and functions from the focus groups (Table 1), as well as findings from the literature review. We conducted two design sessions, in January and February 2014, for this phase.

Design Session I:

Recruitment

We recruited our study participants from those who had participated in our earlier focus group sessions. During recruitment, we explained to potential study participants that all study activities were confidential and that we would not share their identity outside of the focus group or design session.

Sample

For our first design session, we recruited six MSM who had previously participated in our focus group sessions. Focus group methodology, in place of individual interviews, was chosen because it permits participants to use their own words to answer open-ended questions and to freely react to each other’s responses [16]. This interaction between participants was expected to elicit the most information. We selected participants who were available when our design session was scheduled and who were active participants in our earlier focus group sessions. Participants were ages 20-25 years and three identified as being Latino and three as being African American. They were given $40 as a token of appreciation for their time.

Procedures

The goal of the first design session was to identify optimal features for improving HIV prevention in high-risk MSM. Each design session was audio recorded and lasted a little over 1 hour. Study team members were present during the session and took notes. Using the findings from the focus group sessions, we developed an initial list of content, features and functions prior to the session. We asked participants to imagine each of the broad categories, identified from our focus group analysis, as a separate screen on a mobile app. We presented this information in a PowerPoint and asked participants to discuss the content and features that would be included as functionality in a mobile app. Because we did not want to stifle creativity, we did not show existing apps or prototypes except in instances when participants had difficulty envisioning the content or features of the app. Participants were encouraged to revise the information presented and discussion was stimulated by a set of probing questions, including: “What information do you need from a mobile app related to your staying healthy?” and “What would improve some of the current apps that are available?”

The study team met and reviewed the transcripts, notes and drawings from the first design session. Following the review, the study team met and agreed on the content (i.e., “What”) and features (i.e., “How”) of the mobile app to present to the potential end-users during the second design session. The study team also developed a set of use cases based on the end-users plans for use of this app.

Findings from Design Session I

Findings from the first design session are categorized by topic area, derived from the Relevance Cycle focus group sessions, and reported in Table 2. In addition to the content and features, participants also identified usability factors that would make it more likely for them to use this app. Participants mentioned it would be important for the app to be tailored to different age groups. Participants made specific recommendations about the user interface and suggested that it be “sexy, eye catching, up to date, and welcoming.” They also recommended the inclusion of a search function within the app. In addition, the study team developed a set of use cases (Table 3) based on the design session and Focus Group participants’ reported plans for use of the app.

Table 2.

Results from Design Session I

Topic Area	What	How
My Information Management	Log of Past Partners	Date, HIV Status, Rating of experience
Staying Healthy	HIV Information	Videos
		Current Scams
	Prevention	HIV Risk Assessment Tool
		Updates on Pre-Exposure (PrEP) Studies
		Condom Size and Type Selector
	Diet/ Fitness	Body Mass Index Calculator
		Exercise Tracker
HIV Testing	HIV testing site info	Global Positioning System (GPS) Location
		Rating
		Cost
		Waiting Time
	Testing Log	Picture of Test Results
		Last Date Tested
Chat/ Communication	Medical Providers	Contact for PrEP
		Link to Emergency Contact/Resources
		Live Hotline
	Social/ Peer	Forums for Social Support
		Social Media Links
Resources	Support group locations	Voice activated “siri”
		GPS mapping
	Condom distribution locations	GPS mapping and information on distribution sites
	Latest HIV news	Newsfeeds

Table 3.

Use Cases for MSM Usability Testing:

1. Please find the contact information for your doctor

2. Please check your STI results

3. Please find a myth about condom usage

4. Please find information on Services for Sex Workers

5. Please enter information on one of your sexual partners

6. Please find out if you can get HIV from hugging

7. Please locate an HIV testing buddy in the 10033 zip code

8. Please locate the food pantries in 10001

9. Please find information on when to take Pre Exposure Prophylaxis (PrEP).

Design Session II:

Sample

The group of six study participants from design session I also participated in the second design session.

Procedures

The goal of the second design session was to gain information about the user interface so that prototypes could be created at the end of this design session. We audio recorded this session, which lasted approximately 2 hours. We divided our six study participants into two groups and asked each group to sketch a user interface of the desired mobile app. In this session, we presented the findings from the first design session as categories along with pictures of existing apps. Sticky post-it boards, size 25”×30”, were posted on the walls of the room. Each post-it board was designated for one of the topic areas listed in Table 1.

Participants were asked to identify the components to include under each topic area. Once again, participants were reminded that each of the categories would be a screen on the app and were asked to describe the organization of the content and features and the desired “look” of the user interface they would want to see in an app.

After participants shared their ideas, we asked probing questions to stimulate discussion about the existing apps and the need for refined content, features, and interface design. For example, questions included: “What information do you need from a mobile app related to Pre Exposure Prophylaxis (PrEP)/Post Exposure Prophylaxis (PEP)?” and “Which information from your lab reports should be viewable on your app?” Participants were also asked to identify app preferences including platform requirements and navigational features. Following these questions, pictures of existing apps were presented. Sample apps are in Figure 2. Participants were asked to comment on the features and interface of the existing apps.

Findings from Design Session II

After about 20 minutes, each group presented their design of a single mobile app screen. Participants described the desired content and interface and also commented on the other groups’ designs. Participants made suggestions on the content as well as the interface of their desired app.

Following this Design Session, study team members reviewed the sketches on the post-it board designs, PowerPoints from Design Session I and all of the transcripts. The study team reached consensus on the content and features that would be included in the mock-ups. Using this information, we prototyped user interface screens using PowerPoint based upon the results of the design sessions. The PowerPoint prototype of our MSM app developed during our build phase can be found in Figure 3. In addition, findings from our design sessions informed the creation of a map of the screen order of the mobile app. Once mock-ups were created in PowerPoint, study team members reviewed them as well as the map before being evaluated during usability testing. Use of a low-fidelity prototype enabled us to explore and understand our potential end-user’s preferences related to content functionality (e.g., mapping to find an HIV testing site), as well as their ability to achieve tasks associated with the prototype content (e.g., scheduling a medication reminder).The design sessions’ findings provided information for drafting formal functional requirements, which were further refined during The Design Cycle: Evaluate.

Fig. 3.

Mock-up of home screen and STD report – version 1.

The Design Cycle: Evaluate

The goal of this cycle was to improve the design, and to increase the likelihood of technology acceptance [17]. To achieve this goal, we evaluated the user interface and system functions of the prototype and assessed whether they were consistent with the end-users’ needs. We conducted two types of usability assessments from June –August 2014: 1) a heuristic evaluation of the prototype using informaticians with experience in interface design and/or human-computer interaction, 2) and end-user usability testing with high-risk MSM. We had a total of 5 heuristic evaluators and 10 end-user usability testers. We used an iterative process in which one heuristic evaluator and 2 end-users evaluated a mock-up and then changes were made to the app design in accordance with the recommendations. In total, we had 5 versions of the mock-ups each building on an earlier version.

Heuristic Evaluation

Sample

Five informaticians participated as usability experts. Nielsen recommends using three to five evaluators since one does not gain that much additional information by using larger numbers[18]. Each expert had training in human-computer interaction, held at least a master’s degree and was published in the field of informatics. We recruited our experts through direct contact from the Columbia University Department of Biomedical Informatics and the Weill Cornell Center for Healthcare Informatics.

Procedures

The usability experts were provided with a description of the full functionality of the prototype app. Each expert in human-computer interface tested the prototype user interface independently with the use case scenarios for approximately 45-90 minutes. The use cases are listed in Table 3. The process was recorded using Morae software™ (Techsmith Corporation, Okemos, MI) [19]. Recording the users’ interactions and vocalizations provides additional feedback to highlight problems that would not be identified with static screen shots [20]. The experts were asked to evaluate the app using a think-aloud protocol as well as complete the Heuristic Evaluation Checklist to evaluate the extent to which the user interface violates a set of usability heuristics by indicating areas of usability problems and rating the severity of each problem using a Likert scale from 0 (not a usability problem) to 4 (usability catastrophe) [21].

Instrument

Bright et al. [22] developed a Heuristic Evaluation Checklist based on Nielsen’s ten heuristics [23]. Each heuristic was evaluated by one or more items and the overall severity of the identified heuristic violations was rated.

Data Analysis

Mean severity scores were calculated for each heuristic principle from Nielsen’s checklist. Evaluators’ comments about usability problems on the evaluation form and the Morae recordings were reviewed [24] and prioritized. Based on these findings, we refined the prototype app.

Findings from Heuristic Evaluation

Mean scores and sample comments for each usability factor related to the MSM app are reported in Table 4. A total of 112 changes were made to the MSM mock-up based on the heuristic evaluators’ recommendations. There were a number of recommendations related to consistency and standards. For example, one heuristic evaluator suggested taking out the google maps icon where it is not relevant. Another expert suggested that the Ask a Doctor Forum and the Peer Forum should have the same color scheme. One of the experts noted that the same icon was used for law and advocacy and HIV transmission laws on a later screen, which was confusing. In other cases, the experts explained that the flexibility of the system could be enhanced. For example, one expert suggested that if the user is meant to make a selection then response key should be added. To improve flexibility of use, the food pantries/soup kitchens should have a box to enter zip code and results should be ranked based on proximity to the zip code entered.

Table 4.

Mean Scores and Sample Comments of Heuristic Evaluation of MSM App

Usability Factor	Mean (S.D.)	Sample Comment
Visibility of System Status	1.40 (0.89)	Home icon was too subtle
Match between System and the Real World	0.80 (1.30)	I though the question mark icon was the help button. I was surprised it was the “about me” button
User Control and Freedom	1.20 (1.10)	Sometimes the “back” button went to an unexpected place
Consistency and Standards	1.00 (1.00)	Lines, color and font have not been followed consistently in all screens within the system
Help Users Recognize, Diagnose, and Recover From Errors	1.00 (1.41)	Some of the areas of the system should provide prompts or error messages
Error Prevention	0.80 (0.84)	Menus are too deep; increases cognitive load and working memory to remember where things are located
Recognition Rather Than Recall	0.80 (0.84)	Color highlighting is not used to get the user’s attention
Flexibility and Efficiency of Use	1.00 (0.45)	Users do not have the option of clicking directly on the menu item or using a shortcut
Aesthetic and Minimalist Design	1.40 (0.89)	Visual layout is not well designed
Help and Documentation	1.00 (1.41)	There should be a general help button for the 6 major buttons

Usability Testing

Sample

We recruited 10 MSM who did not participate in the design sessions to evaluate the prototype user interface screens. Eligibility criteria were the same as that described above for the focus group sessions. We used similar recruitment methods as those used for the focus group sessions and also included MSM who were eligible for our focus group sessions but could not participate due to scheduling conflicts. Ten participants were selected because past research has shown that the minimum percentage of problems identified rose from 55% to 82% and the mean percentage of problems rose from 85% to 95% when the number of users was increased from 5 to 10 [25]. Eight of the participants were smartphone users.

Procedures

Participants were provided with the same use cases as the heuristic evaluators and a description of the full functionality of the prototype system. After the usability evaluation, participants were asked to rate the prototype’s usability using a standardized questionnaire.

Instruments

Each participant completed the Post Study System Usability Questionnaire (PSSUQ), which is a 19-item survey instrument developed at IBM to assess user satisfaction with system usability on a scale ranging from 1 (strongly agree) to 7 (strongly disagree). Items relate to efficiency, ease of use and likability of the system interface. The PSSUQ has strong evidence of reliability (α = .95), and content and construct validity [26]. A lower score on the PSSUQ indicates a more usable system.

Data Analysis

The analysis was based on the Morae recordings of user sessions, transcriptions, notes and the user survey. The team searched for critical incidents characterized by comments, silence, and repetitive actions. We reviewed these incidents in detail using Morae software. The incidents identified and the users’ written comments were summarized. Results from the PSSUQ were analyzed using SPSS version 22.0 (IBM, Armonk, NY) to calculate the descriptive statistics

Findings from Usability Testing

Our usability testing resulted in 5 versions of our prototype. At each version, we refined the content, potential functionality and interface of the mobile app. A total of 55 changes were made to the mock-ups based on the end-users’ recommendations. During usability testing, end users suggested that the button named “Communications” be renamed. This button was renamed “Connect” and the icon was changed to reflect the new name. Another user recommended that the category “other” be included under race since not everyone identifies with a specific race. In another usability session, one participant said that the glucose graph needed a key at the bottom with an explanation of each color, which was later added. Two participants who reviewed the same version of the mock-up independently said that the home icon needs to be made brighter and centered, which we corrected on the next version. One participant suggested that the superman icon be changed to reflect a real person on the HIV testing buddy locator screen. This icon was changed to an icon reflecting a real person. PSSUQ scores (Figure 4) trended towards a lower score over time for the MSM app indicating that end-users perceived the app to be more useful as we refined the app based on usability testing. By the end of our usability testing process, the score ranged from a 1- 1.26 among smartphone users, with a 1 indicating a perfectly usable tool. The scores given by non-smartphone users continued to be worse than the scores of smartphone users.

Fig. 4.

PSSUQ scores of MSM end-users over time.

Final Functional Components of the App

Findings from the user-centered iterative design process resulted in the Design Document of a mobile app for HIV prevention for high-risk MSM.Our findings suggest the inclusion of the following six broad functional categories in the mHealth app: communication, my personal health information, health and wellness information, HIV information, settings and general resources. Examples of the content within each of these broad functional categories are listed in Table 5.

Table 5.

User-centered design app for MSM at high risk for HIV: functionalities and sample details.

Functionality	Sample Details
Communication	Ask a Doctor Forum, Chat with a Provider or HIV testing counselor, Interactive Blogs
my personal health information,	My Medication Information, Lab Reports, My Providers Contact information, Health Appointments, Partner Logs
health and wellness information,	Myths and Facts, Nutrition & fitness,
HIV information	HIV/AIDS medical care, Testing Information
Settings	Profile picture, password, and alerts
General Resources	Medical Care, General Services (eg. LGBTQ Centers), Substance Use, Law/ Advocacy

Discussion

Current approaches for the delivery of health information, in particular HIV prevention through the use of information technologies, such as mHealth, have not been well tested and there has been limited work especially using a mobile platform [10]. While some research work has shown promising results [27], other studies using mHealth in chronic disease patients have produced only modest or mixed results [28 29] and of those that have been effective, their effects are poorly sustained [30].

Guidelines for Future Mobile App Development

The ISR framework prescribes an iterative process, which included needs assessment, functional requirement identification, user interface design and rapid prototyping. Each of these techniques has been used in technology development in the past [31-33]. Nonetheless our approach provides a new model for the design of mHealth apps which suggests that a holistic approach of incorporating these methods into a single process has the potential to provide a greater sum that a siloed approach to app design. This study supports the use of these methods guided by the ISR framework as a proof of concept for the usefulness of this approach. Although the specific methods illustrated in the paper are well known to the human-computer interaction community and broadly known to the informatics community, the integrated approach to Health apps is somewhat novel and constitutes a genuine contribution to knowledge.

At each stage of this project, important changes were made to the app, demonstrating that each iteration provided meaningful improvements in development of the app. We iteratively refined the app to meet the end-users’ needs while remaining true to the goals of the project. At the same time, the end-user’s recommendations were vetted through our study team, which included experts in HIV prevention, sexual minority groups’ health needs, human-computer interaction and mHealth. The synthesis of end-user feedback with content expert advice provides the foundation for the development of a potentially usable and useful app. importantly findings from this work provide a proof of concept for the use of the ISR framework for guiding mHealth app development.

Another noteworthy contribution of this work is that is offers an approach to evaluating the usability of the mobile app user interface before the app is on a mobile device. Despite the proliferation and almost ubiquitous use of mobile devices, the usability of mobile websites and applications is well behind the usability of desktop websites. Our approach to mobile app usability testing can be helpful in addressing some of barriers to assessing the usability of mobile devices which have been previously noted such as lack of memory, small screen space and poor graphical displays{Harrison, 2013 #37}. In our previous work, we found that differences in technology (device) do not significantly contribute to users’ ability to complete a task but does effect the time it takes [17]. Assessing the user interface of a mobile app on a desktop for design principles may be appropriate however further assessment to assess the time to task completion should be completed using a mobile device.

Inclusion of End-Users in App Design

One reason for the failure of health information technology interventions is poor design that does not meet the requirements of the users of the technology [34]. It is important that patients be engaged in the use of the technology that can best be achieved through the design of tools that are perceived as easy to use and useful [35]. If mHealth apps do not meet end-users’ needs then they will be misused or underutilized and ultimately fail to meet their original objectives [36]. With a user-centered design model, mHealth apps can meet consumers’ needs and result in the creation of a suitable user interface [37]. To develop effective mHealth tools, it is critical to start with the needs of the intended end-users. To achieve our objective we used a series of iterative user-centered design process and active engagement of target users in the design process. Many mHealth apps are designed with minimal input of users whom they are supposed to support. The continued proliferation of mHealth apps without end-user feedback in the design process is inherently flawed especially since in our past work, we found that identification of end-users’ information needs can improve system usability [32 38] . The techniques described in this paper allowed us to focus specifically on what our intended users felt was most important.

Including target users at all stages of the design process, from formative work to usability evaluations of the resulting prototype, doubtlessly resulted in a more useful prototype than if the design process happened with limited user engagement.

Findings from this study illustratethe usefulness of the ISR framework in guiding the design of a mHealth app. Our iterative design process guided by the ISR framework allowed for improvement in the usability of the mHealth app as is evidenced by an overall improvement in usability as indicated by improved PSSUQ scores over time.

While overall the PSSUQ scores improved over time, the scores given by non-smartphone users continued to be worse than the smartphone users, which can be expected. Even so, it is important to garner feedback from both groups since the design needs of these two groups are different because the non-smartphone users do not have the mental model of the user controls and interface rules that an experienced smartphone user has developed. It was important to include both users and experts because our heuristic evaluators can frequently overlook problems that more novice technology users will notice. For example, our users made recommendations to change the name of a button to a more colloquial term that was not identified by the heuristic evaluators. On the other hand, heuristic evaluators noted the importance of Positive test results being denoted in Red and Negative in Green as displayed in Figure 3 to meet the heuristic criterion of “Consistency and Standards.” Our inclusion of both end-users and experts is well-grounded in the informatics literature since end-users are the experts in their own lives and needs and human-computer interaction experts provide expertise in the interface design.

Limitations

Our study participants were drawn from a pool of individuals who participated in previous research, which may be a self-selecting group. Our study participants may be more willing to share information about their sexual behavior and health status than other MSM and this may limit the generalizability of our findings. Moreover, we did not collect information on literacy, education or cognitive functioning of our study participants. Variability in each of these factors is likely to affect participants’ use of mobile technology and should be included in future work. Our findings are also limited because only so much can be determined from evaluation of a not fully-functional prototype for a mobile app. Future research is needed to understand the effectiveness of this app and other mHealth tools that are developed using this approach. There is a need for collection of more extensive usage and outcome data, to enable cost/benefit analyses that will further validate this approach for developing mHealth tools. In addition the collection of our data from a non-generalizable sample of high-risk MSM is a limitation, but the findings from our work are likely relevant for HIV prevention behaviors across MSM, although not more broadly for prevention behaviors related to other diseases. At the same time, apps are being rapidly developed and marketed with little end-user feedback and many are quickly disappearing from the mobile marketplace. It is critical in the development of all HIT but in particular in the case of mobile apps that users perceive their information needs are met and the app is easy to use.

Conclusions

In this paper, we presented the use of the ISR framework for designing a mHealth app. Our work was specifically focused on development of an HIV prevention app for high-risk MSM. The operationalization of the ISR framework in the context of mHealth can provide foundational guidance for future mHealth app development. The iterative nature of this framework allows for the incorporation of end-user feedback and expert opinion in the development process. The multistage approach when using this framework can be time consuming and costly. However, past research has demonstrated that dedicating resources during development can result in technology that is easy to use and useful once it is implemented. Use of this framework by future app developers has the potential to lead to the creation of highly usable apps that may improve health behaviors. Given the findings from this work, the ISR framework is a useful model to guide future mHealth app development.

Highlights.

• The iterative nature of the ISR framework allows for the incorporation of end-user feedback and expert opinion in the mobile app development process.

• Findings from this study support the use of the ISR framework as a useful model to guide the design of mHealth apps.

• User-centered methodologies have been independently applied in technology development in past research.

• The holistic approach, reported in this manuscript, incorporates a series of user-centered design methods (focus groups, participatory design sessions, and usability evaluation methods) and end-user feedback, which are iteratively applied across the process, to inform the design of a mHealth app.