Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Jul 25.
Published in final edited form as: Appl Clin Inform. 2026 Jan 30;17(1):39–45. doi: 10.1055/a-2786-0291

Optimizing HIV Care Engagement: Usability of a mHealth App for Identifying and Retaining Individuals with Non-Viral Suppression in Digital Cohort

Fabiana Cristina Dos Santos 1, Sophia Mclnerney 1, Miya C Tate 2, Aadia Rana 3, D Scott Batey 2, Rebecca Schnall 1
PMCID: PMC12858313  NIHMSID: NIHMS2189169  PMID: 41617201

Abstract

Background:

Drive to Zero is a mobile health application (app) designed to identify and retain people with HIV (PWH) who have experienced challenges with achieving or maintaining viral suppression. The app targets PWH who have lacked documented HIV care in the past months and are experiencing medication adherence barriers. Features include an interactive chat for communicating with the study team and access to educational resources to support care engagement and health management.

Objective:

This usability study aimed to assess the Drive to Zero app’s ease of use and interface design through expert heuristic evaluation and end-user testing.

Methods:

Usability was evaluated through two approaches: heuristic evaluations conducted by five informatics experts following Nielsen’s usability principles, and end-user testing with 20 PWH using the validated Post-Study System Usability Questionnaire and qualitative interviews to collect feedback on app functionality and user experience.

Results:

Heuristic experts and end-users demonstrated satisfaction with the app’s appearance, reporting that it has a simple and intuitive interface for identifying and retaining PWH, which will assist them with study engagement and ultimately re-engage with HIV care. However, participants highlighted areas needing improvement, suggesting better accessibility of “home” and “help” buttons to improve user control and a more detailed explanation of the incentive program to enhance user engagement and retention.

Conclusions:

Usability evaluations provided valuable insights into the Drive to Zero app’s design. Areas for improvement were enhancing user controls and improving the readability of the incentive program. These findings will guide iterative refinements, ensuring that future versions of the app improve the usability and acceptability of its target audience.

Keywords: HIV, consumer health informatics, human-computer interaction, interfaces and usability, smartphone

Background and Significance

HIV continues to present important public health challenges across the United States despite advances in treatment and care. Despite improved overall outcomes for people with HIV (PWH), geographic disparities persist, with the Deep South having higher rates of PWH who have not achieved viral suppression.1–4 Research has shown that achieving viral suppression within three months of an HIV diagnosis is an important indicator of HIV care success.5 However, the time to viral suppression (defined as the interval from HIV diagnosis to the first recorded viral load of fewer than 200 copies/mL) is substantially longer among PWH living in the Deep South compared to those in other regions of the country.2,6 In response, to understand the “drivers” of delays in viral suppression and non-sustained viral suppression, we developed a digital cohort study targeting the enrollment of PWH in Alabama, Louisiana, and Mississippi who have not achieved viral suppression, aligning with the 2022–2025 National HIV/AIDS Strategy Goals.7 The study specifically targets individuals diagnosed with HIV for 3 months or more who have not achieved viral suppression, defined as: absence of viral load testing within the past 36 months, viral load more than 200 copies/ml as confirmed by medical records or surveillance system, or self-reported difficulties with daily HIV medication adherence.

As retention is paramount in this challenging population, we designed the Drive to Zero mobile health (mHealth) application (app) to support study engagement and retention. The Drive to Zero app enables end-users to complete health surveys, tracking their general health status, communicate directly with the study team via personalized and interactive chat, access educational resources, and visualize their progress and completed tasks through charts (Figure 1). To promote consistent engagement, the app features an incentive program in which end-users who engage with the study team and complete monthly assigned tasks earn rewards followed by congratulatory messages, thereby supporting both study retention and ultimately improved HIV care outcomes.

Figure 1. Illustration of the Drive to Zero app:

Figure 1.

Open in a new tab

A mHealth app that offers a comprehensive end-user experience featuring (A) a Home dashboard displaying completed task status with a congratulatory message; (B) a personalized messaging system through a chat feature, which allows end-users to communicate with study team; (C) Learn function featuring educational contents including health tips flyer and video, and (D) Progress tracking with visual representation of completed tasks over time through a bar chart.

A notable issue in the proliferation of mHealth apps is the development of digital technologies with minimal input from end-users, leading to poor design, inadequate consideration of user needs, and, ultimately, poor usability.8,9 Recent evidence suggests that poorly designed apps, lacking in usability, are prone to misuse, underutilization, and failure to achieve their intended objectives.10,11 Similar challenges have been observed in other health-focused applications, which demonstrated the importance of user-centered design in developing effective digital health tools for patient education and engagement.10–14

Usability measures the quality of the end-user’s experience when interacting with a system.15 Prior to deployment of the Drive to Zero app, we conducted comprehensive usability evaluations using two complementary methods: expert-based evaluations (including heuristic evaluations) and end-user-based evaluations (which focus on a user’s experience and interaction with the system to measure acceptability).16,17 This dual approach ensures that the app meets both technical standards and end-user needs, maximizing its potential to support study retention and HIV care engagement.

Objectives

Describe the usability evaluations through expert heuristic evaluation and end-user testing and refine the user interface of the Drive to Zero app.

Methods

The Drive to Zero app’s usability testing included a heuristic evaluation with informatics experts and end-user testing conducted with PWH. The Columbia University Institutional Review Board reviewed [approval number: AAAU8227] and approved all usability testing activities. Participants consented prior to participating in usability testing.

Study Measures

The primary measure of this study was usability problem assessed from (1) heuristic evaluation, using Nielsen’s ten usability principles with scores ranging from 0 (not a usability problem) to 4 (usability catastrophe),16 and (2) end-user acceptability was measured using the 16-item Post-Study System Usability Questionnaire (PSSUQ) with scores ranging from 1 (strongly agree/high acceptability) to 7 (strongly disagree/low acceptability).18 Lower scores indicate satisfaction with the system. To achieve a deeper understanding, we included qualitative feedback from both expert evaluators and end-users regarding specific usability concerns and user experience.

Heuristic Evaluation

Five experts were recruited as evaluators following Nielsen’s recommendation to include three to five evaluators in usability testing.16 Each expert had between 8 and 24 years of experience in health informatics, with expertise in human-computer interaction, interface design, and usability testing.

Experts were provided with the app and asked to complete a session in the app through case scenarios that represented the main functions of the system and think-aloud methodology.18 Participants were instructed to describe what they were thinking, seeing, and doing as they completed the eight scenarios: 1. Log into the Drive to Zero app. 2. Scroll through to see the To Do list. 3. Complete a survey about your current health status. 4. View History to get more details about your completed tasks. 5. View the Chat message on the dashboard and click on the chat button to send a message to the study team. 6. Click the Learn button to get information about hand hygiene. 7. View the points you earned from tasks over one month. 8. Log out (Figure 1). Following their use of the app, experts completed an online heuristic evaluation checklist to assess how a system adheres to Nielsen’s ten usability principles19 using Qualtrics XM, an online survey software. A member of the research team analyzed evaluators’ comments regarding usability problems to identify areas of usability concern that could be targeted for improvement. Mean usability problem severity scores were calculated for each of the ten usability heuristics.

End-users Testing

Twenty end-users were recruited for the usability testing through convenience sampling methods. The eligibility criteria were PWH aged 18 years or older to ensure the Drive to Zero app was evaluated by its intended target audience, providing a diverse usability evaluation. End-users were contacted using a research database and word-of-mouth referrals and invited to participate in the study. During usability testing, the end-users were provided with a brief explanation of the Drive to Zero app and its functions. The end-users were directed to complete the same eight scenarios completed by the experts in the heuristic evaluation. Sessions took place remotely via Zoom meetings (Zoom Communications, Inc.), and a member of the team was present during sessions to provide guidance when an end-user was unable to move through a task independently and to take notes. Following their use of the app, the end-users completed a survey using the Qualtrics XM software. The survey included demographics and a validated version of the PSSUQ. Through validated measures, the end-users contributed feedback on the app’s functionality and user experience.

We utilized Dedoose, a specialized qualitative research software, to analyze interview data and identify emerging themes related to end-user reactions to the Drive to Zero app. The primary researcher (FDS) conducted the initial analysis by developing, refining, and reorganizing codes. A second researcher (SM) performed double-coding on all data to ensure rigor. For each transcribed interview, both researchers independently coded the content before conferring to validate their interpretations. Any coding discrepancies were thoroughly discussed until a consensus was reached. The final thematic structure was developed and supported with representative interview quotes. Quotes from end-users are presented, followed by their identification number. Inter-coder reliability was assessed using Cohen’s Kappa, demonstrating substantial agreement (k = 0.90) between researchers.

Data analysis was performed using R statistical software (version 4.1.2; R Foundation for Statistical Computing) to analyze descriptive statistics and usability measures. The mean scores were calculated for the post-survey.

Results

Heuristic Evaluation

Experts identified 13 unique usability issues across the heuristic evaluation, with the Help button, App information, and Home button being the most frequently cited concerns among the five evaluators. The overall mean severity score was 1.0 (SD±1.1), with individual heuristic items ranging from 0.2 to 1.4, where scores closer to 0 indicate better usability. As shown in Table 1, most technical issues identified by experts were classified as cosmetic or minor usability problems, reflecting the overall strong design of the Drive to Zero app. Two heuristics emerged as primary areas needing improvement (higher severity): User Control and Freedom 1.4 (SD±1.3) and Help and Documentation 1.4 (SD±1.1). Experts noted that the absence of a consistent “Home” button across all screens limited users’ ability to navigate the app efficiently and intuitively. Additionally, experts underscored that having a “Home” button visible would support better direction, especially for users unfamiliar with the app’s structure.

Table 1.

Mean Scores and Sample Comments from Heuristic Evaluation

Nielsen’s Usability Heuristics Mean (±SD) Heuristic Evaluator Identified Issues and Suggestions
Visibility of System Status 1.2 (1.1) “Some headings are not consistent with the other language in the app.” (Expert 1)
“The welcome message button was not clickable.” (Expert 2)
Match Between the System and the Real World 1.2 (1.3) “There needs to be more information about what the app is for, what points can be used for, and why one would want to chat.” (Expert 1)
User Control and Freedom 1.4 (1.3) “Need a home button on all screens.” (Expert 3)
“Home button always visible helps.” (Expert 5)
Consistency and Standards 1.0 (1.0) “Providing more context for the app and a description of the points at the beginning would be helpful.” (Expert 1)
Help users Recognize, Diagnose, and Recover from Errors 0.4 (0.9) “Selecting an answer for the second health status question triggers an automatic save, preventing immediate corrections.” (Expert 4)
Error Prevention 1.2 (1.1) “The survey auto-saves response. Instead of the next button, there could just be a save button that could prevent automatic pop-ups.” (Expert 1)
Recognition Rather Than Recall 0.2 (0.5) “I see a minor problem. In the To-do list, the only field marked as optional, verify your phone number, should not be optional.” (Expert 1)
Flexibility and Efficiency of Use 1.2 (1.6) “In the learn function, there is a search bar, but it needs to be able to search more than just the title of the items.” (Expert 4)
Help and Documentation 1.4 (1.1) “Help button is not accessible. You must go to the menu.” (Expert 4)
“I did not notice a help function.” (Expert 1)
Aesthetic and Minimalist Design 1.2 (1.1) “Alias is most likely not familiar.” (Expert 3)
Overall score 1.0 (1.1)
Open in a new tab

Regarding Help and Documentation, experts observed that the “Help” feature was not readily accessible. The “Help” button was inserted within the main menu, requiring multiple clicks to access support. This design may prevent users from quickly finding assistance when technical issues arise, potentially reducing the app’s overall usability.

Five additional usability concerns with mean severity scores of 1.2 were noted: Visibility of System Status (SD±1.1), where experts reported inconsistent headings and non-clickable interface elements such as the welcome message button; Error Prevention (SD±1.1), where experts recommended replacing automatic pop-ups with explicit save buttons in the health survey function; Aesthetic and Minimalist Design (SD±1.1), where experts identified potentially unfamiliar terminology such as “Alias” that might confuse end-users; Match Between the System and the Real World (SD±1.3), with experts suggesting more explanatory information about app functionality and purpose; and Flexibility and Efficiency of Use (SD±1.6), with suggestions to enhance search functionality beyond title-based searches in the Learn function. In contrast, the Recognition Rather Than Recall principle received the best usability score of 0.2 (SD±0.5), with the lowest variability among expert ratings, indicating consensus that this was not a usability concern compared to other heuristic principles. Experts provided minor suggestions to include only required tasks on the “To-do” list rather than optional tasks.

Usability Testing

The sample of end-users comprised 20 individuals diagnosed with HIV, with a mean age of 48.2 years (SD±14.8). Forty percent of participants (8/20) self-identified as heterosexual, 30% (6/20) as homosexual, 15% (3/20) as queer, 10% (2/20) as bisexual, and 5% (1/20) as other. In terms of race, 70% (14/20) identified as African American/Black, 20% (4/20) as White, 5% (1/20) as Asian, and 5% (1/20) as Native Hawaiian or another Pacific Islander. Of these participants, 65% (13/20) had completed some college or a graduate degree as their highest level of education. All participants (100%) reported using a smartphone several times per day, with 55% (11/20) reporting smartphone use of five hours or more daily. Ninety percent of participants (18/20) reported feeling comfortable downloading and using apps.

Overall, end-users rated the Drive to Zero app highly usable across all usability measures. On the PSSUQ, all subscales, system usefulness, information quality, and interface quality scored 2 or below, indicating high usability (Table 2). The overall mean PSSUQ score across all 16 items was 1.7 (SD±0.6), indicating consensus among end-users that the app was highly usable.

Table 2.

Usability Measures

Post-Study System Usability Questionnaire (16 items) Mean (±SD)
System Usefulness 1.4 (0.5)
Information Quality 2.0 (0.8)
Interface Quality 1.9 (1.2)
Overall PSSUQ 1.7 (0.6)
Open in a new tab

Post-Study System Usability Questionnaire (rating the score from 1 being the best score to 7 being the worst score).

System Usefulness

This category refers to an app that is simple to navigate, enables quick task completion, and allows end-users to become productive. When asked to identify the most useful function after interacting with the Drive to Zero app, end-users reported diverse preferences. The Learn function was frequently cited as the most useful, 45% (9/20), followed by users who valued all functions equally, 25% (5/20), the Chat function, 20% (4/20), and the History function, 10% (2/20).

The Learn function provides participants with access to health education resources, including tips and videos from the Centers for Disease Control and Prevention (CDC). End-user feedback also suggests expanding to educational materials such as seasonal health information (e.g., Influenza, COVID), vaccine options, LGBTQ+ specific resources, and treatment information from other organizations, such as local healthcare departments. Most end-users noted that the app was informative and useful for health education, with statements such as:

“I like the health tips of what appears to be a CDC flyer. So, I think maybe add seasonal flyers like what is going around now, like influenza, COVID, ensure the vaccine options that you have available, something like that, just to keep general health in mind. Maybe some LGBTQ+ resources, like patient care information, or maybe some resources can guide people.” End-User30

“I love the way it says it gives you those two categories, health tips, and videos from the CDC. I really like it; you do not have to go to different places. If this function has a link or a connection to our local health department, where we can find information, that would be wonderful.” End-User4

Participants also expressed the accessible interface of the Chat function for PWH with visual impairment:

“The chat is nice and simple. I like that. It does not have complicated features, and it is even for me, I am visually impaired, and I find it accessible and easy to use. The colors and icons are great, and I really like them. I like the fact that it is different, and different is good. Different keeps people engaged.” End-User4

End-users commented that History was the less useful function because of the chart format and legend.

“I do not like the bar chart. I do not like how today starts on the right. Normally, I feel like I understand that today, we would slowly move over to the left as more days came, and that date would get shifted to the left. I think maybe just redesign it.” End-User30

Information Quality

Information quality refers to the clarity, accessibility, and effectiveness of the app that enables end-users to complete tasks and easily recover from errors. After participants used the app, one end-user recommended including explanations about the challenge (incentive structure):

“I found the wording a bit unclear. I am not sure what challenge refers to. Am I being challenged by a doctor, myself, or the app? Maybe using the word goal instead would make it clearer. I do like seeing the total points earned and the update timing. That helps track my last check-in. I noticed I received points after completing a task, but it would help to include a description of what the points mean and what the reward is.” End-User30

Furthermore, another end-user noted that the organization of information, particularly in the welcome message, was unclear, suggesting a description to guide end-users on its goals and tasks since this feature represents an introduction to the app.

“In the welcome message, perhaps include brief information on what is the goal of this app.” End-User26

Interface Quality

This category represents an app that meets end-user expectations for functionality and capabilities, contributing to overall satisfaction. Most end-users found the interface of the Drive to Zero app to be intuitive, approachable, and easy to use:

“I think this app will play a pivotal role in ending HIV. It is informative, user-friendly, and modern, exceeding the standard of the apps I use daily. I love how thoughtfully designed it is and appreciate the effort that went into creating it. This app becomes customizable and is designed to support and empower you in managing your health. So, I think it is amazing.” End-User11

“That was really easy. It was very straightforward. I had no trouble finding it or figuring out where I was supposed to go next. It was, it was simple.” End-User25

“I really like the color. The icons are nice. I like that you can get rid of them if you want to, so that your screen’s not super busy.” End-User8

End-users’ Engagement

Most end-users found app functions, such as congratulatory messages after completing a task, motivational and engaging. A representative quote is as follows:

“The congratulations little message, I think, just promotes empowerment, even in the little of littlest of ways. Just congratulating somebody on completing a task makes you want to complete more tasks and more tasks. I love the little congratulations icon with the little trophy.” End-User11

Another end-user reflected:

“Congratulations on having completed a task. I think it is good and motivates me. I feel good that what I do, what I respond to, and my activity with the app are recognized.” End-User31

Participants also discussed alternatives to improve their engagement with the app. For instance:

“To motivate, maybe create a system where there are features that you can unlock as you reach certain points, so we complete all tasks in the app.” End-User17

Discussion

Our study evaluated the usability of the Drive to Zero app. Overall, heuristic experts and end-users demonstrated satisfaction with the Drive to Zero app, reporting that it has a simple and intuitive interface with minor areas needing improvement. PSSUQ scores demonstrated that higher user satisfaction and satisfactory overall experience were lower than 2, as a successful usability evaluation score, indicating that the Drive to Zero app users are satisfied with the system. These findings were supported by other studies that used PSSUQ measures to test the usability of mHealth applications for HIV prevention10 and disease management.20

Through end-user testing, we also gained insight into what would promote maximum app usage and help PWH re-engage with HIV care. In this study, we found that the Learn function emerged as the most popular functionality within the Drive to Zero app. Potential reasons include the ability to empower end-users through reliable information available in a private, digestible (e.g., flyer, video, figures), and accessible format when questions arise, potentially reinforcing information retention and practical application in their daily lives. The preference for educational content aligns with previous mHealth research that has identified learning features as particularly attractive to end-users across apps in different health contexts.21–24

Another feature that contributed to the app’s engagement was congratulatory messages and trophies for completing the required tasks linked to incentive programs. These features in the Drive to Zero app may activate intrinsic motivation by creating a sense of achievement and progress that many end-users described as gratifying, similar to the findings reported in Tran’s study.25 In addition, the visual representation of success through interactive icons such as trophies can offer evidence of accomplishment that can be particularly meaningful for individuals navigating the challenging journey of HIV care engagement. Previous studies indicated that the success of mHealth apps with incentive programs depends on their ability to engage with their end-users and align with their needs and preferences.25,26

Although end-user testing effectively identifies important usability concerns that affect end-user experience and performance, focusing on end-user testing alone may lead to missed usability violations that end-users who do not have expertise may not recognize.27,28 Experts provided a complementary evaluation with a more technical viewpoint, highlighting where standard app formatting should be adhered. For instance, experts were more likely to find usability concerns from a general end-user perspective, such as the “Home” button should be added across all screens. Experts also suggested improving the accessibility of the “Help” button to quickly find assistance when technical issues arise. Previous studies recommended combining usability evaluations from both end-user and expert perspectives as an effective and thorough approach for evaluating mHealth apps and identifying usability problems prior to real-world deployment.10,27

While end-users and heuristic evaluations revealed distinct themes, important areas of overlap emerged that enhanced the validity of our findings. Both experts and end-users identified the need for clearer introductory content when opening the app, explicitly suggesting instructions about the Drive to Zero app’s purpose and goals. In addition, both groups emphasized that a more thorough explanation of the incentive program would improve user engagement and retention.

The difference between usability evaluation methodologies allowed us to fill knowledge gaps and enhance the app’s overall design. Previous studies in health technologies have demonstrated that combining multiple usability evaluation methods assists in more comprehensively identifying design problems.10,27,29–32 These experts’ and end-users’ perspectives were essential in developing a user-friendly and visually appealing mHealth app. Findings suggest that rigorous user-centered design of the Drive to Zero app supported a usable product. The results of this usability evaluation will guide our current work in improving and refining the user interface design.

This study presents some limitations. First, the convenience sample was recruited from a database and word-of-mouth referrals, including participants who had participated in past research and expressed interest in future studies. Given that this study uses convenience sampling, which may have resulted in a cohort that does not fully represent the diversity of PWH. To address this, future iterations of usability testing will recruit a more diverse sample, including individuals from varied demographic and socioeconomic backgrounds. Second, the end-user sample included PWH regardless of viral suppression status. While this broader recruitment provided valuable insights into general app usability testing, it limits the generalizability of our findings to the app’s primary target population, PWH with non-viral suppression. Individuals who are not virally suppressed may have distinct needs and engagement barriers that could influence their interaction with, and perception of app features, differently from virally suppressed individuals. Future usability studies should specifically recruit and analyze feedback from the intended end-user population to ensure the app adequately address their unique barriers to care engagement. Third, while team member interaction with end-users was minimal throughout the testing process, their presence during usability sessions may have influenced participant responses. Despite these limitations, we used quantitative measures and qualitative data to identify areas of improvement and refine the Drive to Zero app. In addition, this study enriched the body of research on evaluating the mHealth usability for HIV care engagement.

Conclusion

The usability study of the Drive to Zero app combined rigorous evaluations through heuristic evaluations and end-user testing. The findings indicated that the app has a simple, user-friendly interface and optimal functionality, demonstrating its potential to influence end-user engagement and support HIV care efforts. Usability and acceptability evaluations of the Drive to Zero app were favorable overall. Key recommendations identified for future development include improving introductory content clarity, enhancing user navigation controls (particularly accessible home and help buttons), and improving explanation of incentive program. These findings will guide iterative refinement of the app’s user interface before deployment.

Clinical Relevance Statement.

Understanding the usability and acceptability of a mobile health application is important for designing effective tools to support HIV care engagement, particularly PWH who are not virally suppressed, a population at high risk for poor health outcomes and continued HIV transmission. This study demonstrates a rigorous user-centered approach to evaluating and refining the Drive to Zero app interface before real-world deployment. Assessing usability with both experts and end-users provides insights into whether the app aligns with user needs and preferences, which is important for informing future implementation efforts in clinical practice.

Multiple-Choice Questions.

  1. What methodology was used in this study?
    1. Heuristic evaluation only
    2. End-user testing only
    3. Both heuristic evaluation with informatics experts and end-user testing
    4. None of the above

    Correct Answer: The correct answer is option c.

  2. Which app function did end-users report as most useful during their interaction with the Drive to Zero app?
    1. Chat function
    2. Learn function
    3. History function
    4. All functions were equally useful.

    Correct Answer: The correct answer is option b.

Acknowledgments:

The authors acknowledge the participation of the heuristic evaluators and end-users.

Funding:

Research reported in this study was funded by the National Institute of Allergy and Infectious Diseases through grant number: UG3 AI176566. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the National Institute of Allergy and Infectious Diseases.

Footnotes

Conflict of Interest: None declared.

Protection of Human and Animal Subjects: This study was approved by the Institutional Review Boards. Informed verbal consent was obtained from interview participants who were notified of their right to refuse to participate and the study team’s procedures for de-identifying data.

Data Availability:

The data underlying this article will be shared on reasonable request to the corresponding author.

References

  • 1.Gant Z, Dailey A, Wang S, et al. Trends in HIV care outcomes among adults and adolescents in the U.S. South, 2015–2019. Annals of Epidemiology. 2022;71:15–22. doi: 10.1016/j.annepidem.2022.04.011 [DOI] [PubMed] [Google Scholar]
  • 2.Bassler JR, Ostrenga L, Levitan EB, et al. Redlining and Time to Viral Suppression Among Persons With HIV. JAMA Intern Med. 2024;184(11):1329. doi: 10.1001/jamainternmed.2024.5003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Dos Santos FC, Batey DS, Kay ES, et al. The effect of a combined mHealth and community health worker intervention on HIV self-management. Journal of the American Medical Informatics Association. Published online January 11, 2025:ocae322. doi: 10.1093/jamia/ocae322 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Centers for Disease Control and Prevention. Estimated HIV Incidence and Prevalence in the United States, 2018–2022. HIV Surveillance Supplemental Report 2024.; 2024. https://www.cdc.gov/hiv-data/nhss/estimated-hiv-incidence-and-prevalence.html [Google Scholar]
  • 5.Dombrowski JC, Baeten JM. It’s Time to Make the Time to Viral Suppression After HIV Diagnosis a Metric of HIV Care Success. The Journal of Infectious Diseases. 2019;219(6):845–847. doi: 10.1093/infdis/jiy539 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of HIV/AIDS Prevention. Diagnoses of HIV Infection in the United States and Dependent Areas, 2016.; 2017. https://stacks.cdc.gov/view/cdc/50233 [Google Scholar]
  • 7.The White House. National HIV/AIDS Strategy Federal Implementation Plan.; 2022. https://files.hiv.gov/s3fs-public/2022-09/NHAS_Federal_Implementation_Plan.pdf
  • 8.McCurdie T, Taneva S, Casselman M, et al. mHealth Consumer Apps : The Case for User-Centered Design. Biomedical Instrumentation & Technology. 2012;46(s2):49–56. doi: 10.2345/0899-8205-46.s2.49 [DOI] [PubMed] [Google Scholar]
  • 9.Chang IC, Hwang HG, Hung WF, Li YC. Physicians’ acceptance of pharmacokinetics-based clinical decision support systems. Expert Systems with Applications. 2007;33(2):296–303. doi: 10.1016/j.eswa.2006.05.001 [DOI] [Google Scholar]
  • 10.Dos Santos FC, Brin M, Tanner MR, Galindo CA, Schnall R. The mChoice App, an mHealth Tool for the Monitoring of Preexposure Prophylaxis Adherence and Sexual Behaviors in Young Men Who Have Sex With Men: Usability Evaluation. JMIR Hum Factors. 2025;12:e59780–e59780. doi: 10.2196/59780 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Adedoja D, Kuhns LM, Radix A, Garofalo R, Brin M, Schnall R. MyPEEPS Mobile App for HIV Prevention Among Transmasculine Youth: Adaptation Through Community-Based Feedback and Usability Evaluation. JMIR Form Res. 2024;8:e56561. doi: 10.2196/56561 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Li Y, Li J, Li M, et al. VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions. JAMIA Open. 2024;8(1):ooaf005. doi: 10.1093/jamiaopen/ooaf005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Romero S, Alvarez KS, Nijhawan AE, Nethi A, Bistransin K, King HL. Association of an HIV-Prediction Model with Uptake of Preexposure Prophylaxis. Appl Clin Inform. 2025;16(03):507–515. doi: 10.1055/a-2524-4993 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Roveta A, Canonico M, Francese A, et al. Developing Mobile Apps to Support Clinical Management of HIV Patients: An Observational Study. Appl Clin Inform. 2025;16(03):544–555. doi: 10.1055/a-2543-7444 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Abran A, Khelifi A, Suryn W, Seffah A. Usability meanings and interpretations in ISO standards. Software Quality Journal. 2003;11(4):325–338. [Google Scholar]
  • 16.Yen PY, Bakken S. Review of health information technology usability study methodologies. Journal of the American Medical Informatics Association. 2012;19(3):413–422. doi: 10.1136/amiajnl-2010-000020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Jaspers MWM. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics. 2009;78(5):340–353. doi: 10.1016/j.ijmedinf.2008.10.002 [DOI] [PubMed] [Google Scholar]
  • 18.Bertini E, Gabrielli S, and Kimani S. Appropriating and assessing heuristics for mobile computing. In: Proceedings of the Working Conference on Advanced Visual Interfaces. 2006. [Google Scholar]
  • 19.Nielsen J and Molich R. Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1990:249–256. [Google Scholar]
  • 20.Young R, Sage K, Broom D, Hext A, Snowdon N, Smith C. Evaluating the usability of a co-designed power assisted exercise graphical user interface for people with stroke. J NeuroEngineering Rehabil. 2023;20(1):95. doi: 10.1186/s12984-023-01207-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Mendiola MF, Kalnicki M, Lindenauer S. Valuable Features in Mobile Health Apps for Patients and Consumers: Content Analysis of Apps and User Ratings. JMIR mHealth uHealth. 2015;3(2):e40. doi: 10.2196/mhealth.4283 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Dos Santos FC, Garofalo R, Kuhns L, Scherr T, Schnall R. Evaluating the Effectiveness of a Mobile Health Intervention on Enhancing HIV Knowledge in Sexual and Gender Minority Men. JAIDS Journal of Acquired Immune Deficiency Syndromes. Published online November 19, 2024. doi: 10.1097/QAI.0000000000003562 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Guo SHM, Lin JL, Hsing HC, Lee CC, Chuang SM. The Effect of Mobile eHealth Education to Improve Knowledge, Skills, Self-Care, and Mobile eHealth Literacies Among Patients With Diabetes: Development and Evaluation Study. J Med Internet Res. 2023;25:e42497. doi: 10.2196/42497 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Rowland SP, Fitzgerald JE, Holme T, Powell J, McGregor A. What is the clinical value of mHealth for patients? npj Digit Med. 2020;3(1):4. doi: 10.1038/s41746-019-0206-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Tran S, Smith L, Carter S. Understanding Patient Perspectives on the Use of Gamification and Incentives in mHealth Apps to Improve Medication Adherence: Qualitative Study. JMIR Mhealth Uhealth. 2024;12:e50851. doi: 10.2196/50851 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Cotton V, Patel MS. Gamification Use and Design in Popular Health and Fitness Mobile Applications. Am J Health Promot. 2019;33(3):448–451. doi: 10.1177/0890117118790394 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Cho H, Yen PY, Dowding D, Merrill JA, Schnall R. A multi-level usability evaluation of mobile health applications: A case study. Journal of Biomedical Informatics. 2018;86:79–89. doi: 10.1016/j.jbi.2018.08.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Schnall R, Higgins T, Brown W, Carballo-Dieguez A, Bakken S. Trust, Perceived Risk, Perceived Ease of Use and Perceived Usefulness as Factors Related to mHealth Technology Use. Stud Health Technol Inform. 2015;216:467–471. [PMC free article] [PubMed] [Google Scholar]
  • 29.Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. Journal of the American Medical Informatics Association. 2017;24(e1):e55–e60. doi: 10.1093/jamia/ocw100 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Khajouei R, Farahani F. A combination of two methods for evaluating the usability of a hospital information system. BMC Med Inform Decis Mak. 2020;20(1):84. doi: 10.1186/s12911-020-1083-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for Usability Assessment of Mobile Health Technology: Validation Study. JMIR Mhealth Uhealth. 2018;6(1):e4. doi: 10.2196/mhealth.8851 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Alwashmi MF, Hawboldt J, Davis E, Fetters MD. The Iterative Convergent Design for Mobile Health Usability Testing: Mixed Methods Approach. JMIR Mhealth Uhealth. 2019;7(4):e11656. doi: 10.2196/11656 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The data underlying this article will be shared on reasonable request to the corresponding author.

RESOURCES