Table 1.

Studies evaluating eHealth HIV interventions using usability evaluation methods

First Author & Reference	Title	Year	Country	Intervention Name	Technology	Purpose	Target Population	Usability Evaluation Type(s)	Results
Beauchemin [21]	A multistep usability evaluation of a self-management app to support medication adherence in persons living with HIV	2019	USA	WiseApp	Mobile app and electronic pill bottle	ART adherence and Health management	PLWH aged 18+	Cognitive walkthrough, Think -aloud, Heuristic evaluation, Questionnaire (Health-ITUES), Task analysis	Researchers created a mobile app, in conjunction with an electronic pill bottle, to encourage ART adherence among PLWH. A think-aloud was performed by end-users and revealed the need to increase the visibility of app components, which was supported the heuristic evaluations. A cognitive walkthrough confirmed that the app and pill bottle were usable for end-users.
Cho [17]	Understanding the predisposing, enabling, and reinforcing factors influencing the use of a mobile-based HIV management app: A real world usability evaluation	2018	USA	mVIP	Mobile app	Health management	PLWH aged 18+	Focus groups	A mobile app was developed to provide self-care strategies to aid in symptom self-management. Focus groups with end-users revealed the predisposing, enabling, and reinforcing factors that influenced the app‟s usability in a real-world setting.
Cho [20]	A mobile health intervention for HIV prevention among racially and ethnically diverse young men	2018	USA	MyPEEPS Mobile	Mobile app	Risk reduction	Healthy males aged 15–18	Heuristic evaluation, Think-aloud, Questionnaire (PSSUQ), Scenarios	Researchers translated an HIV-prevention curriculum into a mobile app. The heuristic evaluation revealed that user navigability within the app needed the most refinement. End-users generally were able to complete the use case-scenarios. The end-users‟ PSSUQ scores indicated strong user acceptance.
Cho [18]	A multi-level usability evaluation of mobile health applications: A case study	2018	USA	mVIP	Mobile app	Health management	PLWH aged 18 +	Think-aloud, Heuristic evaluation, Semi-structured interviews, Questionnaire (Health-ITUES), Scenarios, Task analysis, Eye-tracking	A mobile app was developed to aid with self-management of HIV symptoms for PLWH. Researchers employed a 3 level usability evaluation. Through this multilevel evaluation, the content, interface, and functionality of the app was iteratively refined at each level. End-users rated the app as highly usable.
Cho [19]	Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation	2019	USA	MyPEEPS Mobile	Mobile app	Risk reduction	Healthy males aged 15–18	Eye-tracking, Think-aloud, Questionnaire (Health-ITUES)	Researchers translated an HIV-prevention curriculum into a mobile app. eye-tracking retrospective think-aloud enabled us to identify critical usability problems as well as gain an in-depth understanding of the usability issues related to interactions between end-users and the app.
Coppock [22]	Development and usability of a smartphone application for tracking antiretroviral medication refill data for human immunodeficiency virus	2017	Botswana		Mobile app	ART Adherence	Pharmacists	Questionnaire (CSUQ), Contextual interview	Researchers developed a pharmacy-based ART adherence mobile app. End-user evaluators provided recommendations such as a feature for rapid adherence calculations and wanted ongoing technical improvements. End-users rated the app as highly usable.
Cordova [23]	The usability and acceptability of an adolescent mHealth HIV/STI and drug abuse preventive intervention in primary care	2018	USA	Storytelling 4 Empowerment	Mobile app	Risk reduction	Healthy youth aged 13–22	Focus groups, Semi-structured interviews	A mobile app was adapted from a face to face intervention for HIV/STI and drug abuse prevention among adolescents. Qualitative results indicated favorable results for the usability and acceptability of the app‟s format, content, and process.
Danielson [24]	SiHLEWeb.com: Development and usability testing of an evidence-based HIV/STI prevention website for female African American adolescents	2016	USA	SiHLEWeb.com	Website	Risk reduction	Healthy African American females aged 13–18	Think-aloud, Questionnaire (WAMMI)	A website was developed for risk-reduction among female African American adolescents. During the evaluation, users reported issues with website navigation. Users found that the intervention was helpful, easy to use, and generally attractive. In addition, it improved knowledge and learning of HIV prevention.
Hightow-Weidman [25]	HealthMpowerment.org: Development of a theory-based HIV/STI website for young black MSM	2011	USA	HealthMpowerment.org	Website	Risk reduction	Healthy African American males aged 18–30	Think-aloud	An internet-based intervention was developed with input from young African American men who have sex with men. The evaluations indicated acceptable usability and provided potential innovations to the website.
Himelhoch [26]	Pilot feasibility study of Heart2HAART: A smartphone application to assist with adherence among substance users living with HIV	2017	USA	Heart2HAART	Mobile app	ART adherence, Substance use treatment	PLWH aged 18–64	Questionnaire (non-validated instrument)	A mobile app was developed to support ART adherence in conjunction with directly observed treatment.. Although the study participants generally found the app usable, some counselors found the participation required on their part burdensome.
Horvath [27]	eHealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men	2017	USA	Get Connected!	Website	Risk reduction	Healthy males aged 15–24	Questionnaire (non-validated instrument)	An online intervention was developed to increase HIV testing among YMSM. The study explored the effect of tailoring the intervention to a participant‟s background and the effect of eHealth literacy on HIV/STI testing among YMSM and intervention usability. Tailoring and participant health literacy did not have an impact of overall user satisfaction and perceived usefulness, but they did alter perceived quality.
Kawakyu [28]	Development and implementation of a mobile phone-based prevention of mother-to-child transmission of HIV cascade analysis tool: Usability and feasibility testing in Kenya and Mozambique	2019	Kenya & Mozambique	mPCAT	Mobile app	Risk reduction	Health workers and facility managers	Think-aloud, Focus groups	Researchers created a mobile phone based PMTCT cascade analysis tool for health workers. Following training on the use of the app, two rounds of usability testing was conducted among facility workers and managers. Health facility workers found that the app was useful and wanted to implement it in their clinics.
Luque [29]	Bridging the digital divide in HIV care: a pilot study of an iPod personal health record	2013	USA	MyMedical	Mobile app	Health management	PLWH aged 18+	Questionnaire (non-validated instrument)	A personal health record (PHR) for potential users, who also participated in a 6-week self-management training curriculum, evaluated an iPod touch. Usability for both the iPod and the PHR was assessed post-training. Both aspects of the intervention were highly rated in terms of usability, but most participants expressed that they did not intend to use the PHR to remind them to take their medications.
Mitchell [30]	Smartphone-based contingency management intervention to improve pre-exposure prophylaxis adherence: Pilot trial	2018	USA	mSMART	Mobile app	Risk reduction	Healthy males aged 18–30 years	Semi-structured interviews, Questionnaire (SUS)	A mobile app was developed to encourage PrEP adherence among MSM. The intervention received a moderate usability rating, but future studies are recommended due to the sample‟s homogeneity.
Musiimenta [31]	Acceptability and feasibility of real time antiretroviral therapy and adherence interventions in rural Uganda: Mixed method pilot randomized control trial	2018	Uganda		SMS and electronic pill bottle	ART adherence	PLWH aged 18+ and their social supporters aged 18+	Semi-structured interviews	The intervention was developed to provide real-time ART adherence monitoring via text messaging and an electronic pill bottle. PLWH and their supporters generally approved of the intervention, but usability was restricted by environmental factors such as shared phones and limited electricity.
Sabben [32]	A smartphone game to prevent HIV among young Africans (Tumaini): assessing intervention and study acceptability among adolescents and their parents in an RCT	2019	Kenya	Tumaini	Mobile app	Risk reduction	Healthy adolescents aged 11–14 years & their parents	Questionnaire (non-validated instrument), Focus groups	A narrative-based smartphone game was created for risk reduction among healthy youth aged 15–24. The eHealth intervention scored high for usability. There is opportunity for improving game mechanics and communication to parents about their role in the intervention.
Schnall [33]	Usability evaluation of a prototype mobile app for health management for persons living with HIV	2016	USA		Mobile app	Health management	PLWH aged 18+	Heuristic evaluation, Questionnaire (PSSUQ), Scenarios, Think-aloud	A mobile app was created to support self-management and improve health outcomes. The usability evaluation process led to 77 intervention changes recommended by the expert evaluators and 83 changes recommended by the end users.
Schnall [34]	A user-centered model for designing consumer mobile health application (apps)	2016	USA		Mobile app	Risk reduction	Healthy males aged 13–64	Heuristic evaluation, Questionnaire (PSSUQ), Scenarios, Think-aloud	A mobile app for risk reduction among high-risk MSM. Usability evaluation methods were used to iteratively develop the mobile app. The study demonstrated the value of the ISR (Information Systems Research) Framework as a method to incorporate end-user’s preferences when creating mHealth interventions.
Shegog [35]	“+Click”: pilot of a web-based training program to enhance ART adherence among HIV-positive youth	2012	USA	+CLICK	Web-based application	ART adherence	PLWH aged 14–22	Questionnaire (non-validated instrument)	A web-based application was created for ART adherence among youth PLWH. Evaluation ratings indicated that the intervention as easy to use, trustworthy, and easy to understand.
Shegog [36]	“It’s your game”: an innovative multimedia virtual world to prevent HIV/STI and pregnancy in middle school youth	2007	USA	It’s Your Game, Keep It Real	Desktop-based curriculum	Risk reduction	Healthy adolescents aged 12–14	Questionnaire (non-validated instrument)	The multimedia desktop-based HIV/STD and pregnancy prevention intervention for middle school students was highly rated in terms of ease of use, credibility, understandability, and acceptability.
Skeels [37]	CARE+ user study: usability and attitudes towards a tablet PC computer counseling tool for HIV+ men and women	2006	USA	CARE+	Desktop-based curriculum	ART adherence	PLWH aged 18+	Semi-structured interviews, Contextual interview	A computer-counseling tool was created to support ART medication adherence. The study found the intervention was usable by participants of varying experience with computers. Most participants found computer counseling useful, but still wanted to maintain consultations with a human provider.
Stonbraker [38]	Usability testing of a mHealth app to support self-management of HIV-associated non-AIDS related symptoms	2018	USA	VIP-HANA	Mobile app	Health management	PLWH aged 18+	Heuristic evaluation, Think-aloud, Scenarios, Questionnaire (Health-ITUES & PSSUQ)	The mobile app was created to support PLWH to manage symptoms associated with HIV. Experts and end users rated the application as highly usable. Experts and end-users had different feedback regarding essential components for usability. Experts had focused more on the app design compared to end-users.
Sullivan [39]	Usability and acceptability of a mobile comprehensive HIV prevention app for men who have sex with men: A pilot study	2017	USA	HealthMindr	Mobile app	Risk reduction	Healthy males aged 18+	Questionnaire (SUS)	A mobile app was created as a risk reduction intervention for MSM. Participants gave the mobile application an above average usability rating. Nearly ten percent of the PrEP eligible participants-initiated PrEP over the course of the trial.
Vargas [40]	Usability and acceptability of a comprehensive HIV and other sexually transmitted infections prevention app	2019	Spain	Preparadxs	Mobile app	Risk reduction	Healthy males and females aged 18+	Questionnaire (SUS)	The mobile app was created to improve HIV/STI prevention among adults. Evaluators rated the application having good usability and believed that the app had the potential to reduce future incidence of HIV and other STIs.
Widman [41]	ProjectHeartForGirls.com: Development of a web-based HIV/STD prevention program for adolescent girls emphasizing sexual communication skills	2016	USA	ProjectHeartForGirls.com	Website	Risk reduction	Healthy females aged 15–19	Think-aloud	An interactive web-based program for risk reduction among adolescent girls. All recommendations served as part of an iterative process of development and usability testing.
Williams [42]	An evaluation of the experiences of rural MSM who assess an online HIV/AIDS health promotion intervention	2010	USA	HOPE project	Website	Risk reduction	Healthy males aged 18+	Questionnaire (non-validated instrument)	A website was created for risk reduction among MSM residing in rural areas. Participants who completed the intervention found it easy to navigate and access the intervention from home. Perceived usefulness increased from the first to the last module.
Winstead-Derlega [43]	A pilot study of delivering peer health messages in a HIV clinic via mobile media	2012	USA		Mobile app	Health management	PLWH aged 18+	Questionnaire (non-validated instrument)	A mobile app consisting of peer health videos regarding health management for PLWH. Participants found the intervention acceptable in a clinic setting and found the app easy to use. After the intervention, there were no significant changes in participant perception regarding engagement in care or HIV status disclosure.
Ybarra [44]	Usability and navigability of an HIV/AIDS internet intervention for adolescents in a resource limited setting	2012	Uganda	CyberSenga	Website	Risk reduction	Healthy adolescents	Think-aloud, Focus groups, Contextual interview	A website was created for risk reduction among secondary school students in Uganda. Participants were most interested in the interactive activities. From the field-testing, researchers found that those wanting to implement this intervention should consider user preparedness, user literacy, a source of internet connection, bandwidth, and electricity.

ART: Antiretroviral Therapy; CSUQ: Computer System Usability Questionnaire; Health-ITUES: Health Information Technology (IT) Usability Evaluation Scale; HIV: Human Immunodeficiency Virus; MSM: Men who have Sex with Men; PLWH: People living with HIV; PMTCT: Prevention of mother-to-child transmission; PSSUQ: Post-Study System Usability Questionnaire; SMS: Short message service; STI: Sexually Transmitted Infections; SUS: System Usability Scale; WAMMI: Website Analysis and MeasureMent Inventory; YMSM: Young Men who have Sex with Men