Table 4.

Improvements to the program concept after each stage of research

Improvements after stage 1

Substantial refinements were made to the program concept using findings from the focus groups with YBGBM and interviews with HIV providers. These included additional content areas and new or improved tools and features further explored in stages 2 and 3. Improvements included: • Multiple ways to protect user confidentiality. Confidentiality was a concern among many YBGBM. We learned that simple steps could allay such concerns, such as making the program name discreet, having both an app and a website (for anyone reluctant to download an HIV app), and allowing anonymous registration • Including a wide range of content and tools to meet the needs of diverse users. The findings indicated that different people would interact in different ways with the app. We included various tools and content to improve usability and uptake • Adding videos for families. YBGBM and providers discussed stigma from family members. We added videos that YBGBM can share with family, such as a video of a medical doctor clarifying that HIV cannot be transmitted through casual contact (e.g., eating from the same dish) and videos of family members discussing how they have changed to be more supportive • Motivational content. YBGBM and providers suggested that the program could integrate motivational statements and affirmations (e.g., a daily affirmation upon opening the app), and include content that is empowering and informational (e.g., an explanation of how taking HIV medications every day can help one live a long, healthy life) • New tools. Additional tool suggestions included a pharmacy refill reminder and a notepad to store questions or to record side effects to discuss with their provider at their next visit • Peer mentors. Providers and YBGBM responded favorably to the proposed community features of the program and felt this would help address the social isolation experienced by many YBGBM living with HIV. An additional feature suggested was to have trained mentors on the app to answer questions and provide support

Improvements after stage 2

Numerous improvements were made to the program concept using data collected during rapid prototype sessions with YBGBM living with HIV and with HIV providers. Changes were made iteratively and were subsequently tested with the next participants. Improvements included: • Making medication reminders motivational. YBGBM and providers wanted motivational elements in the program. In the focus groups, many mentioned that adherence struggles were due to not wanting to take medications rather than forgetting to take them. We added a message in adherence reminder congratulating users for the number of days in a row that they had taken their medications • Virtual support groups. One YBGBM participant suggested that the program could include ongoing support groups for different topics (e.g., using substances, dealing with a diagnosis, and thriving with HIV). This idea tested very well with subsequent YBGBM and provider participants • Videos for partners and friends, in addition to family members. Focus group participants suggested creating videos for families to help them support (and not stigmatize) people with HIV. One YBGBM participant early in rapid prototype testing suggested including videos of how partners or friends should react when disclosed to. We tested both ideas extensively and they were well received • Tools to help users set and attain goals. One YBGBM participant suggested adding a tool to help set and reach goals, whether related to HIV directly or to other issues (e.g., eating healthy). This tested well with YBGBM and provider participants • Provider dashboard. A provider suggested creating a dashboard that organization staff could use to engage with clients as they use the program. This idea tested well with all subsequent providers and nearly all YBGBM participants

Improvements after stage 3

Stage 3 aimed to test the program concept from the perspective of a larger, more geographically diverse sample and to conduct user testing to identify any issues prior to building the program. In usability testing, most participants were able to complete their tasks very quickly and gave very positive feedback on tools, community features, resources, inclusion of both video and text content, and ease of use, including that the content and tools felt comprehensive (i.e., nothing was missing). Thus, stage 3 reinforced that program concept elements developed in stages 1 and 2 were feasible and accessible, rather than identifying areas for improvement. Several important program elements confirmed included: • That the program should include representation of people living with HIV who are diverse in terms of race/ethnicity, gender, gender identity, sexual orientation, and age, rather than focusing exclusively on YBGBM • An app was preferred over a mobile website (although we will add a mobile website version when funding permits)