Developing and testing a web-based platform for antiretroviral therapy (ART) adherence support among adolescents and young adults (AYA) living with HIV

Abstract

Objective

Describe the development and testing of a web-based platform for antiretroviral treatment (ART) adherence support among HIV+ adolescents and young adults (AYA) in a randomized controlled trial (RCT).

Methods

A seven-member multi-disciplinary team operationalized the flat, password protected, web-based platform. Manualized protocols guided the objectives and content for each of the eight web-based sessions. Team members evaluated usability and content validity. Client satisfaction and perceived ease of use was evaluated with the first ten HIV+ AYA participants.

Results

The web-based platform was developed, evaluated, refined, implemented and pilot tested between September 2020 to April 2022. Usability was rated as high; the evaluation of content validity showed an excellent fit between session content and objectives. HIV+ AYA participants (mean age = 24.2 years) were satisfied with the quality, type, and amount of support/education received, and found the platform easy to use, operate, and navigate. Average time spent per session was 6.5 min.

Conclusion

Findings support the usability, validity, acceptability, and feasibility of this web-based platform for ART adherence support among HIV+ AYA.

Innovation

Our research and findings are responsive to research gaps and the need for transparency in the methodological development and testing of web-based control arms for ART adherence support among HIV+ AYA.

Highlights

• •Development of a robust control condition is key for HIV behavioral trials.
• •We describe a web-based, theory informed, ART adherence control arm.
• •Engaging members of the HIV community in all stages of development is essential.
• •Sufficient resources are needed to develop and test a web-based, control arm.
• •Our experience provides a protocol for developing future web-based platforms.

1. Introduction

In the United States (US), an estimated 1,070,604 people (aged ≥13 years) were living with human immunodeficiency virus (HIV) at year end in 2020, with 30,635 new diagnoses reported [1]. Black/African American and Hispanic/Latine people account for 42% and 27% of HIV infections, respectively [1], with more than half of new infections occurring among individuals aged 13 to 34 years [1]. As such, Black/African American and Hispanic/Latine adolescents and young adults (AYA) are disproportionately affected by HIV in the US.

Achieving viral suppression, or a plasma HIV RNA level below the lower limit of detection (<20 copies/mL depending on the assay used) [2], is a vital component of HIV treatment and prevention, as research shows that people who adhere to antiretroviral therapy (ART) and achieve an undetectable viral load cannot transmit HIV [3]. However, adhering to ART is challenging [4,5] and there are many barriers to participating in regular and sustained HIV health care services [6], known as retention in care. Strategies to improve ART adherence, viral suppression, and retention in care are integral components in achieving a 90% reduction in the number of new HIV infections by 2030, the goal in the national HIV/AIDS strategy for Ending the HIV Epidemic [7,8].

Adherence behavior is complex, shaped by a multi-level [[9], [10], [11]] socioecological (SE) context [[12], [13], [14], [15]]. Among Black/African American and Hispanic/Latine HIV+ AYA, SE influences at the individual level include self-efficacy [16,17], HIV knowledge [18], self-stigma [19], psychological distress [19,20], and substance use [16,19,20]; those at the interpersonal level include personalized HIV stigma/‌disclosure concerns, and social support [16,[20], [21], [22]]. SE influences from the HIV health system include the availability of peer navigators and engagement with HIV healthcare providers [13,23], and structural barriers include decreased access to care (e.g., travel difficulty), HIV discrimination, and public stigma [13,[24], [25], [26], [27]].

Mobile Health (mHealth), or the implementation of mobile technologies, such as desktop or smartphone apps, social media, gaming, texting or calling, or video conferencing [[28], [29], [30], [31]] to promote access to health care services has the potential to improve ART adherence and reduce HIV health disparities among Black/African American and Hispanic/Latine HIV+ AYA [24]. mHealth engages underserved populations [32] – including minoritized and historically underrepresented groups – mitigates structural barriers by increasing access to HIV services [33,34], and decreasing the stigma of accessing HIV care [35]. Yet there are few technology-enabled ART adherence interventions for HIV+ AYA [23], including interventions that leverage theory and conceptualization of developmental context [24,28], and patient specific needs [36]. Among five studies that have utilized technology-enabled, web-based platforms to address ART adherence [[36], [37], [38], [39], [40]], the inclusion of Black/African American and Hispanic/Latine HIV+ AYA has been limited. For example, two studies included samples of men ages 40 years or older [37,38]. Though the remaining three studies included HIV+ AYA participants, the majority of the samples were composed of Black/African American perinatally HIV-infected females [39] or Black/African American gay males [[40], [41], [42]].

More recently, proof-of-concept was demonstrated for a peer-led, technology-enabled ART adherence intervention with promising virologic and adherence data findings, entitled Adherence Connection for Counseling, Education, and Support (ACCESS) [43]. Peers are persons with similar characteristics as the target population, who provide support and education through their shared experience, and in the context of a formal intervention [[43], [44], [45], [46]]. Informed by the findings from the ACCESS proof-of-concept study [43], the ACCESS II randomized controlled trial (RCT) (R01NR019535) was designed to test the efficacy of the peer-led, technology-enabled adherence intervention, compared to a web-based adherence control (ACCESS II web-based platform) on primary outcomes of HIV viral load and self-reported ART adherence for Black/African American and Hispanic/Latine HIV+ AYA 18 to 29 years of age.

The hallmark of an RCT is to establish efficacy of the experimental arm relative to the control arm using randomized assignment [47,48]. However, there is considerable heterogeneity in the design and selection of behavioral health control arms for RCT's [47,48], with potential for statistically significant and large effects on study outcomes [48]. The CDC compendium of evidence-based interventions and best practices for HIV prevention [49] details 19 US-based RCTs of interventions aiming to improve ART adherence, three of which are technology-enabled [[50], [51], [52]]. Among these three technology-enabled RCT's, the control arms varied: one provided standard of care approaches encompassing adherence education and health system engagement [50], the second, general health and well-being counseling [52], and the last, an animated tutorial on ART adherence [51]. Although the research methods for these interventions are well described and rated as good quality, few details are provided on the selection and development of the control arms [49]. Dissemination of the methods used to develop and implement control arms in RCT interventions is greatly needed. This transparency will increase the rigor and reproducibility of intervention design and evaluation strategies for efficacy and effectiveness, lending to subsequent implementation and scalability. Therefore, the purpose of this paper is to describe the methodological development and testing of the ACCESS II control arm, a web-based platform designed for ART adherence support among HIV+ AYA.

2. Methods

2.1. Overview of the ACCESS II RCT

ACCESS II is an innovative approach to engage Black/African American and Hispanic/Latine HIV+ AYA in a much-needed ART adherence support intervention, justified with prior research findings [43] and current gaps in the evidence-base. Development of the ACCESS II intervention was informed by Bronfenbrenner's SE model [[9], [10], [11], [12]] and tailored for specific support of ART adherence behavior among HIV+ AYA using the following levels: individual, interpersonal, health system, community, and structural (Fig. 1). Our approach is a longitudinal (12 and 24-weeks), two-arm RCT with HIV + AYA, ages 18–29 years (initial target enrollment: N = 120). Participants randomized to the intervention arm use study-funded smartphones to connect synchronously (Zoom videoconferencing) with trained peer health coaches (peers) who deliver eight weekly, 60-min cognitive behavioral motivational sessions (sessions 1–8). Participants randomized to the control arm are also provided with study-funded smartphones and receive similar educational content in an asynchronous web-based platform to test the hypothesis that the synchronous delivery of content by supportive peers is the important influence on outcome variables. We hypothesize that participants randomized to the synchronous peer-led, technology-enabled intervention arm will demonstrate significant improvements in 12 and 24-week self-reported ART adherence, HIV viral load (primary outcomes), and self-efficacy, stigma and disclosure concerns, social support, psychosocial distress, and substance use (secondary outcomes), when compared to the asynchronous web-based adherence control arm (ACCESS II web-based platform).

Fig. 1.

Targets of ACCESS II intervention sessions.

2.1.1. Phase I: Development of the ACCESS II web-based platform (control arm)

The development and evaluation of the ACCESS II web-based platform was conducted in four phases (September 2020 to April 2022). We used a multi-disciplinary team-based approach in all phases including software selection, protocol development, evaluation, and implementation of the ACCESS II web-based platform. Team members were content experts in nursing and medical care of HIV+ AYA, psychology (substance use/mental health), research methods, biostatistics, and mHealth. Peers were hired members of the ACCESS II study team with the title Assistant Research Scientist, and thus were compensated for their time across all phases of the study, including the development and evaluation phase. University-based experts in information technology (IT) were also consulted regularly.

2.1.2. Software selection

Our initial work during the fall of 2020 entailed selecting an affordable and secure web-based platform with demonstrated ease of use, and minimal maintenance requirements. Following review of platforms by team members, Rise (https://rise.com/) was selected for the creation of a flat, secure, password protected website (content hidden from the public domain). This selection was based on considerations of data security (Health Insurance Portability and Accountability Act HIPAA), ease of use, and accessibility using a personal computer (via webpage) and/or smartphone (via downloadable application (app)).

Rise allowed for participants to use a self-led/paced approach for asynchronous completion of the ACCESS II study activities (sessions 1–8) with no direct study team supervision. However, the study team was able to track user engagement such as number of times each session was viewed, and time spent on each session [53]. The cost associated with a 1-year subscription of this web-based platform (time of purchase January 2021) was $4788 with capacity for 100 users.

2.1.3. Developing written protocols

The content for each of the eight ACCESS II sessions in the web-based platform was developed during a seven-month time span (January to August 2021) and matched with the goals and intervention content delivered by peers in each session. As such, an iterative process with comparison of content between study arms was conducted, lending to the development of eight distinct written protocols for the eight sessions. Each protocol specified language that was non-judgmental to impart a tone of respect, conversational in style with motivational phrases, and provided opportunities for participant reflection. We selected short videos to engage participants and to include more personalized perspectives on the session content (e.g., experience of HIV stigma). An overview of these sessions including exemplars of session videos is provided in Table 1.

Table 1.

Overview of ACCESS II intervention and control sessions 1–8.

Session #	Session Overview	Session Video	Video Source
1*	Study Engagement	CDC Antron's Story- Let's Stop HIV Together	Act Against AIDS (video no longer available)
2	HIV Knowledge	Heart2Heart - Understanding HIV Basics	Heart2Heart https://vimeo.com/208140568
3	Self-Efficacy	CDC HIV Journey to Undetectable	CDC YouTube Channel https://youtu.be/QCudOrlw5BA
4*	HIV Stigma/Disclosure Concerns	CDC Behind the Scenes with Let's Stop HIV Together	Act Against AIDS (video no longer available)
5*	Managing HIV Stigma/Disclosure Concerns	CDC Let's Stop HIV Together	Tarrant Country HIV Administrative Agency (video no longer available)
6	Psychological Distress and Associated Substance Use	HIV & Mental Health - Marissa	Greater Than HIV YouTube Channel https://youtu.be/LGGKeqUbCZY
7	Healthcare Partnerships	Partnership for Health - Sam's Initial Visit	CDC YouTube Channel https://youtu.be/dBXJzCnbeOE
8	Study Closure, Social Support in Community & Health Care System	No video	No Video

Note. *Videos from Session 1, 4, 5 were removed by CDC ∼ April 2021.

Graphics were selected from websites that provide royalty-free stock art (Adobe Stock; stock.adobe.com) and videos were curated from multiple online federal HIV resources [54]. As per the Fair Use exemption in copyright law, excerpts of nonfictional materials not readily available for license may be posted for a limited period on password-protected educational sites, and in support of research, scholarship, and teaching purposes [55].

2.1.4. Translating written protocols to web-based content

Narrative content from the eight written protocols was used to develop the ACCESS II web-based platform via Rise and the eight separate “courses” (Rise uses the term “courses”). Each course was created using a series of instructional prompts for a range of content, including text (heading, subheading, paragraph), image(s), video(s), and continue buttons to provide breaks in the session and promote user engagement (Appendix A). Relevant images and videos were embedded to reinforce the text, and as an additional method for engaging with the educational material. These eight courses (hereafter referred to as sessions) were developed chronologically during an eleven-month time frame (June 2021–April 2022).

2.2. Phase II: Evaluation by ACCESS II study team members

2.2.1. ACCESS II web-based platform evaluation tool

Guided by published criteria from Jones et al. [56] and Hightow-Weidman & Bauermeister [57], we developed a 16-item tool (Appendix B) to rate the following items on a 5-point Likert scale: (1) overall usability, (2) validity (match of session content with session objectives), (3) developmental and cultural considerations, (4) feasibility, and (5) acceptability. Two survey items were used to examine acceptability: one designed to assess session length (too short, too long, or just about right), and the second, to elicit subjective comments/recommendations for change. We also collected data on user metrics (number of log-in attempts, length in minutes for session completion, day of week/time of day session completed), and free space to report subjective comments.

Members of the ACCESS II study team (N = 7) used the 16-item tool to evaluate each of the eight sessions over the span of a month (Dec 2020 - Jan 2021). The study team members were granted access to the ACCESS II web-based platform and instructed to evaluate one session per day using the 16-item Evaluation Tool. Evaluations were completed on personal computers and each ACCESS II web-based session was viewed in mobile portrait mode to simulate the experience of participants on study-funded smartphones. Data were collected using a fillable form in Microsoft Word®, and then extracted and entered into Microsoft Excel® for analysis by a senior research scientist. These data and subjective comments were summarized and reviewed by the study team (see Table 2).

Table 2.

Study team evaluation of the ACCESS II web-based platform (N = 7).

	Session 1	Session 2	Session 3	Session 4	Session 5	Session 6	Session 7	Session 8	Total
Measure Items	Mean	Mean	Mean	Mean	Mean	Mean	Mean	Mean	Mean (SD)
Usability
Ease of use with smartphone	4.14	4.14	4.29	4.14	4.14	4.14	4.14	4.29	4.18 (0.07)
Ease of reviewing narrative content, graphics, videos on smartphone	4.14	3.57	4.29	3.71	4.00	4.00	4.14	4.29	4.02 (0.26)
Ease of use in session navigation or moving through the different sections	4.14	4.86	4.43	4.43	4.43	4.86	4.43	4.43	4.50 (0.24)
Content Validity
The content matches the session's goals	4.57	5.00	4.71	4.86	4.86	4.29	4.86	4.57	4.71 (0.23)
Language in session matches the session's goals	4.60	5.00	4.71	4.71	4.86	4.43	4.71	4.57	4.70 (0.18)
Graphics and videos match the session's goals	4.40	4.67	4.86	4.86	4.86	4.57	4.71	4.50	4.68 (0.18)
Developmental and Cultural Considerations*
Content in the session is developmentally appropriate and culturally relevant for research participants	4.50	4.67	4.83	5.00	4.60	4.60	4.83	4.50	4.69 (0.18)
Language in the session is developmentally appropriate and culturally relevant for research participants	4.40	4.83	4.67	4.83	4.60	4.80	4.83	4.67	4.70 (0.15)
Graphics and videos are developmentally appropriate and culturally relevant for research participants	4.60	4.83	4.83	5.00	4.80	4.80	4.83	4.67	4.80 (0.12)
Feasibility
Using the information in this session will help improve medication adherence for a young person living with HIV	3.00	4.60	4.40	4.40	4.75	4.25	4.25	4.50	4.27 (0.54)

Note. Items were rated on a 5-point Likert scale across 7 team members. *Evaluated by only 6 team members, 1 rated Not Applicable.

2.3. Phase III: Refinement of the ACCESS II web-based platform

User metrics revealed that study team members previewed each of the eight sessions for an average of 15 min and 45 s, with a range of 12 min and 34 s (Session 8) to 19 min and 26 s (Session 5). All users reported one log-in attempt, with no reported challenges related to access. Length of each ACCESS II web-based platform session was rated as “just right” by nearly all study team members (n = 6, 85.71%). One study team member rated sessions 1, 4, and 8 as “too short” while two study team members rated sessions 5 and 6 as “too long.”

Overall usability of the ACCESS II web-based platform was rated high, including ease of use with a smartphone to navigate the platform and view the written content/graphics and videos. However, study team members suggested the content was best viewed in landscape, rather than portrait mode, particularly when viewing graphics and videos. In addition, we implemented the study team's recommendation to increase the text size.

Assessment of content validity showed an excellent fit between the ACCESS II web-based platform session content (including language, graphics, videos) and session objectives (as outlined in the written protocols). Based on reviewer feedback, we removed content that the study team members perceived as repetitive. Our assessment of content validity also included evaluation of the developmental and cultural relevance of the ACCESS II web-based platform (e.g., appropriate representation in content, graphics, and videos for HIV+ AYA). Ratings showed that the selected content and graphics provided excellent representation, and were developmentally appropriate and culturally relevant for HIV+ AYA. However, subjective reviewer feedback from our peers resulted in the replacement of some videos and illustrations for greater representation of people of color and young adults. We were also alerted to the presence of a microaggression (i.e., actions that unintentionally or unconsciously reflects a prejudiced attitude) in a video entitled, “Anyone could become addicted to drugs” (video no longer available on YouTube). This video was omitted from the web-based platform because it showed a male person of color selling substances to a white female. Overall, study team members rated the ACCESS II web-based platform content as having the potential to improve ART adherence among HIV+ AYA.

In summary, all recommended changes were discussed and categorized into two primary categories: (1) format and (2) cultural and developmental considerations. Format recommendations included increased text size to improve readability on a study-funded smartphone. The narrative content was simplified to create a cohesive message, removing unnecessary or repetitive language, and to maintain session length between 20 and 30 min. We also ensured that the language was consistent across all 8 sessions of the ACCESS II web-based platform to avoid any confusion of frequently used terms (e.g., ART adherence, HIV care). This recursive and extensive evaluation process from January to April 2022, resulted in eight developmentally and culturally tailored ACCESS II web-based sessions for the control arm.

2.4. Phase IV: Implementation and pilot testing with ACCESS II study participants

2.4.1. Study implementation procedures

The ACCESS II study protocol and related study procedures are approved by a single Institutional Review Board (sIRB) at a large metropolitan academic medical center. Informed consent was obtained electronically by the project manager using Research Electronic Data Capture (REDCap) [58,59].

The first step of the implementation phase was for a member of the university IT team to download the Rise app [60,61] on the home screen of study-funded smartphones. Since participant randomization was not known at the time of study-funded smartphone distribution, all smartphones were enrolled as “learners” in the Rise app. To maintain participant confidentiality and securely log into the Rise app as a “learner,” an email (i.e., username) and password was created for each participant. This was completed using the “Gmail task-specific email addresses” feature (i.e., username+anynumber@gmail.com) [62]. These emails were directed to the ACCESS II project manager who has a university-based email protected by dual factor authentication. Upon completion of baseline testing, participants randomized to the ACCESS II web-based platform received their specific username and password with instructions to access and view the sessions using the Rise app. Participants randomized to the peer intervention arm were never provided a username and password to access to the Rise app.

Study participant enrollment began in February 2022 using the following inclusion criteria: English speaking, HIV seropositive (perinatally or behaviorally infected) individuals, ages 18–29 years, currently prescribed ART regimen, and evidence of virologic failure (HIV plasma viral load >200 copies/ml). Exclusion criteria included neurocognitive deficits that would impede participation in videoconferencing sessions or completion of study measures. Participants randomized to the control arm were enrolled in one session per week for a total of eight weeks and received a text message reminder prior to each new session enrollment. Each session was accessible for the duration of their participation, and participants were enrolled in subsequent sessions, irrespective of completion of the previous session.

2.4.2. Pilot testing

In addition to an evaluation of the web-based platform by study team members (see 2.2 Phase II: Evaluation), we conducted a pilot test to evaluate participant satisfaction and perceived ease of use of the ACCESS II web-based platform with the first ten participants enrolled in the control arm (i.e., initial feasibility group). Testing was conducted with the Client Satisfaction/Ease of Use tool, a 14-item tool rated on a 4-point Likert scale with higher scores indicating greater satisfaction and ease of use (Appendix C). This tool was developed by modifying items from two existing measures – Davis [63] and Larsen [64] – so that the language was applicable to the content of the ACCESS II web-based platform. Participants were asked to complete the Client Satisfaction/Ease of Use tool at the end of session 8 using an embedded REDCap link.

3. Results

3.1. Participant satisfaction and perceived ease of use

As shown in Table 3, we present participant satisfaction and ease of use data for our initial feasibility group, which was comprised of 10 of the first 11 participants who completed all eight sessions. One participant did not complete the Client Satisfaction/Ease of Use tool, and therefore their data was not included. All ten participants (mean age of 24.2 years (SD = 2.86) were perinatally HIV-infected, and identified as Black/African American (80%), Hispanic/Latine (30%), and female (70%).

Table 3.

Participant satisfaction and ease of use (N = 10).

Measure Items	Mean (SD)
Participant Satisfaction
Quality of support and education	3.9 (0.32)
Type of support and education desired	3.7 (0.67)
Support and education met needs	3.1 (0.74)
Recommend to a friend	3.9 (0.32)
Amount of support and education	3.5 (0.53)
Helped to handle medication adherence	3.6 (0.70)
Overall Satisfaction	3.4 (0.70)
Willingness to use again	3.5 (0.53)
Perceived Ease of Use
Ease in learning to operate	3.8 (0.42)
Easy in getting what I want it to do	3.7 (0.48)
Clear and understandable	3.8 (0.42)
Flexible to interact with	3.8 (0.42)
Easy to become skillful	3.6 (0.52)
Overall ease of use	3.8 (0.42)

Note. Items were rated on a 4-point Likert scale (1–4).

Participants (n = 10) were generally satisfied with the support and education provided by the ACCESS II web-based platform (M = 3.4; SD = 0.70), including the quality (M = 3.9; SD = 0.32), type (M = 3.7; SD = 0.67) and amount of support and education received (M = 3.5; SD = 0.53). Participants found the ACCESS II web-based platform easy to use (M = 3.8; SD = 0.42), operate (M = 3.8; SD = 0.42), and navigate (i.e., get it to do what they wanted it to do) (M = 3.7; SD = 0.48). Overall, participants reported that the ACCESS II web-based platform was clear and understandable (M = 3.8; SD = 0.42) and they would recommend the platform to a friend (M = 3.9; SD = 0.32). The average time spent per session was 6.5 min (range 2.5–15.3 min) with the most time spent on sessions covering the topics of HIV knowledge, HIV stigma/disclosure concerns, and managing HIV stigma/disclosure concerns.

4. Discussion and conclusion

4.1. Discussion

We describe a protocol for the methodological development, systematic evaluation, refinement, implementation and initial pilot testing of a web-based platform (control arm) in an RCT designed to support ART adherence among HIV+ AYA. Our findings support the usability, validity, acceptability, and feasibility of the ACCESS II web-based platform. The content was developmentally appropriate and culturally relevant, highlighting the importance of engaging members of the community (i.e., peers) in all stages of development. Forthcoming findings from ACCESS II and other active trials [40] will provide important evidence regarding the potential efficacy of web-based platforms and mHealth interventions in addressing ART adherence and HIV prevention. Additionally, through the development of a robust control arm, final results of the ACCESS II trial will help to better elucidate the role of the peer for ART adherence support. Such evidence has important implications for peer training and implementation of peer-led interventions, which require resources for staff and funding [65].

User engagement is a critical component to evaluate the success or failure of any mHealth intervention; therefore, we recommend that study teams carefully consider their methods and tools to evaluate user engagement (e.g., metrics of duration, frequency, and depth of intervention use) when designing mHealth interventions [[66], [67], [68]]. Although we were able to collect important user metrics, we were not able to measure participant level of interaction with the questions posed for reflection on the platform (e.g., experiences with HIV stigma, and ART adherence). At present, the data for level of engagement is limited to session completion and time spent per session. Therefore, future mHealth interventions might consider the option to include a free text form for participant reflection.

Many considerations went into the development and implementation of the ACCESS II web-based platform, including how to best maintain participant confidentiality while ensuring ease of access to the platform. We therefore consulted with university-based IT and Rise content experts to ensure that participant access was secure and confidential. Since Rise requires an email to generate and access accounts, our initial plan was to use vtext (Verizon Wireless: 123456789@vtext.com) and send an email invitation (via text) with the URL to the participant study-funded smartphones. However, vtext only supports 140 characters per text, and the session invitation URL exceeded this limit. Ultimately, per the suggestion of the university-based IT department, “Gmail task-specific email addresses” [62] were generated, which allowed for all study participants to be enrolled from a single (and confidential) email address.

In the selection of graphics/videos for the ACCESS II sessions (phase I), we encountered a limited selection of developmentally and culturally appropriate images and educational videos for HIV+ AYA. Peers provided essential feedback on the cultural and developmental fit of all educational content, including audiovisuals, and graphic illustrations. Regarding the inclusion of videos in a web-based platform, we recommend that study teams embed video links to prevent scenarios in which participants need to exit and re-enter the app to view the content.

Our experience lends support for the implementation and testing of web-based platforms in ART adherence interventions for HIV+ AYA, compared to other mHealth technologies [[28], [29], [30], [31]]. Other recent studies include a multi-site RCT efficacy trial supported by the Adolescent Trials Network (ATN 138) for HIV+ AYA. This efficacy trial compares a mobile-enhanced private social networking website with a control arm of static informational content (electronic newsletter) on the primary study outcome of HIV viral load; results are forthcoming [40].

4.2. Limitations and future considerations

We acknowledge the limitations of this work. First and foremost, the development, evaluation, refinement, implementation procedures, and pilot testing of the ACCESS II web-based platform co-occurred with the COVID-19 public health crisis. As such, the majority of this work was completed remotely and may have lengthened the development and implementation process. However, we believe that the rigorous methods described in this paper will support forthcoming findings and elucidate the role of the peer in intervention research aimed to improve ART adherence among HIV + AYA.

Considerations for the future are the costs of staff, software such as Rise (https://rise.com/), and study-funded smartphones with access to unlimited data and/or Wi-Fi (required to implement the ACCESS II web-based platform). These considerations are especially relevant for lower income settings where there is a need for mHealth supported ART adherence interventions [69,70].

4.3. Innovation

This research is responsive to current gaps, and our approach and findings extend existing paradigms by demonstrating feasibility of community engagement and representation in the science of building a web-based platform. Thus, the active collaboration of peers in the design, testing, and implementation of this web-based platform offers a model to cultivate greater inclusivity within academic settings. Team members were diverse and multi-disciplinary, leading to important dialogue and transparency of decisions made throughout the research process. This approach supported the successful implementation of the web-based platform for an RCT, a first step in minimizing the critical barriers to clinical trial participation among Black/African American and Hispanic/Latine HIV+ AYA. Implementing community-informed approaches is essential for the uptake and long-term sustainability of these interventions [71].

We are also among the first to develop and integrate a mobile peer model in the pre-COVID-19 era to support ART adherence among Black/African American and Hispanic/Latine HIV+ AYA [43]. In our current research, we are comparing this mobile peer model with the web-based platform (control arm) described in this study. The validity of RCTs is in part dependent on the rigor and reproducibility of the control arm, and can be used to elucidate the “active ingredient” of an intervention, and minimize the risk of overestimating the effect of an intervention. By applying rigorous and transparent methods in the design, testing, and implementation of this web-based platform, we aim to favorably impact health outcomes among Black/African American and Hispanic/Latine HIV+ AYA, and ultimately advance health equity. Additionally, as the intervention and control arms are technology-enabled, our approach and methods offer a prototype for addressing many of the critical barriers limiting access to participation in clinical research trials for Black/African American and Hispanic/Latine HIV+ AYA.

4.4. Conclusion

Our experience provides support for a multi-disciplinary team-based approach with content experts actively engaged in the field of HIV, including nursing, public health, medicine, psychology, informatics, and community stakeholders. Consistent with current evidence [28], we recommend integrating theory and leveraging developmental/cultural considerations when designing mHealth ART adherence interventions for HIV + AYA. The development and operationalization of a carefully formulated timeline that allows for multiple rounds of testing and sufficient resources (financial and project staff) is needed to facilitate completion of a web-based platform for implementation in a behavioral trial. Finally, we recommend the inclusion of community members and peers in all stages of research design and evaluation. Such active participation is key to developing and implementing novel interventions that are culturally tailored and engage HIV+ AYA for enhanced healthcare access and improved health outcomes.

CRediT authorship contribution statement

Ann-Margaret Dunn Navarra: Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Maurade Gormley: Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Resources, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Eva Liang: Writing – review & editing, Writing – original draft, Validation, Resources, Formal analysis, Data curation, Conceptualization. Claire Loughran: Writing – review & editing, Writing – original draft, Validation, Supervision, Project administration, Methodology, Formal analysis, Data curation. Allison Vorderstrasse: Writing – review & editing, Writing – original draft, Validation, Supervision, Methodology, Funding acquisition, Formal analysis, Conceptualization. David R. Garcia: Writing – review & editing, Writing – original draft, Methodology, Conceptualization. Michael G. Rosenberg: Writing – review & editing, Validation, Project administration, Methodology, Investigation, Funding acquisition, Conceptualization. Jason Fletcher: Writing – review & editing, Methodology, Formal analysis, Data curation. Lloyd A. Goldsamt: Writing – review & editing, Writing – original draft, Validation, Supervision, Project administration, Methodology, Investigation, Formal analysis, Data curation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Sample user interfaces of the ACCESS II web-based platform: (A) home screen of session on smartphone; (B) content of the ACCESS II web-based platform; (C) reflective questions of an ACCESS II web-based platform session

Appendix B. ACCESS II Web-Based Platform – 16-item evaluation tool for study team members

Date:
ACCESS Session Number:
Stated Objectives/Goals of Session:
Name and title of the Reviewer:

Review criteria

User Metrics:

1) Number of log-in attempts to access the session
2) Time spent to complete the session in minutes:
3) Day of week session completed:
4) Time of day session completed:
Other Comments:

Overall Usability:

*On a scale of 1–5, please only check off 1 answer per statement	Could not use access or review content	Challenging to access or review content	Fairly easy to access or review content	Easy to access or review content	Very easy to access or review content
	1	2	3	4	5
1) Ease of use with smartphone	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5
2) Ease of reviewing narrative content, graphics, videos on smartphone	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5
3) Ease of use in session navigation or moving through the different sections	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5
Other Comments:

Validity:

Match of content with session goals:

*On a scale of 1–5, N/A, please select only 1 answer per question.	Content does not fit with session goals	Poor fit between content and session goals	Fair fit between content and session goals	Good fit between content and session goals	Excellent fit between content and session goals	Not applicable to my specific content expertise
	1	2	3	4	5	N/A
1) Match of session objectives and content – (does the content match the session goals)?	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A
2) Language in session - developmentally appropriate and culturally relevant for research participants (18–29-year-old young people living with HIV)	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A
3) Graphics and videos - developmentally appropriate and culturally relevant for research participants (18–29-year-old young people living with HIV)	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A
Other Comments:

Developmental and Cultural Considerations:

Developmental and Cultural Considerations *On a scale of 1–5, N/A, please select only 1 answer per question.	Content/graphics does not represent participant developmental stage and or culture	Content/graphics poorly represents participant developmental stage and or culture	Content/graphics with fair representation of participant developmental stage and or culture	Content/graphics with good representation of participant developmental stage and or culture	Content/graphics with excellent representation of participant developmental stage and or culture	Not applicable to my specific content expertise
	1	2	3	4	5	N/A
1) Match of session objectives and content – (does the content match the session goals)?	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A
2) Language in session - developmentally appropriate and culturally relevant for research participants (18–29-year-old young people living with HIV)	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A
3) Graphics and videos - developmentally appropriate and culturally relevant for research participants (18–29-year-old young people living with HIV)	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A

Acceptability:

1) Does the length of the session seem too long, too short, or just about right?
2) Do you have any recommendations for change?
Other comments:

Feasibility:

*On a scale of 1–5, N/A, please select only 1 answer per question.	Content will not help to improve medication adherence	Content offers little help to improve medication adherence	Content offers some help to improve medication adherence	Content is helpful to improve medication adherence	Content is very helpful to improve medication adherence	Not applicable to my specific content expertise
	1	2	3	4	5	N/A
1) Using the information in this session will help improve medication adherence for a young person living with HIV.	❑ 1	❑ 2	❑ 3	❑ 4	❑ 5	❑ N/A

Appendix C. Participant satisfaction/ ease of use tool

• 1.
  How would you rate the quality of the support and education you received from the ACCESS II web-based platform?

  • a.
    Excellent
  • b.
    Good
  • c.
    Fair
  • d.
    Poor
• 2.
  Did you get the kind of support and education you wanted from the ACCESS II web-based platform?

  • a.
    No, definitely not
  • b.
    No, not really
  • c.
    Yes, generally
  • d.
    Yes, definitely
• 3.
  To what extent has the support and education from the ACCESS II web-based platform met your needs?

  • a.
    Almost all of me needs have been met
  • b.
    My needs have been met
  • c.
    Only a few of my needs have been met
  • d.
    None of my needs have been met
• 4.
  Would you recommend the ACCESS II web-based platform to a friend?

  • a.
    No, definitely not
  • b.
    No, I don't think so
  • c.
    Yes, I think so
  • d.
    Yes, definitely
• 5.
  How satisfied are you with the amount of support and education you received from the ACCESS II web-based platform?

  • a.
    Quite dissatisfied
  • b.
    Indifferent or mildly dissatisfied
  • c.
    Mostly Satisfied
  • d.
    Very satisfied
• 6.
  Have the support and education you received from the ACCESS II web-based platform helped you to handle medication adherence more effectively?

  • a.
    Yes, they helped a great deal
  • b.
    Yes, they helped somewhat
  • c.
    No, they really didn't help
  • d.
    No, they seemed to make things worse
• 7.
  In an overall, general sense, how satisfied are you with the support and education you received from the ACCESS II web-based platform?

  • a.
    Very satisfied
  • b.
    Mostly satisfied
  • c.
    Indifferent or mildly dissatisfied
  • d.
    Quite dissatisfied
• 8.
  If you were to seek help again, would you come back to the ACCESS II web-based platform?

  • a.
    No, definitely not
  • b.
    No, I don't think so
  • c.
    Yes, I think so
  • d.
    Yes, definitely

Perceived Ease of Use

• 9.
  Learning to operate the ACCESS II web-based platform was easy for me.

  • a.
    Strongly disagree
  • b.
    Disagree
  • c.
    Agree
  • d.
    Strongly agree
• 10.
  I found it easy to get the ACCESS II web-based platform to do what I want it to do.

  • a.
    Strongly disagree
  • b.
    Disagree
  • c.
    Agree
  • d.
    Strongly agree
• 11.
  My interaction with the ACCESS II web-based platform was clear and understandable.

  • a.
    Strongly disagree
  • b.
    Disagree
  • c.
    Agree
  • d.
    Strongly agree
• 12.
  I found the ACCESS II web-based platform to be flexible to interact with.

  • a.
    Strongly disagree
  • b.
    Disagree
  • c.
    Agree
  • d.
    Strongly agree
• 13.
  It was easy for me to become skillful at using the ACCESS II web-based platform.

  • a.
    Strongly disagree
  • b.
    Disagree
  • c.
    Agree
  • d.
    Strongly agree
• 14.
  I found the ACCESS II web-based platform easy to use.

  • a.
    Strongly disagree
  • b.
    Disagree
  • c.
    Agree
  • d.
    Strongly agree