A Pilot Study of a Mobile App to Support HIV Antiretroviral Therapy Adherence among Men who have Sex with Men who Use Stimulants

Abstract

APP+ is a theoretically-grounded mobile app intervention to improve antiretroviral (ART) adherence among men who have sex with men (MSM) who use stimulants. We assessed the feasibility and acceptability of APP+ in a six-month randomized controlled trial among a national sample of 90 MSM recruited online; secondarily, we examined changes in self-reported ART adherence by study arm. Retention at the final assessment was 82%, and acceptability ratings were comparable to other technology-based interventions. MSM in the APP+ group reported higher self-reported percentage ART adherence in the past 30 days at the four-month timepoint compared to a no-treatment control group (89.0% vs 77.2%). However, once access to the app was removed after month four, group differences in ART adherence diminished by month six. APP+ may be a potentially promising intervention approach for MSM living with HIV who use stimulants but would require enhancements to optimize acceptability and demonstrate more sustained effects.

INTRODUCTION

Human immunodeficiency virus (HIV) rates in the US, for the recent years where data are available, have declined slightly from 13.1 new HIV diagnoses per 100,000 in 2012 to 11.8 in 2017 (1). Despite decreasing HIV infection rates among injection drug users, women, and heterosexual men, infection rates among men who have sex with men (MSM) remained stable (1). In 2017, 70% of all new HIV infections in the US are attributed to male-to-male sexual contact (including 3% attributed to male-to-male sexual contact and injection drug use) (1). Opportunities to improve HIV treatment outcomes for MSM diagnosed with HIV are evident, particularly for Black MSM (2). Novel behavioral strategies to mitigate barriers to optimal HIV care are needed.

Illicit drug use is consistently and strongly associated with increased risk for HIV, Hepatitis C (HCV), and other sexually transmitted infections (STIs) (3–5). Drug use potentiates HIV risk directly through blood exchange that may occur during injection drug use or indirectly through psychological processes (e.g., myopia, disinhibition, escapism) operating within sexual encounters involving injection and non-injection drug use (6–8). A nationally-representative sample of MSM in the US (n=10,217) (9) showed high rates of marijuana (25% of MSM living with HIV and 23% of HIV-negative/unknown serostatus men) and other (29% of MSM living with HIV and 18% of HIV-negative/unknown serostatus men) illicit substance use.

Stimulant use - which includes use of methamphetamines, cocaine, ecstasy, and other substances that have stimulant properties (10) - remains high among MSM. Studies show continued high rates of methamphetamine use among MSM. For example, from 2008–2011, methamphetamine use among MSM in Los Angeles ranged from 23% to 27%, with a mean of 25% (11). MSM living with HIV are more likely to report methamphetamine use than HIV-negative MSM, with one study showing that men living with HIV residing in Miami, Florida reported higher methamphetamine use (32%) than HIV-negative respondents (15%) (12). A study in Los Angeles, California showed that the mean days of methamphetamine use in the past 30 days among MSM living with HIV was 5.7 (v. 1.8 for HIV-negative MSM), compared to 0.3 days of cocaine use (v. 0.2 among HIV-negative MSM) and 0.2 days of ecstasy use (v. 0.2 among HIV-negative MSM) (13).

Stimulant use and HIV/AIDS are intertwining epidemics among MSM. In HIV-negative MSM, stimulant use consistently predicts a markedly increased risk for HIV seroconversion (6, 14). Among MSM living with HIV/AIDS, stimulant users are more likely to experience difficulties with ART adherence and persistence that contribute to difficulties with achieving and maintaining viral suppression that amplify risk of onward HIV transmission (15–17). The clinical relevance of these difficulties with HIV disease management is further supported by multiple cohort studies demonstrating that stimulant users display hastened clinical HIV progression (18–20). For these reasons, novel interventions to address the unique HIV treatment and care needs of MSM living with HIV who use stimulant drugs are needed.

There are 14 good or best evidence medication adherence behavioral interventions,(21) although only four are tailored to the unique needs of MSM (22–25) and only two are tailored to people who use drugs (26, 27), with no overlap. Both evidence-based medication adherence interventions for drug users involve time and resource intensive directly administered antiretroviral therapy (DAART). Adherence interventions for MSM include one in-person individual intervention (23), one couples-based intervention (25), a provider-delivered intervention (24), and an in-clinic computerized intervention (22). These interventions, all delivered in-person or in the clinic, can be costly and hard to deliver to hidden or stigmatized drug using populations. In-person access to these interventions can be restrictive with respect to the time and location of their delivery.

mHealth (i.e., mobile health technologies that leverage mobile applications [apps] and text [SMS] messaging) and other technologies open new opportunities for intervention delivery through a growing number of devices (e.g., laptops, tablets, mobile phones) and channels (e.g., online, mobile apps, SMS, and social networking platforms) to overcome the limitations of in-person and clinic-based interventions (28). A review (January 2016-March 2017) of technology-based interventions for drug-using persons (29) showed that three of the eight interventions addressed either ART adherence or engagement in HIV care (30–32). None of the three interventions for drug users were specifically tailored to MSM. A review of mHealth ART adherence interventions for MSM conducted between January 2016 to May 2017 found six mHealth interventions, although none were specifically tailored for substance users (33).

Based on the unique challenges (e.g., sexual minority stigma and stimulant-related nonadherence [19, 34, 35]) that MSM living with HIV who use stimulants experience in achieving optimal ART adherence, the potential for technology-based interventions to reach this stigmatized group, and the lack of mHealth interventions tailored to this high-risk population, we developed the “APP+” mobile app intervention and pilot tested it in a pilot randomized controlled trial (RCT) with 90 MSM living with HIV who use stimulants. The primary objectives of this study were to examine the feasibility and acceptability of the APP+ mobile app; secondarily, we assessed whether use of the APP+ mobile app was associated with changes in self-reported ART adherence and stimulant use.

METHODS

Procedures and Participants

An RCT of the APP+ intervention was conducted between August 2016 and March 2017. The target sample size of 90 participants was established to ensure that there were at least 30 participants per group at the end of the study after attrition. The majority of participants (87%) were recruited with 10 ads on Grindr, a widely-used mobile app for men seeking other men from August 10, 2016 to September 10, 2016. Each ad was viewable for a 24-hour period to persons who opened the app during that timeframe in the following cities: Miami, Orlando, Washington DC, Charlotte, Houston, and New Orleans. The remaining participants (13%) were recruited through other recruitment methods including flyers and palm cards at various community-based organizations and clinics that served the targeted population - such as The StoneWall Project in San Francisco, California and the Red Door Clinic in Minneapolis, Minnesota – and posted information about the study on their social media websites (Facebook).

Those interested in the study were required to complete an online screening survey to determine eligibility for the study (Figure 1). Self-reported eligibility criteria were: (1) male gender; (2) sex with a man in the past five years; (3) US residency; (4) diagnosis of HIV by a physician or healthcare provider; (5) currently taking HIV medication; (6) reporting less than excellent (on a 6-point scale from “very poor” to “excellent”) on the question “In the last 30 days, how good a job did you do at taking your HIV medications in the way you were supposed to?” (36); (7) reporting any use of illicit stimulant drugs, including methamphetamines, powder or crack cocaine, ecstasy, and/or amphetamines in the past 6 months; and (8) having an iPhone or Android mobile phone.

Figure 1. APP+ Study Enrollment and Retention.

* % of individuals who had downloaded APP+ at each follow up: Baseline (n = 34; 75.6%); 2 Month Follow-Up (n = 33; 78.6%); 4 Month Follow-Up (n = 32; 78.0%); 6 Month Follow-Up (n = 29; 76.3%)

Once men were determined to be eligible for the study, they were sent an e-mail to complete the baseline survey. Completion of the baseline survey indicated that the participant was enrolled in the study. We used a spreadsheet developed by the study statistician (RM) prior to the study launch to block randomize participants in a 1:1 allocation to intervention (the APP+ mobile app) arm or the control arm. As each participant was enrolled, the study coordinator (TD) unmasked the cell that indicated study arm assignment, as well as the unique study identification number.

Participants randomized to the control condition were not provided with any intervention activities, and were sent an e-mail to let them know that they would receive an email when it was time for them to complete the next assessment in approximately two months. We chose a no treatment intervention for the control condition because this was an experimental study of which the benefits of the app were unknown, many men received clinical care through their HIV clinic, and the no treatment control maximized the opportunity to detect possible intervention effects. Men randomized to the APP+ intervention arm were sent an e-mail that provided a link to study webpages that described how to download the app from either the Apple App store or from Google Play store, a description of the features of the mobile app (see below for a description), and trouble-shooting advice. Contained in that same e-mail was the participant’s unique study ID number and authorization code. Once participants downloaded the app from the app store and opened it for the first time, they were required to enter in the study ID number and authorization code to access APP+ contents. The first screen participants encountered prompted them to enter a unique 4-digit personal identification number (PIN) that they used to open APP+ from that point forward. To protect privacy, participants were required to enter their 4-digit PIN each time they opened APP+. Next, participants were shown multiple instruction screens for how to set up and change their medication dosing information and dose timing. Men were given access to the APP+ mobile app to use in any way they wanted for up to 4 months after they entered their study ID number and authorization code, but no longer than 4-months. Participants completed follow-up assessments at 2-, 4- and 6-month time points after the baseline assessment. Remuneration amounts for each survey was set at $50. All study procedures were approved by the University of Minnesota Institutional Review Board.

Intervention Description

The APP+ mobile app had three main components. First, when the app was opened, users viewed a homepage that contains up to five new pieces of IMB-based HIV or ART adherence content each day which were presented as vertical scrollable cards (Figure 2). The IMB-based HIV and ART adherence content was developed by study staff to address informational, motivational, and behavioral skills barriers the following categories: a) adherence side effects; b) adherence drug interactions; c) adherence and mental health; d) motivational messages to adhere; and e) adherence strategies. In addition, cards were developed that reflected general issues of living with HIV and tips on how to use the APP+ mobile app. In total, 286 cards were developed and presented over the course of the active intervention period.

Figure 2.

Screenshot of APP+ Homepage with Information-Motivation-Behavioral Skills HIV and ART information

Second, study staff developed a story line of a fictional character (named “JR”) who is living with HIV, uses substances, and is sexually active. Under the CYOA (“choose your own adventure”) tab, a small piece of JR’s storyline was presented to participants and they were asked to choose one of two options for how to proceed. The next installment of the storyline was available six hours after the participant made his response. Storylines were created around six scenarios: 1) having a “party lifestyle” that may interfere with adherence; 2) mixing drugs and HIV medications; 3) medication side effects; 4) depression and adherence; 5) forgetting to take HIV medication; and 6) not having HIV medications easily available. All storylines contained specific medication adherence tips, such as “Some people think it’s safer to skip their HIV meds when they use blow. Mixing the two can have side effects, but it is still better to take your medication when you are using recreational drugs. One thing you can do is to take your ART before you go out for the night or take your medication a couple hours before your regular time. Then the HIV meds will have time to process and get in your system before you start using. Another thing you can do is reduce the amount of blow or meth you do that night.” Participants who were not comfortable with the content of JR’s storyline or did not want to view the storyline were given the option to go directly to the medication adherence tips.

Third, the “My Meds” tab provided a way for men to self-monitor whether or not they took a dose of their medication (Figure 3). Using information that participants inputted on the number of daily doses of HIV medication they were prescribed, a card appeared for each scheduled dose that provided a way for participants to report whether they took that dose or “skipped it.” Men could retrospectively report past doses for the previous 3 days. Doses that were reported were displayed on the “Medication Weekly Challenge” (Figure 4) where participants viewed a graph of how many doses they reported taking both last week and the current week. The graph was color coded such that the bar was green if 90% or more of reported doses were taken, yellow if 80–89% of reported doses were taken, and red if less than 80% of reported doses were taken. Under each graph, the number of doses reported and doses taken and percentage of doses taken (from those reported) were displayed.

Figure 3.

Screenshot of ART Self-Monitoring Component

Figure 4.

Screenshot of Medication Weekly Challenge

Measures

ART medication use and adherence

Participants were asked to report the number of daily doses of their HIV medications they were prescribed. Responses were grouped into one of three categories: a) one dose per day; b) two doses per day; and c) three or more doses a day.

Based on best practices for measuring self-reported ART adherence (37) and a study showing strong associations between similar self-report measures and more objective ART adherence measures (38), three primary self-reported adherence outcomes were assessed for the purpose of this study using the Visual Analog Scale (VAS): a) the percentage of time ART was correctly taken as prescribed in the past 30 days; b) the percentage of time ART was taken within two hours of the scheduled dose in the past 30 days; and c) the percentage of time ART was taken while using stimulant drugs in the past 30 days. For each item, participants responded using a pull-down menu with response options that ranged from 0 to 100 in 1-point increments. In addition, one item was taken from an existing adherence measure developed by Wilson and his colleagues (36): How often have you taken your doses correctly in the past 30 days? (response options: Never, rarely, sometimes, usually, almost always, and always).

Demographic characteristics

Demographic characteristics included age (in years); how long participants had been living with HIV (in years); education (high school diploma; some college or associates degree; or college or post graduate work); race (White, Black, Other); ethnicity (Hispanic identity); sexual orientation (gay/homosexual or other); current employment status (full-time work, part-time work, disabled, unemployed, retired); and whether the participant was currently in school (Table I).

Table I.

Descriptive characteristics of the APP+ intervention and control arm participants

Variable	Total (n = 90)	Control (n = 45)	Intervention (n = 45)
Age (years), mean (sd)	37.4 (9.9)	38.6 (10.2)	36.2 (9.6)
Race Ethnicity, n (%)
White, Non-Hispanic	40 (44.4)	21 (46.7)	19 (42.2)
White, Hispanic	15 (16.7)	5 (11.1)	10 (22.2)
African American, Non-Hispanic	22 (24.4)	10 (22.2)	12 (26.7)
African American, Hispanic	1 (1.1)	1 (2.2)	0 (0)
Other, Non-Hispanic	7 (7.8)	4 (8.9)	3 (6.7)
Other, Hispanic	5 (5.6)	4 (8.9)	1 (2.2)
Education, n (%)
High School Diploma	14 (15.6)	10 (22.2)	4 (8.9)
Some College, Associates Degree	45 (50.0)	23 (51.1)	22 (48.9)
College, Post Graduate	31 (34.4)	12 (26.7)	19 (42.2)
Employment Status, n (%)
Full-Time Work	45 (50.0)	17 (37.8)	28 (62.2)
Part-Time Work	16 (17.8)	10 (22.2)	6 (13.3)
Unemployed	20 (22.2)	12 (26.7)	8 (17.8)
Other (Retired, Disabled)	9 (10.0)	6 (13.3)	3 (6.7)
Sexual Orientation, n (%)
Homosexual / Gay	80 (88.9)	39 (86.7)	41 (91.1)
Years since HIV Diagnosis, mean (sd)	8.3 (7.1)	7.9 (6.5)	8.6 (7.8)
Baseline Adherence, mean (sd)	80.0 (27.5)	78.1 (29.6)	81.8 (25.4)
Doses Per Day*, n (%)
Once a Day	84 (93.3)	43 (95.6)	41 (91.1)
Twice a Day	5 (5.6)	1 (2.2)	4 (8.9)
Three + a Day	1 (1.1)	1 (2.2)	0 (0)
IMB-AAQ (Mean)
Information	4.0 (0.6)	4.0 (0.6)	3.9 (0.6)
Motivation	3.3 (0.7)	3.4 (0.7)	3.1 (0.7)
Behavioral Skills	3.5 (0.7)	3.5 (0.7)	3.4 (0.7)
AUDIT Score, mean (sd)	7.7 (6.9)	7.0 (7.1)	8.4 (6.8)
CES-Depression 10 Score, mean (sd)	12.6 (7.0)	11.1 (6.4)	14.2 (7.3)
Depressive Symptoms, n (%)	61 (67.8)	27 (60.0)	34 (75.6)
HIV Stigma Measures, mean (sd)
Internalized HIV Stigma	2.4 (1.2)	2.3 (1.2)	2.5 (1.1)
Anticipated HIV Stigma	2.3 (1.0)	2.3 (0.9)	2.3 (1.1)
Enacted HIV Stigma	1.7 (0.8)	1.7 (0.8)	1.6 (0.8)
Life Chaos Scale Score, mean (sd)	18.1 (2.8)	17.8 (2.8)	18.4 (2.9)

^*A dose is considered one or more HIV medication taken at the same time.

Psychosocial characteristics

Depressive symptoms were measured with the 10-item Center for Epidemiologic Studies – Depression Scale (CES-D; 39), a widely used measure of depression in research studies (alpha=0.87 for this sample). A score of 10 or higher (range 0–30) suggests that significant depressive symptoms may be evident (40). The Life Chaos Scale is a 6-item measure (item range = 1–5) of whether someone has a stable and predictable lifestyle and has been shown to be psychometrically adequate among people living with HIV (PLWH) in a prior study (alpha=0.71 for this sample; 41). HIV stigma was assessed using the stigma scale developed by Earnshaw and Chaudoir (2009). Designed to measure HIV stigma mechanisms defined by the HIV Stigma Framework, the measure includes 3 subscales; Internalized HIV stigma (alpha = 0.94 for this sample), Anticipated HIV stigma (alpha = 0.92), and Enacted HIV stigma (alpha = 0.92 for this sample; 42). Items are rated on a 5-point Likert-type scale with higher scores indicating greater stigma.

The Alcohol Use Disorders Identification Test (AUDIT; 43) was used to determine whether participants were at risk for alcohol dependency or hazardous alcohol consumption. Prior research showed that 92% of persons who were diagnosed as having harmful or hazardous alcohol use scored 8 or more on the AUDIT (the cut-off score for harmful and hazardous alcohol use; 43). In addition, participants were asked if they had used the following illicit drugs in the past 2 months: marijuana, pain killers purchased on the street, downers, powder cocaine, crack cocaine, amphetamines, methamphetamines, GHB, ketamine, ecstasy, heroin, speedballs (heroin and cocaine mixed together), and hallucinogens. Recent stimulant use was defined as use of methamphetamine, powder cocaine, ecstasy, crack cocaine, or amphetamine in the past 2 months.

Information, Motivation, and Behavioral Skills Adherence Barriers

To assess theoretically-derived ART adherence strengths and barriers, the information and motivations scales from the Information, Motivation, and Behavioral Skills ART Adherence Questionnaire (IMB-AAQ; 44) was completed by participants. The IMB-AAQ assess adherence-related information (9 items; alpha = 0.69 for this sample), personal and social motivation (10-items; alpha = 0.74 for this sample), and behavioral skills (14-items; alpha = 0.86 for this sample) on a 1–5 Likert scale.

Intervention navigation and acceptability

Participants in the APP+ intervention arm completed the System Usability Scale (SUS; 45). The SUS is a 10-item measure that asks users to rate on a one (strongly disagree) to five (strongly agree) scale how much they agree with statements about the ease with which they were able to navigate the intervention (e.g., “I found APP+ unnecessarily complex”; “I found the various functions in APP+ to be very well integrated.”). The SUS has been used in over 500 studies to assess intervention usability, with an mean score of 68 across studies (45, 46). In addition to administering the SUS, men were asked to provide qualitative feedback on the following: 1) three things they liked best about the APP+ mobile app; 2) three things they liked least about the APP+ mobile app; and 3) any suggestions for features or functions that should be added to the APP+ mobile app to make it a better experience.

Analysis Plan

We defined intervention feasibility as the percentage of participants who were recruited and retained at each of the four assessment points (47). Based on prior reviews of retention outcomes in clinical trials (48, 49), a benchmark of 80% or higher retention was set as indicating adequate feasibility. Demographic characteristics, psychosocial characteristics, recent drug use, and ART adherence outcomes were summarized using descriptive statistics (means and percentages). A SUS score of 68 or above was set as indicating satisfactory acceptability based on a review of SUS scores from 500 technology-based intervention studies (45). Intervention acceptability and APP+ intervention utilization were summarized using descriptive statistics.

Study participants randomized to the APP+ interventions were asked to report up to three features they liked best and three features they liked least, as well as up to three features they would like added to the APP+ intervention. We report themes in which at least 25% of participants who downloaded the app) provided responses reflecting that theme. Two independent reviewers identified the major themes of the responses and coded each response. Discrepancies between the two independent reviewers were discussed with a third reviewer and finalized.

All available participant data were included in the analyses of APP+ use on self-reported ART adherence regardless of the level of engagement with the app or survey completion. Differences in past 30-day adherence at the 2, 4, and 6-month follow-ups between the APP+ intervention group and control group were estimated using t-tests. Response options to the question “How often have you taken your doses correctly in the past 30 days?” were dichotomized into (1) Almost Always / Always or (2) Never / Rarely / Sometimes / Usually; difference between intervention arms at 2, 4 and 6-month follow-up were assessed using chi-square tests. Differences in recent stimulant use were compared using chi-square tests at each of the assessment periods. Sensitivity analyses (t-tests) were performed among the subgroup of the intervention participants who downloaded the APP+ app compared to those randomized to the control group at each of the study follow-ups.

Statistical analyses were performed using STATA version 14. No adjustments were made for multiple comparisons.

RESULTS

Study Enrollment and Retention

Through all recruitment methods, 6,380 men were screened for eligibility, 134 met inclusion criteria and expressed interest in the study, and 90 individuals consented and were randomized to the APP+ intervention (n=45) or no treatment control arm (n=45; Figure 1). Overall, among this sample of MSM who use stimulants, retention at the final 6-month assessment timepoint was acceptable (n = 74; 82%): 38 (84%) completed the 6-month assessment among the intervention arm and 36 (80%) completed the assessment in the control arm. There were no differences in retention between the intervention and control groups over the 6-month follow-up period.

Demographic Characteristics

Baseline demographic characteristics for the total sample and by intervention and control arm are presented in Table I. The mean age of participants was 37 and participants had been living with HIV for eight years. The majority of men were persons of color, with nearly one-quarter reporting that they are African American, non-Hispanic (24%) or Hispanic/Latino (23%). This was a highly educated sample, with more than four-fifths having some college education; however, half of all men were not fully employed. Nearly one-quarter of study participants (24%) resided in Florida and nearly 20% resided in Texas (18%) and most participants (91%) reported living in a medium or large city (50,000+ people; not shown in Table I). Men in the APP+ intervention arm reported lower IMB-AAQ motivation scores (3.1 vs 3.4) and higher depression scores on the CES-D (14.2 vs 11.1) at the baseline assessment.

Recent drug use (past two months) was assessed at baseline and presented in Table II. Use of methamphetamines (60%), marijuana (60%), and poppers (60%) was high among this population. At baseline, 82.2% reported any stimulant use in the past two months. Less than 10% of study participants reported recent use of downers, ketamine, heroin, speedball, or hallucinogens (data not shown). There were no substantively meaningful differences in recent drug use between the intervention and control arms.

Table II.

Baseline Self-Reported Drug Use in Past 2 Months

Drug Use	Total (n = 90)	Control (n = 45)	Intervention (n = 45)
Any Stimulant Use*	74 (82.2)	38 (84.4)	36 (80.0)
Specific Stimulant Use
Methamphetamine	54 (60.0)	28 (62.2)	26 (57.8)
Powder Cocaine	20 (22.2)	11 (24.4)	9 (20.0)
Ecstasy	12 (13.3)	5 (11.1)	7 (15.6)
Crack cocaine	2 (2.2)	1 (2.2)	1 (2.2)
Amphetamine	14 (15.6)	7 (156)	7 (15.6)
Marijuana Use	54 (60.0)	24 (53.3)	30 (66.7)
Poppers Use	54 (60.0)	23 (51.1)	31 (68.9)
GHB	32 (35.6)	18 (40.0)	14 (31.1)
Pain Killers	18 (20.0)	7 (15.6)	11 (24.4)

^*Use of one following stimulants: methamphetamine, powder cocaine, ecstasy, crack cocaine, amphetamine

APP+ Intervention Use and Acceptability

Participants had access to the 4-month intervention for a median of 110 days (IQR: 102 – 115 days of access). Among the 45 individuals randomized to receive the APP+ intervention, 34 (76%) downloaded the app from the app stores and logged into it; 11 participants did not download the APP+ mobile app. APP+ was opened 2,444 times among the 34 participants who downloaded it during the intervention period (mean = 71.9; median = 13; IQR: 8 – 124). Upon opening the app, men viewed IMB-based HIV and ART adherence content (Figure 2) on the home page. Although we did not track whether an individual piece of HIV and ART informational content was viewed, the high number of app openings suggests that IMB-based HIV and ART adherence content was viewed often. Twenty-one intervention participants utilized the Choose Your Own Adventure component a total of 268 times (mean = 12.8; Range: 1 – 80). Twenty-nine participants tracked their medication at least once over the active intervention period. Participants tracked 61.7 individual medication events (sd = 48.5; range: 2 – 196); 58.1 medications were reported as taken and 3.6 medications were reported as missed.

Intervention acceptability results are reported only for the 31 participants who were randomized to the APP+ intervention, downloaded the app, and completed the 2-Month intervention survey (Table III). Overall, the mean SUS score for APP+ was 68.2 (SD = 19.6), which is the mean of SUS scores of 500 technology-based intervention studies (45). More than three-quarters of the study participants agreed or strongly agreed that the APP+ was easy to use (77%) and that most people would learn how to use APP+ very easily (77%). Nearly 81% and 74% disagreed or strongly disagreed with the statement that they would need the support of a technical person to use APP+ and that they would need to learn a lot of things before they could use APP+, respectively.

Table III.

Acceptability of the APP+ intervention among Men who Downloaded the APP (n=31^a)

System Usability Scale Component*	Strongly Disagree / Disagree	Neither Agree or Disagree	Strongly Agree / Agree
Would use App+ Frequently	8 (25.8)	7 (22.6)	16 (51.6)
Found App+ difficult to use	18 (58.1)	8 (25.8)	5 (16.1)
App+ was easy to use	2 (6.5)	5 (16.1)	24 (77.4)
Would need support of technical person to use App+	25 (80.6)	4 (12.9)	2 (6.5)
Found App+ was well integrated	7 (22.6)	11 (35.5)	13 (41.9)
Too much inconsistency with App+	18 (58.1)	7 (22.6)	6 (19.3)
Most people would learn App+ very quickly	2 (6.5)	5 (16.1)	24 (77.4)
Found App+ cumbersome to use	14 (45.2)	11 (35.5)	6 (19.3)
Felt confident using App+	3 (9.7)	10 (32.3)	18 (58.1)
Need to learn a lot of things before I could use App+	23 (74.2)	5 (16.1)	3 (9.7)

^a.3 participants did not answer the SUS Scale

Study participants were asked to list up to three features they liked best about the app, up to three features they liked least about APP+, as well as up to three features they would like added (Table IV). Five major themes were found about features liked best about the APP+ intervention. The most common reasons for liking APP+ was its ease of use and because it was user friendly. Many men also appreciated the daily reminders to take their HIV medications, the daily new information about HIV and ART, their perception that the app provided some level of acceptance and support, and the weekly tracker that allowed men to log and track their adherence. However, men also reported a number of aspects that they disliked about APP+, of which three themes emerged. First, men reported experiencing app malfunctions, which impacted their experience with the app. Also, men reported a number of issues that they had with the daily reminder notifications, suggesting that streamlining the process for setting up and receiving the notifications is needed. About one-third of men who downloaded the app reported having a problem with or not liking an aspect of the medication reminder function on the app. For example, a participant noted that the reminder function could be accomplished with existing features on most mobile phones. Some men also had general negative opinions about APP+ that included concerns that it may not fill a unique role and it was time consuming. Few men provided feedback about what additional features that they would like to see in the app; among those who did, feature requests included adding information about HIV-related events in their community, simplifying the app and including more information about sexual risk and harm reduction strategies, integrating the app with their calendar to make reminders more effective, providing a better tutorial for how to use the app, and making the app more discreet by not including a “+” sign in the app logo and taking greater steps to disguise the notifications.

Table IV.

Categories and examples of APP+ likes and dislikes

Theme	Frequency	Example Quote
APP+ Likes
Ease of Use / User Friendly	25	“Easy to navigate” “The ease to log in “It is very user friendly” “The simplicity of the layout”
Daily Reminders	19	“Notification remains until you take your meds, unlike an alarm that can be turned off and forgotten. “It really reminds me with the notification the time I need to take my pill” “Somehow having to record if I took my dose daily reminded me to take it daily”
Information	17	“The additional information it shows about HIV is very useful and informative” “Updated news on HIV” “Information regarding HIV”
Acceptance and Support	17	“It was very comforting and accepting in regards to my HIV” “Motivational comments after logging”
Weekly Tracker	10	“Gave me a percentage of my meds each week” “The ability to log data prior to the present day” “Tool to use with my doctor”
APP+ Dislikes
App Malfunctions	15	“App must remain open in order to function instead of running in the background” “Sometimes it crashes” “Notifications sometimes did not work”
Daily Reminders	11	“My alarm never goes off to alert me to take the meds” “Setting the alarms was cumbersome” “The reminders were not well disguised so someone could see them on your notifications”
General Negative Opinions	9	“Easily done by using phone functions” “Redundant questions” “Time consuming”

Change in Self-Reported ART Adherence by Study Arm

Past 30-day overall adherence, taking ART within two hours of the scheduled time, adherence while using stimulants, and taking ART within two hours of the scheduled time while using stimulants at baseline, 2 months, post-active intervention assessment (up to month 4), and 6 months are shown in Table V. Overall percentage of ART taken in the past 30 days at baseline was high (80.0%; sd = 27.5) and did not differ between those randomized to the APP+ intervention (81.8% [95% CI: 74.2%, 89.4%]) and those randomized to the control arm (78.1% [95% CI: 69.2%, 87.0%]). Overall 30-day adherence scores began to diverge between the intervention and control at the 2-month survey (87.8% [95% CI: 82.4%, 93.2%] vs 81.9% [95% CI: 73.4%, 90.4%], respectively; difference = 5.9% [95% CI: −3.7, 15.5], p=0.23), and at the 4-month assessment men randomized to the intervention group reported higher overall ART adherence compared to those in the control group (89.0% [95% CI: 83.4%, 94.6%)] vs 77.2% [95% CI: 66.7%, 87.7%]; difference = 11.8% [95% CI: 0.34%, 23.2%], p = 0.04). However, these improvements in adherence were not sustained at the 6-month assessment (85.3% [95% CI: 80.0%, 90.6%] intervention vs 89.0% [95% CI: 83.2%, 94.9%] control, difference = −3.7% (95% CI: −11.4%, 4.0%), p = 0.34). Similar results were found at each follow-up period when we examined the association between group assignment and overall 30-day adherence scores adjusting for baseline adherence (data no shown).

Table V.

Mean baseline and follow-up ART adherence outcomes between App+ intervention and control arm participants

Continuous ART Adherence Variable, mean (sd)	Baseline	2 Month	4 Month	6 Month
% ART taken (past 30 days)
APP+ Intervention	81.8 (25.4)	87.8 (16.7)	89.0 (17.0)	85.3 (15.1)
Control	78.1 (29.6)	81.9 (23.9)	77.2 (30.6)	89.0 (15.9)
% ART taken on time (2 hours of scheduled dose) (past 30 days)
APP+ Intervention	72.8 (29.3)	80.0 (22.4)	80.0 (24.9)	79.8 (25.5)
Control	72.2 (33.6)	77.0 (24.6)	70.7 (30.9)	83.0 (21.4)
% ART taken on stimulants (past 30 days)
APP+ Intervention	66.0 (28.3)	65.0 (36.8)	68.0 (34.8)	63.6 (36.5)
Control	66.7 (31.5)	68.7 (33.5)	65.4 (39.1)	65.8 (35.8)
% ART taken on stimulants on time (past 30 days)
APP+ Intervention	62.6 (31.0)	60.4 (34.0)	58.1 (36.0)	64.9 (35.6)
Control	61.6 (30.9)	60.1 (34.2)	57.7 (36.3)	68.0 (33.3)
Categorical ART Adherence Variable, n (%)
How often have you taken your doses correctly? (Past 30 Days)*
APP+ Intervention
Never / Rarely / Sometimes / Usually	16 (35.6)	11 (28.9)	8 (21.6)	11 (33.3)
Almost Always / Always	29 (64.4)	27 (71.1)	29 (78.4)	22 (66.7)
Control
Never / Rarely / Sometimes / Usually	14 (31.1)	10 (30.3)	14 (40.0)	6 (19.4)
Almost Always / Always	31 (68.9)	23 (69.7)	21 (60.0)	25 (80.6)

^*Those who refused to answer were excluded

There were no significant differences at any assessment timepoints for taking ART within two hours of the scheduled time, adherence while using stimulants, and taking ART within two hours of the scheduled time while using stimulants. Similarly, although a marginally higher percentage of men in the APP+ condition reported almost always or always taking their doses correctly in the past 30 days than men in the control condition at the 4-month assessment timepoint, the difference was not significant (p<.09)

The results of an analysis that restricted the sample to those men who were randomized to the intervention arm and downloaded the app (n = 34) is shown in Figure 5. Similarly, among those participants who downloaded the app, overall 30-day adherence scores diverged between the intervention and control at the 2-month survey (90.6% [95% CI: 87.2%, 93.9%] vs 81.9% [95% CI: 73.4%, 90.4%], respectively; difference = 8.6% [95% CI: −0.5., 17.8], p=0.06), and at the 4-month assessment men randomized to the intervention group reported higher overall ART adherence compared to those in the control group (88.6% [95% CI: 81.6%, 95.7%] vs 77.2% [95% CI: 66.7%, 87.7%]; difference = 11.4% [95% CI: −1.4,%, 24.3%], p = 0.08). At the 6-month assessment, the improvements in adherence were similarly not sustained, (85.5% [95% CI: 79.5%, 91.5%] intervention vs 89.0% [95% CI: 83.2%, 94.9%] control, difference = −3.5% [95% CI: −11.8%, 4.7%], p = 0.39).

Figure 5. Percentage 30-Day self-reported ART adherence among men in the APP+ intervention for each randomized group, and for men who downloaded the APP+ mobile app.

Number of participants completing each assessment:

Intervention: Baseline = 45; 2 Month Follow-Up = 42; 4 Month Follow-Up = 41; 6 Month Follow-Up = 38

Intervention (Downloaded): Baseline = 34; 2 Month Follow-Up = 33; 4 Month Follow-Up = 32; 6 Month Follow-Up = 29

Control: Baseline = 45; 2 Month Follow-Up = 39; 4 Month Follow-Up =38; 6 Month Follow = 36

A secondary exploratory analysis was performed to evaluate the effects of the APP+ intervention on stimulant use over the follow-up period. No differences in stimulant use in the past two months were found at baseline between those randomized to the APP+ intervention (80% [95% CI: 65.2%, 89.5%]) compared to those in the control group (84.4% [95% CI: 70.1%, 92.6%]). Over the course of six-month follow up, use of stimulant drugs among the entire cohort of men declined to 63.5% at the final assessment. Fewer men in the APP+ intervention group reported past two-month stimulant use than those in the control group at the 4- month (56.1% [95% CI: 40.2%, 70.9%] vs 81.6% [95% CI: 65.2%, 91.3%], respectively; p = 0.02) and 6-month (52.6% [95% CI: 36.3%, 68.4%] vs 75.0% [57.6%, 86.9%], respectively; p = 0.05) assessment timepoints (Figure 6).

Figure 6. Past 2-month stimulant use among men in the APP+ intervention for each randomized group, and for men who downloaded the APP+ mobile app.

Number of participants completing each assessment:

Intervention: Baseline = 45; 2 Month Follow-Up = 42; 4 Month Follow-Up = 41; 6 Month Follow-Up = 38

Intervention (Downloaded): Baseline = 34; 2 Month Follow-Up = 33; 4 Month Follow-Up = 32; 6 Month Follow-Up = 29

Control: Baseline = 45; 2 Month Follow-Up = 39; 4 Month Follow-Up =38; 6 Month Follow = 36

DISCUSSION

MSM living with HIV who use stimulants require tailored interventions to address their unique ART adherence barriers, including sexual minority stigma and stimulant-related nonadherence (19, 34, 35). To our knowledge, this is the first study to assess the feasibility and acceptability of a mobile app to improve ART adherence among MSM living with HIV who use stimulants. Given the small overall sample size, a secondary objective was to examine changes in self-reported ART adherence by study group. Overall, results showed that it was feasible to conduct the study and retain participants, that APP+ was moderately acceptable, and men in the APP+ intervention arm reported higher overall ART adherence during the active intervention period than men in the control condition; however, those differences did not persist through the end of follow-up. These findings are discussed in more detail below.

Just over 80% of men in this study were retained in this study in which men were recruited and completed the study remotely at the final 6-month follow-up assessment time point. Retention rates of methamphetamine-using MSM recruited and who complete follow-up visits in person vary from 96% retained at a 2-month follow-up (50) to 82% retained at a 6-month follow-up (51). Retention rates in studies of MSM recruited online are often lower; for example, in a study of 3,092 MSM recruited online and randomized to five remotely-delivered study arms, 44% were completed the 60-day follow-up assessment (52). The retention rate at 6-months post-baseline of men in the current study, in which men were recruited and completed study components remotely, suggests that this approach is feasible and may be used in future similar studies to reach a broad diversity of MSM living with HIV who use stimulants. The majority (56%) of men in this study were men of color, who are disproportionately at risk for HIV and poorer outcomes after HIV diagnoses (53, 54). Remote recruitment and conducting of study activities, therefore, may have the potential to engage those most in need along the HIV prevention and treatment continuum.

The overall acceptability of APP+ was comparable to other online-delivered interventions (45). The finding that half or more of the men who downloaded the app found it easy to use and perceived the app to provide additional supports and acceptance of them is encouraging. In addition, half or more men who downloaded the app reported that they appreciated the medication reminders and the IMB-based HIV or ART adherence content, which were two major features of the app. These findings are supported by the high percentage of men who viewed or used the HIV and ART adherence content and the medication reminders during the intervention period. Although medication reminders were not tailored to MSM living with HIV who use stimulants, the IMB-based HIV or ART adherence content was and may have been seen as engaging for men in this study. A meta-analysis of HIV prevention interventions for African Americans showed that culturally-tailored interventions were more effective than those that were not culturally tailored (55), which may also be the case for men in the current study. Studies are needed that vary the degree of tailoring to MSM living with HIV who use stimulants to provide a more definitive assessment of the importance of tailoring intervention content to this group.

Men in this study also provided feedback on aspects of the app that they disliked, as well as features that they would include in future iterations. App malfunctions and improperly working technology was the most reported aspect of the app that was disliked. Issues with technology may be resolved through continued usability testing (56) and mobile app refinement. Several men also expressed a variety of issues that they had with the app, such as the app being perceived as time consuming. These are reminders that continuing to refine APP+ to include features that have added value to MSM living with HIV who use stimulants will be important in future versions of the app. Men in this study provided several suggestions for new or improved features, including a space for community events, simplifying the app and integrating it into current mobile device functions, and increasing its privacy features. Future versions of APP+ should incorporate this feedback, which may also be considered for inclusion in future mHealth interventions to improve ART adherence for MSM living with HIV who use stimulants.

Men in the APP+ intervention condition reported improved ART adherence during the active intervention period (Month-2 and Month-4) compared to men who did not receive any intervention. While caution should be used given this small sample size in this study, these results add to the growing evidence that technology-based interventions may improve prevention and treatment outcomes (29). For example, Reback and her colleagues demonstrated that theory-based text messages reduced serodiscordant condomless anal sex partners and recent methamphetamine use among fifty-two non-treatment-seeking methamphetamine-using MSM (57). A recent review of technology-based interventions addressing drug-using populations showed that between January 2016 and March 2017 there were five published in-progress or completed mobile app-based interventions (29). Mobile app interventions addressed illicit drug use (58, 59), improving HIV healthcare engagement (30), and HIV prevention outcomes (60). Only one mobile app study was reported that addressed ART adherence (31); the Heart2HAART mobile app was an adjunct to directly observed therapy that provided medication reminders, information about medication adherence, real-time self-monitoring, and tailored education, recommendations, and encouragement based on data entered. Among the 28 persons (32.1% female) whose data were available, there were no differences in ART adherence after 3 months, as assessed by unannounced pill counts.

The present pilot study also used a mobile app to address ART adherence, but the results showed an improvement in self-reported ART adherence in the APP+ treatment arm after two months. The difference between the APP+ and the control arms was most prominent at the 4-month assessment, which was administered at the end of the active intervention period. These findings suggest that the APP+ mobile app may assist MSM living with HIV who use stimulants to improve their ART adherence so long as they continue to have access to the app. As with many ART adherence behavioral interventions, intervention effects appear to diminish once participants no longer have access to intervention components. Overall, these results suggest that the APP+ mobile app is a potentially promising intervention approach to improve ART adherence among MSM living with HIV who use stimulants. However, further feature development (e.g., more content to address substance use; integration with clinic health records) may be needed to refine APP+ in order to realize sustained intervention effects.

The primary purpose of APP+ was to improve adherence-related behaviors, however we assessed whether the intervention achieved concurrent reductions in self-reported stimulant use. The majority (80%) of men in the APP+ condition reported recent stimulant use at baseline, which fell to just over half (52.6%) at the final assessment and was notably lower than the three-quarters of men in the control condition who reported stimulant use at 6 months. Men may have been responsive to the combination of IMB-related messages that also reflected a harm reduction approach (61). For example, informational content provided information on interactions between illicit drug use and HIV medications, and behavioral skills content focused on reducing the frequency of drug use and ways to continue to take HIV medications when using drugs. Although encouraging, APP+ may be able to achieve greater reductions in stimulant use if it were combined with other evidence-based interventions to reduce substance use (34).

This study has several important limitations. First, since this is a small pilot study, the results should be interpreted with caution because of the imprecise effect estimates. Second, we assessed ART adherence using commonly-used self-report items (36, 62). Self-reported adherence often overestimates adherence compared to adherence data captured through more objective measures (37). However, the magnitude of difference in self-reported adherence between study conditions in this pilot study of men recruited from geographically-diverse regions of the US nonetheless suggests that APP+ may be of benefit in improving adherence. Similarly, only self-reported drug use measures were collected for this pilot trial. Future iterations of APP+ or similar app-based interventions should collect both self-reported drug use and biological data (e.g., urinalysis) to more decisively determine the effects of this approach on stimulant use. Second, we recruited a relatively small number of participants for this pilot study, and not all men randomized to the APP+ intervention arm downloaded the app. Although we did not observe any appreciable differences in baseline covariates between the arms or in the estimates of the effect of APP+ on adherence in “as treated” analyses with only those who downloaded, the app, future RCTs should include expanded efforts to support app download and provision of tech support to enhance engagement.

Despite these limitations, the results of this pilot RCT suggest that an app-based approach to improving ART adherence among MSM living with HIV who use stimulants is feasible and acceptable, and that further efforts to assess the efficacy of mobile apps for this purpose is warranted. Although a number of recent studies have been conducted or are in progress to understand how to use technology to address HIV and other health outcomes among persons who use drugs (29), relatively little is yet known of its potential in this area. Given recent reports of increases in methamphetamine injection practices and high rates of mixing substance use and sex among MSM in the US and other countries (63, 64), creating effective, sustainable, and widely scalable interventions that leverage technology is a priority.