DOT Diary: Developing a Novel Mobile App Using Artificial Intelligence and an Electronic Sexual Diary to Measure and Support PrEP Adherence Among Young Men Who Have Sex With Men

Abstract

Young men who have sex with men (YMSM) are highly vulnerable to HIV. While pre-exposure prophylaxis (PrEP) has demonstrated effectiveness, adherence has been low among YMSM and difficult to measure accurately. In collaboration with a healthcare company, we configured an automated directly-observed therapy (aDOT) platform for monitoring and supporting PrEP use. Based on interest expressed in focus groups among 54 YMSM, we combined aDOT with an electronic sexual diary to provide feedback on level of protection during sex and to motivate app use. In an 8-week optimization pilot with 20 YMSM in San Francisco and Atlanta, the app was found to be highly acceptable, with median System Usability Scale scores in the “excellent” range (80/100). App use was high, with median PrEP adherence of 91% based on aDOT-confirmed dosing. Most (84%) participants reported the app helped with taking PrEP. These promising findings support further evaluation of DOT Diary in future effectiveness studies.

INTRODUCTION

Young men who have sex with men (YMSM) are among the most vulnerable to HIV in the United States (US), accounting for over one-quarter of new infections among MSM.¹ Annual new HIV infections increased by 47% among MSM aged 25-34 between 2010-2016 and were the highest of any age group in 2016, followed by 13-24 year olds.² Young Black and Latino MSM are particularly impacted by HIV, accounting for over three-quarters of new diagnoses among YMSM.¹ To achieve the US Ending the HIV Epidemic goals of reducing incident HIV infections by 90% by 2030,³ high uptake and adherence to effective prevention approaches are critically needed in these key populations.

Pre-exposure prophylaxis (PrEP) with daily oral tenofovir-disoproxil-fumarate/emtricitabine (TDF/FTC) has been proven to be safe and highly effective in a number of clinical trials,^4-8 open-label studies, ^9,10 and demonstration projects.^11,12 While youth have great potential to benefit from PrEP, adherence rates have been low and difficult to measure accurately. In the Adolescent Trials Network (ATN) 110 study of PrEP use among YMSM aged 18-22 years, adherence declined as visit frequency decreased from monthly to quarterly, with only 34% of participants having protective drug levels by week 48.¹³ Similarly, in the ATN 113 study among 15-17 year old YMSM, the proportion of participants with protective drug levels dropped from 54% at week 4 to 22% at week 48.¹⁴ Despite no-cost provision of PrEP, annualized HIV incidence was 3% and 6% respectively in these cohorts. Additionally, self-reported measures overestimated PrEP adherence in these and other studies.¹⁵ In the ATN 082 (Project PrEPare) study among young MSM aged 18-22 (53% African-American, 40% Latino), self-reported adherence average 62%, while detectable tenofovir levels in plasma ranged from 20-63%.¹⁶ High rates of nonadherence have also been observed in PrEP trials among young African women^17,18 and in clinical trials across a range of conditions and populations,^19-21 impacting the accurate assessment of drug efficacy in these studies. Clearly, tailored approaches to better monitor and support medication adherence in HIV prevention trials, particularly among Black and Latino YMSM, are needed.

There has been growing interest in the use of novel technologies to monitor and support medication adherence, including the use of electronic drug monitoring,^22-24 ingestible sensors,^25-27 breathalyzers,^28-30 and artificial intelligence which can provide more objective confirmation of dosing and opportunity for real-time monitoring and intervention, and can be integrated into clinical and patient workflows.^31-33 This evolution has occurred in tandem with the rapid adoption of smartphone technology for monitoring and supporting health-related behaviors.^34-36 In a recent systematic review, mobile apps that provided feedback on one’s health status and monitored indivdiual health status or behavioral change were common and associated with better health outcomes.³⁶ Mobile phone technologies are highly used by young people and may be particularly useful in supporting preventive health behaviors among youth and sexual and gender minorities.^37-41 An innovative approach to monitor and support medication adherence is the use of an artificial intelligence platform deployed through a smartphone that uses computer vision and deep learning algorithms to confirm the right person is taking the right medication at the right time.³¹ This automated directly-observed therapy (aDOT) approach allows for real-time monitoring, as staff are notified of missed doses through a dashboard and can reach out to participants by phone or text to provide timely support. The technology is being used to measure and maximize adherence in phase I-IV drug development trials, as well as in population health settings, including patients in neuropsychiatry,³² infectious disease,^42,43 and cardiovascular disease.³³ In a pilot study among commercial members with self-funded insurance who initiated hepatitis C virus (HCV) treatment (with ledipasvir/sofosbuvir, sofosbuvir/velpatasvir, or glecaprevir/pibrentasvir), the large majority of participants using the smartphone-based AI platform were able to successfully use it without challenges with a reported adherence of 98%. The observed percentage of days covered (PDC) for HCV medications was statistically higher in the subset of members participating in the pilot program with AiCure (96%) compared to those who did not participate (88%) (p=0.025).³² In another program among patients receiving treatment for active tuberculosis in Los Angeles County and using the AI platform for confirmation of dose adherence, the platform was determined to be a cost-effective alternative to DOT.³³ The technology has also been validated for accuracy against serum drug levels.⁴⁴

Online sexual diaries have been used in a number of studies of MSM to track sexual encounters and reduce risk behaviors,^45-48 and use of self-monitoring functions within apps has been associated with sustained app engagement over time.⁴⁹ In the context of PrEP, an integrated adherence monitoring and pill reminder system coupled with a sexual diary could assist YMSM in tracking their pill-taking and sexual behaviors and times when it is particularly important to take PrEP. To increase engagement with aDOT when used by YMSM taking PrEP, we sought to combine it with an electronic sexual diary to create an integrated “DOT Diary” smartphone-based app. DOT Diary is grounded in the self-management theory which posits that increasing the ability of individuals to take responsibility for and personally manage their health conditions can improve health outcomes.⁵⁰ Here we describe a two-phase study to develop and refine the DOT Diary. In the developmental phase, we conducted qualitative formative work to understand interest in integrating aDOT and the electronic sex diary and elicit and address concerns to inform technology development. This was followed by an optimization phase in which we conducted a pilot study to evaluate acceptability and ease of use of the app. We hypothesized that the integrated DOT Diary app developed iteratively through formative work would be highly acceptable and easy to use among YMSM in Atlanta and San Francisco Bay Area, two metropolitan regions heavily impacted by HIV.¹

METHODS

Intervention components

In collaboration with AiCure (New York, New York), an artificial intelligence (AI) and advanced data analytics company that builds and deploys clinically-validated AI technologies to optimize patient behavior and medication adherence, we configured an aDOT platform for monitoring and supporting PrEP use. The aDOT application is downloaded onto the participant's smartphone and is Health Insurance Portability and Accountability Act (HIPAA)-compliant. The platform captures data through the front-facing camera of the device, processes and analyzes that data using computer vision and deep learning algorithms. The computer vision and algorithms have been optimized over time through machine learning and annotation of the visual and audio data collected of participants dosing on the platform on an ongoing basis. An additional human review step is included as needed to confirm the process of medication intake.

Participants receive daily dosing reminder alarms prompting them to go into the application and visually confirm taking their medication at the time of their choosing. If the participant is late taking a given dose, they will receive an automated system generated text message asking “Are you late taking any medication today?” Site staff can also select from a list of pre-approved text messages to send participants to follow up on any missed or late doses. Dosing data are encrypted at motion and at rest and transmitted to a centralized, cloud-based system and displayed on dashboards to allow for real-time review and intervention by study staff. In addition to a date and time stamp of each dosing event, doses are automatically classified as visually confirmed as taken or missed (non-response); participants can also self-report doses through a button within the application if they forgot or were unable to use the system to confirm their dosing. Missed doses can be reclassified by the site through their dashboards as site reported (indicating the participant told the site that the dose was taken) or clinic reported (indicating the site saw the participant take their medication). Additionally, visually confirmed doses through the system may be flagged for further review for intentional nonadherence (e.g. presenting a different pill, spitting out the medication or hiding it in the mouth without swallowing) – if it is confirmed that medication ingestion did not take place, study staff are alerted with a “red alert”.

The electronic sexual diary component of DOT Diary was adapted from our LYNX mobile app designed to support HIV testing and PrEP uptake.⁵¹ The sexual diary allows participants to track sexual encounters (partner type and HIV status), sexual behaviors that occurred in each encounter (insertive/receptive oral, anal, or vaginal sex, and whether condoms were used), and rating characteristics of partners (overall, chemistry, personality, parts of body). Participants were instructed to enter nicknames of sex partners to maintain confidentiality. Partner-specific information (e.g. ratings) entered by participants is encrypted and not transmitted to the study team to further protect privacy. The app provides a calendar displaying all days in which PrEP medication was taken, and all days in which sexual activity occurred, allowing participants to see coverage of sexual encounters with PrEP (Figure 1). Based on pharmacokinetic and pharmacodynamic data from prior PrEP trials,^52-54 the app indicates the estimated level of protection achieved from PrEP (e.g. low, medium, high) on the home screen, along with a record of their pill-taking in the past week, and motivating personalized messages on the number of additional PrEP doses needed to maximize or maintain protection (e.g. “Take PrEP for 4 more days to reach high protection!”). For the calendar page, a pop-up screen with a legend for dosing, sexual activity, and how protection levels are calculated was added to facilitate use and understanding of information provided by the app.

Figure 1:

Screenshots of DOT Diary App

Study population

Study participants for the focus group discussions (FGDs) (N=54) and optimization pilot (N=20) included YMSM residing in the San Francisco Bay Area or Atlanta Metropolitan Area between the ages of 18 and 35 who were male-identified, HIV-negative, English-speaking, and reported sex with a man in the past 12 months. For the optimization pilot, additional eligibility criteria included currently taking or interest in initiating PrEP, smartphone ownership compatible with DOT Diary (iOS or Android), and reporting at least one of the following behavioral criteria in the past 12 months: 1) any condomless anal sex (not in a mutually monogamous relationship with an HIV-negative partner); 2) ≥2 anal sex partners; 3) self-reported sexually transmitted infection, or 4) having a known HIV-positive sexual partner. Participants also had to be medically eligible to take PrEP, with a creatinine clearance ≥60 ml/min, and be hepatitis B surface antigen negative. For the optimization pilot, we aimed to enroll approximately half of participants already on PrEP and half who were de novo PrEP users.

All phases of this study were approved by the University of California San Francisco Institutional Review Board and the trial was registered on clinicaltrials.gov [NCT03387462].

Developmental Phase: Qualitative Formative Research

Six FGDs (n=34) were conducted in the formative phase of the project, three at each site with 5-7 participants in each, to explore overall interest in the use of a clinical trial tool that combines a sexual diary and PrEP adherence app and acceptability of specific features of each component. FGDs also elicited specific feedback on concerns about privacy, sharing results with clinic staff, and preferred communication modalities from clinic staff about adherence. Using the findings from these FGDs, the two components (aDOT and sexual diary) were integrated into one app (DOT Diary). Detailed notes were taken at each FGD, expanded immediately afterwards, and sessions were audio-recorded. Summaries of findings were produced at each site and guided the development of the integrated app, incorporating the app features that were both desirable and feasible to program. The designs were taken back to 3 additional FGDs (n=20). Quotes from focus groups are reported in the results in italics without individual characteristics, as we did not have an accurate record of which focus group participants were associated with specific quotes.

Optimization Phase: Pilot Study

Upon completion of development, the technology was tested in an 8-week open optimization pilot to evaluate the acceptability and ease of use of DOT Diary. All participants attended a screening visit to determine eligibility, including 4^th generation HIV testing, creatinine, and hepatitis B surface antigen testing, and returned for an enrollment visit if eligible. FGD participants were allowed to participate in the optimization pilot if eligible. Participants completed an online questionnaire via computer-assisted self interview (CASI) at enrollment to gather data on baseline demographics (age, race/ethnicity, sexual orientation, income, and insurance status), drug use (alcohol, popper, cocaine, and methamphetamine use) and sexual behaviors (number of anal sex partners, condom use with anal sex, and history of STIs), knowledge and use of PrEP (current, previous, or never used), and experience using technology (number of apps used, type of smartphone). A study clinician performed a medical history and symptom-directed physical exam to confirm medical eligibility for PrEP; participants had a rapid 4^th generation test at enrollment, along with 3-site STI testing. Enrolled participants were dispensed study drug (TDF/FTC) and provided adherence counseling for taking PrEP, including the PrEP Basics.⁵⁵ Additionally, participants were given instructions to download the app to their smartphone; study staff demonstrated how to use the app, and participants completed a tutorial and demonstrated taking their first dose of TDF/FTC using the app.

Participants had a phone check-in at 1 week to assess for any issues with using the app or taking PrEP. Participants returned to the clinic at 4 and 8 weeks for additional medication dispensing (4 week only), HIV and STI testing (8 week only), and a CASI on sexual and drug use behavior in the past 3 months, use and acceptability of the app assessed via an adapted System Usability Scale (SUS)⁵⁶ and Client Satisfaction Questionnaire (CSQ-8),⁵⁷ and ratings of different components of the app. Frequencies of dosing classifications were calculated, and variation in rates of visually confirmed dosing were evaluated using Pearson’s rho for age and Kruskal-Wallis tests for other demograhics, income, health insurance status, and living situation. Participants (N=20) also completed an in-depth interview at weeks 4 and 8 with staff trained in qualitative methods, to elicit additional feedback on the acceptability of various components of the app and how they could be improved. At each visit, participants were provided PrEP and benefits navigation to ensure access to PrEP after study completion and were paid $50 for completion of study procedures. If needed, participants were provided an additional 3 months of PrEP after study completion to allow more time for linkage to a PrEP provider.

RESULTS

Developmental phase: focus groups discussions (FGDs)

FGDs were carried out in the initial phase of the project to help guide the integration of the aDOT and the sexual diary (mean age 28, 43% African American, 22% Latinx). Overall participants were interested in the integration of the two apps, especially in being able to see a record of their pill taking and to understand their protection levels. Because of these interests, participants brainstormed ways of displaying protection levels, including a shield, a pill, and a circle with colors. The team developed several visual images depicting PrEP protection levels and received iterative feedback leading to a simple, color-coded circle on the home page clearly displaying level of protection (Figure 1). Participants indicated that they would use the aDOT application in a clinical trial and preferred it to other measures of adherence such as blood draws or dosing in the clinic.

Participants also provided feedback on how to make the aDOT more appealing for use with young MSM. They reported that the original tutorial involving three rounds of practice dosing in the system was too long. As one participant expressed in response to the lengthy tutorial session that repeated multiple times, “Make it not feel like a punishment.” Further, they preferred the tutorial to have a male voice and a casual tone. Based on this feedback, the team adapted the app to streamline the tutorial by combining its instructional guidance with the participant’s first dose at home, and changed the voice to be male. FGDs also guided the protocol for staff follow-up in the case of non-adherence. We found that participants preferred the idea of a non-judgmental and casual approach to check-ins if they had not used the app, e.g. someone they knew at the clinic to reach out to them, so that communication would serve as a “wellness check-in,” as opposed to a call that felt punitive. They also said it was appropriate to start with minimal contact and then escalate if necessary. Based on this feedback, we developed a follow-up protocol in which initial contact for non-adherence involved asking participants if they were experiencing any difficulties using the app and whether staff could assist with any issues; for recurrent non-adherence, staff informed participants that they noticed they may be having difficulty using the app, and explored barriers and facilitators to app use. The proposed integration of the sexual diary component was well-received. Participants expressed mixed feelings about the ratings, which one called “aggressive,” indicating he would not like to rate other people or be rated. Others liked the rating feature and in fact had been tracking partners on their own in similar ways. Regarding privacy, most felt comfortable with sharing dosing and sexual behavior information with clinic staff in the context of a study. There were concerns, however, about other people seeing information about partners or about PrEP use on their phone and preference was for both discrete messaging and password protection for the app.

Optimization pilot

From February to June 2018, 20 MSM enrolled into the DOT Diary optimization pilot, and 19 participants completed the 8-week follow-up visit. The median age was 28 (range 20-32, Table 1). Overall, three-quarters were African-American, and one-quarter was Latino. Over half (55%) reported some college education or higher; 35% reported an income less than $20,000, and 60% had health insurance. At baseline, 70% of participants reported condomless anal sex in the past 3 months, and 40% were diagnosed with a sexually transmitted infection during this period. Overall, 35% were currently taking PrEP, 30% had previously taken PrEP, and 35% were PrEP naïve.

Table 1:

Baseline demographics and risk behaviors^* (N=20)

Characteristic	N (%)
Age (median, IQR)	28.0 (26.0-30.0)
Race/ethnicity†
Black	15 (75%)
Latino	5 (25%)
Sexual Orientation
Gay	18 (90%)
Queer	4 (20%)
Bisexual	3 (15%)
Education
College graduate or higher	9 (45%)
Some college	8 (40%)
High school/GED	3 (15%)
Income
<$20,000	7 (35%)
$20,000-49,999	6 (40%)
>50,000	5 (20%)
Has health insurance	12 (60%)
Alcohol use
≥5 drinks/day on typical drinking days	3 (15%)
Substance use (past 3 months)
Popper use	8 (40%)
Cocaine use	5 (25%)
Methamphetamine use	0 (0%)
# anal sex partners, past 3 months (mean, SD)	7.9 (9.9)
Anal sex without a condom, past 3 months	14 (70%)
Receptive anal sex without a condom, past 3 months	7 (35%)
Diagnosed with an STI, past 3 months	8 (40%)
Prior knowledge of PrEP	20 (100%)
PrEP use	7 (35%)
Currently taking PrEP	6 (30%)
Previously taken PrEP	7 (35%)
Never used PrEP
Smartphone
iPhone	16 (80%)
Android	4 (20%)
# cell phone apps used weekly (mean, SD)	5.7 (0.8)

^*Collected at the enrollment visit

^†Based on two-part question asking about Latino/Hispanic origin and multiple-choice item on race (participants could select multiple options)

The median SUS score was 85/100 (IQR 70-98) at week 4 and 80 (70-93) at week 8. At study exit, 32% of participants had SUS scores in the “best imaginable” (90-100) range, 26% “excellent” (80-89), 26% (60-79) "good", and 16% “OK” (40-59).⁵⁸ Acceptability measures of the DOT Diary app assessed at weeks 4 and 8, including response categories, are shown in Table 2. At week 8, nearly two-thirds (63%) were very satisfied with the app, and the remainder (37%) were mostly satisfied. Most (84%) participants reported the app helped with taking PrEP, while one participant (5%) reported the app “didn’t really help” and 2 (11%) reported the app “made things worse”. Over two-thirds (68%) would definitely recommend the app to a friend, and another 26% responded “I think so.” Overall, 84% reported they were likely to use the app to take PrEP in the future.

Table 2:

Acceptability measures of DOT Diary app using the Client Satisfaction Questionnaire-8 and System Usability Scale

Acceptability measure	Week 4 (N=20, %)	Week 8 (N=19, %)
Quality of DOT Diary app
Poor	0 (0%)	0 (0%)
Fair	2 (11%)	0 (0%)
Good	11 (58%)	10 (53%)
Excellent	6 (32%)	9 (47%)
Overall satisfaction with app
Quite dissatisfied	0 (0%)	0 (0%)
Indifferent or mildly dissatisfied	0 (0%)	0 (0%)
Mostly satisfied	6 (32%)	7 (37%)
Very satisfied	13 (69%)	12 (63%)
App provided support you wanted	0 (0%)	0 (0%)
No	0 (0%)	0 (0%)
I think so	8 (42%)	7 (37%)
Definitely	11 (58%)	12 (63%)
Helped with taking daily PrEP
Made things worse	1 (5.3%)	2 (11%)
Didn’t really help	0 (0.0%)	1 (5%)
Helped	6 (32%)	4 (21%)
Helped a great deal	12 (63%)	12 (63%)
Would recommend to a friend
I don’t think so	0 (0%)	1 (5%)
I think so	7 (37%)	5 (26%)
Definitely	12 (63%)	13 (68%)
Would use app in future
I don’t think so	2 (11%)	3 (16%)
I think so	6 (32%)	7 (37%)
Definitely	11 (58%)	9 (47%)
Would use app in the future to help take PrEP
Extremely unlikely		1 (5%)
Unlikely		2 (11%)
Neutral		0 (0%)
Likely		6 (32%)
Extremely likely		10 (53%)
Would use app in a future study
Extremely unlikely		2 (11%)
Unlikely		0 (0%)
Neutral		2(11%)
Likely		3 (16%)
Extremely likely		12 (63%)
System Usability Scale (median, IQR)	85.0 (70.0-97.5)	80.0 (70.0-92.5)
OK	2 (11%)	3 (16%)
Good	7 (37%)	5 (26%)
Excellent	3 (16%)	5 (26%)
Best imaginable	7 (37%)	6 (32%)

Overall, among 1198 (mean 59.8 per participant) expected doses in the optimization pilot, 80% were classified as visually confirmed, 6% were self-reported via a self-report button, 14% were missed, 0.2% were self reported by the participat to the site, and 0.2% were taken at the clinic. Rates of visual confirmation did not differ significantly by age (rho 0.04, p=0.86), race/ethnicity (chi-square 0.55, p=0.46), education (chi-square 0.73, p=0.87), employment status (chi-square 1.59, p=0.45), income (chi-square 13.12, p=0.11), health insurance status (chi-square 0.66, p=0.42), or living situation (chi-square 0.51, p=0.48). In terms of ease of use of the aDOT component, 6 participants (30%) reported it was “quite a bit” or “very” difficult taking a pill with the application, 2 had difficulty entering a new partner in the diary component, 1 had difficulty entering a new sexual encounter, and 1 had difficulty viewing the monthly calendar (1 participant had difficulty with 2 items and 1 had difficulty with 3 items); no participants reported significant difficulty viewing the home screen. The perceived helpfulness of different app components is shown in Figure 2. The most useful components were the circle showing level of protection on the home screen, calendar showing days pill was taken and sex occurred, and the pop-up screen showing protection, dosing, and sexual activity. The least useful components were the notification to update the diary and the ability to rate partners; nonetheless, over 50% of participants rated them as “very helpful.”

Figure 2:

Helpfulness of different app components

Based on real-time dosing data captured by the technology, the median percentage of doses taken with visual confirmation of pill ingestion by the app was 91% (range 35-100%), with a median of 0% (range 0-40%) doses taken (per person) using the self-reported dosing option. The median percentage of doses missed was 5.5% (range 0-65%). The most common reasons reported for not using the app for dosing included not having phone accessible (6 participants), forgetting (4 participants), not having time (4 participants), and having a broken phone/phone problems (4 participants). There were 4 “red alerts” noted in 3 participants. Study staff sent 26 text message and conducted 10 phone interventions in 9 participants in response to non-adherence or red alerts. Additionally, there were 12 messages in 6 participants notifying them that the app was having trouble sending data, advising them to move to a location with a better network connection. All participants entered at least one partner into the sex diary component of the app, with a median of 3 (range 1-10) partners entered.

Qualitative findings from the 4- and 8-week in-depth interviews corroborated the quantitative findings in that the most popular feature of the app was seeing the protection level and a record of their pill-taking. Seeing the rising protection level encouraged use of the app and enhanced participant confidence that the app was science-based. In using the app, one participant assumed “the app will know more than me,” i.e. that the data around protection levels are grounded in pharmacology. Participants also appreciated the messaging that let them know how to get back to full protection. One participant said he was committed to “never dropping to yellow,” indicating that he was below maximum protection, while another said he was very motivated to take his medication after getting into the red. Participants appreciated the reminders to take their pills and seeing the green checks for pills taken on the weekly calendar was helpful for keeping track of their adherence. The process of confirming pill-taking with DOT was found to be time-consuming by some, although a few recognized that the extra step also provided confirmation that the pill was actually taken. Some participants were confused as to why they would need to be monitored to take a pill that benefitted them: “If you know you’re supposed to take it, why not take it?” On the other hand, some participants who were new to taking PrEP said that the app kept them accountable, and helped them to establish a routine. There were also complaints about the processing speed of the app. Participants reported occasionally forgetting to take their pill and noted it would be helpful if they could snooze the reminder for later in the day. While snooze functionality exists on smartphones within short increments (e.g. up to an hour), a reminder for a time later in the day would require changing the original alarm time - existing functionality within the aDOT application. Most participants liked receiving daily reminders, and a few participants noted that they probably would have missed more doses without the reminders and follow-up text messages. Some participants indicated that they used the self-report option if dosing steps were taking too long, and one participant self-reported a dose when he was socializing because it was easier. After using the app for 8 weeks, the majority of participants did not have any privacy concerns. Most indicated that the adherence reported on the app was an accurate reflection of their pill-taking and all participants confirmed they would use the app in a clinical trial.

Use of the sexual diary varied among participants. Some were very enthusiastic about tracking their partners and types of sex. One man indicated that he used the rating system to decide which partners to see again, while others felt that the rating system was “judgey” or objectifying, and preferred not to use it. Participants who were in a relationship or had a greater number of repeat partners did not use the diary as frequently, because the diary did not allow you to enter multiple sexual encounters with the same partner – each encounter was entered as a discrete event. Some participants thought they needed to fill out all of the information in the diary, and were pleased when it was pointed out that they could skip several parts if they wished, such that the diary kept track of the information they were most interested in. Participants stated they would be more motivated to enter information into the diary if they were to receive feedback from the app such as graphs or charts of their partners or sexual encounters. Privacy did not emerge as a major concern during the in-depth interviews. Some participants reported that their diary records did not always represent their actual sexual encounters – some only recorded certain types of encounters (new partners, exciting encounters), and some recorded only when they remembered or not at all.

DISCUSSION

Strategies to support PrEP adherence among youth are critical to preventing new HIV infections in this vulnerable population. We describe the development and optimization pilot study of a novel mobile app using artificial intelligence integrated with an electronic sexual diary to monitor and support PrEP adherence among YMSM. Our formative qualitative work revealed high levels of interest in a combination pill-taking / sexual diary app. In particular, YMSM wanted feedback on their level of protection from PrEP based on data entered into the app. Additionally, they recommended streamlining the tutorial and dose verification steps for the aDOT component, greater customization of the snooze functions and reminders, and preferred step-wise, nonjudgmental communication from study staff in the event of missed or irregular dosing detected by the app.

Based on feedback from end-users in the developmental phase, our team developed and refined the DOT Diary app and tested it in an 8-week open optimization pilot among African American and Latinx YMSM in the San Francisco Bay and Atlanta metropolitan regions. These results demonstrated initial evidence of the feasibility and acceptability of the app, with high levels of app use over the 2 month duration, excellent app usability scores, and very high oral PrEP adherence based on visual confirmation using aDOT. Most participants reported that the app helped with taking PrEP daily. The most helpful components of the app were the visual display of level of protection, and the personalized messaging on the number of doses needed to reach or maintain high protection. Most participants would use DOT Diary to take PrEP in the future and would recommend it to others.

Most focus group participants and all open pilot participants reported they would be willing to use the aDOT component as part of a clinical trial, and the majority of optimization pilot participants experienced no challenges with using aDOT for dosing. Some participants did report the processing speed of dosing to be slow, which is a feature that is continually being upgraded and improved in new releases of the aDOT platform and can be impacted based on the smartphone model. Because of its ability to ensure treatment adherence, directly observed therapy (DOT) has been used for decades to measure and maximize adherence for treatment of tuberculosis infection,⁵⁹ and more recent studies have found DOT to be successful in improving adherence to antiretroviral therapy in HIV patients,^60-62 including in African American and Latino populations.^63,64 While the cost and logistical complexity of DOT in large clinical trials can be prohibitive, aDOT offers a promising, streamlined strategy for real-time monitoring and support of medication adherence.

Mobile apps that record pill-taking and sexual behavior data are a promising tool for monitoring PrEP adherence and behavioral risk in HIV prevention studies. For example, in the recently completed Amsterdam PrEP Project of daily or event-driven PrEP, an app to measure PrEP adherence and sexual behavior based on self-reported data was used frequently over the first year, and data collected by the app were consistent with questionnaire data among those who used the app consistently.⁶⁵ In this study, the sexual diary component of the app was used at least once by all participants in the open pilot, although use was less in those who had a primary partner or who had multiple encounters with the same sexual partner. This was due in part to the requirement to re-enter partner information for repeat sexual partners. Additionally, some participants did not want to complete all data fields for each partner. In future studies, participants will be informed that not all fields need to be completed in the diary, and we will add functionality to allow multiple encounters for a single partner entry. As participants reported that they would be more motivated to complete the diary if the app provided additional feedback and insights into their sexual trends, we have added data visualizations to the next version of the app, including a breakdown of adherence by days of the week, sexual encounters classified by protection level, and a summary of the HIV status of partners and most frequent sexual activities (Figure 3).

Figure 3:

Data Visualizations Added to New Version of DOT Diary App

This study had several limitations. First, the optimization pilot was small, and participants were followed for a short duration. Second, social desirability bias may have influenced participants to preferentially provide positive feedback about the app in the in-depth interviews, although participants were reminded that there were no right or wrong answers at the beginning of the interview and were encouraged to provide both positive and negative feedback. Finally, this pilot study used an open, non-randomized design and therefore did not have a control condition where adherence was measured as a comparator. Despite these limitations, this study also had numerous strengths, including the iterative developmental phase to elicit feedback from youth on the design of the integrated app, and enrollment of a highly diverse population of YMSM from two regions of the US heavily impacted by HIV. Our enrollment of largely Black and Latino YMSM into our formative work and pilot study ensured that the app was tailored to the support needs of youth most vulnerable to HIV infection and will increase the generalzability of our findings.

In summary, young MSM are in great need of tailored strategies to support PrEP adherence. Through qualitative focus groups, an optimization pilot and in-depth-interviews, we developed and refined a novel mobile app integrating artificial intelligence-confirmed dosing with a sex-positive sexual diary to support ongoing PrEP engagement among YMSM. Our pilot study demonstrated high levels of use and acceptability of the app, with high PrEP adherence over 8 weeks. These encouraging findings support further testing in a larger, 6-month randomized controlled trial to evaluate the impact of DOT Diary on PrEP adherence among YMSM and the accuracy of adherence measured by aDOT compared with pharmacokinetic measures, currently underway [NCT03771638]. If found to be effective, DOT Diary could be a highly scalable PrEP adherence intervention for YMSM and help address HIV disparities in this vulnerable population.