Leveraging Technology to Blend Large-Scale Epidemiologic Surveillance With Social and Behavioral Science Methods: Successes, Challenges, and Lessons Learned Implementing the UNITE Longitudinal Cohort Study of HIV Risk Factors Among Sexual Minority Men in the United States

Abstract

The use of digital technologies to conduct large-scale research with limited interaction (i.e., no in-person contact) and objective endpoints (i.e., biological testing) has significant potential for the field of epidemiology, but limited research to date has been published on the successes and challenges of such approaches. We analyzed data from a cohort study of sexual minority men across the United States, collected using digital strategies during a 10-month period from 2017 to 2018. Overall, 113,874 individuals were screened, of whom 26,000 were invited to the study, 10,691 joined the study, and 7,957 completed all enrollment steps, including return of a human immunodeficiency virus–negative sample. We examined group differences in completion of the steps towards enrollment to inform future research and found significant differences according to several factors, including age and race. This study adds to prior work to provide further proof-of-concept for this limited-interaction, technology-mediated methodology, highlighting some of its strengths and challenges, including rapid access to more diverse populations but also potential for bias due to differential enrollment. This method has strong promise, and future implementation research is needed to better understand the roles of burden, privacy, access, and compensation, to enhance representativeness and generalizability of the data generated.

Abbreviations

HIV

human immunodeficiency virus

OR

odds ratio

PrEP

preexposure prophylaxis

SMM

sexual minority men

STI

sexually transmitted infection

UNITE

Understanding New Infections Through Targeted Epidemiology

 

Public health surveillance relies heavily on data collected within clinical settings (1), with prospective cohort designs traditionally relying on in-person, site-based sampling to longitudinally follow a cohort with common characteristics to better understand risk for various outcomes (2–4). Although these methods represent the gold standard for understanding epidemics resulting from discrete biological processes or environmental exposures, social and behavioral science has been crucial for understanding epidemics like human immunodeficiency virus (HIV), which typically include a primary behavioral risk factor and have significant disparities resulting from social and structural factors (5–8). Until recently, however, these designs required distinct recruitment and assessment approaches, timelines, and site-based infrastructures, limiting their ability to integrate and capitalize on the unique benefits of each.

Advances in technology now critically enhance the potential to fuse social and behavioral techniques with large-scale epidemiologic surveillance and cohort designs for complex epidemics like HIV, including populations such as sexual minority men (SMM) (9, 10). Although ongoing surveillance provides key insights regarding trends in the HIV epidemic, surveillance data are significantly limited in terms of the amounts and types of information gathered (11). Conversely, extensive social and behavioral work has detailed a range of modifiable HIV risk factors but has been limited primarily to looking at behavioral risk and proxies for HIV infection given low base rates of HIV incidence that require large sample sizes to be adequately powered to examine infection rates as a primary outcome (12, 13). Moreover, both approaches often rely on site-based procedures that geographically restrict the studies and limit the populations included (14).

Advances in technology and proliferation of their use among high-priority and historically “hard-to-reach” populations allow the integration of rigorous social/behavioral science approaches within the context of large-scale epidemiologic studies to provide unique contributions to our understanding of HIV risk (15–19). Large-scale HIV surveillance by the Centers for Disease Control and Prevention has been critical for establishing and understanding trends in disparities, whereas social/behavioral research has been instrumental in providing insights into the predisposing and mechanistic factors associated with such disparities (20–23). The use of digital technology to conduct large-scale research with comprehensive measures, objective outcomes (e.g., biological testing), and limited interaction (i.e., no in-person contact) have significant potential, but limited research has been published on the successes and challenges of such approaches (17, 24–26).

Our goal is to detail recruitment and enrollment procedures of a longitudinal remote cohort of SMM. In doing so, we focus on detailing the study’s primary procedures and highlighting key implementation issues to inform future use of this methodology, including recruitment costs, enrollment challenges among subpopulations that might introduce bias into the sample, and logistical barriers and facilitators, with concrete recommendations for enhancing these approaches.

METHODS

Understanding New Infections Through Targeted Epidemiology (UNITE) is one of several cohorts funded by the National Institutes of Health, designed to leverage technology to enroll and follow large samples of individuals from populations with a high burden of HIV and be powered to examine new HIV infections as an outcome. To ensure representation of those most affected by the domestic HIV epidemic, UNITE was designed to include at least one-quarter SMM aged 16–24 years and half men of color; in line with this, we limited enrollment to participants who identified as Black, Latino, or multiracial midway through recruitment. UNITE was designed using a framework grounded in the syndemics model and the theories of minority stress and intersectionality (27–29). Procedures include routine surveys, available in both Spanish and English, and annual HIV testing.

Participants and procedures

Recruitment occurred from November 2017 through September 2018. We used online strategies across a range of venues, including geosocial networking applications, social media sites, website referrals, and e-mail blasts. Potential participants were shown an ad that included text indicating they could receive a free at-home HIV test mailed to them or receive up to $275 for joining a study if shown to be eligible for a given study. Individuals were directed to click through the ad to take a brief screening questionnaire to determine eligibility for multiple paid research studies. Subsequently, participants were asked to report their age, with those reporting an age under 13 years being immediately skipped to the end of the survey. Participants were then routed to a brief assent or consent form (we obtained a waiver of parental permission for those under 18) before completing questions about sociodemographic characteristics, sexual behaviors, substance use, HIV and sexually transmitted infection (STI) testing behaviors, and HIV prevention and care.

Participants were deemed eligible if they: 1) were ≥16 years of age; 2) reported HIV-negative or unknown status; 3) identified as male, including transgender men; 4) identified with a sexual minority identity (i.e., gay, queer, bisexual); 5) had a mailing address at which packages could be received in one of the 50 US states or Puerto Rico; 6) were recruited from or reported using geosocial networking applications to meet partners in the past 6 months; 7) reported willingness to complete at-home HIV and STI testing; 8) allowed their contact information to be shared with distributors for the purposes of testing kit and compensation delivery; and 9) reported risk for HIV.

We slightly modified the Centers for Disease Control and Prevention criteria for preexposure prophylaxis (PrEP) indication at the time of the study to define the criterion regarding risk for HIV infection as being the presence of at least 1 of the following in the past 6 months: 1) self-reported diagnosis with an STI; 2) having been prescribed postexposure prophylaxis (PEP); 3) reported condomless anal sex with a casual male partner of any HIV status or an HIV-positive or status-unknown main partner; 4) reported condomless anal sex with an HIV-positive or unknown status main partner; and 5) reported condomless anal sex with their main partner in the context of a nonmonogamous partnership in which their male partner was having condomless anal sex with other partners (22, 23). Finally, participants who reported current PrEP use were deemed eligible only if they reported suboptimal adherence that would place them at risk for HIV infection, defined as missing 4 or more days of dosing in a row or suboptimal adherence (fair, poor, or very poor) using a validated measure of adherence behavior (30).

Eligible participants received a brief description of UNITE and were asked to provide contact information including name, cell phone number, and e-mail address. Eligible participant contact information was securely imported into a study database and automatically verified to ensure it did not duplicate data of an existing participant. At least 5 days per week, the database was queried to automatically generate e-mails to eligible participants containing a unique link to the enrollment survey in Qualtrics (Qualtrics International Inc., Provo, Utah). The e-mail notified participants that they can start and stop across multiple occasions and that joining the study made them eligible for random iPad (Apple, Cupertino, California) drawings. Additional steps were taken to ensure the safety of those participants under age 18 for whom parental permission was waived, including additional screening and discussions of privacy concerns, use of password-protected study pages, and the option of having testing kits sent to a location other than their home.

Before beginning the survey, participants watched a 5-minute video that detailed key information about the study; following the video, participants completed reCAPTCHA verification (31). Participants next proceeded to the full study’s informed consent or assent documentation—this longer form (approximately 1,500 words) focused on the elements of the study in its entirety, including at-home testing for HIV, and compensation. Participants were provided with an opportunity to download a digital copy of the document, after which they were required to respond in the affirmative to 8 statements verifying their age, understanding of procedures and voluntary nature of participation, and willingness to complete HIV and STI testing, receive text messages, and have their contact information used to mail testing kits and provide electronic gift-card compensation. Upon providing affirmative consent or assent, participants completed a 6-item comprehension quiz and received corrective feedback as needed. The 1-hour survey consisted of a range of questions about sociodemographic characteristics, elements of the syndemics and minority stress models (e.g., depression, stigma), and HIV and STI testing and prevention practices. Upon completion of the survey, participants were asked to confirm their contact information and provide a mailing address for the home-based testing, which was automatically validated with the US Postal Service Web Tools. Before being sent an at-home testing kit, participants were required to respond to an automated text message to verify the authenticity of their number and minimize multiple attempts at participation.

Participants were automatically added to the next available wave of testing kits to be distributed after completing text-message verification. Participants were randomly assigned to receive or not receive a rectal STI testing kit along with their HIV test. Due to costs, we randomly selected half of the sample to complete this objective measure of sexual risk. We used an HIV-1 Oral Specimen Collection Device (OraSure Technologies, Bethlehem, Pennsylvania) to gather oral samples (32) and an Aptima multitest swab specimen collection kit (Hologic Inc., Marlborough, Massachusetts) for rectal samples (33). Participants were provided with a link to a survey that contained an instructional video on how to perform the oral fluid collection and rectal swab.

Upon completion of eligible screening, the enrollment survey, contact info, and text-message verification, and receipt of a sample for HIV testing by the lab (regardless of successful rectal STI sampling), participants were compensated with a $25 Amazon electronic gift card and enrolled in the cohort.

Measures

During screening, participants were asked to self-report their age, ethnic and racial self-identification, sexual identity, zip code (which was converted to regions), whether they had a main partner, insurance status, PrEP status, how recently they had received an HIV test, and the history and recency of any STI diagnoses. Participants also responded about the use of 14 different drugs (alcohol, cocaine, crystal methamphetamine, ecstasy, “GHB/GBL/etc.,” heroin/opiates, ketamine, marijuana, crack, poppers, sildenafil/vardenafil/tadalafil, prescription stimulants, prescription sedatives, prescription pain killers) in the past 6 months, which was recoded into a dichotomous indicator of recent drug use. For prescription drugs, participants were instructed to respond with instances of using them without a prescription, using more than prescribed, or using them for a recreational purpose. All variables of interest were pulled from the screening survey to ensure we could look at factors associated with enrollment among all participants who had completed screening.

Statistical analyses

Analyses were conducted using SPSS, version 24 (International Business Machines, Armonk, New York), and focused on group differences in proceeding through the UNITE enrollment process. We were interested in examining achievement of each milestone among those who completed the prior milestone to inform how study procedures affected sample makeup and any points at which bias might be introduced into the study sample. We began by using descriptive statistics to characterize success metrics among each recruitment venue and χ² analyses to compare the sociodemographic characteristics of enrolled participants across recruitment venues. Next, we assessed bivariate group differences using χ² analyses examining completion of each of the following enrollment steps: 1) providing contact information after being deemed eligible; 2) completing the enrollment survey; and 3) completing testing with an HIV-negative result. After examining unadjusted completion rates for each enrollment milestone nested within one another, we conducted multivariable analyses to examine the independent effects (i.e., adjusted odds) of each factor on completion of every enrollment milestone among all eligible participants. We entered all factors into a multivariable logistic regression predicting completion of enrollment to assess the independent impact of each factor. Across analyses, we sought to minimize the interpretation of statistically significant results with low practical significance that might be due solely to the large sample sizes and only describe group differences as significant if they fell below the threshold of P < 0.001.

RESULTS

The flowchart in Figure 1 displays the progression of participants through the process from initial screening through to completing enrollment and Figure 2 displays the geographic distribution of the enrolled cohort. Table 1 displays sociodemographic and behavioral characteristics of both the full sample screened for eligibility and the final enrolled cohort.

Figure 1.

Flowchart showing the enrollment process from initial screening through to completion of testing and full enrollment, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018. HIV, human immunodeficiency virus.

Figure 2.

US map indicating aggregated location information (based on home zip code) of fully enrolled participants at both the state (shading) and local (circles) levels, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018. Alaska, Hawaii, and Puerto Rico not to scale.

Table 1.

Characteristics of the Sample Reached for Screening and the Final Enrolled Cohort, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018

Characteristic	Completed Screening   (n = 113,868a)		Fully Enrolled   (n = 7,956b)
	No.	%	No.	%
Age group, years
<18	1,559	1.4	80	1.0
18–24	26,809	23.5	2040	25.6
25–29	22,885	20.1	1849	23.2
30–49	46,205	40.6	3,119	39.2
≥50	16,410	14.4	868	10.9
Race/ethnicity
Black	14,775	13.0	868	10.9
Latino	20,806	18.3	1,570	19.7
White	59,673	52.4	4,103	51.6
Multiracial	12,114	10.6	954	12.0
Other	6,500	5.7	461	5.8
Region
Northeast	21,680	19.0	1,338	16.8
Midwest	20,165	17.7	1,460	18.4
South	40,356	35.4	2,806	35.3
West	30,366	26.7	2,289	28.8
Puerto Rico	1,201	1.1	62	0.8
Military overseas or invalid zip code	104	0.1	0	0.0
Sexual identity
Gay	88,994	78.2	6,505	81.8
Straight	1,568	1.4	– c	– c
Queer	3,323	2.9	223	2.8
Bisexual	19,962	17.5	1,228	15.4
Lesbian	21	0.0	– c	– c
Relationship status
Single	79,222	69.6	5,800	72.9
Partnered	34,646	30.4	2,156	27.1
Insurance status
No insurance	22,788	20.0	1,626	20.4
Private insurance	70,694	62.1	5,190	65.2
Public insurance	20,386	17.9	1,140	14.3
PrEP status
Currently on PrEP	14,645	12.9	410	5.2
Previously on PrEP	5,610	4.9	817	10.3
Never taken PrEP	77,029	67.6	6,729	84.6
Participant is HIV-positive	16,595	14.6	– c	– c
Most recent HIV test
Within the past 6 months	51,677	45.4	5,086	63.9
Within the past 7–12 months	11,713	10.3	1,550	19.5
Over a year ago	20,127	17.7	857	10.8
“I’ve never been tested for HIV”	13,756	12.1	463	5.8
Participant is HIV-positive	16,595	14.6	– c	– c
Recent STI, last 6 months
No	85,997	75.5	6,765	85.0
Yes	11,276	9.9	1,191	15.0
Participant is HIV-positive	16,595	14.6	– c	– c
Drug use last 6 months
No	92,204	81.0	6,553	82.4
Yes	21,664	19.0	1,403	17.6

Abbreviations: HIV, human immunodeficiency virus; PrEP, preexposure prophylaxis; STI, sexually transmitted infection.

^a Six eligible participants were excluded from analysis due to missing data.

^b One enrolled participant was excluded from analysis due to missing data.

^c Several cells for fully enrolled participants are empty by design due to inclusion criteria.

Recruitment and screening

As shown in Figure 1, a total of 113,874 unique screening surveys were completed. Tables 2 and 3 contain a breakdown of our digital recruitment venues and metrics of their reach and success as well as key sociodemographic characteristics we sought to reach among enrolled participants broken out by venue. Overall, approximately one-quarter (24.0%) were deemed eligible and provided contact information, and 29.0% of eligible contacts fully completed enrollment. Advertising costs ranged substantially across venues. In terms of reach for screening, social media and the 2 networking applications showed substantially lower costs, which was also true when considering costs per eligible contact. Due to differing enrollment rates, however, Networking App 1 proved most successful both in terms of volume and cost per enrolled participants. Despite lower overall volume and higher costs, social media (i.e., Facebook and Instagram) and BGCLive performed best in terms of achieving racial and ethnic diversity among enrolled participants, and social media performed best at enrolling the youngest age groups.

Table 2.

Digital Recruitment Venue Success Metrics, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018

Success Metric	Social Media			Networking App 1			Networking App 2			Adam4Adam			BGCLive			Other
	No.	Cost, $	%	No.	Cost, $	%	No.	Cost, $	%	No.	Cost, $	%	No.	Cost, $	%	No.	Cost, $	%
Screened	2,636			71,880			32,581			2,533			965			3,273
Eligible (screened)	631		23.9	19,989		27.8	5,137		15.8	683		27.0	205		21.2	713		21.8
Enrolled (eligible)	115		18.2	5,698		28.5	1,565		30.5	220		32.5	41		20.0	318		44.6
Advertising volume, days	101			59			84			30			45			N/A
Estimated ad costs per completed screener		0.86			0.89			0.77			1.97			1.69			1.68
Estimated ad costs per eligible contact		3.61			3.20			4.87			7.32			7.95			7.71
Estimated ad costs per enrolled participant		19.83			11.23			15.97			22.72			39.76			17.30

Abbreviations: BGC, Black Gay Chat; N/A, not available,

Table 3.

Digital Recruitment Among Key Subpopulations, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018

Population Characteristic	Social Media Enrolled (n = 115)		Networking App 1 Enrolled (n = 5,698)		Networking App 2 Enrolled (n = 1,565)		Adam4Adam Enrolled (n = 220)		BGCLive Enrolled (n = 41)		Other Enrolled (n = 318)		χ 2	P Value
	No.	%	No.	%	No.	%	No.	%	No.	%	No.	%
Race/ethnicity													390.4	<.001
Black	16	13.9	595	10.4	155	9.9	24	10.9	38	92.7	40	12.6
Latino	39	33.9	1,082	19.0	350	22.4	30	13.6	0	0.0	69	21.7
White	33	28.7	3,022	53.0	743	47.5	142	64.5	3	7.3	160	50.3
Multiracial	23	20.0	627	11.0	252	16.1	18	8.1	0	0.0	34	10.7
Other	4	3.5	372	6.5	65	4.2	6	2.7	0	0.0	15	4.7
Age group, years													481.0	<.001
<18	5	4.3	71	1.2	3	0.2	0	0.0	0	0.0	1	0.3
18–24	48	40.9	1,650	29.0	254	16.2	13	5.9	6	14.6	69	21.8
25–34	32	35.5	1,375	24.4	337	21.6	27	12.3	213	31.7	65	20.6
35–49	27	18.3	2,106	37.0	754	48.2	87	39.5	19	46.3	126	39.7
≥50	3	2.6	496	8.7	217	13.9	93	42.3	3	7.3	56	17.7
HIV status													87.0	<.001
Negative	70	60.9	4,383	76.9	1,336	85.4	160	72.7	28	68.3	268	84.0
Unknown	45	39.1	1,315	23.1	229	14.6	60	27.3	13	31.7	49	16.0
Region													689.1	<.001
Northeast	26	22.6	991	17.4	263	16.8	35	15.9	6	14.6	47	14.7
Midwest	13	11.3	1,049	18.4	269	17.2	54	24.5	7	17.1	60	18.8
South	31	27.0	2,073	36.4	512	32.7	73	33.2	25	61.0	114	35.7
West	20	22.2	1,558	27.5	509	32.8	57	26.0	3	7.3	96	30.3
Puerto Rico, military overseas, or invalid zip code	25	21.7	25	0.4	12	0.8	1	0.5	0	0.0	2	0.6

Abbreviations: BGC, Black Gay Chat; HIV, human immunodeficiency virus.

One notable challenge was that one-quarter (n = 9,785, 26.3%) of people deemed eligible (n = 37,149) to participate did not provide contact information. The left-most column in Table 4 compares eligible participants who did and did not provide contact information, and we observed significant differences across nearly all metrics, with several notably different rates. In particular, we observed comparatively low rates of eligible participants providing contact information among both the youngest and oldest age groups (71.4% of those aged 16–17 years; 71.9% of those aged 18–24; 70.3% of those aged ≥50 years), bisexually identified SMM (68.8%), those who reported having private health insurance (71.4%), those who had never taken PrEP (72.9%), those who had not used drugs in the past 6 months (71.9%), and those who had never had taken PrEP (72.9%).

Table 4.

Sociodemographic and Behavioral Differences in Achieving Each Study Milestone From Eligibility Through Completion of Enrollment^a, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018

Characteristic	Eligible Contacts (n = 27,358)   of All Eligible (n = 37,143)b				Completed Enrollment Survey   (n = 10,687c) of All Sent (n = 26,000)				HIV-Negative Test Result (n = 7,956 d ) of Sent Test Kits (n = 10,685)
	No.	%	χ 2	P Value	No.	%	χ 2	P Value	No.	%	χ 2	P Value
Age group, years			67.10	<0.001			137.6	<0.001			19.91	0.001
<18	506	71.4			114	40.3			80	84.8
18–24	8,297	71.9			2,832	36.1			2040	72.0
25–34	6,384	75.4			2,512	41.0			1849	73.5
35–49	9,636	75.1			4,100	44.3			3,119	76.0
≥50	2,535	70.3			1,129	45.4			868	77.0
Race/ethnicity			20.09	<0.001			205.17	<0.001			130.67	<0.001
Black	3,982	73.3			1,266	34.5			868	68.6
Latino	6,514	73.9			2,270	38.0			1,570	69.2
White	11,616	73.1			5,195	45.6			4,103	79.0
Multiracial	3,886	75.9			1,364	37.8			954	69.9
Other	1,360	71.9			592	44.3			462	77.9
Region			17.67	0.003			20.41	0.001			12.68	0.03
Northeast	4,838	72.8			1806	39.9			1,338	74.1
Midwest	4,670	73.3			1936	43.3			1,460	75.4
South	9,787	73.8			3,780	40.7			2,806	74.2
West	7,739	74.4			3,061	41.3			2,289	74.8
Puerto Rico	316	73.0			103	34.8			62	60.2
Military overseas or invalid zip code	7	14.0			0	00.0			0	00.0
Sexual identity			118.05	<0.001			42.67	<0.001			16.80	<0.001
Gay	21,576	74.7			8,660	42.1			6,505	75.1
Queer	731	78.6			288	41.6			223	77.4
Bisexual	5,040	68.8			1739	36.9			1,228	70.6
Relationship status			2.16	0.09			1.25	0.26			0.09	0.76
Single	20,031	73.9			7,799	40.9			5,800	74.4
Partnered	7,327	73.0			2,888	41.7			2,156	74.7
Insurance status			144.13	<0.001			118.99	<0.001			85.74	<0.001
No insurance	6,892	76.8			2,356	36.5			1,626	69.0
Private insurance	15,914	71.4			6,701	43.8			5,190	77.5
Public insurance	4,552	77.3			1,630	41.1			1,140	69.9
PrEP status			80.40	<0.001			8.38	0.02			2.08	0.35
Currently on PrEP	1,268	79.8			533	43.5			410	76.9
Previously on PrEP	2,640	78.4			1,109	43.1			817	73.7
Never taken PrEP	23,450	72.9			9,045	41.1			6,729	74.4
Most recent HIV test			121.01	<0.001			216.21	<0.001			32.37	<0.001
Within the past 6 months	16,260	72.0			6,747	42.7			5,086	75.4
Within the past 7–12 months	4,554	74.1			2036	45.6			1,550	76.1
Over a year before	3,878	79.5			1,235	35.7			857	69.4
“I’ve never been tested”	2,666	75.2			669	29.7			463	69.2
Recent STI, past 6 months			4.26	0.04			6.49	0.01			5.26	0.02
No	22,969	73.5			9,037	41.4			6,765	74.9
Yes	4,389	74.7			1,650	39.3			1,191	72.2
Drug use, past 6 months			259.66	<0.001			41.57	<0.001			61.89	<0.001
No	21,527	71.9			8,614	42.1			6,553	76.1
Yes	5,831	81.2			2073	37.3			1,403	67.7

Abbreviations: HIV, human immunodeficiency virus; PrEP, preexposure prophylaxis; STI, sexually transmitted infection.

^a Completion of enrollment was defined as returning an HIV-negative sample.

^b Six eligible participants were excluded from analysis due to missing data.

^c Four eligible participants were excluded from analysis due to missing data.

^d One enrolled participant was excluded from analysis due to missing data.

Enrollment procedures

Of the 27,364 who left suitable contact information, 26,000 were sent the enrollment survey and 10,691 (41.1%) completed it (6 eligible participants were missing data and excluded from analyses reported in the tables, resulting in an analytical sample size of n = 27,358). A small fraction (2.9%) withdrew participation, did not provide contact information, or never completed text-message verification after completing this enrollment survey. In total, 10,385 test kits were sent out, and 79.7% of samples were returned (Figure 1). Among those returning a valid testing sample, the point prevalence estimate for undiagnosed HIV infection was 1.9% (n_cases = 154, n_tested = 8,105). In total, 7,957 SMM were verified HIV-negative and fully enrolled in the UNITE cohort, and their characteristics are displayed on the right-hand side of Table 1 (1 of these participants is missing data and excluded from numbers within all tables, resulting in an analytical sample size of n = 7,956).

Table 4 shows rates and comparisons of the completion of the enrollment survey (the middle column) and full enrollment with a valid, HIV-negative test result (the right-hand column). Among those who were invited to complete the enrollment survey, we identified significant differences across a range of characteristics. In particular, age, race/ethnicity, insurance status, and recency of HIV testing showed noteworthy differences suggesting lower rates of survey completion among the youngest age groups; participants who identified as Black, Latino, and multiracial; those without insurance; and those who had never tested for HIV. Among those who did complete the survey, completion of an HIV-negative test also showed numerous significant differences largely consistent with those for the enrollment survey, with lower rates among those who identified as Black, Latino, or multiracial; those without insurance; and those who had not tested within the past year for HIV, as well as a notable difference showing lower rates of completed testing among recent drug users.

Table 5 displays a model with adjustments examining factors independently associated with full enrollment among participants invited to complete enrollment (i.e., collapsing across the previously disaggregated enrollment steps), and several meaningful differences emerged according to age, race/ethnicity, region, sexual identity, insurance status, HIV testing recency, and drug use. Overall, participants aged 16–17 years had significantly lower odds of enrollment than those of all age groups ≥25 years. Black men had the lowest odds of enrollment and significantly differed compared with participants who identified as White (adjusted odds ratio (OR) = 1.72), multiracial (adjusted OR = 1.21), or another race (adjusted OR = 1.59). SMM from the Midwest (adjusted OR = 1.17) and West (adjusted OR = 1.19) had significantly higher enrollment than those from the Northeast. SMM who identified as bisexual had significantly lower enrollment (adjusted OR = 0.76) than gay-identified men. SMM with private insurance (adjusted OR = 1.25) had significantly higher odds of enrolling compared with those who had no insurance. SMM who had been tested within the past 6 months (adjusted OR = 1.61) or within the past 7–12 months (adjusted OR = 1.83) had higher odds of enrolling compared with those who had never tested. Finally, SMM who reported drug use (adjusted OR = 0.85) had significantly lower odds of enrollment.

Table 5.

Adjusted Odds of Full Enrollment (n = 7,956^a) According to Sociodemographic and Behavioral Characteristics Among All Eligible Participants (n = 37,143^b), Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018

Characteristic	Adjusted OR	95% CI	P Value
Age group, yearsc
18–24	1.34	1.05, 1.71	0.02
25–34	1.69	1.32, 2.17	<0.001
35–49	1.88	1.47, 2.40	<0.001
≥50	1.66	1.29, 2.14	<0.001
Race/ethnicityd
Latino	1.15	1.05, 1.26	0.004
White	1.72	1.59, 1.87	<0.001
Multiracial	1.21	1.09, 1.34	<0.001
Other	1.59	1.44, 1.81	<0.001
Regione
Midwest	1.17	1.07, 1.27	<0.001
South	1.14	1.06, 1.23	0.001
West	1.19	1.10, 1.28	<0.001
Puerto Rico	0.83	0.63, 1.10	0.19
Military overseas or invalid zip code	0.24	0.03, 1.80	0.17
Sexual identityf
Queer	1.15	0.99, 1.35	0.08
Bisexual	0.76	0.71, 0.81	<0.001
Relationship statusg
Partnered	0.95	0.89, 1.00	0.05
Insurance statush
Private insurance	1.25	1.17, 1.33	<0.001
Public insurance	1.10	1.01, 1.20	0.03
PrEP statusi
Previously on PrEP	0.90	0.78, 1.04	0.14
Never taken PrEP	0.82	0.72, 0.92	0.001
Most recent HIV testj
Within the past 6 months	1.61	1.44, 1.80	<0.001
Within the past 7–12 months	1.83	1.62, 2.06	<0.001
Over a year ago	1.19	1.04, 1.35	0.01
Recent STI, past 6 monthsk
Yes	0.92	0.85, 0.99	0.02
Drug use, past 6 monthsk
Yes	0.85	0.79, 0.91	<0.001

Abbreviations: CI, confidence interval; HIV, human immunodeficiency virus; OR, odds ratio; PrEP, preexposure prophylaxis; STI, sexually transmitted infection.

^a One enrolled participant is missing data necessary to be included within analyses.

^b Six eligible participants were excluded from analysis due to missing data.

^c Referent: under 18 years of age.

^d Referent: Black.

^e Referent: Northeast.

^f Referent: gay.

^g Referent: single.

^h Referent: no insurance.

ⁱ Referent: currently on PrEP.

^j Referent: never been tested for HIV.

^k Referent: no.

Duplicate case identification

All analyses reported above exclude any responses that were deemed as repeats from a prior respondent. Using an algorithm based on Internet Protocol address and demographic information, 5,078 screening surveys were flagged as likely duplicates and removed prior to analyses, suggesting that only 4.3% of participants completed the screening survey more than once despite routine advertising. Eligible contacts were automatically screened for duplication in e-mail and phone number, and we automatically flagged 2,919 contacts comprising 1,178 unique individuals, with only the first entry producing an invite to join the study. Following the automated detection of duplicate contacts, 65 instances of duplicate participation among participants invited to join the study were identified during study procedures based on mailing address and other factors—23 of these instances originated from only 2 individuals, representing the only suspected cases of intentionally enrolling multiple times.

DISCUSSION

This present work reported procedures for and enrollment data from a nationwide cohort of SMM that relied on technology-mediated limited-interaction methods. Enrollment of the proposed cohort was successful, screening 113,874 SMM and meeting the target of completing survey data collection and HIV testing with ≥8,000 SMM in less than 1 year. The enrolled sample was diverse, including more than one-quarter under age 25 years, nearly half men of color, more than one-third from the South, one-fifth who were uninsured, more than 15% who had not tested for HIV in the past year, 15% with a self-reported STI diagnosis in the past 6 months, and nearly 18% who were actively using drugs; moreover, the sample was geographically dispersed and represented SMM from every state and Puerto Rico, including highly remote and rural regions and those completely disconnected from HIV prevention services. Enrolling these populations, while difficult for traditional cohort and clinic-based studies, was facilitated by the online nature of the study design and allowed for the sampling of diverse, dispersed, and disconnected participants who are hard to reach with traditional methods. As such, this study adds to prior work to provide further proof-of-concept for this nascent methodology, and also highlighted some of its strengths, challenges, and strategies for further optimization (17, 26). Table 6 contains a series of challenges identified during the process of designing and implementing the study and a description of lessons learned and recommendations for future research.

Table 6.

Key Challenges, Lessons Learned, and Recommendations for Future Researchers, Understanding New Infections Through Targeted Epidemiology, United States, 2017–2018

Challenge	Lessons Learned And Recommended Practices
Identifying venues for ongoing high-volume recruitment poses challenges	The leading mobile application and website companies that are most popular among sexual minority men have a large reach, but advertising multiple times leads to duplicate participants as well as advertising fatigue. Recommendations include limiting the number of advertisements per venue, spacing advertisements in the same venue by several weeks, and targeting fewer, higher-volume advertisement options versus a more evenly distributed and longer advertising program.
Marketing trends and advertising costs present a challenge for recruitment	Different types of advertisement within the same venue have differing effectiveness, and our work demonstrated that “inbox” style advertisements were more effective than pop-ups that could be easily and quickly dismissed. For social media where there is significant corporate advertising, costs per impression go up substantially around holidays and other high-volume shopping periods, and use of these should be limited during such times. Finally, due to high costs per eligible/enrolled participant, screening for multiple studies at the same time significantly enhances the value and diminishes overall costs of advertisements.
Avoiding duplicate and fraudulent enrollments is logistically challenging	Several studies have been published that document best practices for identifying fraudulent and duplicate participants. Strategies successfully used in the present study included nonincentivized screening, requiring completion of multiple steps before receiving compensation, and requiring verification by multiple means (i.e., completing links within study e-mails, responding to an automated text message, and receiving a package at a residential address).
Few devices available to conduct at-home sampling for lab-based HIV testing	There is a strong ethical imperative to inform research participants of their HIV status; thus, an FDA-approved clinical test is the preferred option even within the context of research. Few such devices exist and, as of the start of this study, none were FDA-approved for at-home sample collection. This requires a validation study to be conducted by a CLIA-approved clinical laboratory before use, which also makes finding a laboratory more difficult.
Compensation challenges to adequate recognition of participants’ time and effort while minimizing coercion and fraudulent enrollment	Compensation values participants’ time and efforts and thus should be set at an appropriate level that encourages completion of study tasks without being coercive or incentivizing multiple enrollments. Based on our experiences with enrollment and feedback received, future studies should include higher compensation (e.g., $50), splitting up payments (e.g., compensating for the survey and HIV test separately), but also incentivizing full completion by building in bonus structures. These strategies implemented over time in our study have proven fruitful.
Reducing participant burden	Self-report surveys, in particular, are a source of potential frustration at longer durations. The field would benefit strongly from the development of standardized, comprehensive, and brief survey measures that encompass a range of social, psychological, and behavioral factors that serve as barriers and facilitators to HIV prevention. Surveys should be limited to 30–45 minutes, on average, and should focus on limiting perceived redundancy, highlighting the importance of the data, and providing an explanation of why different questions are being asked (particularly if sensitive in nature).
Minimizing participant confusion	One of the largest difficulties in a remote study with a substantial sample size recruited in a brief period of time is effectively responding to participant e-mails and other communications. Beyond individual issues that warranted one-on-one communication, the largest reason for participant e-mails was confusion about where they were in the enrollment process, what they had and hadn’t completed, and when they would be compensated. To minimize this, 2 strategies implemented over time in our study have proven significantly helpful. First, using informational videos is a cost-effective way of explaining different procedures, and our participants indicated a preference for animated rather than live-action videos. Second, login-based home pages or study portals should be provided that allow a participant to see a breakdown of all activities, their completion status, and associated compensation (if relevant).
Preparing at-home test kits for shipment	Preparation of mailers with at-home test kits and supplemental material (e.g., a card with participant’s test kit identification, a card with important information) must be performed well in advance for rapid distribution. Over time, we discovered the importance of making small but important adjustments to the contents based on participant e-mails and common issues reported by the lab, including the modification of our information card to ensure proper procedures were followed. Additionally, rate changes over time for shipping led to the need to modify the postage of our return envelopes that had already been packed and sealed within the outgoing envelope. To minimize personnel effort and supply cost, sealing of mailers should be postponed until the date of shipment, and preprinted postage should only be applied for kits expected to be used within 6 months to avoid postage changes.

Abbreviations: CLIA, Clinical Laboratory Improvement Amendments; FDA, Food and Drug Administration; HIV, human immunodeficiency virus.

Given the disproportionate burden of the HIV epidemic, our study specifically aimed to have substantial representation of SMM who identified as Black, Latino, and multiracial as well as those aged 16–24 years who are at risk for HIV infection. Regarding racial and ethnic makeup of the sample, Black, Latino, and multiracial men completed both the enrollment survey and the at-home HIV testing at significantly lower rates than men who identified as White or another race. We also had to continue recruitment of Black, Latino, and multiracial men for a few months after we stopped enrolling men of other racial/ethnic backgrounds in order to ensure that they made up approximately half of the sample. The method proved viable for adolescent SMM in terms of completion rates, which were comparable to older age groups for the survey and the highest of all age groups for testing, although these rates were lowest for the group aged 18–24. In terms of overall numbers, few SMM under age 18 were enrolled in the cohort, although we achieved approximately one-quarter of the sample being under age 25 without closing recruitment to older groups, as was done in terms of race/ethnicity. Most notably, we reached a population with a high burden of HIV that was comparable to or higher than population-based national samples (34–36)—specifically, a self-reported HIV prevalence of 14.6% in the screening sample and a prevalence of 1.9% previously undiagnosed HIV infections detected using remote testing procedures during enrollment.

The remote nature of our study meant that we could rely fully on digital technologies for both recruitment and enrollment procedures, which led to significant cost savings and the ability to enroll participants traditionally underrepresented or absent from site-based research. We successfully met our enrollment targets within 1 year at an estimated advertising cost of $12.99 per fully enrolled participant, which is well below the compensation levels participants receive once joining the study, and our findings can be used to develop advertising campaigns in future research that will be more cost-effective for specific populations. For example, the networking applications were particularly costly but delivered high volumes of enrolled participants, which is critical for large studies, whereas other approaches, such as social media, were somewhat more costly per enrolled participant but would be optimal for studies needing lower numbers but higher access to younger and more racially and ethnically diverse participants while also being able to fine tune the exact spending amount. Moreover, the geographic dispersion of our sample across rural areas, the South, and Puerto Rico, along with the relatively high proportions who were uninsured and who had never been tested for HIV, demonstrate the reach of these methods among populations who are historically much more difficult to reach in site-based research, particularly when recruitment is done with clinic patients.

Although there were numerous successes to the methodology, there are also challenges that must be addressed to enhance its impact. We reached but witnessed more difficulty enrolling SMM with several other important features, including those who were not gay-identified, those without insurance, those who had never tested for HIV, and those who were actively using drugs. Rates of completion and full enrollment were also lower among men residing in Puerto Rico, although it is worth noting that the study was implemented not long after Hurricane Maria. Taken together, these findings highlight that although this methodology can enhance representation of these groups, particularly in terms of numbers, they nonetheless fall short at achieving equitable enrollment rates. Our significant findings regarding barriers at different milestones of enrollment highlight areas of bias that are introduced into the sample. One key approach to addressing this will be the development of sampling weights or propensity scores that account for each of these sources of bias in achieving full enrollment among the underlying—and observed—sample of eligible participants. It remains critical to identify the barriers and facilitators to participation that can enhance these methods and further our ability to represent more marginalized groups within epidemiologic and social/behavioral research on HIV prevention, given that these limited-interaction, technology-mediated methods represent one of the best strategies for engaging many of these groups.

Logistically, these fast-paced studies with little in-person contact require significant advanced planning to avoid numerous pitfalls. For example, there is a substantial risk of duplicate and fraudulent participation when face-to-face interaction is removed, although our relatively straightforward and largely automated procedures proved successful for substantially limiting the number of repeat enrollments in the present study. We have also outlined numerous additional considerations that future researchers might consider within Table 6 to plan and implement studies of this nature that are the culmination of best practices established in this and prior similar studies.

In summary, this study demonstrated further proof about the utility and feasibility of technology-facilitated limited-interaction methods to conduct hybrid social/behavioral and epidemiologic surveillance studies of HIV prevention with SMM. Although this method has strong promise, future implementation research is needed to better understand how procedural burden, privacy concerns, research mistrust, access to technology, compensation, and other factors act as facilitators and barriers to enhance the use of these methods to improve representativeness and generalizability of the data generated.