What works? An empirical perspective on how to retain youth in longitudinal HIV and substance risk reduction studies

Abstract

Background

Low retention rates are a problem for longitudinal studies involving adolescents, and this is particularly true for justice-involved youth.

Methods

This study evaluates: (1) strategies used to retain high-risk adolescents participating in a longitudinal research project, (2) the extent to which retention efforts were different in a justice-involved versus a non-justice involved (school-based) sample, and (3) differential characteristics of justice-involved versus school-based adolescents that might explain differences in retention difficulty.

Results

Compared with the school-based youth, justice-involved youth required significantly more phone calls to be successfully reached. Additionally, baseline substance use (alcohol and marijuana use frequency) was higher in the justice-involved sample and significantly related to retention difficulty.

Conclusions

High retention rates for justice-involved and substance-using youth are possible with focused efforts on frequent communication and effortful contact.

Introduction

Adolescence is a unique developmental period¹. One feature that makes adolescence unique is that natural exploration often takes the form of risk behavior². This is concerning, as many behaviors that emerge and develop during adolescence, such as the initiation of sexual behaviors and substance use^3,4, can have serious unintended health consequences. For example, unprotected sexual activity can result in the transmission of sexually transmitted infections (STIs), and adolescents are one of the groups for which the incidence of STIs and HIV are not decreasing⁵. Substance use not only plays a role in risk behavior in its own right⁶, there is some evidence to suggest that substance use is associated with greater sexual risk consequences, particularly in high risk adolescents^7,8.

As compared with their mainstream, school-attending peers, youth involved with the justice system have even greater rates of health risk behavior. Most justice-involved adolescents are sexually active, have had intercourse with multiple partners, and have a high incidence of unprotected sex⁴. Furthermore, recent studies of justice-involved youth indicate high rates of alcohol and marijuana use⁷. Therefore, developing prevention programming to reduce risky sexual behavior and related substance use among adolescents generally, and justice-involved adolescents in particular, is urgently needed.

One critical factor that complicates research on the long-term effects of behavioral risk reduction efforts in adolescents is retention. While studies actively highlight the difficulties around retaining high-risk and underserved youth⁹, few studies have empirically evaluated which approaches facilitate retention, and/or disseminated those methodological details to other teams in the field. This is particularly important because attrition may be greatest among those engaged in the highest levels of health risk behavior. Ultimately, this may result in our having the least information about how our prevention programming works for those at the greatest need. Not only does this affect statistical power, it impacts our understanding of the real-world potential effectiveness of these interventions^9,10,11. Therefore, it is requisite for prevention researchers to understand not only who might have a more difficult time returning for follow-ups, but also how to better engage these youth in prevention programming and related follow-ups (i.e., are different or extraordinary measures required to retain higher risk youth than more “traditional” youth?).

This study sought to directly address this gap in the literature by examining retention efforts and rates of retention for a six-month study period across two disparate samples (justice-involved youth and school-attending youth) in the same metropolitan area. Consistent with the literature^12,13, we posited that justice-involved youth would report greater rates of substance use and sexual activity, as compared with school-attending youth. In addition, we hypothesized that higher levels of substance use and sexual activity would be associated with lower levels of youth follow-up participation, and greater staff efforts toward retaining those youth.

Methods

As a component of a larger, longitudinal HIV risk reduction intervention evaluation¹⁴ (R01 AA017390; PI: AB) this sample was comprised of youth from local juvenile justice centers who were not actively enrolled in school (justice involved N = 244; male = 176, female = 68), along with a comparison sample of school attending youth from several high schools in the same metropolitan region (school N = 40; male = 23, female = 17). All participating youth were between the ages of 14-18. All youth consented to be re-contacted for follow-up appointments. Data for this study were taken from the baseline, 3 and 6-month behavioral assessments.

All study procedures were paralleled across both study samples. Specifically, all youth were approached by trained research staff at their respective participation center (justice programs, school programs) and provided with a brief description of the study. In order to be eligible for participation, youth had to be 14-18 years of age, demonstrate proficiency in English, and provide their own written informed assent. Youth under the age of 18 also had to have informed consent from a parent/guardian (obtained via tape-recorded telephone conversation). Because this particular study also included a neurocognitive assessment (not included in these analyses), all youth were required to be free of fMRI contra-indications (i.e., absence of metal in the body, and not currently taking neuroleptic/anti-convulsant medication). With university Institutional Review Board approval and a federal Certificate of Confidentiality, in order to improve youth reporting of risk behavior, we maintained complete distinction from the facility staff. Specifically, to ensure youths' privacy and the distinction from facility procedures, all study procedures were completed independently of facility (justice/school) staff. Thus, facility staff did not participate in recruitment, nor were they aware of youth's participation status. All measures were completed on a laptop computer using ACASI¹⁵ (Audiocomputer-assisted Self-interview), which facilitates participation for youth with reading difficulties, and/or navigating complicated skip patterns. Youth were informed that they could skip any question without penalty.

Retention Procedures

Follow-up Timing

During the consent, we informed all youth that they would be re-contacted throughout the course of the upcoming year to complete behavioral follow-ups. Trained RAs informed youth that we would be in touch with them (and their relevant contacts if needed) to facilitate their return. And, trained RAs informed youth that they would receive payment in cash for completing each follow-up. In order to effectively reach youth for their follow-up appointments staff were alerted to contact youth two weeks prior to the follow-up date (see Tracker Database). Staff were given 1 month after the youth's designated follow-up appointment to get them in, thereby providing a full 6 week window for contact and completion.

Adequate Staffing

Consistent with our prior work, we had 2 full time staff members dedicated to retention. This team included at any time at least one full-time research assistant (RA) dedicated to staying in touch with youth. This person was responsible for overseeing all follow-up procedures, including getting in touch with youth, updating and maintaining retention databases, getting participant incentives for the follow-ups, and directing the follow-up RAs in the field. Two part-time male RAs went into the field to complete follow-ups. For the field component, we selected male RAs due to the experienced adverse social consequences (e.g., public harassment, name calling) observed for females in our prior work in high-risk neighborhoods. (see Telephone Calls/Postcards).

Confidential Locator Information

Based upon published studies in this area⁸, and the broader research team's prior experience working with high-risk youth (SFE; AB), we utilized the following approaches to obtain the highest likelihood of retaining youth. Trained RAs had youth complete confidential locator forms at every in-person meeting (see Figure 1). Due to the highly transient nature of justice-involved youth, these forms were highly detailed, including residence data, at least five active phone contacts, contact information for at least one individual “who would be able to find them” in three months, justice information, and social networking information. If we had evidence that youth were no longer residing at the same location, the RAs telephoned youth to update their confidential locator phone information via telephone conversation or text message.

Follow-Up Tracking Database

One full time RA entered the primary location information (e.g., current residence address, current cell phone number) from confidential locator forms into a confidential follow-up tracking database (using Microsoft Access). The rest of the information on the locator forms remained in the participant's hardcopy file, which was kept in a locked cabinet inside the project staff's laboratory. We found that an interactive database was requisite to effectively maintain participant information in a 5-year large-scale, longitudinal study where the research team naturally changed throughout the course of the study. This database was crucial to alerting the on-board retention RA about important dates for each participant. This database provided several alerts. First, it highlighted when youth were 2- and 1-month prior to the follow-up date, so that the team could send reminder postcards at each time point. This was an important interim step, as returned postcards signaled that a youth had moved or that the listed address was incorrect, thereby indicating that new contact information was needed for that youth.

Second, the Access database highlighted when to make telephone calls to youth. All participants were contacted by telephone at 8, 6, and 4 weeks prior to their follow-up date to update their contact information in the Access database for the pending follow-up. Third, the Access database highlighted when a participant was within 30 days of their follow-up due date. Once in this window, the full-time RA began calling once per day. Often, numerous phone calls were required to schedule an in-person follow-up with the youth. Once youth were reached by phone, follow-ups were arranged around participants' schedules.

Contact Decision Tree and Related Staff Supervision

Due to the history of differential attrition in juvenile justice-involved youth versus those who are not involved in juvenile justice^12,13, our team was very concerned about ensuring that we had data from all youth, in order to ensure our project and the subsequent findings had the broadest reach and the greatest generalizability. Thus, it was vitally important to our team to effectively retain all enrolled youth. We therefore utilized the following decision tree to maintain contact with youth throughout the course of this study.

Because we had promised youth during the consent phase that we would make every effort to reach them throughout the project, the contact goal was explicitly to reach the youth to complete a follow-up every three months and receive payment for their efforts.

Trained RAs who could not engage youth using the first effort were immediately directed to begin contacting youth using the next effort. In order to review and ensure appropriate retention strategies, the retention staff participated in weekly meetings, supervised by a senior study staff member (SFE), to evaluate progress and address and problem solve difficult situations. All efforts to reach youth were tracked using a tracking log.

(1) Postcards

All youth were mailed postcards to their provided address for each of the preceding months before their follow up (at 2 months prior to follow-up date, at 1 month prior to their follow-up date).

(2) Telephone Calls to Youth

Telephone calls to youth were made in three contexts. Youth were telephoned at 8, 6, and 4 weeks prior to their follow-up date to update their contact information in the Access database for the upcoming follow-up. Similarly, if a youth failed to be reached via postcard at any point within the course of the study, he or she was contacted via telephone in order to obtain current contact information. Finally, in order to effectively reach youth as their appointment neared, the trained RA began to call youth once per day once they were within the 30-day window of their follow-up in order to schedule their follow-up assessment session. We did not limit the number of phone calls that trained staff could make. We therefore observed a large range of phone calls required to effectively schedule a follow-up with youth (number of contact calls varied from 1 – 187; see Table 1).

Table 1. Retention Efforts: Differences between Justice- and School-Based Adolescents.

Retention Effort	Justice-Based (n=244)	School-Based (n=40)	t-value
No. of calls at 3-month follow-up	10.04 (19.64)	5.87 (5.64)	2.54*
No. of calls at 6-month follow-up	8.63 (11.19)	4.73 (5.60)	3.26**
No. of reschedules at 3-month follow-up	.16 (.37)	.05 (.23)	2.35*
No. of reschedules at 6-month follow-up	.18 (.73)	0.00 (0.00)	3.53**
No. of home visits at 3-month follow-up	.19 (.83)	.11 (.66)	.70
No. of home visits at 6-month follow-up	.17 (.58)	.13 (.83)	.21

Note. Means and standard deviations in parentheses.

^***p<.001, two-tailed.

^**p< .01, two-tailed.

^*p< .05, two-tailed

(3) Telephone Calls to Other Contacts

If the trained RA could not reach youth directly, they move to calling each of the individuals that the youth had given permission for our team to contact. All calls to the juvenile justice system were limited to only requesting any additional contact information on the participant as a last and final option. Trained RAs only stated the youth had signed-up for our study; they were reaching a follow-up period, and asked for updated contact information for the youth.

(4) Visits to Youth's Home and Other Relevant Locations

If the retention RA could not reach the participant or their provided contacts by phone, field staff visited their last listed home address. If trained staff found that youth no longer resided at that address, they asked family members about youth's most recent residence. Importantly, field staff carried all assessment materials (e.g., laptop computers) to the home visits, to facilitate the immediate completion of that follow-up onsite. If youth were not at any known residence, following other work in the field¹⁶, field staff visited the workplaces and/or locations youth reported frequenting on their confidential locator form.

Scheduling the Follow-Up

Once the retention RA reach youth, the RA accommodated the youth in any way possible to facilitate the effective completion of the follow-up appointment. To that end, the RA organized the time, date, and location of the follow-up around the youths' schedule. The RA directed the youth to pick a time, on any day of the week, at a location convenient to them (e.g., McDonalds, the University Psychology Department). Trained staff were permitted to drive no more than 1 hour from the metropolitan area to complete follow-ups. And, youth who were beyond this location were allowed to complete follow-ups via telephone. In addition, youth were reminded how much they would receive for completing the upcoming follow-up, and were informed that we would provide food from the location if they met at an eatery.

Contact Efforts between Scheduled Date and Follow-Up Participation

Once the trained RA established an official follow-up appointment they called participants one week, 3 days, 2 days, 1 day and 1 hour prior to the appointment simply reminding the youth the date, time, and location of their appointment.

Additional Study Measures

In addition to our retention efforts, to evaluate study hypotheses, we also examined a number of behaviors that we believed might be related to retention in a longitudinal HIV prevention intervention evaluation.

Substance Use and Risky Sexual Behavior

We evaluated the incidence of substance use and sexual behavior for each time point using the Time Line Follow Back (TLFB)¹⁷ (baseline, 3 months, 6 months). Specifically, this is an interviewer-administered measure that queries the quantity and frequency of each behavior during the past 30 days. This measure facilitated an evaluation of the amount and frequency of alcohol, marijuana, tobacco use, frequency of sexual intercourse, and number of sexual partners.

Analysis Plan

Potential overall retention rate differences were examined between the justice-based and school-based youth. Next, differential attrition by condition and group were assessed. Then we evaluated staff retention efforts (i.e., phone calls, reschedules, and home visits) across groups. To explore factors that could be associated with follow-up difficulties, we examined a number of risk behavior variables upon which the two groups were hypothesized to differ. Finally, an exploratory analysis examined the relationship between substance use and various retention strategies.

Results

Overall Retention

We first examined overall retention rates between the two samples of youth. Of the 240 justice-based youth eligible for 3-month follow-up, 84.6% (n = 203) completed follow-ups. Of the 40 school-based participants eligible for 3-month follow-up, 92.5% (n = 37) completed follow-ups. This difference was not statistically significant (χ² (1, 280) = 1.755, p = 0.185). Similarly, 88.4% (198 of the 224) justice-based and 97.4% (37 of the 38) school-based youth completed the 6-month follow-up. This difference was also not significant (χ² (1, 262) = 2.832, p = 0.092).

Factors that May Impact Retention in Longitudinal HIV Prevention Interventions

Source of Recruitment: Justice vs. School

We used logistic regression to evaluate differential attrition by condition within the two groups. Retention at both 3 and 6-month follow-ups (1=yes, 0=no) was regressed on group, intervention condition, and their interaction. There was no significant effect of intervention condition on retention, β = .241, SE = .318, p = 0.45, indicating that, on average, retention rates were not influenced by intervention condition. There was also neither a significant effect of group (justice-involved versus school-based youth), β = .774, SE = 1.75, p = 0.66, nor a condition × group interaction, β = .095, SE = 1.10, p = 0.93. Ultimately, this suggests that none of these factors were directly associated with retention.

Adequacy of Retention Efforts

We evaluated the staff efforts in order to obtain a quantitative evaluation of “what may be needed” in order to retain youth, and to evaluate whether those efforts differed across our samples. At the 3-month follow-up, staff made more phone calls and rescheduled more follow-up appointments for justice-based youth than school-based youth (see Table 1). This same pattern was found at the 6-month follow-up. However, we observed no difference between samples in number of home visits required.

Youth Risk Behavior

In terms of factors that may make retention of these samples even more important, we found that both samples had similar ages of first alcohol use (see Table 2), but justice-involved youth used alcohol more frequently than their school-based peers. Similarly, justice-involved youth were younger at first marijuana use, and used marijuana more frequently (see Table 2). Compared with their school-based peers, justice-involved youth were significantly younger at first intercourse and had more sexual partners.

Table 2. Risk Behaviors: Differences between Justice- and School-Based Adolescents.

Behavior	Justice-Based (n=244)	School-Based (n=40)	t-value or χ2 value
% ever had intercourse	79.30	39.50	26.89***
Age at first intercourse	13.15(2.08)	15.50(.759)	-9.33***
Number of lifetime sexual partners	7.54(8.71)	3.07(3.24)	4.36***
% ever used alcohol	90.10	68.40	13.69***
Age at first alcohol use	12.56(2.30)	12.38(3.26)	.27
Days of alcohol use past 30 days (baseline)	3.24(4.75)	.73(1.71)	6.16***
% ever used marijuana	84.10	38.90	37.26***
Age at first marijuana use	11.84(2.61)	14.12(1.62)	-5.26***
Days of marijuana use past 30 days (baseline)	11.61(12.31)	1.58(5.22)	8.77***

Note. Means and standard deviations in parentheses.

^***p<.001, two-tailed.

^**p< .01, two-tailed.

^*p< .05, two-tailed

Relationships between Substance Use and Retention

Table 3 illustrates the correlation between the hypothesized risk behaviors and various retention strategies utilized collapsing across group. There were significant positive correlations between number of calls made at the 6-month follow-up and frequency of marijuana use in the last 30 days at baseline and 6-month follow-up, and between number of reschedule attempts made at the 6-month follow-up and frequency of alcohol use in the last 30 days at baseline and frequency of marijuana use at the 6-month follow-up. Finally, at the 6-month follow-up, youth who had begun drinking at an earlier age required a larger number of home visits. No other correlates were significantly associated with retention efforts.

Table 3. Correlations between retention strategies and health risk behaviors (3 month follow-up/6 month follow-up).

	Number of Calls	Reschedule Attempts	Home Visits
Ever had intercourse	.07/.06	.07/.01	.11/.05
Age at first intercourse	-.03/-.07	-.10/-.04	-.05/-.01
No. of sexual partners	.06/.07	-.01/.02	.04/.06
Ever used alcohol	.04/-.02	.05/.02	.07/-.05
Age at first alcohol use	.03/-.04	.12/.08	.08/.15*
Days of alcohol use past 30 days (baseline)	.09/.08	-.09/.20**	.11/.07
Ever used marijuana	.08/.09	.05/.06	.11/.05
Age at first marijuana use	-.04/.04	.02/.04	.07/.13
Days of marijuana use past 30 days (baseline)	.11/.13	.01/-.02	.05/-.02
Days of alcohol use past 30 days (3 months)	.08/.07	.09/.12	.08/-.01
Days of marijuana use past 30 days (3 months)	-.05/-.03	.03/.17*	-.02/-.01
Days of alcohol use past 30 days (6 months)	.07/.02	.10/-.06	.10/-.08
Days of marijuana use past 30 days (6 months)	-.05/.20**	.04/.08	-.02/-.001

Note.

^***p<.001, two-tailed.

^**p< .01, two-tailed.

^*p< .05, two-tailed

Discussion

The results suggest that our staff was able to achieve excellent retention rates with both school-based and justice-based adolescent samples. However, importantly, justice-based youth required significant effort above and beyond school-based youth. Justice-based youth evidenced earlier initiation of most health risk behaviors. Youth who had higher/more frequent alcohol and marijuana use were found to be more difficult to retain, suggesting that extra efforts may be required to retain justice-involved and substance-abusing youth.

Retention Strategies

We found that frequent positive, energetic, and friendly phone calls/text messages made to the youth, their friends and family, facilitated retention. We found that telephone contact (texting and/or phone conversations), as opposed to home visits, served as the cheapest, easiest, and most effective strategy to successfully reach and retain both samples of youth. Anecdotally, we also found that making the follow-up assessment process as easy as possible for the youth expedited participation and enhanced retention. These strategies included conducting the assessments at a place of the youth's choosing, providing cab fare when necessary, and providing snacks during the follow-up assessment. We suggest that these steps may bolster youths' trust in the research team, and their willingness to work with and communicate with our staff. Cottler et al. maintain that participant retention is largely due to efforts employed by research staff, and our results support that assertion¹⁸.

Substance Use and Retention

Of particular interest is the relationship between both past and current (past month) alcohol and marijuana use and retention efforts, such as reschedule attempts and home visits. Youth who used more frequently required more reschedule attempts. It appears to be ongoing substance use, rather than just substance use at baseline that is related to increased retention efforts. These data suggest that without the additional staff efforts necessary to continuously interact with and retain this subsample, studies that do not take these steps are likely to subsequently lose this critical subsample to follow-up. The greater concern is that this could result in biased estimates of treatment effects, with the end result being that errors are made in determining how impactful a prevention intervention is for youth at the highest need.

Strengths

In terms of study strengths, this study included two different youth populations and a panoply of retention strategies. The two samples were different in terms of demographic characteristics and risk behavior, yet importantly, were of the same age, lived in the same region, and participated in an identical study protocol. However, while it is possible to retain both groups, our study suggests that justice-involved youth require an additional level of staff rigor and attention, in order to successfully keep them enrolled in long-term participation.

Conclusions and limitations

Due to funding limitations, the size of the school-based group was significantly smaller than the justice-involved sample in the parent study. While smaller sample sizes may make it more difficult to detect group differences, we observed a number of significant differences underscoring that these groups differ on important correlates of attrition, and that the size of these differences is large. It is clear that retention of all youth requires persistence, time, adequate staffing, and commitment. However, our data demonstrate that justice-involved youth, particularly substance-using youth, are not easy to retain. Different follow-up methods do not appear to be required, it is simply that more of the same efforts appear to be necessary to successfully retain justice-based youth. Further investigations into the individual differences that facilitate or impede retention, determining the expected cost and effort needed to retain participants, and helping research staff to flag those participants who might require extra effort represent critical next steps.