Ensuring Retention in Longitudinal Studies: A Practical Evaluation of an Intensive Follow-up Protocol and Suggested Adaptations

Abstract

Ensuring retention in longitudinal studies of individuals with substance use disorders (SUD) is a continual challenge for researchers. This study made several modifications to a highly intensive follow-up protocol (Scott, 2004), originally designed for adults with SUD, in order to adapt it to a group of adolescents in low-intensity outpatient SUD treatment (N = 127, M age 16.7 yrs) and to accommodate limitations in the financial resources available for study staffing and transportation. In the present sample, adolescent participants generally found it unreasonable for study staff to request to contact people outside their immediate family in order to locate them and to attempt to schedule interviews 3–6 months in advance, as specified in the original protocol. Changes were made to accommodate these concerns and follow-up rates remained high (85–91%). Even though this study is limited by its non-experimental nature, it provides a replicable example of a scaled-down, less costly version of a highly intensive follow-up protocol that can be used to achieve high follow-up rates in studies of adolescents with SUD. We hope this will be encouraging for researchers and program evaluators who have limited resources or who work with participants who express concerns about privacy or study burden.

1. Introduction

Substance use disorders (SUDs) are recognized as chronic conditions that last for many years, result in a variety of short- and long-term consequences, and require multiple treatment episodes (Dennis, Scott, Funk, & Foss, 2005). As such, longitudinal study designs are the best method of evaluating substance use outcomes and treatment efficacy and effectiveness over time. Despite the benefits of longitudinal studies, participant attrition is a common problem associated with this design, particularly with adolescent samples (Stephens, Thibodeaux, Sloboda, & Tonkin, 2007).

1.1 The Problem of Attrition in Longitudinal Research and Researchers’ Response

High attrition rates threaten a study’s internal validity, as it becomes unclear whether outcomes are caused by or related to the independent variable or due to selection bias, wherein participants who are followed up over time differ systematically from those who are lost. External validity may also be compromised, as it is unclear whether results would generalize to the population of interest as a whole (Meyers, Webb, Frantz, & Randall, 2003). A study with a high rate of attrition could yield findings based on a biased sample. In some adolescent samples, study dropouts are more likely than study completers to be involved in drug use and deviant behavior, get in trouble more at school, and have poorer grades (Boys et al., 2003; Snow, Tebes, & Arthur, 1992; Stinchfield et al., 1994). In order to maintain the integrity of study results, it is crucial to lose as few participants over the course of the study as possible (Boys et al., 2003; Stephens et al., 2007), as large losses will reduce the generalizability and weaken the interpretation of the findings. However, this is not an easy task, particularly with substance-involved adolescent samples.

Researchers have addressed this difficult task through various methods, such as collecting large amounts of contact information for participants and their friends and family members, maintaining frequent contact with participants over the phone or through mail, and providing financial incentives for participation (Boys et al., 2003; Meyers et al., 2003; Scott, 2004). Scott (2004) developed a systematic and intensive follow-up protocol specifically for longitudinal studies of individuals with SUD, which has since been used in a number of studies with both adolescent and adult samples. The protocol, referred to as “EVMC” for its four stages of Engagement, Verification, Maintenance, and Confirmation, emphasizes proactive methods of locating and maintaining contact with participants by addressing common behavioral patterns among individuals with SUD, such as residential instability, multiple social networks, estrangement from friends and family members, and frequent entries into treatment facilities or prison (see Method section for details). Under the protocol, study staff systematically track all attempted and actual contact with participants and periodically verify and update each participant’s contact information. Studies using this protocol have achieved short- (i.e., within 12-months) and long-term (i.e., 1–5 years) follow-up rates around 95%, including two studies with adolescent populations that achieved short-term follow-up rates of 94–100% (Dennis et al., 2002; Godley, Godley, Dennis, Funk, & Passetti, 2002).

Scott (2004) points out that the traditional “rule of thumb” of retaining at least 70% of the original sample in longitudinal research is likely to be inadequate. In a reexamination of two longitudinal studies with adult samples (both of which used the EVMC protocol and achieved actual follow-up rates of 95%), Scott (2004) compared outcomes among the first 70% of completers (those who completed follow-up sooner and required fewer contacts) with the remaining 30% (those who completed later and required more contacts). There were significant differences between the initial 70% and the remaining 30% of the sample in substance use, incarceration, employment, and criminal activity. Additionally, accounting for covariates such as gender, age, and substance use did not eliminate these differences at the multivariate level and increased the differences at the univariate level. Thus, a retention rate of 70% would have introduced significant biases that would have threatened the internal and external validity of these studies.

Meyers and colleagues (2003) produced similar findings using a sample of adolescents with SUD. Using 6-month follow-up data, youth were characterized as easy-to-retain (5 or fewer contact attempts) or difficult-to-retain (6 or more contact attempts). Difficult-to-retain youth accounted for just over half the sample and were more likely to report using marijuana and opiates and engaging in delinquent behavior (e.g., carrying a weapon), and were less likely to be attending school. If difficult-to-retain participants had been lost from follow-ups, outcomes would have seemed more favorable, a result that has been reported in earlier adolescent research as well (Stinchfield et al., 1994).

Attrition rates in longitudinal studies of adolescents and young adults with SUD vary greatly. For example, Becker, Curry, and Yang (2009) retained, on average, only 70% of study participants over a 12-month period (although this was partially due to a problem with timing, as participants who enrolled later in the study were not eligible for all follow-ups), whereas Dennis and colleagues (2002), who employed Scott’s (2004) protocol in a large randomized controlled trial, retained an average of nearly 98% of study participants over the same duration with a similar sample. Other recent studies of adolescents in SUD treatment have achieved retention rates of 85–95% (Chi, Kaskutas, Sterling, Campbell, & Weisner, 2009; Edelen, Slaughter, McCaffrey, Becker, & Morral, 2010; Meyers et al., 2003; Sabri, Williams, Smith, Jang, & Hall, 2010). While attention to follow-up rates and the methods used to follow study participants appears to have increased in recent years, most studies do not provide detailed descriptions of their follow-up procedures, making it difficult to compare methods across studies.

1.2 Balancing Attrition Prevention Efforts with Practical and Ethical Concerns

While high follow-up rates are crucial to the scientific merit of a longitudinal investigation, there are some practical and ethical issues associated with the use of a highly intensive follow-up protocol like Scott’s EVMC method. Under such a protocol, participants are asked to provide a large amount of contact information (e.g., multiple phone numbers for themselves and those close to them) and experience frequent contact from the researcher, including possible visits to their home if they cannot be reached by phone or mail. As a result, participants might feel as though their privacy has been compromised or that they have to continue participating in the study, even if they are informed by study staff that their information is confidential and their participation is voluntary. Participants may have difficulty distinguishing between researchers’ words (i.e., that consent may be withdrawn at any time) and actions (i.e., calling friends and family members to locate the participant, visiting the participant’s home unannounced). This may conflict with American Psychological Association (APA) ethical standards, which dictate that researchers must respect the “privacy, confidentiality, and self-determination” of participants (APA, 2010, Principle E). These concerns may be further complicated by the vulnerable status of adolescents, who may believe they are required to conform to the demands of adult researchers. One study of outpatient adolescents that used the EVMC protocol found that, when asked to rate reasons for their continued study participation, roughly half the sample agreed with the statement “because I felt obligated to finish the study” (Garner, Passetti, Orndorff, & Godley, 2007). Furthermore, as information spreads among study prospects in the target community regarding the intensive nature of study participation, eligible individuals who are bothered by this level of intensity may choose to not enroll in the study, causing the enrolled sample to be less representative of the population of interest. As such, attempts to track study participants must be carefully balanced with sensitivity to participant concerns about privacy, time, voluntariness, and effort.

Carrying out a highly intensive follow-up protocol requires a great deal of resources. Funds must be available to cover the costs of the time spent by study staff tracking participants and to pay for the travel costs of street outreach. In a study of approximately 200 adolescents from residential SUD/mental health treatment facilities that used a highly intensive follow-up protocol, Meyers and colleagues (2003) estimated that they used an extra $17,000 at each follow-up time point to cover the costs of follow-up activities, over and above their base budget. This degree of funding simply may not be available to smaller-scale naturalistic observation studies or program evaluations, but high follow-up rates nevertheless remain important. Again, it would be ideal to find a balance between using systematic, rigorous follow-up procedures that increase follow-up rates and minimizing costs.

1.3 The Present Study

Just as clinicians make changes to evidence-based therapeutic practices when translating from manuals to practice, researchers are likely to modify evidence-based research protocols to fit the particular population, context, and budget of their study. It is important to know how these modifications relate to subsequent follow-up rates. The present study sought to address this issue in a naturalistic, observational, longitudinal study of adolescents in a community SUD treatment program. The study was a prospective assessment of the relationships among adolescents’ participation in outpatient treatment and mutual-help groups (e.g., Alcoholics Anonymous) and their substance use outcomes over a 12-month period. We initially attempted to implement the highly intensive EVMC follow-up protocol in its original form (Scott, 2004). However, participants repeatedly expressed concerns about some of the demands we were placing on them and we had concerns about the resources needed to fund street outreach. This prompted us to make several adjustments to the protocol. These adjustments were made at different points throughout the study in response to participant concerns and thus were not experimentally manipulated. Nevertheless, the aim of this article is to describe the modifications we made to the EVMC protocol, our experience-based rationale for the changes, and the follow-up rates that we were able to achieve with a less intensive version of this evidence-based follow-up protocol.

2. Method

2.1 Participants

Participants were 127 adolescents who presented for treatment at a private outpatient SUD treatment facility in the Northeastern US between August, 2006 and May, 2009. Individuals were eligible to participate if they (a) were within their first month of treatment at this facility, (b) were between the ages of 14 and 19 at the time of study entry, (c) had a parent/guardian who gave consent (for those under 18), (d) gave assent to participate (or consent if over 18), and (e) were English-speaking. Adolescents were excluded from the study if they were actively psychotic or had a cognitive deficit affecting their ability to comprehend the study and its risks and benefits. Of the 178 adolescent patients who presented for treatment at the facility during the study enrollment period, 160 (90%) were eligible to participate. Of the eligible patients, 95% (n = 152) agreed to be contacted by study staff and 127 (84%) were enrolled in the study. Of those who did not enroll in the study (n = 25), reasons for non-enrollment included (a) study staff unable to contact patient within the first month of treatment (24%), (b) patient unable/unwilling to schedule an appointment within the first month of treatment (24%), (c) patient did not attend treatment and chose not to participate in the study as a result (24%), (d) parent declining to give consent for their child (20%), (e) patient declining participation (4%), and (f) transportation difficulties (4%).

2.2 EVMC Follow-up Protocol: Main Elements

The EVMC protocol consists of four phases: Engagement, Verification, Maintenance, and Confirmation. Below is a brief description of the original protocol (see Scott, 2004 for more details), followed by descriptions of our modifications and the rationale for each change (see Table 1).

Table 1.

Modifications to the EVMC protocol

Element and Phase	EVMC Protocol	Modified Protocol	Timing of Modification	Rationale
Collateral contact information (Engagement)	Collect contact information for at least three collaterals beyond immediate family	Did not collect contact information for collaterals beyond immediate family	After enrollment of first 38 participants	Participants often suspicious about how such information would be used Most participants living with parent(s) Fear of jeopardizing rapport
Interview scheduling (Engagement)	Schedule the date and time of next appointment at the end of each interview	Delayed scheduling until 1–4 weeks before target date	After enrollment of first 10 participants	More feasible for participants to schedule closer to target date Gave participants and parents more flexibility and control over scheduling
Interview confirmation (Confirmation)	Six weeks prior to next appointment, call participant every 24–48 hours until appointment is confirmed	Four weeks prior to next appointment, called once to remind participant One week prior, called every 24–48 hours until appointment was scheduled	After enrollment of first 10 participants	Kept participants and parents from feeling overwhelmed by phone contact Maintained flexibility in scheduling More feasible in terms of time and resources available to research staff
Lost participants (Confirmation)	If a participant is lost, first contact the three collaterals and then mobilize street outreach	Did not use street outreach	From start of study	Felt street outreach was too intrusive for this population Study lacked resources for travel to suburban areas (staff used public transportation)

2.2.1 Engagement phase

During the first meeting with the client, study staff (a) educate and motivate participants about follow-ups, (b) complete consent and locater (i.e. contact and collateral information) forms, and (c) schedule the next appointment. Directly following the baseline interview, study staff enter the completion date for documents needed to track the participant, place consent and locater forms in a physical file, and mail a thank you card to the participant, in order to both thank the participant and verify their address.

2.2.2 Verification phase

Within 7–10 days after the baseline interview, study staff verify contact information for at least three collaterals (i.e., contact persons). If a collateral cannot be verified, study staff immediately re-contact the participant by phone to clarify. Study staff also take into account the geographic stability of the collateral, the degree of communication between the collateral and the participant, and whether the collateral is using or in recovery.

2.2.3 Maintenance phase

Between interviews, study staff send letters to the participant in order to remind them about the next interview, keep them engaged, and continue to verify their addresses. Letters are sent every 4–6 weeks.

2.2.4 Confirmation phase

Six weeks before the next interview, study staff attempt to make direct phone contact with the participant. They call the participant or their collaterals every 24–48 hours until direct contact with the participant and appointment confirmation is made. They then mail a confirmation letter to the participant with the date, time, and location of the next interview and follow up with a series of reminder calls 28, 7, and 1 day before the scheduled interview.

2.2.5 Monitoring, tracking, and review

Throughout these four phases, study staff take steps to maintain the integrity of the protocol by (a) tracking the dates of attempts and completion of each element of the protocol, (b) following a standardized procedure for participants who become lost (e.g., using street outreach, checking with social services or local correctional facilities), and (c) completing a weekly review of, and planning action needed for, unverified and unconfirmed cases.

2.3 The Present Study: Modifications to the EVMC Follow-up Protocol

2.3.1 Collateral contact information

Scott recommends collecting contact information for at least three collaterals per participant (Engagement phase). Many of our adolescent participants were reluctant or unable to provide contact information for collaterals outside their immediate family and expressed concerns about how this information would be used (e.g., whether we would disclose the nature of the research). Therefore, we decided not to actively collect contact information for collaterals beyond participants’ parents/guardians. Even so, for the majority of cases (83%), we were able to obtain at least three phone numbers (Mode = 4) just for the participant and his or her parents/guardians. Participants sometimes spontaneously provided contact information for people other than their parents, which increased the percentage with at least three phone numbers to 90%. We were able to obtain a primary address in all cases, as well as additional addresses as needed (e.g., if participants entered an inpatient facility). Contact information was updated at each follow-up. We believe this degree of information was adequate for our sample, given that the majority was living with their parent(s) and/or legally under parental control throughout the study. This modification also seemed to assuage participants’ concerns about privacy and confidentiality, as few participants expressed concerns about being asked to provide contact information for their parents after being informed that their information could not be shared with their parents.

2.3.2 Interview scheduling and confirmation

Scott recommends scheduling the participant’s next appointment at the end of each interview (Engagement phase). We found that our adolescent participants preferred to schedule their next appointment much closer to the target date (e.g., between one month and one week before). Accordingly, at the end of each interview, we gave the participant a card containing the target week of their next appointment, as well as the amount of money they would receive and study staff contact information. We also found that beginning phone contact (Confirmation phase) six weeks prior to the target week of the next appointment was still too early for participants to schedule, so we delayed this until four weeks prior. At that time, we called once (as opposed to every 24–48 hours) to offer to schedule the upcoming appointment, but did not necessarily schedule the appointment until the week before the target week, during which time we called one or more of the available phone numbers every 48 hours. Once the interview was scheduled, we performed the 7- and 1-day reminder calls as indicated by Scott. This modified timeframe seemed to be preferable for participants, in that it allowed them to have more flexibility and control over scheduling and likely kept them and their parents from feeling overwhelmed by phone contact by study staff.

2.3.3 Lost participants

Participants were eligible to complete the 3- and 6-month follow-ups for up to one month after the due date, and were eligible to complete the 12-month follow-up for up to three months after the due date. Thus, including the week before the due date when study staff began attempting to schedule the appointment, study staff would spend 5–13 weeks attempting to contact a participant by calling one available phone number every 48 hours. If participants and/or parents could not be contacted by phone, mail, or e-mail during this time, participants were considered lost and study staff did not attempt to locate them via street outreach. Street outreach seemed to be too intrusive for this sample, as participants often expressed concerns about privacy. The study also lacked resources to fund travel to suburban areas (staff relied on public transportation to access study offices and the treatment clinic).

2.4 Procedure

The study consisted of four assessments, including a baseline interview and 3-, 6-, and 12-month follow-ups. The study employed two research assistants who generally followed the same participants from start to finish. At each timepoint, participants were asked to provide information on their substance use, use of formal and informal treatment services, justice system involvement, school/work status, and social supports. Participants also completed several questionnaires at each timepoint, addressing their motivation, self-efficacy, spiritual/religious beliefs, psychological symptoms, and family environment. The baseline interview took approximately 2–3 hours to complete and the follow-up assessments took between 45 and 90 minutes. Assessments usually took place at the outpatient treatment facility (86.3%), as this was often most convenient for participants, but interviews were also completed over the phone (0.8% at baseline; 6.9% at 3 months; 10.3% at 6 months; 18.0% at 12 months) or in person at a location other than the treatment facility (e.g., inpatient facilities, offices of the study staff; 1.6% at baseline; 3.4% at 3 months; 5.6% at 6 months; 9.9% at 12 months). Interviews were not completed in participants’ homes, as most participants lived in suburban areas accessible only by car. Participants were paid $50 for both the baseline and 12-month interviews and $40 for both the 3- and 6-month interviews. When interviews were conducted in person, saliva toxicology screens were performed to verify cases of reported abstinence during each follow-up period. Study staff employed the modified EVMC protocol throughout, although the timing of modifications varied (see Table 1). See Figure 1 for details on the modified protocol.

Figure 1.

Modified EVMC follow-up protocol (figure adapted from Scott, 2004)

3. Results

3.1 Sample Characteristics

The final sample was 75.6% male, 86.6% White, and M = 16.7 years old (SD = 1.2) at the time of study entry. At baseline, most participants were living with at least one parent (93.7%), enrolled in school (75.6%), not employed (56.8%), and justice system involved (50.4%). The main reasons for entering the current treatment program were parent(s) wanted them to (42.5%), other treatment provider recommended it (25.2%), or court/probation officer required or recommended it (22.0%). Only 11.8% of participants indicated that they entered the treatment program because they wanted to.¹ Marijuana was the most commonly reported drug of choice at baseline (70.9%), followed by alcohol (11.8%), heroin/opiates (11.1%), and cocaine/amphetamines (3.2%).

Lifetime DSM-IV abuse and dependence symptoms for alcohol and up to three frequently used drugs were separately assessed using the Customary Drinking and Drug Use Record (CDDR; Brown et al., 1998). The vast majority (93.7%) met lifetime DSM-IV criteria for an SUD; 26.8% met criteria for marijuana abuse (without dependence), 57.5% for marijuana dependence, 27.6% for alcohol abuse (without dependence), 31.5% for alcohol dependence, 2.4% for opiate abuse (without dependence), and 11.0% for opiate dependence. The presence of other past-year Axis-I conditions was assessed using the Computerized Diagnostic Interview Schedule for Children, version IV (C-DISC-IV; Shaffer, Fisher, Lucas, Dulcan, & Schwab-Stone, 2000). Approximately 61% of the sample met DSM-IV criteria for one or more past-year Axis I conditions other than SUD, with the most common being conduct disorder (41.3%), major depressive episode (18.9%), and oppositional defiant disorder (18.3%).

3.2 Follow-Up Rates and Timing

Follow-up rates were 91.3% at 3 months, 84.3% at 6 months, and 87.4% at 12 months. Some participants who were unable to be contacted at either 3 or 6 months completed a later assessment (i.e., 6 and/or 12 months). While most assessment material could not meaningfully be covered retrospectively, we did obtain a limited amount of behavioral outcome data, such as treatment experiences, substances used, and the approximate frequency of substance use, for participants who completed a later follow-up. This brought the follow-up rates for those from whom any outcome data was collected to 94.5% at 3 months and 91.3% at 6 months (see Table 2). Chi-square analyses revealed that follow-up rates at each timepoint did not differ by participant gender (ps > .21) or the presence of one or more comorbid Axis I diagnoses (ps > .19). Spearman’s rho correlations revealed that follow-up rates at each timepoint were unrelated to participant age (ps > .10).

Table 2.

Follow-up rates and timing.

Timepoint	Follow-up rate	Follow-up rate including retrospective data collection*	Days from due date (Median [M, SD])
3-month	91.3%	94.5%	3 (7.3, 7.8)
6-month	84.3%	91.3%	5 (8.8, 8.1)
12-month	87.4%	87.4%	6 (12.1, 15.9)

^*Includes participants who missed the 3- or 6-month follow-up, but provided limited outcome data on these time periods at their 6- or 12-month follow-up, respectively.

On average, participants completed their baseline assessment within 10.8 days (SD = 12.1) of their treatment start date, their 3-month follow-up within 7.3 days (SD = 7.8) of the due date, 6-month within 8.8 days (SD = 8.1) of the due date, and 12-month within 12.1 days (SD = 15.9) of the due date. However, outliers at each time point inflated the means, as reflected by the lower median days to completion: 7 for baseline, 3 for 3-month, 5 for 6-month, and 6 for 12-month. Spearman’s rho correlations showed that the length of time to complete each follow-up was not related to participant age (ps > .09) or the presence of comorbid diagnoses (ps > .20). At 3 and 12 months, gender was unrelated to the length of time to complete follow-ups (ps > .50), although at 6 months, girls completed their assessments significantly later than boys (r_s = .27, p = .005).

3.3 Reasons for Non-Completion of Follow-Up

Reasons for non-completion of follow-ups were recorded and participants were categorized as (a) unable to be contacted (i.e., no direct phone contact was made with the participant during their eligibility period), (b) unable to schedule (i.e., some direct contact was made with the participant, but they were unable to schedule an appointment or scheduled an appointment(s) and did not show), (c) declined (i.e., directly declined to participate in the assessment, but did not withdraw from the study), (d) withdrawn (i.e., participant or their parent [if under 18] withdrew consent), (e) incarcerated, or (f) in residential care (see Table 3).

Table 3.

Reasons for non-completion of follow-ups at each timepoint.

Reason	3-month	6-month	12-month
Unable to contact1	36%	40%	38%
Unable to schedule2	36%	35%	31%
Declined	9%	15%	12%
Withdrew consent	0	0	19%
Incarcerated	0	10%	0
In residential care	18%	0	0

¹Participants who could not be reached by phone or mail during their eligibility period.

²Participants with whom researchers had some direct contact (or with their parents if under 18), but who could not schedule and/or keep an appointment.

Of the 11 participants who did not complete the 3-month follow-up, 4 (36%) were unable to be contacted, 4 (36%) were unable to schedule, 2 (18%) were in residential care, and 1 (9%) declined. Of the 20 participants who did not complete the 6-month follow-up, 8 (40%) were unable to be contacted, 7 (35%) were unable to schedule, 3 (15%) declined, and 2 (10%) were incarcerated. Of the 16 participants who did not complete the 12-month follow-up, 6 (38%) were unable to be contacted, 5 (31%) were unable to schedule, 3 (19%) were withdrawn, and 2 (12%) declined. Overall, inability to contact participants accounted for 38% of the missed follow-ups, followed by inability to schedule (34%), and participants declining (13%).

On a more impressionistic level, it seemed likely participants who did not respond to voicemails or letters (i.e., unable to contact) were passively declining further study participation, rather than being unaware that they were due for follow-ups. This was evidenced by a number of rejected phone calls and hang-ups after repeated contact attempts by researchers. Researchers routinely told participants (via voice messages and letters) that if they no longer wished to be contacted, they could withdraw their consent or decline participation at any time. However, it was relatively rare for participants or their parents to withdraw consent or decline participation, accounting for 19% of missed follow-ups. Many of the participants who could not be scheduled, but with whom the researcher had some contact, were in situations that limited their availability for assessments (e.g., running away from home, living in group homes, being in State custody, working many hours in addition to going to school). While many participants who did complete follow-ups were also experiencing difficult circumstances and events, participants’ reasons for missed follow-ups highlight some of the challenges that adolescents with SUD often face.

As an inability to contact participants accounted for a substantial proportion of missed follow-ups, it is reasonable to assume that use of the full EVMC protocol would have resulted in higher follow-up rates, as study staff would have had additional methods of reaching participants (i.e., collaterals). We compared the follow-up rates of those who were asked to provide additional collateral contact information (i.e., the first 38 participants) to those who were not asked to provide this information (i.e., the last 89 participants). There were no differences in follow-up completion rates among these groups at 3-months (χ²(1) = 0.04, p = .84) or 6-months (χ²(1) = 1.15, p = .28). There was a trend at 12-months (χ²(1) = 3.52, p = .06) in the opposite direction than might be expected. Participants who were asked to provide additional collateral information were less likely to complete the 12-month follow-up (78.9%) than those who were not asked to provide this information (91.0%).

4. Discussion

The follow-up rates in the current study, which ranged from 84% to 91% for completion of the full assessment battery, plus an additional 3–6% from whom only substance- and treatment-related outcome data were gathered, are on par with other recent adolescent studies that do not detail their follow-up methods (e.g., Chi et al., 2009; Edelen et al., 2010; Sabri et al., 2010) and are somewhat lower than those in adolescent studies that report using the full EVMC protocol (e.g., Dennis et al., 2002; Godley et al., 2002). Our modified follow-up protocol was less intensive than the original EVMC method, as we did not typically contact collaterals outside the immediate family to locate the participant, ask participants to schedule their appointments three or six months in advance, require that direct confirmation contact be made with the participant six weeks before their appointment, or use street outreach or other agencies to locate the participant. We did, however, utilize many elements of the original protocol, including engaging and motivating participants, obtaining multiple phone numbers for participants and their parents, mailing thank you cards and periodic follow-up letters, performing reminder phone calls before each appointment, using a standardized tracking system, and conducting weekly reviews of ongoing cases. In making these modifications, we believe we were able to scale back the EVMC protocol to a level of intensity that was respectful of the privacy concerns expressed by our adolescent sample, while at the same time allowing us to maintain a standardized and effective system for tracking participants.

In the present sample, the collection of collateral contact information beyond immediate family members was not related to higher follow-up rates at any time. In fact, at the 12-month follow-up, there was a trend in the opposite direction. While it is difficult to know precisely why this occurred, it is important to note that the existence of additional collateral information did not necessarily directly translate into greater likelihood of actually completing a follow-up assessment. For example, in nearly two-thirds of the missed follow-up cases, research assistants were in direct contact with participants and/or their parents and participants had other reasons for not completing a follow-up assessment. In other words, there were few cases where the ability to contact people outside the family would have been the deciding factor in whether participants completed a given follow-up. This limits the statistical comparisons that could meaningfully be performed between (a) those who provided this information and missed follow-ups due to an inability to contact and (b) those who did not provide this information and missed follow-ups due to an inability to contact (i.e., several cells in Chi-square analyses of this nature would have counts < 5). Firm conclusions cannot be drawn because this element of the protocol was not experimentally manipulated and is confounded with time of study entry. Since this modification was made to accommodate the discomfort that was openly expressed by participants, future research should use experimental techniques to examine whether high follow-up rates could be achieved if this element was left out of the protocol, while also tracking the proportion of participants for whom this element was the ultimate deciding factor in their assessment completion.

While the techniques and strategies used to maintain follow-up contact in a longitudinal study play an important role in attrition rates, there are other factors that affect participation over time. Garner et al. (2007) conducted a qualitative study of reasons for and attitudes towards continued study participation in a sample of adolescents (N = 145) participating in an outcome evaluation study of outpatient SUD treatment. Similar to the present study, the majority of participants were male (74%), White (79%), and between the ages of 16 and 17 (58%). After completing the 12-month follow-up, participants were given a 40-item questionnaire regarding reasons for and attitudes toward continued study participation. Adolescents’ top reason for continued participation was financial compensation (77%), with additional reasons for participation being understanding that their participation was important (75%) and that the study didn’t take up much time (71%). Strangely, the EVMC protocol does not include guidelines for the provision of financial incentives, even though that could be one of the most important factors for continued study participation among adolescents (Garner et al. 2007). In the present study as well, it is likely that many other factors (e.g., payment) besides the specific follow-up techniques affected the likelihood of follow-up completion, although these were not systematically assessed.

4.1 Limitations and Generalizability

One limitation of the present study was that it did not maintain detailed records regarding the number of contacts attempted and made, number of missed appointments, or costs per participant. The present study also made several modifications to the EVMC protocol simultaneously and did not manipulate the changes experimentally, thereby preventing conclusions about which alterations (if any) were responsible for the slightly lower follow-up rates in the present study as compared to other adolescent studies that have used the full EVMC protocol. Future studies could use a dismantling design to examine which elements of the original protocol could be left out without a loss in follow-up completion rates, taking into account participants’ concerns (e.g., privacy, contact from researchers, scheduling) and the costs associated with an intensive follow-up protocol.

The present study included a relatively small sample of adolescents who were attending low-intensity outpatient SUD treatment at a small, private clinic in a suburb of a major metropolitan area in the U.S. It is unclear whether the follow-up protocol described here would produce similar results in samples drawn from urban settings, non-profit or publically-funded SUD clinics, clinics outside the U.S., or more intensive types of treatment (e.g., inpatient, intensive outpatient). It is also difficult to know whether similar results would be found in adult samples or samples of individuals with other health or mental health problems. Finally, there was little ethnic diversity in the present sample, possibly limiting the generalizability of the protocol to samples with more ethnic, racial, and/or cultural diversity.

4.2 Conclusion

This study made several modifications to a highly intensive follow-up protocol, which was originally designed for adult SUD populations, in order to accommodate limitations in the available financial resources for study staffing and transportation and to make it more tolerable for a group of adolescents in low-intensity, community-based outpatient SUD treatment. We found that adolescent participants generally found it unreasonable for study staff to request to contact people outside the immediate family in order to locate them and to ask them to schedule their next interview 3–6 months in advance. Changes were made to accommodate these concerns. Even though this study is limited by its non-experimental nature, it does provide a replicable example of a scaled-down, less costly version of a highly intensive follow-up protocol that can be used to achieve high follow-up rates in adolescents with SUD. We hope this will be encouraging for other researchers who have limited resources or who work with adolescent substance-involved populations who express concerns related to privacy or study burden.