A replicable model for achieving over 90% follow-up rates in longitudinal studies of substance abusers

Abstract

The goals of this paper were to: (a) discuss the interface between dominant behavioral patterns of substance users and the development of a follow-up management model, (b) describe the components of the model, (c) present data regarding its effectiveness, (d) estimate the number of contacts for various follow-up rates, (e) explore the generalizability of the model across sub samples, and (f) present client outcome data that underscore the potential negative impact of low follow-up rates. The model has been used to follow-up over 12,000 research participants yielding over a 95% follow-up rate across seven studies (with over 90% completed within ±14 days of their anniversary date). Using data from two of these studies (n = 2010, n = 632), 22 contacts or less captured 70% of the participants while 33 or 38 contacts or less captured 90% in the first and second studies, respectively. When outcome variables were compared based on 70% versus 90% follow-up, the results varied by study and within study. An examination of the effect of attrition on validity in these two samples demonstrated that even the traditionally acceptable 30% level of attrition can result in significant bias and that the nature of the bias is unpredictable.

1. Introduction

The advantages of studying addiction, treatment, and recovery through longitudinal studies have been well documented in the research literature (Farrington et al., 1992; LaPorte et al., 1981; Mowbray and Luke, 1996). While longitudinal designs provide the optimal method for exploring the cyclical nature of addiction, their internal validity is often threatened by high rates of attrition, which have long been cited as a significant threat to their conclusions (Bickman and Foster, 1996; Bootsmiller et al., 1998; Claus et al., 2002; Cottler et al., 1996; Hansen et al., 1990; Howard, 1992; Kessler et al., 1995; Marmor et al., 1991). Because little is known about the bias that may result when conclusions are based on 50–70% of a sample, it is important to minimize attrition in order to further reduce systematic bias and prevent loss of statistical power (Claus et al., 2002; Marmor et al., 1991). When participant attrition is reduced, the investigator establishes the foundation for making strong statements rather than using presumptions based on statistical adjustments (Burgess, 1989; Ribisl et al., 1996).

Substance abusing populations clearly challenge the researcher's ability to minimize attrition rates. Research is often compromised by the transient, chaotic, and clandestine lifestyle that accompanies addiction (Cohen et al., 1993; Hansen et al., 1990; Inciardi et al., 1993; Ziek et al., 1996). Several studies, however, have overcome these obstacles to implementing longitudinal designs and have successfully produced high follow-up rates with substance abusers over a period of 12 or more months (Call et al., 1982; Capaldi and Patterson, 1987; Cohen and Patrick, 1996; Cottler et al., 1996; Dennis et al., 2002, 2003; Godley et al., 2002; Gregory et al., 1992; Meyers, 2002; Scott and Dennis, 1998; Scott et al., 2003). In addition, numerous articles and manuals have been published that describe a vast array of engagement and tracking techniques that successfully increase participant retention and minimize attrition (see, for example, Anglin et al., 1996; Boice, 1978; Call et al., 1982; Cottler et al., 1996; Dennis et al., 2000; Desmond et al., 1995; Eckland, 1968; Fowler, 1991, 1993; Fowler and Mangione, 1990; Moos and Bliss, 1978; Ribisl et al., 1996). Nonetheless, a significant variation in follow-up rates across studies continues to plague the field and potentially jeopardizes the validity of various methods (Cohen et al., 1993; Hansen et al., 1990; Prendergast et al., 2002). A model protocol or set of procedures is clearly needed, especially one that consistently yields high follow-up rates, one that generalizes across studies and sub-samples (e.g., homeless, criminal justice, mental illness), and one that can be successfully adopted by different research teams.

2. Interface between dominant behavioral patterns among substance abusers and a successful follow-up model

After reviewing the tracking literature, Call et al. (1982) concluded that the methodological literature was scarce, technique oriented, non-cumulative, and largely unavailable to other researchers. They attributed the poorly developed theoretical and methodological literature on tracking to: (a) the expense of long-term tracking efforts (which limits the ability to conduct experimental studies on the effectiveness of different techniques), (b) a scarcity of tracking designs outlined in journal articles, and (c) the willingness of the field to settle for tracking techniques used by others. This narrow focus resulted in a “bag of tracking tricks,” and a commonly held belief that high follow-up rates were achieved simply by applying the correct number and type of techniques. This approach led many to claim that tracking is currently practiced more as a craft than a science (Call et al., 1982; Clarridge et al., 1977; Dennis et al., 2000). Call and colleagues suggested that in order to move the field from the “craft approach” to a replicable science, we need to draw more heavily on the behavioral literature to identify ways of maximizing response rates.

Consistent with this approach, the proposed follow-up management model focuses on four dominant behavioral patterns that often characterize the lives of individuals with substance abuse disorders and the related impact of these behavioral patterns on attrition. The model was developed, and has been shown, to effectively leverage these predictable patterns of human behavior.

2.1. Mobility and instability

Sustained addiction often creates chaotic lifestyles that lead to physical and social mobility (Bale et al., 1984; Brown et al., 1989; Goldstein et al., 1977; MacKenzie et al., 1987). Such mobility, coupled with urban renewal and redevelopment of inner cities, generally leads to residential instability, which in turn leads to greater contact difficulty (Bale et al., 1984; Eaton et al., 1992; Moos and Bliss, 1978; McCoy and Nurco, 1991). The nature of substance abusing lifestyles contributes not only to unstable living arrangements, but often alienation from friends and family members (Goldstein et al., 1977; McCoy and Nurco, 1991; Ziek et al., 1996), further increasing the difficulty of follow-up contact. Consequently, information collected at baseline and through various tracking methods is often temporary, with little or no indication of when it will become obsolete. Thus, an effective follow-up model must include a standardized mechanism for: (a) early detection of participant movements, (b) frequent verification of participant location, and (c) constant recycling and repetition of various tracking techniques.

2.2. Chronic cyclical nature of addiction

Addiction is characterized by cycles of abstinence, treatment and relapse (Carroll et al., 1993; Stout et al., 1999; Dennis et al., 2003). Understanding these cycles and related consequences is critical to the development of an effective follow-up model. For example, there is a clear consensus among clinicians and researchers that chances of recovery improve when individuals terminate relationships with drug involved people and build new social networks that include individuals who are in recovery or at least not drug involved (Anglin et al., 1987; Burman, 1997; Chen and Kandel, 1998; Gordon and Zrull, 1991; Humphreys et al., 1999; Labouvie, 1996; Laudet et al., 2000; Margolis et al., 2000; Waldorf, 1983). The impact of rebuilding social networks on attrition can be significant. If, for example, research participants are recruited during intake at a treatment agency, the contact information they provide may primarily include individuals in their “using network” and this information may quickly become irrelevant post-treatment. If they are recruited closer to discharge from treatment and provide names of individuals in their new recovery network, these contacts may be unhelpful in the event of relapse. Given that individuals vacillate between the two networks over time, including individuals from both networks can maximize contact information.

2.3. Systems involvement

Addiction-related problems including mental illness, homelessness, criminal activity and physical illness generate the need for contact with multiple systems of care and control. Without access to the various institutions and organizations that address the multiple needs of this population and house them, researchers may experience attrition levels as high as 50–60% (Desmond et al., 1995) which will bias the results. For example, several institutions require internal review and prior approval before granting access to participants housed in their facilities while others require releases specific to their institution that would need to be incorporated into the informed consent process during enrollment. Therefore, an effective management model needs to proactively incorporate a component to systematically remove these institutional barriers.

2.4. Social disaffiliation

Other circumstances that frequently coexist with substance abuse can result in the opposite situation whereby participants may minimize contact with systems and formal institutions. This is often the case with those who are involved in criminal activity, undocumented individuals, homeless, and adolescents on the run. In addition, addiction is commonly accompanied by two or more co-occurring psychiatric conditions and a range of personality problems. Lack of trust among participants and family members, resulting from the participant's involvement with criminal activities as well as psychiatric issues concomitant with substance abuse, create significant barriers for longitudinal studies. These problems and a host of other issues not only complicate the treatment and recovery processes but also make the participants difficult to maintain in a longitudinal study (Freedman et al., 1980; Grella and Joshi, 1999; Hristova et al., 2003; Kessler, 1994; Kessler et al., 1994; Lamb et al., 1998; Meyers et al., 2003; McCoy and Nurco, 1991; Ribisl et al., 1996; Robins and Regier, 1991; Ross et al., 1988; Tims et al., 2001; Wright et al., 1995).

3. Model

The proposed model focuses on an effective system of managing follow-up, with a particular emphasis on achieving congruency between population characteristics and follow-up techniques, having a clearly defined infrastructure, and utilizing a follow-up protocol that directs and monitors the detailed tracking of participants; all of which should be developed and in place long before the first participant is recruited.

3.1. Staff roles and responsibilities

While no agreed-upon criteria exist for selecting follow-up staff, most researchers agree that interviewers and trackers can affect the quality of the information collected as well as the attrition rate. To establish a solid foundation for the follow-up structure, staff need to receive thorough study-specific training, as well as training on the population being studied and methods for improving personal rapport with the respondents (Marmor et al., 1991; Bootsmiller et al., 1998; Sullivan et al., 1996). While Considine and Nurco (1991) suggest that matching staff and participant characteristics is an effective strategy, Fowler (1991) argues there are no consistent findings regarding the correlates of interviewer characteristics and the quality of the interview beyond adequate reading and writing skills. Fowler (1993) further argues that hiring individuals familiar with a topic area, such as persons in recovery, can be counterproductive because they may assume more about the respondent than is actually reported. Fry and Dwyer (2001) found interviewer characteristics to be one of the reasons that people participate in research, further echoing the importance of staff characteristics.

The flexible nature of the staffing pattern also significantly impacts the team's success to track and complete interviews. Successful follow-up studies require that contacts be made 7 days a week and 12–15 h a day. To minimize costs, the staffing plan may include “on call” hours, hiring of part-time staff to allow for more complete coverage, split shifts for staff, or shifting the workday by having one or two staff begin and end late (e.g., 11 a.m. to 7 p.m.).

3.2. Obtain agency and institutional approval to locate and access participants

Living arrangements and custody status often change during the course of a longitudinal study. While some participants may be incarcerated in federal, state, or local criminal justice facilities at the time of their follow-up interview, others may reside in psychiatric institutions, residential treatment facilities, or other confined environments. Without institutional approval and appropriate consents, staff will likely be unable to legally disclose to the research team the presence of the participant in the facility. Similarly, a participant's collateral may disclose to the research team where the participant is currently housed, but the institution cannot acknowledge it or allow access without the appropriate signed consent forms. With increased interest in privacy and related laws (e.g., HIPAA), it is becoming increasingly difficult to obtain locating information from institutions making access difficult if not impossible.

Failure to complete the appropriate groundwork at the time the project is awarded will force the research team to expend considerable financial resources trying to locate these participants, with many approvals for access granted after the study is completed. In some cases, institutions may require that their IRB approve the study, or institutions may require approval from a less formal internal review committee (Desmond et al., 1995); either often takes 6–12 months. Some researchers argue that this is too time-consuming. However, the upfront costs of gaining access must be balanced with the cost of increased attrition.

Adolescent studies face additional obstacles when custody status shifts from one person to another, or to an institution. Once again, depending on the nature of the sample, some portion of the adolescents who were under parental custody at the time of enrollment will have custodians at follow-up who may not have knowledge of their involvement in the study or who did not provide permission for the adolescent to participate at intake because they were not involved at that point. In addition, adolescents who reach adulthood during the course of the study require the team to access institutions that serve adolescents as well as those serving adults, such as the criminal justice system.

Although the issues related to gaining access can never be fully addressed due to the numerous institutions and agencies that may ultimately be involved, certain steps are effective in minimizing the impact. During the early stages of a new project, the research team can analyze the characteristics of the sample to be drawn paying particular attention to the nature of the study (substance abuse treatment, mental health, prevention, education), and to the characteristics of the target sample including age (adolescents versus adults), rates of homelessness and mental illness, predicted level of criminal justice involvement, seasonality (migration during the winter), undocumented status, and geographic catchment area. Once this task has been completed, the members of the team identify systems, facilities, and institutions in which participants will likely reside at the time of follow-up. Next, a member of the team investigates the required or preferred procedures for gaining access and whether or not agency-specific releases are required.

3.3. Develop and use appropriate follow-up materials

3.3.1. Educate and motivate

Sobell (1978) and others (Marmor et al., 1991) recommended explaining to participants the reasons for conducting follow-up, when participants can expect to be contacted for follow-up, the kinds of information that will be collected and how the information will be used. If the study is recruiting an intent-to-treat sample, enrollment may provide the only opportunity for the research team to educate and motivate participants. Those who do not understand the expectations for participation or who are not properly motivated will more likely refuse to participate, be more difficult to locate, and be unresponsive to calls. Unless study parameters are specifically addressed, participants make assumptions about their participation. Individuals often assume that if they decide not to participate in treatment then their participation in the study is also terminated. Participants commonly believe that dropping out of treatment, using substances again, or moving to a different state will halt their participation in the follow-up.

3.3.2. Secure consent

It is not uncommon for a research team to utilize a locator form but fail to secure the necessary consents and signatures allowing the team to contact the persons and agencies listed on the form. Another missing piece of documentation often relates to the participant giving the agencies permission to disclose their address or whereabouts. When developing this type of consent, it is important that the nature of the study not be included on the form because this would breach confidentiality. At a minimum, the following issues should be addressed by the consent forms: (a) the participant gives members of the research team permission to contact the persons or agencies listed on the locator form to learn about his or her whereabouts; (b) the participant gives permission to collaterals or agencies to disclose their whereabouts; (c) the nature of the study cannot be disclosed unless this is covered explicitly within this or another consent, and should only be done under extreme situations. Even in instances where the research team has permission from the participant to disclose the nature of the study, it is best to keep this confidential and design the tracking consents to eliminate it.

3.3.3. Collect locator information

Aside from the documents that allow agencies and institutions to disclose the whereabouts of participants, the Locator Form with subsequent updates is the most critical document for a longitudinal study (Cottler et al., 1996; Dennis et al., 2002). As noted earlier, given that individuals transition into and out of recovery multiple times, it is important to collect information about persons, places, and things related to both the using and non-using social networks. In addition, collecting information from past, current, and future time frames has also been useful. For example, if a participant is incarcerated at the time of the follow-up interview, it is useful to ask questions about persons, places, and movements prior to incarceration, the names of individuals who visited them while in prison and with whom they corresponded via phone or mail, as well as the person's plans after release. It is important to add questions about seasonal mobility if relevant. Another variable often overlooked in this process is how recently the participants had contact with the persons named on the locator. Filling the locator with names and addresses of individuals who have not been in contact with the participants for months and sometimes years may limit their usefulness.

The procedures used to collect information also impact the quality of the information. In our studies, the interviewer always administers the locator form and records the information, never the respondent. Before asking any questions, the interviewer explains how the information will be kept confidential, what the information will be used for, and what type of script will be used when talking with the contacts on the locator form.

It is rare that a participant cannot provide enough information for tracking. Regardless of whether the individual is homeless, mentally ill, or high, people eat, sleep, bathe, and move from one location to another; they have some routine in their life. The challenge in collecting locating information is asking questions that allow the researcher to capture this information. The participant's routine may only surface when collecting information about them over a month instead of a day or week. When working with individuals who are homeless or mentally ill, these participants are better able to respond to concrete questions and recent time frames. For example, instead of asking participants “Where do you typically live?” researchers ask concrete questions about recent events such as, “Where did you wake up this morning? Where did you sleep last night? How did you get to the office for the interview? Did you eat today? Where did you get the food?” In cases where the information is difficult to elicit, the research team must question: (a) whether or not they have adequately explained the use of the information so the individual is not concerned about disclosing it, (b) whether the questions are framed in a way that makes sense to the respondent, and (c) whether a different staff person, perhaps one with more experience or a different background, would be more successful.

3.4. Develop and implement the Engagement, Verification, Maintenance and Confirmation (EVMC) Protocol

The EVMC Follow-up Protocol described in this paper provides a standardized proactive set of procedures used to manage all cases in a study and can also be used to manage cases across studies. The protocol was designed to meet the following objectives: (a) maintain contact with participants regardless of high rates of mobility, (b) quickly detect when locating information is no longer accurate, (c) allow adequate time to relocate participants prior to the interview date and (d) rely on the most efficient and cost effective strategies. It outlines a specific set of rules and timelines for combining telephone contacts, mailings, and fieldwork. The EVMC protocol begins with the baseline interview and ends with the last follow-up interview. The time frames for completing the tasks can be modified to accommodate different follow-up windows (e.g., annual follow-up, quarterly follow-up), but it is important to understand the potential implications of these types of changes.

3.4.1. Engagement phase

The EVMC protocol for an intake to 3-month follow-up window is shown in Fig. 1. The first set of tasks to be completed during the engagement phase of the protocol include: (a) educate and motivate the participants, (b) collect appropriate consents and locator data, (c) schedule the next follow-up appointment, and (d) provide participant with schedule card. The schedule card includes: (a) the date, time, and location of the next interview; (b) a toll free number the participant can use to update locator information or check-in; (c) the amount of money or type of compensation the participant will receive upon completion of the next interview.

Fig. 1.

EVMC Follow-up Protocol. This figure shows the EVMC protocol for recruiting, verifying locator information, tracking and getting people to a single interview at 3 months. (Modifications for subsequent waves or longer durations are discussed further in the text.)

Next a member of the research team enters the completion date for all consent forms, the locator form, and other documents needed to track the participant. A report to track the completion of documents necessary for successful follow-up can be generated. Next, the follow-up interview documents are separated from all types of assessment information, and the documents needed for follow-up are placed in a physical file. If possible, “the case” is then assigned to a specific staff person who becomes responsible for tracking this participant through the end of the study. This is the optimal method as it provides continuity for the participant as well as the research team by developing the rapport necessary to reduce future barriers often encountered in working with difficult participants (Kline et al., 1991; Mowbray et al., 1993).

The last step in this phase requires that within 7 days after completion of the interview, a thank you card be mailed to participants. This task serves two purposes. The first is to express the team's gratitude to the participant for agreeing to participate, and the second is to indicate that the mailing address on the locator may be incorrect and that appropriate action should be taken. Mowbray (1991) recognized the engagement phase as a pivotal time that must be successfully completed before participants can proceed to the next stages.

3.4.2. Verification phase

The primary goal of this phase is to verify the information on the locator within 7–10 days after a participant has been enrolled, and it requires that contact information for no less than three collaterals be verified. This is consistent with other research teams (Sullivan et al., 1996; Claus et al., 2002) that concluded that obtaining three or more contacts at the recruiting interview were associated with lower attrition. Techniques used to verify locators include telephone directories, phone discs, directory assistance, Internet, and telephone contact with the collateral. Other variables to consider when verifying locating information are: (a) the stability of the collateral (length of time at current address), (b) frequency and recency of communication between the participant and the collateral, and (c) the collateral's inclusion as a member of the participant's using or recovery network. In the event that a staff member is unable to verify information recorded on the locator form, it is useful to contact the participant as quickly as possible either by phone or a face-to-face visit to review the information and ask for additional information. It is helpful in these situations to take responsibility for the problem by simply explaining that the interviewer may have recorded the information incorrectly. The difficulty and costs in locating participants increase as the time between collecting the locating information and using it increases. Even though the case has been transferred to the field for street outreach, the staff member continues to work the case via phone, mail, and other inexpensive techniques.

3.4.3. Maintenance phase

The follow-up window, e.g., 3 months versus 12 months post-intake, drives the timing for a series of mailings and phone contacts designed to provide periodic reminders about the next interview, keep the participant engaged, and provide an ongoing mechanism for identifying cases in which the locating information has become obsolete. Returned mail triggers additional locating steps often allowing the team time to relocate the participant if necessary. If the time between intake and the first follow-up interview is 3 months, the protocol requires mail to be sent 4 and 8 weeks post-intake. In contrast, a protocol for a 12-month window might require flyers to be mailed every other month. The longer the study runs (4–6 years), the more critical it is to change the content, layout, and the mailing schedule of the flyers so that participants do not become bored but rather continue to look forward to receiving correspondence.

3.4.4. Confirmation phase

Six weeks prior to the 3-month follow-up date, the confirmation process begins. The protocol requires that for a case to transfer to “confirmed” status a staff person must speak directly with the participant to confirm the date, time, and location of the follow-up interview. A case is not confirmed if a message is left on an answering machine, left with a significant other, or with a collateral—the participant must speak directly with a staff member. During the confirmation phase, a staff person is required to complete some type of activity on each unconfirmed case every 24–48 h. Beginning this process 6 weeks before the participant is due allows adequate time for: (a) staff to leave messages with collaterals in the network, (b) collaterals to encounter the participant and give him/her the message, and (c) the participant to contact the office to confirm the appointment. Even though the majority of participants in our studies do not have phones, messages left with collaterals during the confirmation phase produce confirmation rates ranging from 70 to 95%. This timeline also provides adequate time to conduct field searches when necessary. When locating information fails to produce a successful confirmation 4 weeks prior to the follow-up appointment date, the case is reviewed by the Field Supervisor and then transferred to the field for street outreach. Staff members continue to work the case using phones and mailings even though the case has been transferred to the field. During the field search, the outreach worker will door knock and leave flyers at the addresses listed on the locator. Once a case is confirmed, a confirmation letter is mailed to the participant to give the date, time, and location and to communicate that the research team is looking forward to their meeting. Lastly, the staff person places a series of reminder calls—28 days, 7 days and 24 h before the interview date.

3.5. Monitor compliance with the follow-up protocol

A comprehensive protocol for managing a follow-up study will not singly or consistently produce follow-up rates over 90%. A mechanism for monitoring protocol compliance is necessary and provides staff and supervisors with the information they need to manage studies and cases effectively and efficiently. The information contained in the tracking system needs to mirror the requirements of the protocol. The tracking management information system (TMIS) should be designed to track completion dates for activities required in the protocol as well as any activities completed in an attempt to satisfy the requirements of the protocol. For the tracking system to be successful, staff must document (either electronically or with paper/pencil) all events they attempt (called a number but no answer) or complete. In order to assure consistent and effective tracking efforts and prevent effort duplication, a system of detailed record keeping must be maintained (Ziek et al., 1996) and should contain information on every step of the follow-up (Desmond et al., 1995). This includes all activities ranging from faxing a release of information to an agency (allowing agency staff to disclose location of participant) to door knocking in neighborhoods and any other type of field searches.

3.6. Create standardized case tracking procedures

A standardized set of case tracking procedures should be used to work cases when participants have become lost. An excellent and detailed list of techniques for tracking participants at the case level is provided in Staying in Touch: A Fieldwork Manual for Tracking Procedures for Locating Substance Abusers for Follow-up Studies (Center for Substance Abuse Treatment: UCLA, 1996). The Case Tracking Checklist includes events like checking with social services (for public aid recipients), the department of corrections or local jails, parole and probation. This list can easily serve as a reminder of many of the things that should be done when working a case and can be helpful to supervisors when they review cases for thoroughness. When a task has been completed, however, it should not be removed from the checklist but completed again and again until the participant has been located.

3.7. Case review meetings

Case review meetings are similar to clinical review meetings. During a follow-up case review meeting, staff and supervisors review each case that is in the verification phase but is not verified, in the confirmation phase but not confirmed, and no shows. They review the activities completed in each case during the prior week to locate participants and the outcomes. The team then identifies activities that will be completed during the next week. All agreed upon next steps are documented on a log and reviewed the next week. Supervisors also use this log to check with staff during the week about their progress.

4. Model effectiveness, generalizability, and validity

4.1. Effectiveness of the model across studies

This model has been used to track and interview over 5000 individual research participants (over 12,000 interviews) across seven separate studies with an overall follow-up rate of 95% and over 90% completed within ±2 weeks of the anniversary date. Follow-up windows or intervals ranged from 3 months post-discharge to 6 years post-intake, and the geographic catchment areas ranged from an entire state (Illinois) to a large metropolitan area (Chicago) to a single rural/small urban county. Data collection across these studies spans a 7-year time frame beginning in September 1995; the largest study is still underway with an estimated completion date of August 2007—10 years after the initial interview.

The samples across these studies represent diverse participant characteristics with four studies including only adults, two studies including only adolescents, and one study including both adults and adolescents (see Table 1). All approached 100% of intake cohorts, with the first four drawing samples stratified by level of care and four targeting a specific subgroup for a specific level of care. Six of the seven studies included both males and females, and the ages ranged from 12 to 66 years. The majority of the samples within five of the seven studies were overwhelmingly minority, ranging from 99% African–American to 61% Hispanic. Participants in the studies were primarily being treated for crack/cocaine, alcohol, opioid, or cannabis-related problems. Approximately one-third of the participants presented to treatment with one or more co-occurring disorders, and the rates of adult homelessness were typically around one-third.

Table 1. Results of studies using EVMC Follow-up Protocol.

Study	Scott et al. (2003a), Chicago Target Cities (CTC)	Scott et al. (2003b), Persistent Effects of Treatment Study (PETS)a	Dennis et al. (2003), Early Re-intervention Experiment (ERI)	Dennis et al. (2002), Drug Outcome Monitoring Study (DOMS)	Scott et al. (2002a), El Rincon Targeted Capacity Expansion	Hristova et al., 2003, Haymarket Capacity Expansion (HayCap)	Godley et al. (2002), Adolescent Continuing Care (ACC)	Dennis et al. (2002), Cannabis Youth Treatment (CYT), Madison
Target population	Adult treatment intake cohort (no DUI or court mandates)	Adult treatment intake cohort (no DUI or court mandates)	Adult treatment intake cohort	Adult and adolescent treatment intake cohorts	Adult Latino at high risk of HIV	Adult African–American women at high risk of HIV	Adolescents discharged form residential treatment	Adolescent cannabis users appropriate for outpatient treatment
Initial treatment setting	Outpatient, intensive outpatient, methadone, halfway houses and residential treatment	Outpatient, intensive outpatient, methadone, halfway houses and residential treatment	Outpatient, intensive outpatient, detox and residential treatment	Outpatient, intensive outpatient, methadone, and residential treatment	Methadone	Intensive outpatient	Residential treatment aftercare	Outpatient
Sample	Intake census stratified by level of care	Intake census stratified by level of care	Intake census	Intake census stratified by level of care	Intake census	Intake census	Intake census	Intake census
Demographics	53% female, 73% African–American, 13% Hispanic, 13%, other	59% female, 87% African–American, 7% Hispanic, 6% other	59% female, 85% African–American 7% Hispanic, 8% Other	30% female, 57% African–American, 6% Hispanic	27% female, 31% African–American, 61% Hispanic, 8% other	100% female, 99% African–American, 0% Hispanic, 1% other	29% female, 20% African–American, 2% Hispanic, 78% other	25% female, 14% African–American, 1% Hispanic, 85% other
Primary substances	Cocaine, alcohol, opioids, cannabis	Cocaine, alcohol, opioids, cannabis	Cocaine, alcohol, opioids, cannabis	Alcohol, cocaine, cannabis, opioids	Opioids, alcohol, cocaine, cannabis	Alcohol, cocaine, cannabis, opioids	Cannabis, alcohol	Cannabis, alcohol
Method	In-person	In-person	In-person	Phone	In-person	In-person	In-person/phone	In-person/phone
Incentives (US$)	25–35	35	30	25	30	20	35	40
Sample size	2862	1326	448	693	183	344	183	150
Short-term follow-up completion rateb	94% at 6 months		97% at 3 months	95% at 3 months	94% at 6 months	98% at 6 months	95% at 3 months	100% at 3 months
			96% at 6 months		93% at 12 months	98% at 12 months	95% at 6 months	99% at 6 months
			95% at 9 months				94% at 9 months	97% at 9 months
			96% at 12 months					95% at 12 months
Long-term follow-up completion rateb	95% at 18 months	94% at 2 years 94% at 3 years	95% at 15 months		90% at 18 months
		96% at 4 years	95% at 18 months		96% at 24 months
		95% at 5 years	94% at 21 months
		90% at 6 years	97% at 24 months

^aPETS funded annual long-term follow-ups for a subsample of the CTC cohort.

^bFollow-up rate based on the number of completed interviews divided by the number of living participants (i.e., the total number due minus those who are deceased); Across studies our overall follow-up rate is 12756/13488 = 94.6%.

As shown in Table 1, the follow-up rates across all studies and all waves are above 90%, with 2 waves at 90%, 1 at 93%, 5 at 94%, 8 at 95%, 5 at 96%, 3 at 97%, 1 at 98%, 2 at 99%, and 1 at 100%. Follow-up rates are calculated by taking the total number of completed interviews divided by the total number of participants enrolled in the study minus those who died. Participants are not deducted from the denominator even if they are incarcerated and inaccessible, if they move out of state, if they cannot be found, if they become incapacitated. In other words, once the person is enrolled in the study they remain in the denominator unless they die. Participants, who have been located, but for a variety of reasons did not complete the follow-up interview, are not included in the numerator. Of those who completed interviews, over 90% completed them within 14 days of the target date (over 80% within 7 days).

Given that the model consistently produced high follow-up rates across studies, the next steps were to determine the amount of effort associated with various follow-up rates and whether a bias in outcomes results from lower follow-up rates. To answer these questions data was used from two studies where staff documented each contact made for each case. The type of contact ranged from dialing a telephone number and getting a busy signal to door knocking and tracking in the field. Contacts did not include interviewing time, participation in case review meetings, or transporting participants. Thus, the total number of contacts until the interview was completed can be used as a proxy measure of the difficulty of completing an interview. Below is a short description of these two studies and their participant characteristics.

4.1.1. Chicago Target Cities (CTC)

The Chicago Target Cities study (Scott et al., 2000) was conducted from March 1995 through January 1999. Staff conducted face-to-face baseline interviews and 6-month post-intake interviews with a total of 2672/2850 (94% completion rate) participants from Chicago who presented to publicly-funded treatment. Participants presented to various levels of care including outpatient, methadone, intensive outpatient, halfway houses and residential treatment. For purposes of this report, only 2010 adults will be included. No documentation on follow-up contacts is available for the first 840 participants because the team did not have formal follow-up procedures in place when they were enrolled and subsequently followed up. Fifty-three percent of the remaining sample was female, 73% African–American, 13% Hispanic, 89% were not married, and the age ranged from 18 to 66 years with a mean of 35 years. Eighty-eight percent were unemployed at intake, and 31% had been homeless during the 6 months prior to intake. Eight percent reported current criminal justice system involvement and 63% reported prior involvement. Weekly substance use during the 30 days prior to intake (5/30 days) included 33% alcohol, 33% crack/cocaine, 27% marijuana, and 30% heroin. All participants relied on publicly funded treatment. Approximately 52% reported one or more major physical problems and 42% presented with co-occurring mental disorders (32% with major depressive disorders, 36% with generalized anxiety disorders, 8% with past-month suicide attempts). The CTC data presented here are based on the number of contacts between intake and a follow-up interview at 6 months post-intake (median of 183 days after intake). Interviews were predominately conducted in-person (though some interviews for people of out the area were done by phone) and participants received US$ 25 for completing the interview. The final completion rate on this group was 96% (with 90% completed within 14 days).

4.1.2. Drug Outcome Monitoring Study (DOMS)

The Drug Outcome Monitoring Study (Dennis et al., 1999, 2000) was conducted between August 1996 and December 1997. It included a total of 410 (65%) adults and 222 (35%) adolescents drawn from a total of 21 treatment units across Illinois. Staff interviewed participants at baseline and again 3 months post-discharge via telephone. Thirty percent of the participants were female, 57% were African–Americans, 31% Caucasian, 6% Hispanic and 5% other. Their ages ranged from 12 to 63 years with a mean of 27 years. At intake, 17% were employed and 19% were involved in school or training. During the 90 days prior to intake 32% had been homeless, 50% reported current criminal justice system involvement, 19% were from a single parent family and 85% were not married or living as married. Weekly substance use during the 90 days prior to intake (13/90 days) included 38% marijuana, 37% alcohol, 33% crack/cocaine, 16% heroin, and 11% used any other drug weekly. Approximately 67% reported major health problems, 38% high levels of internal distress, 44% ADHD and 37% conduct disorder. The DOMS data presented here are based on the number of contacts between intake and a follow-up interview at 3 months post-discharge (median of 183 days after intake). Interviews were predominately conducted by phone and participants received US$ 25 for completing the interview. The final completion rate on DOMS was 93% (with 92% completed within 14 days).

4.2. Number of contacts and related follow-up rates

As shown in Fig. 2, 10 or fewer contacts captured about one-third of the sample, 20 or fewer contacts captured another one-third, and in both studies, 23 contacts or less captured 70% of the participants. It took 33 contacts or less to capture 90% in CTC and 38 in DOMS. Using a polynomial to control for the s-shape of the curve, we can predict over 99% of the variance in the follow-up rate distribution as a positive function of the number of contacts (C).

Fig. 2.

Percent of completed interviews by number of contacts. This figure shows the cumulative percent of interviews completed by number of contacts for each study. The tables that follow compare subgroups on the number of contacts to reach 70% and the maximum; and on the mean of several variables for those requiring more or less contacts than the 70th percentile (marked on the figure).

4.3. Generalizability of the model across sub-samples

To evaluate the relationship between effort (measured by the number of contacts) and follow-up rates (cumulative percent completed) for different subpopulations, we compare the distributions with Cox-Proportional Hazards Regression. The number of contacts to complete the interview was used as the measure of time/effort and participants “without interviews” were treated as right censored. In CTC, 74 (4%) of the observations were right censored, and in DOMS, 45 (7%) were right censored. Specifically, the following sub-population comparisons were made: female versus male, Hispanic versus other (CTC only), adolescent versus adult (DOMS only), past month criminal justice system involvement versus other, past month homeless versus other, any prior substance abuse treatment versus other, weekly (or more frequent) alcohol use versus other, weekly cocaine use versus other, weekly opioid use versus other, weekly marijuana use versus other, and level of care. Rather than providing two dozen more figures, we have summarized the differences in these pair wise comparisons in Table 2. The first pair of columns shows the percent in each subgroup for each study. The next two pairs of columns show the number of contacts it took for each subgroup to get to 70% completion, and 90% completion. To the right of this is the Wilcoxon–Gehen statistic and probability from the Cox-Proportional Hazard model testing whether the distribution functions for the subgroup were significantly different than the one for everyone else in the study.

Table 2. Number of contacts to reach 70 and 90% completion by study^a.

	Subgroup size (%)		70% completion		90% completion		Wilcoxon Gehan statistic (prob.)
	CTC	DOMS	CTC	DOMS	CTC	DOMS	CTC	DOMS
Total	100	100	20	23	33	38	–	–
Male	45	70	21	23	34	39	2.78	0.03
Female	55	30	20	22	22	36
Hispanicb	12	–	27	–	52	–	55.90**	–
Non-Hispanic	88	–	19	–	31	–
Adolescentc	–	35	–	23	–	39	–	0.60
Adults	–	65	–	22	–	38
Past month CJ involved	8	50	21	25	40	43	0.02	9.10*
Non-CJ involved	92	50	20	21	33	33
Past month homelessnessd	30	11	23	23	35	45	38.02**	0.05
Non-homeless	70	89	19	22	32	36
Any prior SA treatment	54	78	20	22	33	38	0.45	2.43
No prior SA treatment	42	22	20	23	33	36
Weekly alcohol used,e	33	37	20	23	34	40	0.36	0.04
Non-weekly alcohol use	67	61	20	23	33	38
Weekly cocaine used	33	33	21	22	34	35	0.13	1.53
Non-weekly cocaine use	67	67	20	23	32	39
Weekly opioid used	30	16	19	20	32	32	1.27	1.59
Non-weekly opioid use	70	84	21	23	33	39
Weekly marijuana used,e	27	38	22	23	34	42	10.42*	0.26
Non-weekly marijuana use	73	62	19	22	33	38
Adolescent outpatient	–	6	–	22	–	36	21.46**	1.06
Adolescent residential	–	29	–	23	–	40
Adult OP	56	10	18	20	30	34
Adult residential	41	55	22	23	35	39

^aInterview completion rates divided by baseline sample—including people who died, were out of community, or located but not interviewed; follow-up rates (done/due-dead) were 96.3% of 2010 with a maximum of 119 contacts in Chicago Target Cities (CTC) and 92.9% of 632 with a maximum of 124 contacts in the Drug Outcome Monitoring Study (DOMS).

^bInsufficient n of Hispanics in DOMS.

^cAll adults in CTC.

^dWeekly AOD use groups are not mutually exclusive. In DOMS, “weekly” was defined as 13+/90 days, for CTC 5+/30 days.

^eN sizes vary due to missing data.

^*P < 0.05.

^**P < 0.0001.

Out of the 20 sub-population comparisons, only 4 were significantly different. In CTC, it took significantly more effort (i.e., contacts) to reach participants who were Hispanic, homeless, weekly marijuana users, or in residential treatment. Most of these differences were related to a subgroup of people from a single short-term residential program that primarily served Hispanic men. In DOMS, it took significantly more effort to reach participants who were involved in the criminal justice system. In each case, the differences were slight at 70% but widened to a gap of 10 or more contacts in order to reach 90%. While these subgroups took more effort, overall the model still produced over 90% follow-up rates in each subgroup in these analyses. Putting all of these subgroup covariates into the Cox regression at the same time did not change the pattern of results for either study. Thus, while the level of effort may vary somewhat, the follow-up management model appears to work reliably across a wide range of subgroups to produce follow-up rates over 90%.

4.4. The impact of attrition on validity

Attrition from follow-up has long been recognized as one of the major threats to the internal validity in research (Campbell and Stanley, 1963; Cook and Campbell, 1979). The general rule of thumb, to get at least 70%, has come under increasing criticism for leaving open the door to biases that are as large or larger than the treatment effects being sought (Dennis et al., 2001; Flick, 1988; Foster and Bickman, 1996; Donovan et al., 2001; Epstein and Botvin, 2000; Fitz and Tryon, 1989; Hedeker et al., 1999; Howard et al., 1986; Ribisl et al., 1996; Stanton and Shadish, 1997). To evaluate the potential bias associated with results based on a 70% completion rate, we ranked people in each study by the number of contacts required to complete the interview. Next, outcomes from the first 70% (those requiring fewer contacts) were compared with the remaining 30% (those requiring more contacts but producing higher follow rates). Table 3 shows the results of the multivariate analysis of variance to evaluate the potential bias of accepting the lower (70%) group on days of alcohol use, cocaine use, opioid use, marijuana use, being incarcerated, working, and committing illegal acts for money. For each study, it gives the mean days for each group, the percentage of the bias ([mean hard – mean easy]/mean easy), a multivariate test of the bias on the joint distribution of all variables using Wilks' lambda, and the results of univariate analysis of variance (ANOVA) for each variable individually. Since the power of some of the variables is lower than 50%, Cohen's effect size d ([hard mean – easy mean]/pooled standard deviation) is provided to the right. Effect sizes of 0.2 or more would generally be considered small but important when evaluating a treatment effect and, hence, represent relatively large bias.

Table 3. Bias in dependent variables associated with different follow-up rates^a.

Dependent variables	Mean days					F	P	d
	1–70% (n = 1389)	71–max% (n = 518)	Total (n = 1907)	%Diff.	%Bias
Chicago Target Cities (CTC)b
Joint distribution—Wilks' lambda						0.99	0.006
Days of alcohol use	3.23	4.05	3.45	26	−7	4.4	0.0360	0.11
Days of cocaine use	1.99	3.38	2.37	70	−16	18.0	0.0000	0.22
Days of opioid use	2.42	3.16	2.62	31	−8	3.8	0.0520	0.10
Days of marijuana use	1.33	1.32	1.33	−1	0	0.0	0.9710	0.00
Days incarcerated	1.63	1.76	1.67	8	−2	0.2	0.6850	0.02
Days of work	7.54	7.03	7.40	−7	2	0.9	0.3310	−0.05
Days of illegal activity for US$	1.50	2.06	1.65	37	−9	3.3	0.0690	0.09
Dependent variables	Mean days
	1–70% (n = 429)	71–max% (n = 158)	Total (n = 587)	%Diff.	%Bias	F	P	d
Drug Outcome Monitoring Study (DOMS)c
Joint distribution—Wilks' lambda						0.972	0.019
Days of alcohol use	5.96	7.86	6.47	32	−8	1.4	0.2330	0.11
Days of cocaine use	2.39	4.22	2.88	77	−17	2.6	0.1100	0.15
Days of opioid use	2.20	6.15	3.26	180	−33	9.1	0.0030	0.28
Days of marijuana use	5.65	6.30	5.82	12	−3	0.2	0.6770	0.04
Days incarcerated	1.62	3.36	2.09	107	−22	3.3	0.0690	0.17
Days of work	26.66	29.12	27.32	9	−2	0.8	0.3830	0.08
Days of illegal activity for US$	3.41	6.78	4.32	99	−21	5.6	0.0180	0.22

^aThe first three columns show the mean days of the variable for the first 70% followed up (based on ranked number of contacts), the rest of the sample, and the total cohort; the %Diff. is difference between the more and less contacts subgroups, divided by the less contacts subgroup; %Bias is the difference between the less contact subgroup and the total, divided by the total.

^bCTC is based on days of use in the 30 days preceding a follow-up interview at 6 months post-intake (median of 183 days after intake); Power for CTC multivariate tests was 93% while power ranged from 5 to 99% for the univariate tests.

^cDOMS is based on the days of use in the 90 days preceding a follow-up interview at 3 months post-discharged (median of 133 days after intake); power for DOMS multivariate tests was 87% while power ranged from 7 to 66% for the univariate tests.

In both studies, there was a significant bias associated with being in the hard-to-complete subgroup—with the bias by variable ranging from −7 to +180% higher rates in the hard-to-complete interview. In CTC, the bias for days of alcohol and cocaine use was significantly different at the univariate level. In DOMS, the bias for days of opioid use and days of illegal activity for money were significantly different at the univariate level. In three instances the effect size (d) of the bias associated with accepting a 70% completion rate exceeded 0.2 and would have been larger than the treatment effects being evaluated. Attempting to control for these biases using the covariates shown in Table 2 did not eliminate these biases at the multivariate level. At the univariate level it actually made the bias worse. For CTC, the days of alcohol use, cocaine use, opioid use, and illegal activity for money were all biased in the univariate analyses of covariance. For DOMS, univariate analyses of covariance reduced the biases for days of illegal activity for money but increased the bias for days of opioid use. Moreover, the multivariate analyses also revealed that the direction of the bias varied by subgroup—in some cases (e.g., for adolescents) the bias was even significant in the opposite direction. Thus, accepting a 70% completion rate would have introduced significant and unpredictable biases that would have compromised the internal validity of these studies.

5. Discussion

The follow-up management model described in this paper is proactive and designed to prevent losing participants over the course of a longitudinal study while at the same time providing early detection of deficient locating information. To plan for a successful follow-up, adequate planning and financial resources need to be reserved and protected. Given that shifts in living arrangements and custody status (adolescents) are the norm in longitudinal studies with this population, obtaining access to institutions and agencies that house the participants at the time of follow-up is critical. Blocked access can create attrition rates as high as 50% depending on the nature of the sample. A proactive follow-up protocol should provide a standardized set of procedures used to manage all cases in a study. It should be designed to: (a) maintain contact with participants, (b) quickly detect, correct, or replace locating information that has become obsolete, and (c) allow adequate time to relocate participants prior to the interview date.

The proposed management model and EVMC Protocol have consistently produced over 90% follow-up rates by specifying tasks and timelines that ensure adequate time for tracking and locating participants and monitoring staff performance throughout every step of the protocol. Additional tools—such as standardized case tracking procedures and frequent case review meetings—provide accountability and maximize the input from members of the team.

An examination of the effect of attrition on validity in these two samples demonstrated that even the traditionally acceptable level of attrition can result in significant bias and that the nature of the bias is unpredictable. When outcome variables were compared based on 70% follow-up versus 90%, the results varied by study and within study, clearly demonstrating the potential impact on the internal validity of both studies. While Hansten et al. (2000) concluded that follow-up rates below 70% may produce valid findings, the classification system on which they based their conclusions differs significantly from this study. Their measure of “effort” was based on how close to the anniversary date the interviews were completed and not on the number of contacts required to complete the interview. Using the model presented in this paper, 90% of 12,000 interviews were completed within 2 weeks of the anniversary date and the results regarding attrition are based on the actual number of contacts needed to complete the interview. We have also found that when the number of interviews completed within 2 weeks of the anniversary date is a staff performance measure, the number of interviews completed within this time frame often increases, but the number of contacts needed to complete the interviews does not necessarily decrease.

The most common approach to assessing the potential bias from attrition in the addiction field has been to use baseline characteristics to compare individuals who were successfully followed-up with those who were not (e.g., Simpson and Savage, 1980; Simpson et al., 2002; Hansen et al., 1985; Harrison and Hoffman, 1989; Biglan et al., 1991; Eaton et al., 1992; Ziek et al., 1996; Hubbard et al., 1989, 1997; Hser et al., 2001). Following this common approach using the data from these two studies, the results from the baseline comparisons yielded an inaccurate prediction of the actual bias from attrition. Specifically, in CTC, instead of a bias resulting from attrition in two variables (alcohol and cocaine), the baseline comparisons correctly predicted differences in only one variable (cocaine) and incorrectly predicted differences in three other variables (heroin, marijuana and illegal acts for money). In DOMS, instead of bias in two variables (opioids, illegal acts for money), baseline comparisons failed to predict both of these and incorrectly predicted differences in two others (incarceration and work). The findings here are consistent with earlier literature that suggests that the difficulty of follow-up is often associated with differences in treatment outcomes in unpredictable ways (e.g., Moos and Bliss, 1978; Cottler et al., 1987; Farrington et al., 1990; Stinchfield et al., 1994; Walton et al., 1998).

The findings from this study should be considered in relation to its limitations. First, the majority of participants in both studies were African–American and presented to publicly funded substance abuse treatment in one state in the US, thereby limiting the generalizability of these results. A sample drawn from a different economic group would likely have more access to telephones and perhaps require a less assertive protocol to limit attrition. In addition, the impact of lower follow-up rates on validity may differ for a sample that is more highly homogenous on the dependent variable and its correlates. We plan to explore this possibility in future studies. Third, the potential negative impact of the required contacts outlined in the EVMC Protocol on the validity of the data is unknown. Frequent contacts might function as an intervention and/or increase demand characteristics (e.g., wanting to please the interviewer) resulting in false reporting of decreased substance use and other related behaviors. This concern increases significantly as the study progresses to later waves of follow-up interviews. However, several steps can be taken to balance contamination and attrition. On-site urine testing and disclosure to the participant of the results from the screen before collecting substance use information can be helpful. Training staff to maintain a positive but not judgmental position on results is critical. Statements like “you're doing a terrific job” or “I'm really happy to know you're still clean” seem positive on the surface but can easily create a desire on the participant to continue pleasing the interviewer. Audiotaping all interviews and completing quality assurance on them is critical in minimizing this type of demand. Also, in many cases, the staff person implementing the EVMC Protocol does not conduct the interview. Last, even though this management model and protocol have consistently produced follow-up rates of over 90% across studies and sub-populations, it is unknown which component or combination of components account for its success.

Future research in this area could prove invaluable to the field of substance abuse. As the field begins to experiment with recovery management models (e.g., Dennis et al., 2003; White et al., 2003), one of the keys to any model's success will be the ability to locate the clients in order to administer the check-up. In addition, there is increasing recognition in the field that addiction is often characterized by cycles of abstinence and relapse, occurring over many years. Longitudinal designs provide more than a snapshot of the chronic nature of the condition; they instead provide an ongoing opportunity to learn about factors that facilitate and prevent recovery. However, the farther out in the time the study evolves, the more difficult it is to maintain high follow-up rates. This research team has begun to identify ways in which the protocol needs modification. In particular, locating and maintaining participation with individuals 12 months after discharge from a program requires different strategies than locating and maintaining participation with individuals 4–7 years after discharge.