Tracking and Incentivizing Substance Abusers in Longitudinal Research: Results of a Survey of NIDA-Funded Investigators

Abstract

Objectives

Increased recognition that addictive behaviors tend to be chronic and relapsing has led to a growing emphasis on longitudinal substance abuse research. The purpose of this study was to identify effective follow-up strategies employed by NIDA-funded investigators who have conducted at least one study involving follow-up data collection from human subjects.

Methods

A web-based survey was administered to a representative sample of NIDA-funded researchers (N=153) with a history of conducting longitudinal research.

Results

Reported study response rates were generally high, although 27% of the studies fell below the 80% benchmark. Face-to-face and telephone-based interviews commanded the largest subject payments—two to three times higher than compensation rates for collection of biological samples. With regard to the presumed impact of low follow-up rates on the generalizability of study findings, one third of investigators who compared baseline characteristics of those who did and did not participate in the follow-up found meaningful differences. Support was found for the hypothesis that follow-up rates and total compensation would be positively related, with the mean compensation amounts between studies achieving less than an 80% follow-up rate versus those achieving rates of 80% or higher, revealing a statistically significant effect in the predicted direction.

Conclusions

The majority of respondents reported difficulty tracking and locating subjects, and study respondents often proved to be quite different from non-respondents. Incentives improved followup rates to a point, though the relationship was not linear. Efforts to improve followup rates may be better spent addressing tracking and locating logistics rather than on strategies to compel participation once the subject has been located.

Although addiction researchers often differ in how they construe the etiology, impact, and optimal treatment of substance abuse, there is a broad consensus surrounding one central point: addictive behaviors tend to be chronic and relapsing (McLellan, 2002). Accordingly, there is a growing emphasis on research that examines substance abuse and treatment outcomes using a chronic, or “careers” perspective (Anglin and Hser, 1990; National Institute on Drug Abuse, 2007). However, for field studies involving repeated measures, successfully tracking, locating, and following up with a representative sample of subjects is a challenge. One meta-analysis of 85 longitudinal studies of substance abuse clients found that nearly one third of subjects were lost to attrition within 36 months (Hansen et al., 1990). This poses a significant threat to the validity of findings in the substance abuse field (and related fields), as follow-up rates below 80% have been shown to produce dramatically biased estimates of drug use and crime (Nemes et al., 2002).

Low follow-up rates in the drug abuse research field raise serious questions regarding the validity and generalizability of results (Cohen et al., 1993; Cranford et al., 2008; Hansen et al., 1990). In general, the more difficult subjects are to find at follow-up, the more impaired they are (with regard to drug use and criminality). Nemes’ study of cocaine users had a 23% follow-up rate from the first contact attempt, increasing to 54% when up to four attempts were made, and up to 80% with nine attempts (Nemes et al., 2002). Comparing outcome results by follow-up percentage showed cocaine use at follow-up was underestimated and employment at follow-up was overestimated for findings derived from less than 80% of the target sample. Interviewing a self-selected subsample at follow-up is likely to bias findings in favor of clients who are located more easily or whose lives are more stable. Hard-to-track individuals, therefore, may be more deviant, sick, or have poorer outcomes than those respondents who could be contacted more readily (Nemes et al., 2002).

One of the goals of this survey was to gather estimates of the costs associated with conducting follow-up research with human subjects.¹ Estimating the costs specifically associated with tracking and locating study subjects is complex. The literature review conducted to prepare for this study did not reveal any empirical studies that directly addressed this topic. The most useful estimate was reported by Hansten et al. in their study of the relationship between follow-up rates and substance abuse treatment outcomes (Hansten et al., 2000). These investigators found that the marginal costs associated with increasing their follow-up rates from 60% to 90% (where N = 654) was approximately $40,000. An internal study at the UCLA Integrated Substance Abuse Programs (ISAP) estimated that the costs directly associated with locating and collecting follow-up data from a sample of substance abusers was approximately $350.00 per subject (in 2007 dollars) (Prendergast and Hall, 2001). This estimate included salaries (but only the proportion spent specifically on follow-up activities), computer use, supplies, travel, telephone use, subject payments, database costs, and mailings. It did not include benefits, space rental, or indirect costs. Based on these assumptions, researchers conducting a longitudinal study of 500 substance abusers would expect to spend $175,000 just in follow-up costs alone for each wave of follow-up contacts. The NIDA-funded Drug Abuse Outcome Studies (DATOS) project required approximately 10 field interviewer hours for each completed interview—and this was to achieve a 62% 12-month follow-up rate (Gerstein and Johnson, 2000). Undoubtedly, given the scale of the DATOS project, the staff time and costs associated with increasing the DATOS follow-up to 80% would have been prohibitive using traditional means of tracking and locating.

Given the importance of achieving high response rates—and the challenges of doing so with substance-abusing populations who are more likely to be homeless, unemployed, and involved with the criminal justice system than the general population—the authors sought to identify effective follow-up strategies employed by NIDA-funded investigators who have conducted at least one study involving follow-up data collection from human subjects. Specifically, the survey was designed to capture information about average follow-up rates, follow-up periods, methods and amount of subject payments, modes of data collection, and most critical elements to be collected to ensure successful tracking and locating of subjects.

Methods

The survey was submitted to a representative sample of NIDA-funded researchers with a history of conducting longitudinal research. The methods for administering this survey are summarized below.

Sample

The sample frame for the survey was drawn from the Computer Retrieval of Information on Scientific Projects (CRISP) database. CRISP is a biomedical database system containing information on research projects and programs supported by the Department of Health and Human Services. As of December 2008, CRISP contained 3,083 NIDA-funded projects. This list was reduced by limiting the projects to those funded between 2000 and 2006, and those that contained the terms “longitudinal” and/or “followup” in their abstract.² This resulted in 459 distinct “hits” in the database. Of these, 209 were deemed ineligible based on a one-by-one review of study abstracts. The reasons for removing these studies were as follows: use of animal subjects; non-longitudinal studies; studies used existing data; study is part of an educational or training grant; non-individual unit of analysis; or general technical errors (either in delivery of the survey or submission of the survey).

Survey Administration

The survey consisted of 16 items covering topics such as obtained follow-up rates for a given project; perceived barriers to improving follow-up rates; types of locator information considered to be most useful; typical modes of subject payments (e.g., vouchers, gift certificates, cash, etc.); number of waves of data collection; types of data collected; and how much subjects are typically paid for participating in these activities.

Investigators in the respondent sample were sent an initial e-mail invitation with a link to a web-based survey form hosted by SurveyMonkey.com. Two follow-up e-mails were sent (at 2-week intervals) to encourage non-respondents to participate. The final response rate was 61%.

All procedures for this study were reviewed and approved by the UCLA Institutional Review Board.

Results

Notwithstanding our efforts to describe general trends based on survey results, perhaps the most striking finding to emerge from our analyses was the considerable variation among studies. The mean sample size (collapsing across study conditions for multiple-group designs) was 847.2 (SD = 1,721.7), ranging from 20–15,000. Likewise, the average number of data collection waves was 5.8 (SD = 5.7), ranging from 1–43. Although the survey did not capture time periods between all follow-up points, respondents were asked to provide the time period between the baseline assessment and the first follow-up contact. The average time to first follow-up was 9.2 months (SD = 16.4), ranging from one week to 10 years. When participants were asked to provide an approximate percentage of the subjects that they were able to contact for the first follow-up point (excluding deceased subjects from the base, but including cases in which subjects were contacted but refused to participate), the average follow-up rate was 84% (SD = 11.9), ranging from 36%–100%. The median follow-up rate was 85%.

To learn more about the issues confronting researchers in their efforts to achieve high follow-up rates, respondents were asked to indicate (1) the types of activities for which subjects were paid, (2) how much they paid for each of these activities, and (3) how difficult they found various aspects of subject tracking and locating to be. In addition, data from this survey allowed us to conduct some simple tests of association between study characteristics and follow-up rates.

As shown in Table 1, the most common research activity required of subjects in the projects surveyed was a face-to-face interview (80%), followed by telephone interviews (36%), and provision of biological samples, such as blood, urine, or hair (31%). The least commonly reported data collection method was a web-based survey, which was reported by only 4% of the investigators. Based on the subject compensation data, the two most commonly used data collection approaches also brought the highest incentives, with face-to-face interviews being compensated at $44.00, and telephone interviews at approximately $33.00. Somewhat surprisingly, the typical compensation for providing biological samples ($17.20) was substantially lower than that of providing self-report data. Respondents were also asked to provide the full expected compensation for a subject who participated in all study activities for the entirety of the study. The average total possible compensation was $328.70 (SD = 600.7), ranging from $10.00–$4,800.00. To estimate the typical subject payment per data collection session, the total possible compensation was divided by the number of data collection waves for each study.³ This resulted in an estimate of approximately $60.00 (SD = 68.8) per data collection session.

Table 1.

Types of Research Activities for Which Subjects Were Paid, and Amount of Compensation (N=153)

RESEARCH ACTIVITY	% REPORTED	COMPENSATION (MEAN (SD))
Face to Face	80.4	$44.30 (34.2)
Interviews
Telephone	36.0	$32.80 (25.5)
Interviews
Mail-in Survey	13.1	$25.40 (12.1)
Web-based Survey	3.9	$29.40 (31.2)
Biological Samples (e.g., blood, urine, hair)	30.7	$17.20 (20.7)
Initiating Follow- up Contact	13.1	$8.50 (8.9)
Traveling to Research Site	17.8	$7.40 (16.6)

To assess the perceived difficulty of conducting longitudinal research, investigators were asked to rate the level of difficulty of six aspects of tracking and locating substance abusers. Response options for these items ranged from 1, “Very Easy,” to 6, “Very Difficult.” For purposes of the present study, responses were collapsed to indicate percentages of respondents reporting these study elements to be “Difficult” or “Very difficult.” Responses to these six survey items are shown in Figure 1. By a substantial margin, “tracking and locating” was the most commonly cited difficulty associated with conducting longitudinal research with substance abusers (56%). The element with the second highest difficulty rating was “maintaining current locator information over time” (41%). Fewer problems were associated with issuing subject payments or confirming receipt of payments, with only 5% of respondents rating these as difficult or very difficult. A somewhat encouraging finding was the relatively high participation rate of subjects whom the researchers were able to contact, with 86% of investigators indicating that obtaining data from contacted subjects was not a serious problem. As a general measure of the costs associated with achieving a representative follow-up rate, investigators were asked, “Considering your study overall, how much of a financial burden (including staff costs, etc.) was tracking and locating research subjects in order to collect follow-up data?” Response options and responses were as follows: “Little or No Cost” (2.7%), “Moderately Costly” (48.7%), “Very Costly” (33.3%), and “A Significant Portion of the Budget” (14.7%), with 48% of the respondents endorsing one of the latter two categories.

Figure 1.

Perceived Difficulty of Various Aspects of Conducting Follow-Up Research with Substance Abusers (N=153)

To examine possible associations between various study characteristics and follow-up rates—particularly the achievement of the conventional standard of 80%, it was hypothesized that three study elements: sample size, time to first follow-up contact, and compensation amount would be correlated with the follow-up rates in the studies represented in the survey. Specifically, it was hypothesized that larger sample sizes and longer time periods before the first follow-up would be associated with lower follow-up rates, whereas compensation amount would be positively related. These hypotheses found limited support in our data. The Pearson correlation coefficients between follow-up rates and the first two characteristics: sample size (r =−.01, p > .10) and time lapse to follow-up (r = .05, p > .10) failed to show any meaningful association.

If investigators set the values of their subject payments, in part, to incentivize participation, it is important to know the expected returns to subject compensation (i.e., do increases in subject compensation yield a significant response?). To estimate subject responsiveness the compensation elasticity of response was calculated. A high elasticity value would indicate that subjects are responsive to levels of compensation (i.e., that compensation is a useful motivator to improve followup), low values would indicate that subjects are relatively inelastic (unresponsive) to compensation (i.e., that changing the amount of compensation would have little effect on response rates).

To estimate the elasticity the dependent variable (followup percentage) was logit transformed and regressed against compensation, controlling for number of waves, sample size, and time to followup. Elasticity was then calculated as ε = (1 − F̄)C̄βwhere F̄ is the average followup rate in the sample, C̄ is the average compensation in the sample, and β is the coefficient on compensation. The estimated elasticity was of the predicted sign (followup percentage is positively related to compensation), but subjects were inelastic to the amount of compensation offered. The estimated elasticity was low (ε=0.02), suggesting that doubling the amount of compensation offered from the mean would only increase followup rates by 2%.

An additional analysis comparing mean compensation amounts between studies achieving less than an 80% follow-up rate versus those achieving follow-up rates of 80% or higher revealed significantly higher payment levels for the high follow-up-rate studies (Means = $207.6 (SD = 223.7) versus $376.2 (SD = 690.2), respectively; t (df = 137) = −2.2, p < .03).

Lastly, to assess the impact of low follow-up rates on the generalizability of study findings, investigators were asked whether they compared baseline characteristics of subjects who did and did not participate in follow-up data collection. Nineteen percent of the respondents reported not making these comparisons. Among the 81% who did, more than one third (34.4%) reporting finding meaningful differences between those who did and did not participate in the follow-up.

Discussion

The chronic nature of addictive disorders should be reflected in clinical interventions as well as research. Unfortunately, as mentioned in the introduction to this paper, study attrition is common in studies involving substance-abusing populations (Hansen et al., 1990), which—as indicated in the present study and elsewhere—can negatively affect the generalizability of study findings.

The goal of this study was to explore some important facets of conducting longitudinal research through a survey of NIDA-funded investigators who have conducted at least one study involving repeated contacts with substance abusers. One of the most striking findings of the survey was the heterogeneity of the study sample. Still, from this diverse body of research, several noteworthy trends emerged.

The overall follow-up rate reported by the investigators was 85%. By far, the most common form of data collection reported was the face-to-face interview, followed by telephone interviews, and collection of biological samples. However, these categories often overlap; respondents reported an average of 2 distinct types of data collection (with one study reporting 7). Face-to-face and telephone-based interviews also commanded the largest subject payments— two to three times higher than compensation rates for collection of biological samples. It may be that the invasiveness of the latter is given less weight than the (likely) longer time commitment associated with collection of self-report data. Overall, subjects are paid approximately $60, on average, for participating in a single data-collection session.

A powerful theme running through several analyses in this paper was the overall difficulty of tracking and locating subjects over time in substance abuse research. About half of the respondents indicated that doing so was either very costly or accounted for a significant portion of their research budgets. On the other hand, certain elements such as issuing subject payments and confirming receipt of those payments did not appear to pose problems to most researchers. Perhaps the most auspicious finding of this survey was the low percentage of researchers indicating that they had difficulty obtaining data from a subject once he or she had been contacted. This suggests that the battle for representative follow-up rates lies more in the ability to track itinerant subjects—or to overcome passive refusals—rather than having to convince subjects of the importance or benefits of participating.

Our attempts to capitalize on these survey data to identify the presence or magnitude of associations between follow-up rates and certain study features, such as sample size, time lapse to first follow-up, and overall compensation levels failed to indicate an influence of either of the first two variables, but did offer some support for the importance of providing sufficient compensation. Although the relationship between the two continuous measures of follow-up rates and total compensation produced a marginally significant (and positive) effect, the mean compensation amounts between studies achieving less than an 80% follow-up rate versus those achieving rates of 80% or higher did reveal a statistically significant effect in the predicted direction. The lack of an association between follow-up window and follow-up rate was perhaps the most surprising result. It may be that studies designed to collect long-term follow-up data are funded accordingly and, therefore, better equipped to overcome difficulties posed by lengthy time delays. Unfortunately, the data from this survey could not directly assess this possibility.

One of the limitations of this study is that—in spite of repeated reminders—only at 61% response rate was obtained. While it is ironic that a study of response rates falls short of its own mark, it is important to keep in mind that, unlike follow-up studies examining psychological traits that are likely to be associated with a tendency to respond to a survey, the connection here is less obvious. Moreover, the response rate was quite high when compared to other electronic surveys. A review of 31 studies relying on web-based or e-mail-based surveys revealed an average response rate of 37% (with rates declining each year) (Sheehan, 2001). Another important limitation is the study’s reliance on self-report. While it was assumed that fellow researchers can be counted on to provide reasonably accurate responses to our questions, few of us are completely immune to the desire to present ourselves—and our research—in the best light. On the other hand, this tendency is likely to have been mitigated by the fact that many of the questions posed in the survey address issues that are of public record—either in published journal reports or in other government-maintained databases, such as CRISP, or www.clinicaltrials.gov.

In spite of these limitations, we believe that the results from the NIDA-Funded Substance Abuse Researcher Survey provide useful information regarding the current state of longitudinal research with substance abusers, where the greatest difficulties lie, and what aspects of longitudinal research methods may warrant more attention.