Abstract
Background
Selection of appropriate outcome measures is important for clinical studies of drug addiction treatment. Researchers use various methods of collecting drug use outcomes and must consider substances to be included in a urine drug screen (UDS); accuracy of self-report; use of various instruments and procedures for collecting self-reported drug use; and timing of outcome assessments.
Objectives
We sought to define a set of candidate measures to 1) assess their inter-correlation, and 2) identify any differences in results.
Methods
Data were combined from completed protocols in the National Institute on Drug Abuse Clinical Trials Network (CTN), with a total of 1897 participants. We defined nine outcome measures based on UDS, self-report, or a combination. Multivariable, multilevel GEE models were used to assess subgroup differences in intervention success, controlling for baseline differences and accounting for clustering by CTN protocols.
Results
There were high correlations among all candidate outcomes. All outcomes showed consistent overall results with no significant intervention impact on drug use during follow-up. However, with most UDS variables, but not with self-report or “corrected self-report”, we observed a significant gender-ethnicity interaction with benefit shown in African American women, white women, and Hispanic men.
Conclusion
Despite strong associations between candidate measures, we found some important differences in results.
Scientific Significance
In this study, we demonstrated the potential utility and impact of combining UDS and self-report data for drug use assessment. Our results suggest possible differences in intervention efficacy by gender and ethnicity, but highlight the need to cautiously interpret observed interactions.
Keywords: Drug addiction, measurement, clinical trials, methodology
Introduction
In drug addiction research, the sensitive nature of the data collected may lead study participants to misreport their drug use. Conversely, relying on urine drug screens (UDS) alone may miss drug use events occurring too far in the past to be detected. The combination of self-report and biologic testing (UDS) could provide a stronger basis for classification. Within both self-report and biologic testing, however, detailed choices regarding the operationalization of drug use and other variables may result in very different outcomes. For example, the choice of relevant time point or period, relevant drugs to be assessed as part of the outcome, and which study participants to include, may all impact the results of the analysis and the appropriate interpretation.
In 2008, the National Institute on Drug Abuse (NIDA) Clinical Trials Network (CTN) Executive Committee formed a task force to address several issues, including the definition of drug use outcome measure to be used in CTN protocols. In 2010, the task force presented its final report recommending a primary outcome measure of the number of days of drug use in the last 30 days of treatment, based on self-report (timeline follow-back) and corroborated with urine drug screening (1). As part of their report, the task force listed several further recommendations, including a) comparing and correlating different outcome measures of drug use, and b) identifying short-term predictors of long-term outcomes. In the current analysis, we focused on comparing and correlating different outcome measures; in particular, combination measures based on self-report corroborated by UDS data as recommended in the task force report.
Very few, if any studies have undertaken a detailed analysis of candidate drug use outcome measures to compare and contrast strategies for operationalizing drug use; in particular, candidate outcome measures addressing the recommendation to combine self-report and UDS data. The goal in the current analysis was to compare candidate outcome measures in a pooled dataset that included many of the early protocols run through the NIDA Clinical Trials Network. By pooling data, we were able to obtain sufficient numbers of participants to examine the intercorrelation of several different possible outcome measures, and differences between measures in multivariable models. By assessing and comparing candidate outcome measures, we sought to identify advantages and disadvantages in choosing a particular operationalization of illicit drug use during study follow-up. We constructed several candidate outcome measures with different strategies for resolving discrepancies between negative self-report data and positive urine drug screens. Designating these variables as the “gold standard”, we sought to examine differences in the results obtained when using self-report or UDS alone, due to the misclassification that would occur when using the more limited measures. We expected that the results of these analyses would provide a clearer understanding of the implications of decisions made when operationalizing drug use during study follow-up.
Methods
This study was approved by the Institutional Review Board of the Medical University of South Carolina. Data from seven protocols conducted through the NIDA CTN were combined. We constructed a series of candidate outcome measures, each designed to reflect drug use at the end of treatment or during follow-up. The goal was to compare outcomes to each other, and to assess the performance of each candidate measure in statistical analyses of intervention efficacy.
2.1. Selection of CTN protocols for inclusion
Based on the following criteria, CTN protocols 01, 02, 04, 05, 06, 07, and 13 were included: 1) databases were locked and had been posted to the CTN data sharing repository (http://www.ctndatashare.org/), 2) primary outcome of interest was illicit drug use during follow-up, comparing randomized treatment and control groups, and 3) primary results were published and posted to the CTN dissemination repository (http://ctndisseminationlibrary.org/). For several of these protocols, illicit drug use and study retention jointly served as primary outcomes of interest; however, in this analysis only illicit drug use was considered. All protocols included testing for amphetamine/methamphetamine, cocaine, morphine/methadone/opiates, and marijuana. In addition, several tested for benzodiazepines (CTN 01, 02, 04, and 13), barbiturates and PCP (CTN 01, 02, and 04).
2.2. Candidate outcome measures
Outcome measures were selected based on the ability to calculate each outcome in a comparable way across all protocols; and based on previous employment of the measure in drug studies or the plausibility of the outcome for use in drug studies. Some of the measures were not entirely comparable across CTN protocols, however, due to differences in factors such as length of follow-up, the number of visits/screens during follow-up, and the set of substances included in the urine drug screen.
2.2.a. Last treatment visit UDS
This outcome measure was coded as a dichotomous variable, based on the presence or absence of illicit drugs detected in the urine drug screen at the last scheduled study treatment visit.
2.2.b. First follow-up visit UDS
This outcome measure was coded as a dichotomous variable, based on urine drug screen results at the first scheduled study follow-up visit.
2.2.c. Last follow-up visit UDS
This outcome measure was coded as a dichotomous variable, based on urine drug screen results at the last scheduled study follow-up visit. With varying lengths of follow-up in each protocol, this measure took on somewhat different meaning across protocols. Follow-up time varied between three and six months (Table 1).
Table 1.
NIDA Clinical Trials Network (CTN) studies included in analysis
| Study title | Primary target drug | Length of Intervention | Length of follow up* | Main findings | |
|---|---|---|---|---|---|
| CTN 01 | Buprenorphine/Naloxone versus Clonidine for Inpatient Opiate Detoxification | Opiates | 13 day detox | 1,3 and 6 months from study start | Buprenorphine/Naloxone superior to Clonidine: 77% vs. 22% treatment success. |
| CTN 02 | Buprenorphine/Naloxone versus Clonidine for Outpatient Opiate Detoxification | Opiates | 13 day detox | 1,3, and 6 months from study start | Buprenorphine/Naloxone superior to Clonidine: 29% vs. 5% treatment success. |
| CTN 04 | Motivational Enhancement Treatment to Improve Treatment Engagement and Outcome in Individuals Seeking Treatment for Substance Abuse | Any substance | 3 sessions MET over 4 weeks | 4 and 12 weeks after study termination | MET similar to control in sustained substance use reduction among primary drug users, but superior to control among primary alcohol users. |
| CTN 05 | Motivational Interviewing to Improve Treatment Engagement and Outcome in Individuals Seeking Treatment for Substance Abuse | Any substance | 1 session of MI at intake evaluation | 4 and 12 weeks post randomization | MI similar to control in substance use outcomes at 28 days and 84 days, but superior to control in study retention at 28 days follow-up. |
| CTN 06 | Motivational Incentives for Enhanced Drug Abuse Recovery: Drug Free Clinics | Stimulants | 12 weeks | 1, 3 and 6 months post study start | MI superior to control in treatment retention, sessions attended, number of negative samples, and continuous abstinence of 4, 8, and 12 weeks. |
| CTN 07 | Motivational Incentives for Enhanced Drug Abuse Recovery: Methadone Clinics | Stimulants | 12 weeks | 1, 3 and 6 months post study start | MI superior to control in submission of negative samples, and continuous abstinence of 4, 8, and 12 weeks. |
| CTN 13 | Motivational Enhancement Therapy to Improve Treatment Utilization and Outcome in Pregnant Substance Users | Any substance | 3 sessions of MET over 4 weeks | 4 and 12 weeks post active phase | MET similar to control overall, but results suggested a possible benefit in decreasing substance use among minority participants. |
Urine drug screen and clinical drug use assessment using variables D1 – D13 on Addiction Severity Index (ASI) performed at each follow-up visit
2.2.d. Percent negative UDS during follow-up
Protocols included in our analysis had varying numbers of follow-up visits with urine drug screens, ranging from 2 to 3 visits. Accordingly, possible values for this variable were limited to 0, 33%, 50%, 67%, or 100%. If a participant had missing urine drug screen data at a follow-up visit, we considered them to be positive at that visit.
2.2.e. Dichotomous self-report
Self reported drug use was captured with the Addiction Severity Index (ASI), variables D1 to D13, asking participants how many days in the last 30 they used specific drugs. These data were collected at baseline and at each follow-up assessment. For the MIEDAR (CTN06 and CTN07) protocols, the ASI was administered at baseline and a study specific follow-up assessment was administered at each follow-up visit. Participants were asked how many days in the last 30 days they had used specific drugs (questions 11–19). We calculated a dichotomous outcome measure as any illicit drug use during follow-up, based on self-report obtained at each follow-up visit.
2.2.f. Continuous self-report
This outcome measure was calculated as the maximum number of days out of the previous 30 days on which the participant reported using drugs, assessed across all follow-up visits.
2.2.g. Corrected self-report #1
This outcome measure was based on the continuous self-report measure (see Section 2.2.f.), with values of 0 days “corrected” by adding 5 for participants whose calculated “percent negative UDS during follow-up” (see section 2.2.d.) was less than 100%.
2.2.h. Corrected self-report #2
Similar to corrected self-report #1, this outcome measure was based on the continuous self-report variable (Section 2.2.f.), but instead of adding 5 days, we added 10 days for participants who reported 0 days, but whose percent negative UDS was less than 100%.
2.2.i. Corrected self-report #3
Similar to corrected self-reports #1 and #2, this outcome measure was based on the continuous self-report variable (Section 2.2.f.), with the following changes made to the self-reported data: we added 10 days for participants who reported no drug use, but whose percent negative UDS showed less than 50% negative screens during follow-up. We added 5 days for participants who reported no drug use, but whose percent negative UDS showed a majority but not all negative screens. Finally, we added 5 days for participants who reported from 1 to 4 days of drug use, but whose percent negative UDS during follow-up showed less than 50% negative screens.
2.3. Statistical analysis
In preliminary analyses, we evaluated each candidate outcome measure with regard to its distribution of values, proportion missing, and its association with the other candidate outcome measures. Unadjusted bivariate analyses were conducted to identify the intervention impact on each outcome measure, and the relationship with selected other predictors. For associations between dichotomous candidate outcome measures, chi-square tests in 2x2 contingency tables were used to assess the concordance and strength of association between outcome measures. For associations between percent negative drug screens and dichotomous outcomes, we fit the dichotomous measure as the outcome in unadjusted logistic regression models, and fit the “percent negative drug screens” variable as the predictor. This allowed us to code “percent negative drug screens” as a five-level indicator variable, with 0 as the reference category, and each other value (33%, 50%, 67%, and 100%) compared individually to the reference category. With this approach, we were able to estimate individual odds ratios for each level of the “percent negative drug screens” variable, in relation to each dichotomous outcome measure. For simple associations between the ordinal percent negative UDS variable and the continuous self-report and “corrected” self-report variables, we fit unadjusted regression models with the continuous self-report variable as the outcome, and the ordinal percent negative UDS variable as the predictor.
Using multivariable models, each candidate outcome measure was assessed in a similar series of models, evaluating the intervention impact controlling for gender, ethnicity, age, type of intervention, major drug of abuse, and the seven subscale composite scores of the Addiction Severity Index (alcohol, drug, employment, family, legal, medical, psychological). Generalized estimating equations (GEE) were used in an effort to account for induced clustering by CTN protocol. For dichotomous candidate outcomes, we fitted multilevel logistic regression models using a logit link for the probability of outcome occurrence; for continuous self-report and “corrected” self-report measures we used GEE with an identity link to estimate the intervention impact on mean days of drug use; and for percent negative screens during follow-up, we fitted a proportional odds model with a cumulative logit link to compare better and poorer outcomes across the five levels of the dependent variable. SAS version 9.1 (Cary, NC) was used for all analyses.
Results
In the seven clinical trials included in this analysis (Table 1), there were 2273 individuals with demographic data including sex and race/ethnicity, and with data for urine drug screens after randomization (i.e., during treatment and/or follow-up). We excluded 148 individuals whose race or ethnicity was not coded as African American, non-Hispanic White, or Hispanic, leaving 2125 individuals. An additional 228 individuals were excluded due to missing baseline data for major drug of abuse or Addiction Severity Index, leaving a total of 1897 individuals for final analyses. Participant demographic and drug use characteristics by CTN protocol are summarized in Table 2.
Table 2.
Participant characteristics by CTN protocol (n(%))
| CTN 01 | CTN 02 | CTN 04 | CTN 05 | CTN 06 | CTN 07 | CTN 13 | Total | |
|---|---|---|---|---|---|---|---|---|
| n=88 | n=206 | n=363 | n=342 | n=401 | n=346 | n=151 | n=1897 | |
| Race/ethnicity* | ||||||||
| African-American | 14 (15.9) | 77 (37.4) | 159 (43.8) | 38 (11.1) | 169 (42.1) | 186 (53.8) | 56 (37.1) | 699 (36.8) |
| White | 57 (64.8) | 81 (39.3) | 153 (42.2) | 283 (82.8) | 159 (39.6) | 95 (27.5) | 64 (42.4) | 892 (47.0) |
| Hispanic | 17 (19.3) | 48 (23.3) | 51 (14.1) | 21 (6.1) | 73 (18.2) | 65 (18.8) | 31 (20.5) | 306 (16.1) |
| Gender* | ||||||||
| Female | 33 (37.5) | 58 (28.2) | 106 (29.2) | 142 (41.5) | 216 (53.9) | 152 (43.9) | 151 (100.0) | 858 (45.2) |
| Male | 55 (62.5) | 148 (71.8) | 257 (70.8) | 200 (58.5) | 185 (46.1) | 194 (56.1) | 0 (0.0) | 1039 (54.8) |
| Primary drug of abuse* | ||||||||
| Heroin, methadone, Opiates | 85 (96.6) | 204 (99.0) | 35 (9.6) | 12 (3.5) | 8 (2.0) | 142 (41.0) | 21 (13.9) | 507 (26.7) |
| Cocaine | 0 (0.0) | 0 (0.0) | 69 (19.0) | 15 (4.4) | 162 (40.4) | 115 (33.2) | 35 (23.2) | 396 (20.9) |
| Cannabis | 0 (0.0) | 0 (0.0) | 54 (14.9) | 46 (13.4) | 9 (2.2) | 1 (0.3) | 57 (37.8) | 167 (8.8) |
| Other drug | 0 (0.0) | 0 (0.0) | 19 (5.2) | 55 (16.1) | 77 (19.2) | 10 (2.9) | 9 (6.0) | 170 (9.0) |
| Polydrug | 3 (3.4) | 0 (0.0) | 30 (8.3) | 14 (4.1) | 58 (14.5) | 64 (18.5) | 6 (4.0) | 175 (9.2) |
| Alcohol-drug combination | 0 (0.0) | 1 (0.5) | 52 (14.3) | 70 (20.5) | 50 (12.5) | 9 (2.6) | 9 (6.0) | 191 (10.1) |
| Alcohol | 0 (0.0) | 0 (0.0) | 104 (28.6) | 130 (38.0) | 29 (7.2) | 4 (1.2) | 12 (8.0) | 279 (14.7) |
| Nicotine/no problem | 0 (0.0) | 1 (0.5) | 0 (0.0) | 0 (0.0) | 8 (2.0) | 1 (0.3) | 2 (1.3) | 12 (0.6) |
p<0.0001.
Primary drug of abuse using variable D14 on the Addiction Severity Index (ASI)
Overall we found very high correlations between the candidate outcome measures. Odds ratios are displayed in Table 3a for the five levels of “percent of negative drug screens during follow-up” in relation to each dichotomous outcome measure, and for relationships among the dichotomous outcome measures. A convincing monotonic increase in odds ratios was observed between each dichotomous measure and the five levels of percent negative screens during follow-up. Due to some overlap in outcome measure construction, a few correlations were extremely high but non-informative: for example, individuals with a positive UDS at the first follow-up visit could never attain 100% negative UDS during follow-up (result omitted from Table 3a); in addition, they could never be coded “no drug use” on the “corrected” dichotomous self-report outcome variable (variable omitted from Table 3a).
Table 3a.
Outcome measure concordance: odds ratios and 95% confidence intervals between percent negative urine drug screens (UDS) during follow-up and candidate dichotomous outcome measures
| Last treatment visit UDS | First follow-up visit UDS | Last follow-up visit UDS | Follow-up self-report: any use yes/no | |
|---|---|---|---|---|
| Percent negative UDS during follow-up | ||||
| 0% | (ref) | (ref) | (ref) | (ref) |
| 33% | 1.0 (0.66, 1.6) | 15.3 (7.9, 29.3) | 6.4 (3.8, 10.8) | 3.4 (2.0, 5.9) |
| 50% | 7.5 (5.1, 11.0) | 51.8 (26.3, 102.2) | 7.0 (4.3, 11.2) | 6.9 (4.2, 11.5) |
| 67% | 8.8 (6.0, 12.8) | 86.1 (43.1, 172.0) | 22.4 (12.5, 40.0) | 10.6 (6.5, 17.2) |
| 100% | 15.8 (11.3, 22.0) | (non-informative) | 69.4 (43.5, 110.7) | 17.5 (11.3, 27.2) |
| Last treatment visit UDS | (not applicable) | 14.2 (10.3, 19.5) | 12.9 (9.7, 17.2) | 5.3 (4.0, 6.9) |
| First follow-up visit UDS | 14.2 (10.3, 19.5) | (not applicable) | 10.6 (7.4, 15.2) | 14.5 (9.1, 23.1) |
| Last follow-up visit UDS | 12.9 (9.7, 17.2) | 10.6 (7.4, 15.2) | (not applicable) | 7.2 (5.1, 10.0) |
In Table 3b we present the uncorrected and “corrected” self-report variables, reflecting the maximum days reported during follow-up for days of drug use during the previous month, stratified by percent negative UDS during follow-up. Overall, we observed the largest “corrections” of self-reported data among participants in intermediate categories of % negative UDS. Among participants with 0% negative UDS during follow-up, relatively few self-reported days of drug use sufficiently low to trigger an increase in the “corrected” value. In contrast, more participants in intermediate categories of % negative drug test reported 0 or only a few days of use, triggering an increase in the “corrected” value. For participants with 100% negative UDS during follow-up, the uncorrected and corrected self-report outcome measures were identical, as dictated by our variable construction.
Table 3b.
Outcome measure concordance: maximum number of self-reported and “corrected” self-reported days reported drug use in the previous month (means and standard deviations) stratified by percent negative UDS during follow-up
| Self-report | Corrected self-report #1 | Corrected self-report #2 | Corrected self-report #3 | |
|---|---|---|---|---|
| Percent negative UDS during follow-up
| ||||
| 0% | 21.2 (11.5) | 21.5 (11.1) | 21.7 (10.8) | 22.4 (9.7) |
| 33% | 18.0 (12.7) | 18.8 (11. 7) | 19.5 (10.9) | 20.1 (10.1) |
| 50% | 7.5 (9.8) | 8.8 (9.0) | 10.1 (8.7) | 11.8 (7.4) |
| 67% | 10.6 (13.0) | 12.4 (11.7) | 14.1 (10.7) | 12.4 (11.7) |
| 100% | 4.3 (8.3) | 4.3 (8.3) | 4.3 (8.3) | 4.3 (8.3) |
In multivariable modeling, we assessed intervention assignment (intervention vs. control) as the main predictor of drug use during follow-up. We included gender, ethnicity, age, treatment type (medication, counseling, contingency management), major drug of abuse, and Addiction Severity Index composite scores as covariates of interest. Several of these covariates showed consistent and significant correlations with the candidate outcome variables. With regard to primary drug of abuse, opiate users had significantly higher levels of drug use during follow-up compared to other users. In a related observation, participants in medication trials (CTN01 and CTN02, buprenorphine trials in opiate users) had more drug use in follow-up in comparison to participants in counseling or motivation therapy. The ASI composite subscale scores for drug and employment problems were associated with more drug use, while higher scores on the family and legal ASI subscales were associated with lower levels of drug use. In a non-significant association, women tended to have lower drug use than men during follow-up. Gender, race, and age were not significant overall predictor of outcome in models with no interaction term; however, we explored a significant three-way interaction between race, gender, and intervention group assignment (described below).
In unadjusted models and in adjusted models with no interaction terms, we found consistent findings across all candidate outcome variables, with no significant intervention effect. No clear pattern of increasing or decreasing intervention effect was observed over time, comparing the strength of effect of UDS measures corresponding to last treatment visit, first follow-up visit, and last follow-up visit. Clear differences were seen across all outcome measures, however, in the gender-ethnicity specific findings. We observed a highly significant gender-ethnicity interaction for intervention impact on first follow-up visit UDS, last follow-up visit UDS, and percent negative UDS during follow-up, with intervention benefits shown for African American women, white women, and Hispanic men. However, results were not consistent across all measures (Table 4a and 4b), with no significant interaction by gender-ethnicity found in analyses of last treatment visit UDS or the self-report constructs.
Table 4a.
Consistency of estimated intervention effect: odds ratios for dichotomous outcome variables using urine drug screen (UDS)1
| LTV UDS | FFV UDS | LFV UDS | % Neg UDS | |
|---|---|---|---|---|
| Model 1 (unadjusted GEE) | ||||
| Intervention vs. Control | 1.1 (0.76, 1.6) | 0.97 (0.70, 1.3) | 1.0 (0.77, 1.3) | 1.0 (0.82, 1.3) |
| Model 2 (adjusted GEE)2 | ||||
| Intervention vs. Control | 1.3 (0.78, 2.1) | 1.2 (0.95, 1.5) | 1.0 (0.76, 1.4) | 1.2 (0.97, 1.5) |
| Model 3 (stratified adjusted) | ||||
| African-American male | 1.1 (0.51, 2.3) | 1.5 (0.88, 2.4) | 0.74 (0.35, 1.6) | 1.1 (0.70, 1.6) |
| African-American female | 2.3 (0.93, 5.5) | 1.2 (0.79, 1.9) | 1.1 (0.53, 2.1) | 1.5 (1.1, 1.9) |
| Hispanic male | 1.1 (0.33, 3.6) | 1.5 (0.60, 4.0) | 1.2 (0.46, 3.1) | 2.2 (1.2, 4.1) |
| Hispanic female | 0.72 (0.27, 1.9) | 0.55 (0.24, 1.3) | 0.99 (0.37, 2.7) | 0.71 (0.39, 1.3) |
| White male | 1.3 (0.90, 1.8) | 0.80 (0.56, 1.2) | 1.3 (0.92, 1.8) | 0.78 (0.52, 1.2) |
| White female | 1.2 (0.85, 1.7) | 1.5 (1.1, 2.1) | 0.95 (0.75, 1.2) | 1.8 (1.3, 2.4) |
LTV, Last Treatment Visit. UDS, Urine Drug Screen. FFV, First Follow-up Visit. LFV, Last Follow-up Visit. % Negative UDS, percent negative urine drug screens during follow-up.
Model 2 adjusted for age, gender, ethnicity, primary drug of abuse, intervention type (medication, counseling, incentives), and Addiction Severity Index subscales.
Table 4b.
Consistency of estimated intervention effect: mean difference in maximum number of “corrected” days reported drug use in the previous month1
| Self-report | Corrected self-report #1 | Corrected self-report #2 | Corrected self-report #3 | |
|---|---|---|---|---|
| Model 1 (unadjusted GEE) | ||||
| Intervention vs. Control | 1.69 (−0.40, 3.78) | 1.64 (−0.42, 3.71) | 1.59 (−0.46, 3.64) | 1.42 (−0.55, 3.39) |
| Model 2 (adjusted GEE) | ||||
| Intervention vs. Control | 0.58 (−0.51, 1.66) | 0.53 (−0.62, 1.69) | 0.49 (−0.75, 1.73) | 0.40 (−0.77, 1.56) |
| Model 3 (stratified adjusted) | ||||
| African-American male | 1.44 (−1.18, 4.06) | 1.75 (−0.54, 4.03) | 2.05 (0.06, 4.04) | 1.65 (−0.26, 3.56) |
| African-American female | −1.22 (−4.82, 2.38) | −1.13 (−4.55, 2.30) | −1.03 (−4.36, 2.30) | −1.47 (−4.74, 1.80) |
| Hispanic male | 1.44 (−0.81, 3.70) | 1.17 (−1.05, 3.38) | 0.89 (−1.31, 3.09) | 1.38 (−0.58, 3.33) |
| Hispanic female | 0.93 (−3.40, 5.27) | 1.17 (−3.19, 5.52) | 1.40 (−3.00, 5.80) | 1.50 (−2.91, 5.92) |
| White male | 0.27 (−0.43, 0.98) | 0.17 (−0.73, 1.06) | 0.06 (−1.07, 1.19) | 0.31 (−0.60, 1.23) |
| White female | 1.10 (−1.11, 3.32) | 0.71 (−1.60, 3.01) | 0.31 (−2.10, 2.71) | 0.12 (−2.43, 2.68) |
Self-report: maximum reported number of days using a given drug during the previous month, assessed across follow-up. Corrected self-report #1: based on the continuous self-report measure, with values of 0 days “corrected” by adding 5 for participants whose calculated “percent negative UDS during follow-up” was less than 100%. Corrected self-report #2: similar to #1, but with 10 days added. Corrected self-report #3: if self-report was 0 days then we added 5 days if participant had some positive urine drug screens and 10 days if participant had mostly positive screens during follow-up; if self-report was 1 to 4 days then we added 5 days if participant had mostly positive urine drug screens during follow-up.
Conclusions
Overall, most reports describing the development or characterization of a new drug outcome measure are in response to a new need or a new area of research (2, 3). However, questions remain regarding the appropriate choice of measures when planning a study, and very few, if any studies have been conducted making direct comparisons of outcome measures in large datasets. In this analysis, a range of candidate outcome measures were created and compared for the operationalization of drug use during follow-up in drug treatment studies. The definition of drug use during follow-up is a critical decision to be made when planning a study. We observed fairly consistent results (showing no statistically significant difference between intervention groups) across all candidate outcome measures when fitting models without interaction terms. However, for several urine drug screen measures, we observed a significant three-way interaction between gender, ethnicity, and intervention group assignment, with non-Hispanic White women, African-American women, and Hispanic men showing a statistically significant beneficial intervention effect. These subgroup effects were not observed when using UDS data from the last follow-up visit for each study, when using self-report measures of drug use during follow-up, or when using constructed measures reflecting self-report corroborated by UDS results. Correction of negative self-report with positive urine drug screen data appeared to have the greatest impact on the results observed for intermediate-level drug users, i.e. participants with some, but not all, positive urine drug screens during follow-up.
Previous studies have found a diminution of treatment effects with longer follow-up periods (4). Our adjusted models with UDS outcomes were consistent with this finding, and showed odds ratios decreasing from 1.3 at last treatment visit, to 1.2 at first follow-up visit, to 1.0 at last follow-up visit; however, none were statistically significant. Also concordant with the finding of results diminishing over time, we found that the observed gender-ethnicity interaction was no longer significant at the last follow-up visit. However, among the subgroups found to potentially benefit most from intervention (African American women, white women, and Hispanic men) there was no clear pattern of increasing or decreasing intervention impact over time. While longer-term outcomes may reflect a more meaningful or significant intervention impact, a comparison of short-term and long-term outcomes is complicated by the higher rates of loss to follow-up for longer-term outcomes, and the coding of missing visits as positive on UDS. Both the expected rate of attrition and decisions for the handling of missing visits are important considerations in determining the timing of assessments during or after treatment.
With regard to consistency of the observed gender-ethnicity interaction, we found that results based on self-report or UDS-corroborated self-report did not show statistical differences in the intervention impact within gender-ethnicity subgroups. In these analyses, all subgroups showed fairly consistent non-significant associations. Therefore, despite the overall high correlations among all measures that were assessed, the results of subgroup analysis differed in a meaningful way when using UDS results versus the self-report, whether or not “corrected” by positive UDS. These results highlight the sensitivity of research findings to decisions made in operationalizing important study variables, and underline the importance of utilizing study measures that maximize theoretical validity and minimize misclassification.
Overall, our results suggest that analysis results are sensitive to choices made in operationalizing the outcome measure; in particular, observed effect modification in subgroups is likely to be particularly sensitive to misclassification of the drug use measures. The combined measures provide a theoretical strength for interpretation of results, with an opportunity to reduce outcome misclassification. The combination of self-report with objective measures to create a composite outcome measure has emerged as a recommendation for state-of-the-art clinical research in the final report of the CTN task force mentioned above (1); however, a clearer understanding of the impact seen in results after implementing different strategies is required before substance abuse researchers can identify the best method for combining these different types of outcome measures.
Previous work in comparing self report and UDS outcome measures has been completed in the context of cocaine treatment studies (5), where discrepancies in self report and UDS have been reported. Specifically, researchers have found that concordance of self-report with urine drug screens was higher among women than among men; and that concordance overall declined during the study period (6). The latter effect may be due to an increasing tendency over time for participants to underreport their drug use in an effort to demonstrate progress. In another study, researchers found much stronger correlations between alternative biological methods than between biological measures and self-report (7), further underscoring the uncertainty associated with self-report, and the advantages that may accrue by combining objective and self-reported data.
Similar work towards defining a common outcome measure has been published in alcohol research. In 2001, alcohol researchers chose a common sentinel measure of alcohol use: the percent of heavy drinking days (8). More recent alcohol studies have evaluated and recommended alternate or supplemental outcome measures including reduction in heavy drinking (9) and the percent of subjects with no heavy drinking days (10). In the study of no heavy drinking days, researchers were able to make a direct comparison with other candidate outcome measures and found that “percent of subjects with no heavy drinking days” performed similarly to alternate measures, especially when a grace period was incorporated into the analysis (10). Active outcome measure development and refinement is also occurring in cannabis research (11).
Our research moves the field forward by demonstrating similarities and differences between several candidate measures using pooled CTN data. The comparison of results using self-report alone, UDS alone, and combination measures, provide some perspective regarding advantages and disadvantages of various operationalizations of drug use in clinical studies, and the impact on study results that may result from misclassification. This work has obvious implications for clinical practice, where a wide variety of drug assessments are in use across the US, many of which have not been rigorously assessed but were developed locally for use in individual treatment centers (12). While the appropriate tools will vary for different populations and various substances of abuse, our results will help lead to an improved ability to identify the best tools and the most appropriate outcome measures for use in clinical settings as well as in research studies. In addition, secondary outcomes not directly reflecting drug use, such as employment and family/social problems, are of interest and similar analyses using these outcomes will further advance our understanding of meaningful construction of outcome variables.
In conclusion, the choice of outcome measures for an observational study or clinical trial on illicit drug use is a difficult and important decision. Drug use measures of interest may include duration of abstinence, quantity and frequency of use, time to relapse, severity of relapse, and a variety of secondary outcomes reflecting aspects of daily functioning such as employment, family relationships, and other factors. While the CTN studies included in the current analysis did not employ the timeline follow back assessment, in future analyses we plan to use data from later CTN studies to extend our comparison of candidate outcomes. The use of timeline follow back data, in which specific days of drug use are reported, will enhance our ability to identify discordance between self-report and UDS results. The outcome measures considered in the current analysis had reassuringly high correlations, suggesting that results from early timepoints may often be good predictors of results based on later timepoints, and suggesting that most “corrections” of self-reported data using UDS results may not result in large changes to the estimated drug use outcomes. Nevertheless, our findings showing significant subgroup effects when using UDS data, which were not present in analyses of self-reported data or the combination measures, underscores the importance of cautious interpretation and verification of results in clinical studies. The theoretical advantages of combining self-report and UDS data are offset somewhat by the difficulty of choosing a strategy for doing so. Studies like the current analysis will help guide implementation of methodological recommendations to construct combined measures. By making direct comparisons of different outcome measures in the same dataset, we can improve our understanding of the utility, importance, and validity of different measures. By planning studies and conducting analyses with greater methodological rigor, we can enhance our ability to evaluate predictors of drug use as well as evaluate the effectiveness of potential treatments, leading to better clinical and public health outcomes. Finally, the findings of gender and ethnicity as potential modifiers of the intervention effectiveness indicate the importance of gender and cultural considerations across treatment approaches.
References
- 1. [accessed August 13 2010]; http://ctndisseminationlibrary.org/display/522.htm.
- 2.Ismail AC, Seneviratne Rde A. Adaptation and validation of a self-report measure to evaluate substance use among Sri Lankan adolescents. Subst Use Misuse. 45(1–2):213–23. doi: 10.3109/10826080902960387. [DOI] [PubMed] [Google Scholar]
- 3.Howard MM, Weiler RM, Haddox JD. Development and reliability of items measuring the nonmedical use of prescription drugs for the Youth Risk Behavior Survey: results from an initial pilot test. J Sch Health. 2009;79(11):554–60. doi: 10.1111/j.1746-1561.2009.00448.x. [DOI] [PubMed] [Google Scholar]
- 4.Tross S, Campbell AN, Cohen LR, Calsyn D, Pavlicova M, Miele GM, Hu MC, Haynes L, Nugent N, Gan W, Hatch-Maillette M, Mandler R, McLaughlin P, El-Bassel N, Crits-Christoph P, Nunes EV. Effectiveness of HIV/STD sexual risk reduction groups for women in substance abuse treatment programs: results of NIDA Clinical Trials Network Trial. J Acquir Immune Defic Syndr. 2008;48(5):581–9. doi: 10.1097/QAI.0b013e31817efb6e. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Somoza E, Somoza P, Lewis D, Li SH, Winhusen T, Chiang N, Vocci F, Horn P, Elkashef A. The SRPHK1 outcome measure for cocaine-dependence trials combines self-report, urine benzoylecgonine levels, and the concordance between the two to determine a cocaine-use status for each study day. Drug Alcohol Depend. 2008;93(1–2):132–40. doi: 10.1016/j.drugalcdep.2007.09.007. [DOI] [PubMed] [Google Scholar]
- 6.Schuler MS, Lechner WV, Carter RE, Malcolm R. Temporal and Gender Trends in Concordance of Urine Drug Screens and Self-Reported Use in Cocaine Treatment Studies. J Addict Med. 2009;3(4):211–217. doi: 10.1097/ADM.0b013e3181a0f5dc. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Winhusen TM, Somoza EC, Singal B, Kim S, Horn PS, Rotrosen J. Measuring outcome in cocaine clinical trials: a comparison of sweat patches with urine toxicology and participant self-report. Addiction. 2003;98(3):317–24. doi: 10.1046/j.1360-0443.2003.00311.x. [DOI] [PubMed] [Google Scholar]
- 8.Allen JP. Measuring outcome in interventions for alcohol dependence and problem drinking: executive summary of a conference sponsored by the national institute on alcohol abuse and alcoholism. Alcohol Clin Exp Res. 2003;27(10):1657–60. doi: 10.1097/01.ALC.0000091223.72517.13. [DOI] [PubMed] [Google Scholar]
- 9.Gastfriend DR, Garbutt JC, Pettinati HM, Forman RF. Reduction in heavy drinking as a treatment outcome in alcohol dependence. J Subst Abuse Treat. 2007;33(1):71–80. doi: 10.1016/j.jsat.2006.09.008. [DOI] [PubMed] [Google Scholar]
- 10.Falk D, Wang XQ, Liu L, Fertig J, Mattson M, Ryan M, Johnson B, Stout R, Litten RZ. Percentage of Subjects With No Heavy Drinking Days: Evaluation as an Efficacy Endpoint for Alcohol Clinical Trials. Alcohol Clin Exp Res. 2010;34(11):1–13. doi: 10.1111/j.1530-0277.2010.01290.x. [DOI] [PubMed] [Google Scholar]
- 11.Adamson SJ, Kay-Lambkin FJ, Baker AL, Lewin TJ, Thornton L, Kelly BJ, Sellman JD. An improved brief measure of cannabis misuse: the Cannabis Use Disorders Identification Test-Revised (CUDIT-R) Drug Alcohol Depend. 2010;110(1–2):137–43. doi: 10.1016/j.drugalcdep.2010.02.017. [DOI] [PubMed] [Google Scholar]
- 12.Gans J, Falco M, Schackman BR, Winters KC. An In-Depth Survey of the Screening and Assessment Practices of Highly Regarded Adolescent Substance Abuse Treatment Programs. J Child Adolesc Subst Abuse. 2010;19(1):33–47. doi: 10.1080/10678280903400578. [DOI] [PMC free article] [PubMed] [Google Scholar]