Assessment concordance and predictive validity of self-report and biological assay of cocaine use in treatment trials

Abstract

Background and Objectives

Cocaine use during randomized clinical trials (RCTs) is typically assessed by participant self-report or biological assay (e.g., urinalysis). There have been few direct comparisons of these assessment methods to investigate their concordance and their predictive validity for cocaine use and psychosocial outcomes following treatment completion.

Method

In a combined sample of 380 participants from 5 cocaine RCTs, the concordance between cocaine use assessment methods was examined. Sequential multiple linear and logistic regression models evaluated the predictive validity of two assessment methods for cocaine use and psychosocial outcomes assessed at one, three, six, and 12 months after treatment.

Results

Concordance for self-report and urinalysis indicators of cocaine use was high within-treatment (k = 0.72) and moderate during follow-up (k = 0.51). Rates of concordance were higher in studies using test cups with immediate urinalysis results. Regression analyses indicated that self-report data within-treatment predicted self-reported cocaine use at all post-treatment points (β 0.22 – 0.30, p < .01), while urinalysis results within-treatment predicted urinalysis results at one, three, and six months post-treatment (OR 3.92 – 20.99, p < .05). Cocaine-positive urinalyses within- treatment were negatively associated with a composite “good outcome” indicator at one and three months post-treatment (OR 0.17 – 0.32, p < .05).

Discussion and Conclusions

These results suggest a significant role of method variance in predicting post-treatment outcomes from within-treatment cocaine use indices.

Scientific Significance

Results support recommendations that cocaine treatment trials should include both biological assay and self-report assessment. Test cups may facilitate increased self-report accuracy.

Introduction

Testing treatments for cocaine use requires the selection of appropriate primary outcome measures^1,2. Cocaine use within the treatment phase is a routinely used outcome measure. While both self-reported cocaine use and urinalysis results have been identified as predictors of end-of-treatment abstinence, ^3,4 the two forms of assessment may provide different results for a given participant or time period. ⁵

Self-report and biological assay of substance use each have advantages and disadvantages. Self-report measures such as the timeline follow-back method (TLFB^6,7) allow participants to report on their daily cocaine use on the assessment day and previous days, using a calendar to prompt recall. These measures allow for consecutive data collection without gaps in time, unlike biological assays which can only cover a specific time frame,⁸ and have the advantage of a flexible time window, as participants can report on daily substance use for a period of several weeks or months.⁶ However, the validity of self-report measures has been called into question: participants may forget whether they used ⁸ or under-report their substance use, possibly to reduce the potential for negative consequences⁹ or to avoid embarrassment ⁸ or the perception of treatment inadequacy.¹⁰

Biological assays such as urinalysis are sometimes considered more accurate¹ and are fairly specific and sensitive for cocaine, with few substances known to cause false results.¹¹ However, because cocaine metabolites are usually detectable only within 1–3 days of use,¹² using urinalysis as an indicator of cocaine use requires frequent testing.¹⁰ Further, urinalysis results can be affected by individual metabolic differences,⁵ systematically missing data when participants are not attending treatment appointments,⁵ chemical problems due to dilution or test adulteration,¹⁰ insufficient urine,¹⁰ different routes of cocaine administration,¹³ or cocaine dose.¹³

When self-report and biological assay of recent cocaine use are obtained and matched, concordance rates calculated with percentage agreement or kappa tend to be moderate. Clinical trials have shown 74–83% agreement (i.e., both self-report of past three days and urinalysis are positive for cocaine, or both are negative⁵) or kappas ranging 0.27–0.56. ¹⁴ In treatment samples, moderate concordance has been reported in methadone treatment clinics (kappa = 0.51) ¹⁰ and criminal justice groups (70–78% agreement),⁹ while a 32% false-negative rate (i.e., no self-reported cocaine use with a cocaine-positive urinalysis) rate was obtained among homeless adults.¹⁵

The timing of assessment may be also associated with concordance between self-report and urinalysis: concordance appears to be greater for patients in early treatment than in later treatment ^5,16 with the same pattern in follow-up assessments.¹⁷ Finally, some newer testing procedures allow for urinalysis results to be immediately available, such that patient and staff can be aware of the urinalysis result at time of interview. The degree to which immediately available results affects assessment concordance is unknown. Given the advantages and disadvantages of both assessment methods, most expert panels recommend that substance use clinical trials include a combination of self-reported drug use and biological assay as primary treatment outcomes. ^1,18,19

A key, but rarely studied question is whether either method is superior to the other in predicting longer-term outcomes. Within-treatment substance use has been identified as a predictor of poorer substance use outcome at long-term follow-up.²⁰ End-of-treatment cocaine abstinence has been associated with within-treatment self-reported use and urinalysis results, ^3,4 and urinalysis results during treatment have been associated with urinalysis results at follow-up. ²¹ However, self-report and urinalysis results have not been directly compared for their ability to predict post-treatment cocaine use. Similarly, little is known regarding which method may more accurately predict longer-term psychosocial adjustment outcomes, such as problems in psychological, employment, and social domains.²²

The present study examines the utility of the two within-treatment cocaine use assessment methods as predictors of longer-term (up to one year) outcome in cocaine use and psychosocial domains. Given enduring concerns regarding the accuracy of self-report measures of substance use, we first examine the concordance between self-report and urinalysis assessment methods of cocaine use within-treatment and during follow-up in a set of five cocaine treatment trials. Based on prior studies,¹⁴ we hypothesize that moderate concordance between urine- and self-reports will occur within-treatment, and we examine kappa levels to generate hypotheses about factors influencing concordance (e.g., immediately available results). Next, we examine the predictive validity of the two methods for cocaine use and psychosocial outcomes up to a year after treatment. As the two methods are different, we predict some method variance, that is, that self-reported cocaine use within-treatment will predict self-reported cocaine use at follow up, and within-treatment urinalysis results will predict urinalysis results at follow-up. Finally, as little prior data exists to suggest which method is a more robust predictor of longer-term psychosocial outcomes, we conduct exploratory analyses examining the predictive validity of the two methods for psychosocial problems at follow-up.

Method

To test these hypotheses, we used pooled data from five independent randomized clinical trials evaluating behavioral and medication treatments for cocaine dependence. All of the studies included urine specimen collection at least weekly during treatment, were conducted by the same laboratory group using similar data collection approaches, and were available for these analyses. Similar assessment batteries were used in each study and administered at in-person interviews. Four studies tested treatments that were 12 weeks in duration with follow-up interview and urinalysis at one, three, six, and 12 months post-treatment. The remaining study tested an eight week treatment with follow-up interview and urinalysis at one-, three-, and six-months post-treatment. Studies are summarized in Table 1 and described in more detail in previous reports.²

Table 1.

Study descriptions.

Study	n	Original study inclusion criteria	Behavioral Treatments	Medication Treatments	Design	Urines per week	Reduction in self-reported cocaine use	Reduction in cocaine use by urinalysis	Urinalysis in trial / follow-up	Papers
A	73	Cocaine dep; alcohol dep	CBT v TSF v ClinM	Disulf v no meds		1	Main effect for disulf, CBT or TSF compared to ClinM	Main effect for disulf, CBT, TSF compared to ClinM	Lab / Lab	Carroll et al., 1998; 2000
B	92	Cocaine dep	CBT v IPT	Disulf v placebo		1	Main effect for time, disulf, CBT	Main effect for time, disulf	Lab / Lab	Carroll et al., 2004
C	106	Cocaine dep, receiving methadone	TSF +TAU v TAU	Disulf v placebo		3	Main effect for TSF	Main effect for TSF when outliers removed	Lab / Lab	Carroll et al., 2012
D	37	Substance dep*	CBT4CBT + TAU v TAU	--		2	Trend for greater length of abstinence in CBT4CBT	Main effect for CBT4CBT	On site / On site	Carroll et al., 2008, 2009
E	72	Cocaine dep	CM + CBT v CBT	Disulf v placebo		3	Interaction effect for CM + placebo	Main effect for CM	On site / Lab	Carroll et al., under review

Note. All studies conducted in outpatient clinics; study C was conducted in a methadone clinic. dep = dependence. CBT = Cognitive-behavioral therapy. TSF = Twelve-step facilitation. ClinM = Clinical management. Disulf = disulfiram. IPT = interpersonal therapy. CBT4CBT = computerized CBT. CM = contingency management.

^*Study D included participants with any substance dependence; only participants with cocaine dependence are included in these analyses. Results for treatment effects are summarized

Behavioral treatments investigated in the studies included cognitive-behavior therapy (CBT²³), twelve-step facilitation (TSF²⁴), clinical management (ClinM²⁵), interpersonal therapy (IPT²⁶), computer-based CBT²⁷, and contingency management (CM²⁸). Medication conditions included disulfiram, no medication and placebo controls. Participants from each study who met criteria for cocaine dependence were included in these analyses if they provided data for at least one follow-up point (75–93% of randomized cocaine-dependent participants across studies).

Measures

Urinalysis

Participants were informed during the informed consent process that urine would be collected at study visits; therefore, urine sample collection for all studies was “announced” rather than random. Urine samples during treatment and follow-up visits were collected by study staff, with the exception of study C during treatment, when urine samples were collected by clinic staff. For studies A, B, and C, and study E at follow-up, urine samples were collected and sent out for analysis by a commercial laboratory. In studies D and E (during treatment), a rapid test cup was used such that participants and study staff knew of the results at the time of the interview. All urine samples were tested for cocaine metabolites using standard cutoff values (benzoylecgonine levels above 300 ng/mL were considered positive). Urinalysis results are reported as dichotomous (positive or negative). Missed urine samples are treated as missing data. The percent cocaine-positive urines during treatment was calculated by dividing the number of cocaine-positive urines obtained during treatment by the number of urine samples obtained during treatment.

Substance Use Calendar

Self-reported substance use was assessed using the Substance Use Calendar.²⁹ This instrument, based on the TLFB, assesses substance use on a daily basis using a calendar format. It was administered at baseline to reflect the 28 days prior to the study, during the study to provide daily indicators of within-treatment substance use, and at each follow-up visit to provide an indicator of daily substance use in the previous 28 days. A percent of days of self-reported cocaine use for each time period was calculated (number of days on which participant reported any cocaine use divided by number of days during treatment or follow-up period). In study A, a weekly report of cocaine use was given rather than a day-by-day report.

Addiction Severity Index (ASI³⁰

This widely used index of psychosocial functioning was administered monthly during treatment and at each follow-up. The ASI has demonstrated good test-rest and inter-rater reliability and good internal consistency.³¹ A dichotomous measure of “good outcome” was derived from the ASI to indicate no cocaine use and no days of problems in family, legal, psychological, or employment areas in the past 28 days.² Mean days of problems in each area was also calculated across follow-up periods using ASI results.

Data Analysis

For the four studies with a daily self-report of cocaine use, each urine result was matched to the participant’s self-report of cocaine use for the three days prior to the urine sample, consistent with the cocaine testing window.¹² Percent agreement (i.e., sum of urine positive and self-report positive and urine negative and self-report negative), and percent disagreement (either urine positive and self-report negative, or urine negative and self-report positive) were calculated for each study within-treatment and at follow-up. Kappas were calculated representing the group level of agreement between urinalysis and self-report. Based on results from similar populations,¹⁴ we hypothesize that the pooled-sample kappa will be in the moderate range (0.41–0.60³²). In study A, urine and cocaine use self-report were collected on a weekly basis. Percent agreement and kappa were calculated as above using weekly data, but not included in the pooled sample kappa given the different data collection procedure.

To examine the association of each assessment method (self-report vs. urines) to selected participant outcomes at follow-up, stepped multiple regression analyses were conducted with the various participant outcomes at follow-up as criterion variables and percent of days during treatment with self-reported cocaine use and percent positive urinalyses within-treatment as predictor variables. Data were screened for multicollinearity and outliers. Multiple linear regression was used for continuous outcomes (percent days of self-reported cocaine use at one-, three-, six, and 12-month follow-ups; mean days of difficulty in ASI problem areas). Multiple logistic regression was used for dichotomous outcomes (urinalysis results; “good outcome” composite). All analyses were run in a forward sequential manner. Baseline self-reported cocaine use for the past 28 days and urinalysis results were entered as the first step to provide an indicator of pre-treatment frequency of cocaine use. Dichotomous indicators of treatment were entered as the second step: treatment received (disulfiram vs. no medication or placebo; CBT vs. no CBT; TSF vs. no TSF; CM vs. no CM) and treatment completion status (attended the end-of-treatment assessment session). Within-treatment cocaine use indicators (percent days of self-reported cocaine use; percent positive urinalyses) were entered as the third step. Significant beta values for within-treatment self-reported cocaine use or urinalyses would indicate independent predictive validity after controlling for baseline frequency of cocaine use and treatment variables.

Results

Sample characteristics

The total sample from all five studies included 434 participants. Of these, 380 participants (88%) provided at least one urine sample during treatment and are included in this report. The sample included 255 men (67.1%). Participants described themselves as Caucasian (n = 200, 53.2%), African-American (n = 138, 36.3%), or Hispanic or other (n = 39, 10.3%). Most individuals had attained at least a high school education (n = 299, 78.7%) and nearly half were employed at baseline (n = 188, 48.9%). ASI composite scores at baseline indicated participants reported problems with employment (M = 0.61, SD = 0.30), and cocaine (M = 0.66, SD = 0.21), with fewer problems reported with family (M = 0.19, SD = 0.19), psychological (M = 0.18, SD = 0.20), medical (M = 0.14, SD = 0.26), or other drugs (M = 0.06, SD = 0.07). The mean number of urine samples provided by each participant during treatment was 14.38 (SD = 10.46). Fifty-nine percent of the urines collected were cocaine-positive for the pooled sample.

Concordance between assessment methodologies

Percent agreement between self-report and urinalysis for each assessment during treatment, calculated as sum of “both urinalysis and self-report positive” and “both urinalysis and self-report negative,” ranged 81.4 - 89.7% across the five studies within-treatment, and 74.7 – 81.4% during follow-up (Table 2). Low rates of discordance occurred for both under-reporting (a cocaine-positive urinalysis with a self-report of no cocaine use) and over-reporting (a cocaine-negative urinalysis with a self-report of cocaine use). Kappa values within-treatment indicated high agreement (.72), while kappas at follow-up indicated moderate agreement (.51).³² For Study A, concordance between weekly urinalysis and self-report of cocaine use was moderate (.58).

Table 2.

Concordance between self-reported cocaine use and urinalysis results.

				Within-Treatment					Follow-Up
Study	n	Lab or test cup	Number of UA	UA and SR both pos n (%)	UA and SR both neg n (%)	UA neg, SR pos n (%)	UA pos, SR neg n (%)	k	Number of UA	Lab or test cup	UA and SR both pos n (%)	UA and SR both neg n (%)	UA neg, SR pos n (%)	UA pos, SR neg n (%)	k
A	73	Lab	617	228 (37.0)	260 (42.1)	58 (9.4)	71 (11.5)	0.58**	-	-	-	-	-	-	-
B	92	Lab	800	235 (29.4)	438 (54.8)	31 (3.9)	96 (12.0)	0.66**	419	Lab	96 (22.9)	217 (51.8)	28 (6.7)	78 (18.6)	0.46**
C	106	Lab†	2459	1527 (62.1)	566 (23.0)	66 (2.7)	300 (16.4)	0.65**	351	Lab	144 (41.0)	119 (33.9)	8 (2.3)	80 (22.8)	0.52**
D	37	Test cup	321	99 (30.8)	189 (58.9)	9 (2.8)	24 (7.5)	0.78**	59	Test cup	20 (33.9)	28 (47.5)	3 (5.1)	8 (13.6)	0.62**
E	72	Test cup	1116	386 (34.6)	604 (54.1)	101 (9.1)	25 (2.2)	0.77**	250	Lab	54 (21.6)	136 (54.4)	3 (1.2)	57 (22.8)	0.49**
B-E total	380	-	4696	2247 (47.8)	1797 (38.3)	131 (2.8)	521 (11.1)	0.72**	1079	-	314 (29.1)	500 (46.3)	42 (3.9)	223 (20.7)	0.51**

Note.

^**= p < .01.

UA = urinalysis. SR = self-report. pos = positive. neg = negative. Study A did not have self-reported use by day during treatment or follow-up. Matching rates during treatment for Study A are based on weekly report and weekly urine toxicology screen. Study A is not included in the pooled sample kappa due to the difference in data collection procedures.

^†Urine samples were collected by clinic staff during Study C treatment phase. Urine samples were collected by research staff during all phases of all other studies.

Predictive validity of self-report and urinalysis for outcomes at follow-up

Results of multiple regression analyses are presented in Table 3. Self-reported cocaine use at one-, three-, six-, and twelve-month follow-up were significantly associated with within-treatment self-reported cocaine use, but these variables were not strongly related to within-treatment urinalysis. Conversely, logistic regression results for dichotomous urinalysis results at one-, three-, and six-month follow-ups showed significant associations with within-treatment urinalyses, and no association to self-reported use within-treatment (Table 4). In other words, self-reported use within treatment was associated with self-reported use at all four follow-up points, and cocaine-positive urinalyses within-treatment were associated with cocaine-positive urinalysis at three of four follow-up points. In the multiple linear regression analyses (Table 3), treatment completion status was negatively associated with self-reported cocaine use at one-, three-, and six-month follow-ups, although no one treatment was associated with significantly better outcomes across all follow-ups. However, urinalysis results at follow-up were not associated with treatment completion status or treatment received, with the exception of CM at the one-month follow-up (Table 4).

Table 3.

Hierarchical multiple regression analyses predicting self-reported days of cocaine use during follow-up.

Month of Follow- up	1				3				6				12
	ΔR2	B	SE B	β	ΔR2	B	SE B	β	ΔR2	B	SE B	β	ΔR2	B	SE B	β
Step 1: Baseline	0.07**				0.06**				0.08**				0.05**
Recent SR use		0.07	0.05	0.08		0.06	0.05	0.07		0.09	0.05	0.10		0.10	0.05	0.12
Urinalysis		0.33	0.98	0.02		0.00	0.99	0.00		0.89	0.98	0.05		0.73	1.09	0.04
Step 2: Treatment	0.03				0.06**				0.06**				0.03
CBT		−0.28	0.89	−0.02		−2.87	0.91	−0.19**		−2.06	0.90	−0.14*		−0.84	0.99	−0.06
TSF		−0.15	1.00	−0.01		−0.78	1.02	−0.04		−2.61	1.01	−0.14*		−0.09	1.09	−0.01
Conting. Mg.		−1.58	1.38	−0.06		−0.60	1.40	−0.02		−2.34	1.38	−0.09		−3.33	1.44	−0.14*
Medication		0.32	0.73	0.02		1.32	0.74	0.09		0.19	0.74	0.01		0.18	0.80	0.01
Completed		−1.84	0.77	−0.12*		−1.74	0.78	−0.11*		−2.00	0.78	−0.13*		−0.74	0.85	−0.05
Step 3: Within-	0.06**				0.07**				0.04**				0.03*
Treatment
% Positive Urines		0.83	1.35	0.04		0.27	1.37	0.01		1.07	1.37	0.05		−1.76	1.52	−0.09
% Days SR		8.17	2.08	0.26**		9.75	2.12	0.30**		6.85	2.09	0.22**		6.95	2.23	0.23**
Total R2	0.15				0.19**				0.19**				0.11*
Total Adjusted R2	0.13				0.17				0.16				0.09
Total R	0.39				0.44				0.43				0.34
Total F(9, n)	6.94**				9.34**				8.59**				4.12**
n	367				365				351				303

Note. Recent SR use = days of self-reported use in last 30. CBT = cognitive-behavior therapy. TSF = twelve-step facilitation. CM = contingency management. Completed = completed treatment. SR = self-reported.

^*p < .05,

^**p < .01.

Table 4.

Hierarchical logistic regression analyses predicting cocaine-positive urine at follow-up.

	Follow-up Month
	1			3			6			12
	Wald 2	OR	95% CI	Wald 2	OR	95% CI	Wald 2	OR	95% CI	Wald 2	OR	95% CI
Step 1: Baseline
Recent SR use	1.52	1.03	0.99–1.07	0.29	1.01	0.98–1.05	1.07	1.02	0.98–1.06	0.69	0.99	0.95–1.02
Urinalysis	0.16	1.16	0.56–2.42	1.83	1.61	0.81–3.19	0.90	1.40	0.70–2.81	2.64	1.85	0.88–3.90
Step 2: Treatment
CBT	0.89	0.70	0.34–1.46	0.13	1.13	0.58–2.18	1.49	0.67	0.35–1.28	1.43	0.67	0.35–1.29
TSF	0.05	1.09	0.49–2.44	1.45	1.55	0.76–3.18	0.22	1.19	0.57–2.48	1.47	0.65	0.32–1.31
Conting. Mg.	8.02	0.17**	0.05–0.58	1.73	0.51	0.19–1.39	0.04	1.11	0.41–2.99	0.14	0.84	0.33–2.12
Medication	0.64	0.79	0.44–1.42	0.27	1.15	0.68–1.95	2.86	1.58	0.93–2.68	1.27	1.35	0.80–2.29
Completed	2.99	0.55	0.28–1.08	3.58	0.58	0.33–1.02	2.73	0.62	0.35–1.09	0.05	1.06	0.60–1.87
Step 3: Within- Treatment
% Positive Urines	23.57	20.99**	6.14–71.74	15.39	7.04**	2.66–18.66	6.87	3.92*	1.41–10.87	2.46	2.19	0.82–5.80
% Days SR (constant)	2.09	0.23	0.03–1.67	0.29	1.54	0.32–7.30	1.14	2.51	0.46–13.65	1.26	2.24	0.55–9.20
	1.48	0.54	-	9.55	0.22**	-	7.80	0.27*	-	2.83	0.43	-
Model Statistics
Step 1 χ2(2)	19.76**			27.61**			24.64**			12.16**
Step 2 χ2(5)	25.82**			15.75**			11.78*			6.68
Step 3 χ2(2)	33.07**			26.99**			17.00**			7.13**
Full Model χ2(9)	78.64**			70.35**			53.42**			25.97**
Full Nagelkerke R2	0.35			0.28			0.23			0.13
% classified correct	75			73			68			66
n	263			294			277			255

Note. Recent SR use = days of self-reported use in last 30. CBT = cognitive-behavior therapy. TSF = twelve-step facilitation. CM = contingency management. Completed = completed treatment. SR = self-reported.

^*p < .05,

^**p < .01

Step 1: days used out of last 30, baseline urinalysis. Step 2: received CBT, TSF, Contingency Management, medication, completed treatment. Step 3: within-treatment percent positive urines, within-treatment percent days of self-reported cocaine use.

Regression analyses for non-drug outcomes (mean days of problems in medical, employment, family, psychological, and social areas across follow-up) were nonsignificant for the most part, showing no relationship between within-treatment cocaine use and follow-up outcomes (model F statistics nonsignificant; tables of analyses available on request). Models for mean days of alcohol and drug problems during follow-up were significant (respectively, F(9, 364) = 2.30, p < .05 and F(9, 363) = 2.55, p < .01; tables of analyses available as supplemental material), but neither self-reported within-treatment cocaine use nor within-treatment urinalyses emerged as significant predictors. For mean days of cocaine problems during follow-up, the overall model was significant (F(9, 364) = 5.91, p < .01; table of analyses available as supplemental material) and self-reported within-treatment cocaine use emerged as a significant predictor (t = 2.58, β = 0.17, p < .05).

To evaluate relationships between within-treatment urine and self-report indicators and general functioning during follow-up, we developed a composite indicator of “good outcome” derived from the ASI (no cocaine use and no reported legal, family, employment, or psychological problems in the 28 days prior to follow-up.² Logistic regression models predicting the composite were significant at one, three, six, and 12-month follow-ups (model χ²(9, n = 362) = 41.37, χ²(9, n = 361) = 26.40, χ²(9, n = 352) 26.37, and χ²(9, n = 302) = 17.44, respectively, all p < .01; table of results available as supplemental material). Self-reported cocaine use during treatment was not associated with the composite measure at any follow-up point. Percent positive urines within treatment were negatively associated with the “good outcome” composite as indicated by significant Wald χ² values and odds ratios below 1.00 at one- and three-months follow-up (respectively, χ² = 9.19, p < .01, OR = 0.17, 95% CI 0.05–0.53; χ² = 4.24, p < .05, OR = 0.32, 95% CI 0.11–0.95), indicating that the odds of good outcome were decreased for participants with cocaine-positive urines during treatment.

Discussion

Data from a pooled sample of five cocaine treatment RCTs indicated that self-reported cocaine use and urinalysis demonstrated good concordance within-treatment, with concordance decreased to some extent at follow-up. Regression analyses indicated that, after controlling for baseline frequency of use and treatment variables, within-treatment self-reported cocaine use was associated with self-reported cocaine use at all follow-up points. Within-treatment urinalysis results were associated with urinalysis results at three of four follow-up points. A “good outcome” composite at follow-up was predicted by within-treatment urinalysis at early follow-up points.

Concordance level results for within-treatment and follow-up cocaine use in this combined group of 5 trials were comparable to prior cocaine treatment trials⁵ and higher within-treatment than rates obtained in clinical settings.^10,16 Concordance within-treatment appeared slightly higher than when assessed during follow-up, consistent with previous studies. ^5,17 These results may reflect a wish to please research staff during follow-up or possibly to reduce length of follow-up interviews.

An intriguing point raised by these results was that concordance rates may also be influenced by technical developments in urinalysis: for studies A, B, and C, and study E at follow-up, urine results were not available at time of interview, but in studies D and E (during treatment), rapid test cup use allowed participants and study staff to know urinalysis results at the time of the interview and study staff could use conflicting urinalysis results to evoke recollection of cocaine use that might have been overlooked. Kappas for within-treatment urines for test cup studies D and E (respectively, 0.78, 0.77) appeared higher than for lab test studies B and C (respectively, 0.66, 0.65). At follow-up, kappa for study D (test cup) remained in the substantial range (0.62), while for study E (lab), kappa dropped to the moderate range (0.49), suggesting the immediate urinalysis results may be a factor in increasing concordance. Finally, in study C, differences in urine sample collection (by clinic staff during treatment; by study staff at follow-up) did not appear to influence concordance: kappa was in the substantial range within-treatment (0.65) and moderate range at follow-up (0.52), similar to study B. Consistent with prior studies, these results suggest concordance between self-report and biological assay of cocaine use may be affected by timing and data collection procedures. Future clinical trials should continue to employ data collection methods designed to enhance participant willingness to report drug use, such as providing recall cues⁸ and using clinically trained interviewers.⁶

Method variance may account for the results of regression analyses (generally, self-report predicts self-report during follow-up, urinalysis predicts urinalysis results during follow-up). These results suggest that biological and self-report assessment methods are not interchangeable and should not be used interchangeably. Using self-reported drug use as the only indicator of drug use during a trial may result in the inclusion of participants who do not meet inclusion criteria: in a recent cognition study, 19% of participants were excluded when biological assays indicated they had engaged in drug use behavior that did not match their self-report. ³³ Other limitations of self-reported drug use (i.e., forgetting,⁸ embarassment,⁸ fear of negative consequences,⁹ a wish to avoid perceived treatment inadequacy¹⁰) indicate that self-report should not be used as the sole indicator of drug use outcomes. Despite the greater perceived accuracy of biological assay,¹ the limitations of urinalysis suggest it should not be used as the sole indicator of cocaine use. Urinalysis represents a “snapshot” assessment representing only up to three days of potential cocaine use at a time. The validity of urinalyses can also be limited by factors including individual biological differences⁵ and cocaine dose.¹³ Efforts to develop biological cocaine use assays with longer detection windows are ongoing; hair testing³⁴ and oral fluid testing³⁵ show some promise. Until biological assays with longer detection windows and fewer limitations are developed, this study’s results support the consensus that cocaine treatment trials should include both biological assay and self-report assessment of cocaine use.

Which assessment method is a better predictor of longer-term client outcome beyond drug use outcomes? The “good outcome” composite was predicted by within-treatment urinalysis results at the one- and three-month follow-ups with modest odds ratios, suggesting that reduced within-treatment cocaine use when assessed with biological assay was associated with greater initial post-treatment stability. However, this relationship did not persist at later follow-up points, suggesting that it remains difficult to predict longer-term outcomes. Further study with longer follow-up periods is required.²⁰

Strengths of this investigation include use of regression models incorporating baseline cocaine frequency and treatment variables; use of data from several studies to reduce the effects of study-specific factors; and careful matching of urinalysis results to self-report data. Limitations include those typically associated with urinalysis, including the three-day cocaine testing window, the potential for several factors to affect urinalysis results, and sensitivity to missing data. The matching procedure used to calculate concordance results includes cocaine use in the three days prior to urinalysis but not the day of urinalysis, such that a participant who used cocaine on the morning of the urinalysis could have a negative self-report for the three days prior but a positive urinalysis, resulting in discordance between the two assessment methods. Further limitations include missing data, particularly at 12-month follow-up and for follow-up urinalyses. Study-specific effects could not be examined without introducing multicollinearity; replication is needed by other research groups to examine generalizability. Regression analyses were conducted in stepped fashion to control for baseline and treatment effects, however, given the exploratory nature of these analyses, no correction was made for multiple analyses. Overall, these data offer further support to recommendations that treatment studies report on both self-report and urine toxicology data, as these remain non-interchangeable and may reflect somewhat different dimensions of outcome among cocaine users.