Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2009 Aug 4.
Published in final edited form as: J Subst Abuse Treat. 2007 Jul 5;34(3):347–355. doi: 10.1016/j.jsat.2007.05.004

Inconsistencies in Self-Reported Drug Use by Adolescents in Substance Abuse Treatment: Implications for Outcomes and Performance Measurement

Katherine M Harris 1, Beth Ann Griffin 1, Daniel F McCaffrey 2, Andrew R Morra1 1
PMCID: PMC2720834  NIHMSID: NIHMS43330  PMID: 17614240

Abstract

This paper analyzes logical inconsistencies in adolescents’ reporting of recent substance use to assess the potential effect of inaccurate reporting on measures of treatment outcomes and program performance. We used data from 1,463 clients at 10 adolescent treatment programs to assess the relationship between inconsistent reports and various factors that contribute to program assignment and treatment outcomes. Our results suggest that inconsistencies do not arise at random. Instead, inconsistencies were associated with program assignment and factors widely considered to influence treatment outcomes, including age at first use, living situation, race/ethnicity, and mental distress. We also found a positive relationship between the level of inconsistent reporting of drug use and self-reports of improvement over time on several well-established treatment outcome measures. Our study highlights the need for greater awareness of the potential impact of inaccuracies in the reporting of substance use on outcomes and performance measurement and for the development of methodologies to improve accuracy.

INTRODUCTION

Measuring and improving the outcomes of substance abuse treatment for adolescents has been an ongoing priority of the Substance Abuse and Mental Health Services Administration (SAMHSA). In 1998–1999, SAMHSA funded the Adolescent Treatment Models (ATM) program in order to understand better the effectiveness of community-based services typically available to youths and their families (Morral et al 2006). At the same time, federal and state governments, as well as many private payers, are applying increasing pressure on treatment programs to collect outcomes data for the purpose of measuring the performance of substance abuse treatment providers. Outcomes-based performance measurement systems being implemented by SAMHSA rely heavily on self-reports of substance use at admission, discharge, and post-discharge follow-up (Substance Abuse and Mental Health Services Administration 2006). In order to assure the integrity of performance measures, policy makers and program administrators need to understand how inaccuracies in self-reported drug use and other outcomes can influence outcomes measurement and inferences regarding program performance.

Inaccurate reporting of substance abuse can arise through a number of mechanisms. Perceived social stigma and social desirability effects are thought to play a particularly important role in the accuracy of substance use reported by adolescents. For example, a greater willingness of adolescents to report drug use in school-based settings than at home suggests that perceived confidentiality of responses and the acceptance of peers influence adolescents’ willingness to report substance use truthfully (Gfroerer et al 1997). In treatment settings, adolescents may perceive that understating use at baseline may result in assignment to less restrictive treatment programs (Knight et al 1998). Conversely, diversion to substance abuse treatment might be an attractive option for adolescents facing criminal justice sanctions, for instance, and thereby encourage exaggerated reports of recent drug use or problems. Similarly, a number of considerations might lead respondents to over or under report the extent of their improvements after treatment is completed, including desires to obtain the approval of treatment program staff, or to avoid social or legal sanctions.

Inaccurate reports of drug use do not necessarily result from intentional deception. For example, adolescents may not recognize the names of drugs as listed on the assessment instrument. Recognition may depend on whether street names, slang terms, and/or chemical names are included in the definitions and may change over time as respondents become more experienced (Johnston & O’Malley 1997; Morral et al 2003). “Telescoping” is the tendency to report events in the distant as occurring more recently than was actually the case (Aday 1996). There is some evidence that adolescents forward telescope age of first substance use more than adults. Legal restrictions on adolescents’ use of alcohol and cigarettes, shorter recall periods, and adolescents’ lack of motivation to provide accurate autobiographical information are two potential explanations for this effect (Johnson & Mott 2001). Because the use or non use of some drugs is more salient and easier to recall compared to others, it is reasonable to expect that telescoping effects could vary by substance, among adolescents and adults.

There is also evidence that mood influences memory, such that individuals who are depressed are more likely to remember negative or unpleasant events (Blaney 1986; Watkins et al 1996). Such a relationship may drive recall of substance use behavior, particularly among individuals at high risk of substance use disorders who are more likely to experience co morbid mood problems. (Kessler et al 2005).

Whether intentional or not, inaccurate reporting of substance use can bias measures of treatment outcomes and program performance. The effect of inaccuracies on treatment outcome measures depends on how inaccuracies contribute to differences in measured changes in substance use over time compared to actual changes. For example, a disproportionate number of false denials of use at baseline or false admissions of use at follow-up can make treatment appear less effective than it actually is. Likewise, a disproportionate number of false denials of use at follow up or false admissions at baseline can make treatment appear more effective than it actually is. For dichotomous and other discrete outcome measures even random inaccuracies can result in bias in estimated treatment effects. For continuous outcome measures, random inaccuracies do not introduce bias, but they have the effect of increasing variance and making it more difficult to detect statistically meaningful change over time in treatment outcome measures.

Inaccurate reporting will be most problematic when the tendency of clients to inaccurately report substance use differs across programs in non random ways that are difficult to observe and measure. For example, biased assessments of program performance might result if a disproportionate share of clients with mental health problems were enrolled in a particular program, mental health problems were related to a tendency to either over- or under-state past substance use, and the health problems underlying these relationships were not adequately measured and controlled for in generating treatment effect estimates.

Despite the potential threats to the integrity and utility of outcome and performance measurement systems, the accuracy of self-reported use among treatment populations has received little attention (Lennox et al 2006; Wish et al 1997). By contrast, the accuracy of self-reported substance use and its effect on prevalence and age at first use estimates has been widely studied by epidemiologists and prevention researchers using both community and school-based samples (Appel et al 2001; Colon et al 2001; 2002; Fendrich 2005; Fendrich et al 1999; Fendrich & Rosenbaum 2003; Fendrich & Vaughn 1994; Johnson & Mott 2001; Johnston & O’Malley 1997; Morral et al 2000; Percy et al 2005; Shillington & Clapp 2000; Tassiopoulos et al 2006). In general, the literature on the reliability and validity of substance use reports is derived from research studies with uncertain relevance to treatment populations.

Research in this area is limited by the lack of a “gold standard” against which to measure accuracy of self-reported drug use. Whereas self-reports of drug use have often been compared to independent assessments obtained through collateral informants (like parents, spouses, or probation officers) or toxicology tests conducted on biological samples (such as urine, breath, blood, hair or saliva), these approaches have important limitations. Drug users often succeed in misleading collateral informants about their drug use or drug-related problems. Biological tests can only confirm recent use, often very recent use such as within the past 4–48 hours, making them not useful for assessing drug related problems or reports of use or abstinence in even the not so distant past.

In the absence of a direct accuracy measure, we analyzed logical inconsistencies in the reporting of recent substance use by adolescent treatment clients to assess the potential effect of inaccurate reporting on measures of treatment outcomes and program performance. Inconsistency measures, commonly used by epidemiologists and prevention researchers, (Fendrich & Vaughn 1994; Johnston & O’Malley 1997; Percy et al 2005) are a potentially valuable source of information about the accuracy of treatment outcome data. However, inconsistency measures must be used and interpreted cautiously. In the context of measures of recent use, inconsistent reporting is evidence of response error, but we cannot infer with certainty which of two logically inconsistent responses is correct, and therefore cannot determine whether the error is leading to a more or less favorable outcome than would otherwise be found. Moreover, the absence of inconsistency does not imply accuracy because clients can be consistently untruthful.

To accomplish our objective, we conducted several analyses using longitudinal survey data collected from 1,463 clients in 10 adolescent treatment programs participating in the ATM initiative. First, we measured the number of substances for which adolescent clients supplied one or more inconsistent reports across three assessments conducted at program intake, and at six, and twelve-month follow-up assessments. We then used this measure to study whether the number of inconsistently reported substances was associated with program assignment and/or baseline client characteristics. Finally, we measured the association between inconsistent reporting of recent substance use and five well-established treatment outcome measures.

METHODS

Data

We used longitudinal data from a cohort of treatment admissions followed over a 12 month period from 10 programs selected by SAMHSA’s Center for Substance Abuse Treatment to participate in the Adolescent Treatment Models (ATM) evaluation conducted between 1998 and 2001. ATM grants were available to well-established treatment programs serving youths age 18 and under and able to provide evidence suggestive of effectiveness. The selected programs included a mix of treatment modalities including short- and long-term residential and outpatient programs. In the analyses we present here, we do not identify individual programs. Descriptions of programs participating in the evaluation can be found elsewhere (Morral et al 2006; Stevens & Morral 2003). A total of n = 1,545 adolescents were followed in the ATM study. Of these, at least 91 percent of adolescents completed the 12 month follow up assessment for each outcome of interest studied in this report.

Data collection included a baseline assessment and follow-up assessments at 6- and 12-months after their baseline interviews. Each site employed the same assessment instrument, the Global Appraisal of Individual Needs (Dennis 1999). The GAIN is a structured clinical interview collecting information in 8 main topic domains (background, substance use, physical health, risk behaviors, mental health, environment, legal, and vocational) (Dennis 1999). Assessments were conducted by staff trained and certified in GAIN administration. Missingness in baseline and outcome variables of interest ranged from 0 to 11%. Missing values were imputed using a regression model hot-deck imputation procedure (Little & Rubin 1987). We note that missing values were not imputed for the recency variables which were used to measure inconsistent reporting.

Inconsistency in self-reported drug and alcohol use

We measured inconsistency across the three assessment periods in the reporting of substance use using GAIN items assessing how recently clients used each of 12 different substances. Substances included alcohol, marijuana, crack, cocaine, heroin, opiates, amphetamines, inhalants, PCP, hallucinogens, tranquilizers, and sedatives. Recency categories included 0–3 months (collapsed from several more detailed categories), 4–12 months, one or more years ago, and never. Examples of inconsistencies found in the data included obvious forms such as a report of past use at one time period followed by a report of never having used at a subsequent assessment and more subtle inconsistencies such as a report of use in the past 3 months followed by a subsequent report 6 months later of use one or more years ago.

Since imputed values were not used in assessing inconsistency, our analysis allowed for one missing recency reports at baseline, 6, or 12 month assessment. For cases with one missing recency report, we identified inconsistencies by comparing the two available reports. Eighty-two adolescents had only one assessment of recent drug use across the three time points for at least one drug and thus, for these adolescents, consistency could not be assessed. These adolescents were excluded from the analysis, yielding a total of sample size of 1,463 adolescents for our study. See the appendix for a list of inconsistent patterns observed in the ATM sample.

First, we created an indicator variable for each substance that took on the value “1” if a response measured at either 6 or 12 months was logically inconsistent with a response given during one or more earlier assessments. Using these substance-specific indicators of inconsistency, we then created a count of the total number of substances for which a client provided at least one inconsistent response (range: 0–12). We also used the indicators to calculate substance-specific inconsistency rates.

Baseline client characteristics

In order to examine the relationship between baseline client characteristics and inconsistent reporting of past substance use, we selected a subset of baseline GAIN items identified by Morral and others (2006) as assessing key patient placement criteria identified by the American Society of Addiction Medicine (ASAM) (Mee-Lee et al 2001). ASAM patient placement criteria represent the best available empirical evidence and clinical judgment about the factors most important for determining patient placement and treatment needs. Our four domains included (1) basic demographic characteristics, (2) mental and physical health, (3) criminal justice involvement, and (4) substance use involvement. Involvement measures include an index of substance related problems, such as first use of drugs before age 15, and ever using heroin or opiates. To the extent that involvement is a measure of the number of substances used, the associations between involvement measures and the number of inconsistencies provide information about the relationship between inaccurate reporting and the number of substances used by a client. This relationship is impossible to observe directly in the absence of an objective measure of drug use. Involvement measures also help to control for confounding relationships between drug involvement and client characteristics in our multivariate analysis. Table 1 provides statistics describing the baseline characteristics of the ATM sample.

Table 1.

Baseline Characteristics of the ATM Sample (n = 1,463)

Client Characteristic Mean (sd) or N(%)
Race
 White 650 (44.43%)
 Black 161 (11.00%)
 Hispanic origin 259 (17.70%)
 American Indian 139 (9.50%)
 Other 254 (17.36%)
Male 1109 (75.08%)
Age 15.85 (1.27)
Living Status
 House/Apt 1125 (76.90%)
 Friend/Relative 138 (9.43%)
 Jail/Correctional 103 (7.04%)
 Other status 97 (6.63%)
Homeless/Runaway 281 (19.21%)
Mental/Physical Health Variables
 Internal Mental Distress Scale 11.07 (8.81)
 Emotional Problem Scale 0.29 (0.20)
 Health Problem Index 0.14 (0.16)
Criminal Justice Involvement
 Total Arrests in past 90 days 0.74 (1.32)
 Currently involved in CJ system 1192 (81.48%)
 Illegal Activities Scale 0.10 (0.05)
 MAXCE 17. 50 (25.90)
Substance Use Problems
 Substance Problem Scale (past month) 4.54 (4.69)
 Substance Frequency Scale (past 90) 0.22 (0.18)
 First Use < 15 1324 (90.50%)
 Ever used heroin or opiates 295 (20.16%)
 Substance Need Treatment for:
  Alcohol 423 (28.91%)
  Marijuana 1001 (68.42%)
  Crack 47 (3.21%)
  Cocaine 141 (9.64%)
  Inhalants 11 (0.75%)
  Heroin 74 (5.06%)
  Opiates 6 (0.41%)
  PCP 5 (0.34%)
  Hallucinogens 66 (4.51%)
  Sedatives 1 (0.07%)
  Tranquilizers 6 (0.41%)
  Amphetamines 134 (9.16%)
 Past Month Use:
  Alcohol 670 (45.80%)
  Marijuana 812 (55.50%)
  Crack 94 (6.43%)
  Cocaine 129 (8.82%)
  Inhalants 29 (1.98%)
  Heroin 59 (3.69%)
  Opiates 108 (7.38%)
  PCP 31 (2.05%)
  Hallucinogens 156 (10.66%)
  Sedatives 68 (4.65%)
  Tranquilizers 59 (4.03%)
  Amphetamines 136 (9.30%)
Open in a new tab

Treatment outcome measures

We examined the relationship between inconsistency in the reporting of recent substance use and five treatment outcome measures, the first four of which represent the difference between baseline and 12 month follow-up measurements: (1) substance use frequency, (2) substance use problems, (3) illegal behavior, (4) emotional problems, and (5) time in controlled environments. None of the outcomes are directly derived from the recency data that were used to determine inconsistent reporting. Substance use frequency is assessed using the GAIN’s Substance Frequency Scale (SFS). The SFS takes the average of responses to seven GAIN items concerning the number of days in the past 90 days during which there was: reported use of alcohol, marijuana, cocaine or heroin, use to the level of intoxication, and a failure to perform routine activities due to substance use. There is no expectation that a common factor contributes to the variance in each item, so inter-item reliability is neither expected nor assessed.

Substance problems were measured using the GAIN’s Substance Problem Scale (SPS). The SPS is a 16-item count of self-reported symptoms of substance abuse and dependence experienced in the 30 days prior to the interview. The SPS has shown good internal consistency (Cronbach alpha=.9) and test-retest reliability (r=0.7) in prior research. (Dennis et al 2002a; Dennis et al In press; Godley et al 2005)

Illegal behavior was assessed using the GAIN’s Illegal Activities Scale (IAS). The IAS is constructed by averaging three GAIN items (recency of illegal activities, days of illegal activities in the past 90, and days supporting self through illegal activities in the past 90) after transforming each to a 0 to 1 range. The IAS was found to have good inter-item reliability (Cronbach alpha=.78) (Dennis et al 2002b).

We assessed emotional problems at baseline and follow-up using the GAIN’s Emotional Problem Scale (EPS). The EPS averages 7 GAIN items assessing the recency, frequency, and severity of psychological distress, after transforming each item to have a range of 0 to 1. The EPS has also been show to have good internal reliability (Cronbach alpha = 0.80) (Dennis et al 2002b).

Finally, we assess time in a controlled environment using the GAIN’s MAXCE variable, which calculates the maximum number of days, in the past 90, which the respondent reports being in any of several different types of controlled environments in which drug use and liberty were substantially constrained for the whole day (e.g., jail, inpatient treatment, group homes or probation camps).

Analytic Approach

We conducted three types of analyses to describe inconsistencies in substance use reporting and to assess its potential influence on outcomes and performance measurement. First, we described rates of inconsistent responses among the 1,463 adolescents in our sample for each of the 12 substance categories. Second, we estimated a negative binomial regression model to measure the relationship between the number of inconsistently reported substances and baseline client characteristics, including program assignment. The regression coefficient in a negative binomial model estimates the log of the incidence rate ratio for a predictor and thus, provides information about the multiplicative effect that a one unit increase in the predictor has on the mean number of inconsistently reported substances. We conducted this analysis in SAS using PROC GENMOD to appropriately account for the clustering of clients within treatment programs.

Third, we report mean change scores (baseline to 12-month follow-up) for the outcomes measures (substance use frequency, substance use problems, illegal behavior, and emotional problems) and mean score for MAXCE by number of inconsistently reported substances. Only adolescents with a 12-month follow-up measure for a given outcome were included in the analysis of that treatment outcome, yielding a sample sizes ranging between 1,396 and 1,403 adolescents for each outcome of interest. We tested the statistical significance of the relationship between each treatment outcome and the number of inconsistent substances using the estimated coefficient of a linear regression model with the outcome as the dependent variable and the number of inconsistently reported substances as the independent variable. Significance tests accounted for the clustering of adolescents into treatment programs via linearization methods implemented in SAS PROC SURVEYREG.

RESULTS

The mean number of substances for which clients provided inconsistent recency responses in the ATM sample was 2.25 (sd = 1.98). Figure 1 shows the distribution of the number of inconsistently reported substances in the ATM data. Approximately 77 percent of the adolescents provided inconsistent responses for at least one of the 12 substances. Table 2 shows the percent of clients providing at least one inconsistent response at a follow-up assessment by substance type. Consistent with the published literature, (Fendrich & Rosenbaum 2003; Fendrich & Vaughn 1994; Johnston & O’Malley 1997; Percy et al 2005) inconsistency rates were lowest for the most and least commonly used substances: alcohol (7.45%), marijuana (5.67%), and heroin (8.75%). Inconsistency rates for other substances are two or more times higher.

Figure 1.

Figure 1

Open in a new tab

Distribution of the Number of Inconsistently Reported Substances across Time in the Reporting of Substance Use Recency (n=1463).

Table 2.

Percent and Number Reporting Inconsistent Drug Use Recency and Type of Inconsistent Reporting by Substance across the whole sample (N=1,463)

Percent inconsistent N inconsistent
Alcohol 7.45 109
Marijuana 5.67 85
Crack 22.62 331
Cocaine 25.84 378
Inhalants 34.54 359
Heroin 8.75 128
Opiates 24.40 357
PCP 18.05 264
Hallucinogens 23.99 351
Tranquilizers 18.59 272
Amphetamines 22.35 327
Sedatives 22.28 326
Open in a new tab

Next, we examined client characteristics measured at baseline to determine which variables were significantly associated with the number of inconsistently reported substances in self-reported substance use. Table 3 shows the result from the final adjusted negative binomial regression model. In contrast to population and community-based studies, (Fendrich & Johnson 2005; Johnston & O’Malley 1997; Shillington & Clapp 2000) blacks on average reported inconsistently on less than half the number of substances than whites (incidence rate ratio = 0.41, 95% confidence interval = (0.31, 0.53)). Clients of Hispanic origin in the ATM sample provided inconsistent responses on only slightly fewer substances on average than did whites (incidence rate ratio = 0.91, 95% confidence interval = (0.85, 0.98)). Age was positively associated with the number of inconsistently reported substances (incidence rate ratio = 1.04, 95% confidence interval = (1.02, 0.1.06)). However, gender was not significantly associated with the number of inconsistently reported substances

Table 3.

Negative Binomial Regression Coefficients

Demographics Characteristics Incident Rate Ratio 95% CI P-valuec
Race
 American Indian 0.96 (0.66,1.40) 0.850
 Black 0.41 (0.31,0.53) <0.001
 Hispanic origin 0.91 (0.85,0.98) 0.013
 Other 0.89 (0.74,1.06) 0.204
 Whitea
Male 1.07 (0.99,1.17) 0.099
Age 1.04 (1.02,1.06) <0.001
Living Status
 Friend/Relative (0.84,1.20) 0.976
 Jail/Correctional 0.94 (0.69,1.28) 0.709
 Other status 1.13 (0.98,1.31) 0.101
 House/Apta
Homeless/Runaway 1.12 (1.03,1.20) 0.007
Mental/Physical Health
 Internal Mental Distress Scale 1.01 (1.00,1.01) 0.001
 Health Problem Index 0.87 (0.73,1.04) 0.118
Criminal Justice Involvement
 Total Arrests in past 90 days 1.01 (0.99,1.03) 0.347
 Currently involved in CJ system 1.03 (0.95,1.12) 0.502
Substance Use
 Substance Problem Index 1.03 (1.01,1.05) 0.003
 First Use < 15 1.22 (1.06,1.39) 0.004
 Ever used heroin or opiates 1.23 (1.13,1.35) <0.001
Treatment Programb
 A 1.32 (1.22,1.45) <0.001
 B 1.03 (0.90,1.17) 0.644
 C 1.15 (1.09,1.21) <0.001
 D 1.46 (1.39,1.52) <0.001
 E 1.12 (1.05,1.19) <0.001
 F 0.57 (0.51,0.64) <0.001
 G 0.80 (0.62,1.05) 0.107
 H 0.75 (0.70,0.79) <0.001
 I 1.38 (1.34,1.42) <0.001
 J 1.21 (0.86,1.70) 0.860
Open in a new tab
a

Reference category

b

Treatment programs are standardized so that coefficient represents difference between a program and the average mean of all programs.

c

P-values test the null hypothesis that the incident rate ratio equals one.

Clients who were homeless or had run away from home had higher rates of inconsistently reported substances than their counterparts, with homeless and/or runaways on average reporting inconsistently on 1.12 times more substances (95% confidence interval = (1.03, 1.20)). On the other hand, the living status of adolescents was not significantly associated with the number of inconsistently reported substances.

Higher levels of mental distress were associated with only slightly greater number of inconsistently reported substances (incidence rate ratio = 1.01, 95% confidence interval = (1.00, 1.01)). Interestingly, neither of our two measures of criminal justice involvement was significantly associated with the number of inconsistently reported substances. Adolescents who reported first substance use under the age of 15 or ever having used heroin or opiates at baseline on average reported inconsistently on a greater number of substances than their counterparts (incidence rate ratios = 1.22 and 1.23, respectively; 95% confidence intervals = (1.06,1.39) and (1.13,1.35), respectively).

Finally, our multivariate results revealed a strong relationship between program assignment and the number of inconsistent responses. Seven of the ten programs were associated with statistically significant differences in the number of inconsistent responses compared to the overall mean number of inconsistent responses among the sample. The incidence rate ratios ranged from a low of 0.57 of the overall mean (95% confidence interval = (0.51,0.64)) to about 1.46 times the overall mean (95% confidence interval = (1.39,1.52)). Thus rates of inconsistent reporting varied by more the 2.5 times across programs even after controlling for other characteristics of the adolescents.

We fit a separate model that substituted three levels of care indicators (outpatient, short-term residential and long-term residential modalities) for program indicators in order to examine the association between program type and inconsistency rates. The model showed that adolescents in long-term residential care on average reported inconsistently on 1.28 times more substances than those in short-term care, controlling for other client characteristics (95% confidence interval = (1.14,1.45)). Parameters estimates for the other variables in this model were similar to those given in Table 3 for the model which included program indicators.

In our third analysis, we assessed the nature of the relationship between the number of inconsistent reports and treatment outcomes. As shown in Table 4, the number of inconsistently reported substances was negatively associated with the changes in (substance use frequency, substance-related problems, and emotional problems. Each of these outcomes measures change from baseline to 12 month follow-up and since each measures undesirable outcomes negative values indicate a desired improvement in outcomes from baseline to follow-up. Thus, as the number of inconsistently reported substances increased, adolescents on average appeared to have greater improvement on their treatment outcome measures. For substance frequency index, each additional inconsistently reported substance was associated with a −0.18 decrease in the change in substance frequency index between 12-month follow-up and baseline (95% confidence interval = (−0.28,−0.82)). Such a decrease corresponds to a small effect size (standardized coefficient = 0.08) difference for each unit increase in the number of inconsistently reported substance but yields a moderate size effect (standardized coefficient ≥ 0.30) when comparing adolescents with no inconsistencies to those with many (eg. 4 or more). Similar effect sizes were found for substance problem index and emotional problems. Table 4 also shows the mean number of days spent in a controlled environment within the prior 90 days at 12 month follow-up by the number of inconsistently reported substances. There was no statistically significant association found between the number of inconsistencies and the number of days spent in a controlled environment.

Table 4.

Mean Change in Treatment Outcomes from Baseline to 12 Month Follow-up by Number of Inconsistencies in the Reporting of Substance Use Recency.

N Substance Use Frequency Scalea Substance Problem Scalea Illegal Activities Index Time in ControlledEnvironmentb Emotional Problems Indexa
Number of Inconsistencies
 0 324 −1.09 −0.55 0.28 20.03 −0.05
 1 312 −1.35 −0.70 0.30 19.97 −0.05
 2 266 −1.75 −0.76 0.29 24.03 −0.05
 3 204 −1.95 −0.87 0.30 24.63 −0.08
 4+ 357 −2.04 −1.02 0.29 27.54 −0.11
Open in a new tab
a

The estimated slope in the linear regression of the number of inconsistencies the given outcome measure was statistically significant from zero (e.g., p-value < 0.05).

b

Mean time in controlled environment is for 12-month follow-up visit only. It is not the mean change from baseline.

DISCUSSION

The purpose of this paper was to evaluate the potential influence of inaccuracies (whether intentional or inadvertent) in adolescent’s reporting of substance use behavior on measures of treatment outcomes and program performance. Our study found that inconsistent reporting of recent substance use was very common among clients in the ATM sample. Our estimates are likely to understate the true level of inaccuracy in the sample, because our inconsistency measures do not include those who report inaccurately in a consistent manner. Our results suggest that inaccuracies in the reporting of recent substance use are not randomly distributed across individual clients or treatment programs and are associated with other well established treatment outcome measures.

We found that the level of inconsistencies was positively related to a constellation of characteristics associated with greater levels of drug involvement and poor treatment prognosis, including use of hard drugs, early initiation of drugs, high levels of mental distress, and fragile living situation. This finding may suggest that these difficult to treat cases may drive any bias resulting from inaccurate reporting of substance use and related problems.

Some readers may feel that the relationship between greater levels of drug involvement and greater numbers of inconsistent reports is just a de facto result of our use of the number of inconsistently reported substances as our measure of inconsistent reporting. However, this conclusion is predicated on the tacit assumption that adolescents who use more substances have a greater risk for inconsistent reporting because denial of true use is more likely than falsely reporting use when abstinent. This is, however, an assumption that has not been empirically investigated in adolescent treatment populations. The literature offers various potential motivations for denial of use (e.g., stigma or fear of negative consequences from others learning of use) and asymmetric errors with greater risk for denial or use than false report of use. Thus the relationship we find between greater levels of drug involvement and reporting errors might be considered empirical support for a difficult to test hypothesis, rather than a forgone conclusion of our measure. Moreover, even if the greater number of inconsistent reports results from greater drug use, this does not change the fact that differential rates of inconsistent reporting could result in bias when making inferences about treatment programs serving adolescents with different use profiles.

Curiously, we found inconsistencies in the reporting of substance frequency unrelated to criminal involvement at baseline or criminal involvement at 12 month follow-up. This finding was somewhat unexpected given that involvement in criminal activities is commonly associated with a propensity to be untruthful. At the same time, this finding is difficult to interpret. We note that while the large proportion of criminal justice involved patients in the ATM sample may affect parameter estimates and significance levels associated with our criminal justice variables, this is of minimal concern for the generality of our results since the populations to which the results apply will likely involve a large number of adolescent involved in the criminal justice system. Additionally, despite the large proportion of adolescents involved in the criminal justice system, our study has sufficient power to detect small effects of criminal justice on the number of inconsistently reported substances.

We also observed a negative relationship between the number of inconsistently reported substances and change over time in three treatment outcome measures related to substance use frequency, substance-related problems and mental distress. This finding suggests that adolescents who had a greater number of inconsistencies in their reported substance use also tended to report having greater improvement in their treatment outcomes. As the number of inconsistencies increased, the magnitude of this effect became moderately large in terms of effect sizes of a given outcome. The social and reporting context surrounding the self-reporting of recent substance use is similar to that surrounding the reporting of substance use behavior and related problems generally. This similarity combined with our finding of a positive association between inconsistencies and observed improvement on three of five treatment outcome measures gives rise to concerns about the effect of reporting inaccuracies on a wide-range of treatment outcome and program performance measures.

Because treatment program performance must usually be evaluated against either other programs, untreated youths, or fixed benchmarks, our finding that inaccuracy rates varied significantly across treatment programs even after accounting for many other potential explanations of this variation raises special concerns for outcomes-based performance measurement systems like SAMSHA’s State Outcomes Measurement and Monitoring System (SOMMS). Specifically, the finding of program-level variation in response inaccuracy means that any observed differences in program performance may not reflect differences in the effectiveness of the treatments offered at those programs, but might instead reflect differences in the bias introduced into each program’s measured outcomes. Unfortunately, our measure of inaccuracy provides limited insights into the nature of this potential bias. This limitation stems from the inability to determine from inconsistency measures the point in time in which a reporting inaccuracy occurred.

Our findings are also limited by uncertainty surrounding their generality. Although the ATM sample was one of the largest and most complete treatment samples ever assembled, including youths from across the country attending a wide range of community based treatment programs, its representativeness has not been established. Because the sample consisted exclusively of adolescents, it is also not clear whether the level of inconsistency observed in the ATM sample is applicable to adults. For example, studies show that adolescents report substance use less accurately than adults and point to differences in recall, stigma, and motivation to explain these age effects (Johnson & Mott 2001; Patrick et al 1994).

Seen as a whole, our study highlights the need for greater awareness on the part of policy makers and researchers on the potential impact inaccuracies in the reporting of substance use have on outcomes and performance measurement. The integrity of self-reported substance use data could be improved through the development of methodologies to reduce the impact of reporting errors through data reduction methods, such as those proposed by Lennox and others (2006) (Lennox et al 2006). Alternatively, it may be possible to reduce reporting error by designing screening tools to reduce cognitive burden or the social environment in which baseline and follow-up assessments are conducted. One such method currently being explored by Dennis and others (2005) (Dennis et al 2005) to reduce nonrandom reporting error is to provide clients prior to asking about their past substance use information about their previously reported substance use and/or the results of urine test administered in the past week.

Acknowledgments

This work was supported by the National Institute on Drug Abuse, grants 1R49CE000574 (Morral), 1R01DA16722 (Morral), 1R01 DA015697 (McCaffrey).

APPENDIX

Inconsistency was measured across the three assessment periods (baseline, 6- and 12-month follow-up) in the reporting of substance use using GAIN items assessing how recently clients used each of 12 different substances. Recency categories included 0–3 months (collapsed from several more detailed categories), 4–12 months, one or more years ago, and never. Below is a complete list of the inconsistent patterns that were found in the ATM sample where 0 = “Never,” 1 = “More than a year ago,” 2 = “4–6 months ago,” and 3 = “In the past 3 months”. Thus, adolescents who had a reporting pattern of 311 were inconsistent about their substance use between the first and second assessment period (e.g. 31) since they report having used 0 to 3 months prior to baseline while at the 6-months follow-up they state having used the substance of question over a year ago. Similarly, the reporting pattern 3NA0 represents an inconsistent report of substance use since adolescents report using within 0 to 3 months prior to baseline but at 12-months report never having used the substance in question.

Inconsistency patterns found in the ATM sample included 311, 31NA, 331, NA31, 231, 021, 031, 313, 131, 312, 100, 10NA, 110, 1NA0, 200, 20NA, 210, 220, 2NA0, 020, 102, 202, 302, 300, 30NA, 3NA0, NA20, 230, 120, 130, 320, 330, 030, 103, 203, 303, 310, 010, 101, 201, 301, and 121.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Aday L. Designing and conducting health surveys. San Francisco (CA): Jossey-Bass; 1996. [Google Scholar]
  2. Appel PW, Hoffman JH, Blane HT, Frank B, Oldak R, Burke M. Comparison of self-report and hair analysis in detecting cocaine use in a homeless/transient sample. J Psychoactive Drugs. 2001;33:47–55. doi: 10.1080/02791072.2001.10400468. [DOI] [PubMed] [Google Scholar]
  3. Blaney PH. Affect and memory: a review. Psychol Bull. 1986;99:229–46. [PubMed] [Google Scholar]
  4. Colon HM, Robles RR, Sahai H. The validity of drug use responses in a household survey in Puerto Rico: comparison of survey responses of cocaine and heroin use with hair tests. Int J Epidemiol. 2001;30:1042–9. doi: 10.1093/ije/30.5.1042. [DOI] [PubMed] [Google Scholar]
  5. Colon HM, Robles RR, Sahai H. The validity of drug use self-reports among hard core drug users in a household survey in Puerto Rico: comparison of survey responses of cocaine and heroin use with hair tests. Drug Alcohol Depend. 2002;67:269–79. doi: 10.1016/s0376-8716(02)00081-9. [DOI] [PubMed] [Google Scholar]
  6. Dennis M, Titus JC, Diamond G, Donaldson J, Godley SH, et al. The Cannabis Youth Treatment (CYT) experiment: rationale, study design and analysis plans. Addiction. 2002a;97(Suppl 1):16–34. doi: 10.1046/j.1360-0443.97.s01.2.x. [DOI] [PubMed] [Google Scholar]
  7. Dennis ML. Global Appraisal of Individual Needs (GAIN) Manual: Administration, Scoring and Interpretation. Lighthouse Publications; Bloomington, IL: 1999. [Google Scholar]
  8. Dennis ML, Foss MA, Scott CK. College of Problems on Drug Dependence. Orlando, FL: 2005. Improving Self-Reported Recency of Drug Use Utilizing Historical Data and Onsite Urine Testing Feedback. [Google Scholar]
  9. Dennis ML, McDermeit M, Funk R. Preliminary Data Tables and ChartsFor Cross-Site Analysis of the Adolescent Treatment Models Study (ATM/MASC) Chestnut Health Systems. 2002b Unpublished working paper. [Google Scholar]
  10. Dennis ML, Titus JC, Diamond G, Donaldson J, Godley SH, et al. The cannabis youth treatment (CYT) experiment: rationale, study design, and analysis plans. Addiction. doi: 10.1046/j.1360-0443.97.s01.2.x. In press. [DOI] [PubMed] [Google Scholar]
  11. Fendrich M. The undeniable problem of recanting. Addiction. 2005;100:143–4. doi: 10.1111/j.1360-0443.2005.00993.x. [DOI] [PubMed] [Google Scholar]
  12. Fendrich M, Johnson TP. Race/ethnicity differences in the validity of self-reported drug use: results from a household survey. J Urban Health. 2005;82:iii67–81. doi: 10.1093/jurban/jti065. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Fendrich M, Johnson TP, Sudman S, Wislar JS, Spiehler V. Validity of drug use reporting in a high-risk community sample: a comparison of cocaine and heroin survey reports with hair tests. Am J Epidemiol. 1999;149:955–62. doi: 10.1093/oxfordjournals.aje.a009740. [DOI] [PubMed] [Google Scholar]
  14. Fendrich M, Rosenbaum DP. Recanting of substance use reports in a longitudinal prevention study. Drug Alcohol Depend. 2003;70:241–53. doi: 10.1016/s0376-8716(03)00010-3. [DOI] [PubMed] [Google Scholar]
  15. Fendrich M, Vaughn CM. Diminished Lifetime Substance Use Over Time. Public Opinion Quarterly. 1994;58:96–123. [Google Scholar]
  16. Gfroerer J, Wright D, Kopstein A. Prevalence of youth substance use: the impact of methodological differences between two national surveys. Drug Alcohol Depend. 1997;47:19–30. doi: 10.1016/s0376-8716(97)00063-x. [DOI] [PubMed] [Google Scholar]
  17. Godley MD, Kahn JH, Dennis ML, Godley SH, Funk RR. The stability and impact of environmental factors on substance use and problems after adolescent outpatient treatment for cannabis abuse or dependence. Psychol Addict Behav. 2005;19:62–70. doi: 10.1037/0893-164X.19.1.62. [DOI] [PubMed] [Google Scholar]
  18. Johnson TP, Mott JA. The reliability of self-reported age of onset of tobacco, alcohol and illicit drug use. Addiction. 2001;96:1187–98. doi: 10.1046/j.1360-0443.2001.968118711.x. [DOI] [PubMed] [Google Scholar]
  19. Johnston LD, O’Malley PM. The Recanting of Earlier Reported Drug Use by Young Adults. In: Harrison L, Hughes A, editors. The Validity of Self-Reported Drug Use: Improving the Accuracy of Survey Estimates. Rockville, MD: National Institute on Drug Abuse; 1997. pp. 59–80. [Google Scholar]
  20. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62:617–27. doi: 10.1001/archpsyc.62.6.617. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Knight K, Hiller ML, Simpson DD, Broome KM. The validity of self-reported cocaine use in a criminal justice treatment sample. Am J Drug Alcohol Abuse. 1998;24:647–60. doi: 10.3109/00952999809019614. [DOI] [PubMed] [Google Scholar]
  22. Lennox R, Dennis ML, Scott CK, Funk R. Combining psychometric and biometric measures of substance use. Drug Alcohol Depend. 2006;83:95–103. doi: 10.1016/j.drugalcdep.2005.10.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Little RJA, Rubin DB. Statistical Analysis with Missing Data. New York: Wiley; 1987. [Google Scholar]
  24. Mee-Lee D, Shulman G, Fishman M, Gastfriend D, Griffith J. ASAM Patient Placement Criteria for the Treatment of Substance-Related Disorders. 2. Chevy Chase, MD: American Society of Addiction Medicine, Inc; 2001. Revised (ASAM PPC-2R) [Google Scholar]
  25. Morral AR, McCaffrey D, Iguchi MY. Hardcore drug users claim to be occasional users: drug use frequency underreporting. Drug Alcohol Depend. 2000;57:193–202. doi: 10.1016/s0376-8716(99)00048-4. [DOI] [PubMed] [Google Scholar]
  26. Morral AR, McCaffrey DF, Chien S. Measurement of adolescent drug use. J Psychoactive Drugs. 2003;35:301–9. doi: 10.1080/02791072.2003.10400013. [DOI] [PubMed] [Google Scholar]
  27. Morral AR, McCaffrey DF, Gregory Kirk, Ridgeway GK, Mukherji A, Beighley C. Rep. TR-346-CSAT. RAND; Arlington, VA: 2006. The Relative Effectiveness of 10 Adolescent Substance Abuse Treatment Programs in the U.S. [Google Scholar]
  28. Patrick DL, Cheadle A, Thompson DC, Diehr P, Koepsell T, Kinne S. The validity of self-reported smoking: a review and meta-analysis. Am J Public Health. 1994;84:1086–93. doi: 10.2105/ajph.84.7.1086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Percy A, McAlister S, Higgins K, McCrystal P, Thornton M. Response consistency in young adolescents’ drug use self-reports: a recanting rate analysis. Addiction. 2005;100:189–96. doi: 10.1111/j.1360-0443.2004.00943.x. [DOI] [PubMed] [Google Scholar]
  30. Shillington AM, Clapp JD. Self-report stability of adolescent substance use: are there differences for gender, ethnicity and age? Drug Alcohol Depend. 2000;60:19–27. doi: 10.1016/s0376-8716(99)00137-4. [DOI] [PubMed] [Google Scholar]
  31. Stevens SJ, Morral AR. Adolescent Substance Abuse Treatment in the United States: Exemplary Models from a National Evaluation Study. New York: The Haworth Press; 2003. p. 296. [Google Scholar]
  32. Substance Abuse and Mental Health Services Administration. National Outcome Measures. 2006 http://www.nationaloutcomemeasures.samhsa.gov/./outcome/index.asp.
  33. Tassiopoulos K, Bernstein J, Heeren T, Levenson S, Hingson R, Bernstein E. Predictors of disclosure of continued cocaine use. Addict Behav. 2006;31:80–9. doi: 10.1016/j.addbeh.2005.04.005. [DOI] [PubMed] [Google Scholar]
  34. Watkins PC, Vache K, Verney SP, Muller S, Mathews A. Unconscious mood-congruent memory bias in depression. J Abnorm Psychol. 1996;105:34–41. doi: 10.1037//0021-843x.105.1.34. [DOI] [PubMed] [Google Scholar]
  35. Wish ED, Hoffman JA, Nemes S. The validity of self-reports of drug use at treatment admission and at followup: comparisons with urinalysis and hair assays. NIDA Res Monogr. 1997;167:200–26. [PubMed] [Google Scholar]

RESOURCES