The validity of drug use responses in a household survey in Puerto Rico: Comparison of survey responses with urinalysis

Abstract

Aims

The available evidence suggests that the validity of drug use responses in general population surveys is low. We have conducted a household survey to examine viral infections in the general population of Puerto Rico employing a number of procedures believed to increase the validity of drug use responses, as well as confidentiality and privacy: Telling participants of toxicological verification of drug use prior to the interview, ACASI self-interviewing, and interviewing outside households in mobile examination units.

Methods

The study employed a stratified cluster sample of 1654 adults 21 to 64 years old, 532 recruited while urine samples were being collected and 1122 recruited after urinalysis was discontinued due to budgetary reasons.

Results

Drug use rates calculated from participants recruited while urinalysis was being conducted did not vary significantly to those derived from participants recruited after urinalysis was discontinued. Sensitivity of responses of drug use during the last three days was 80.0% for marihuana, 76.2% for cocaine, and 40.0% for heroin. The lower validity of heroin reports did not seem to be the result of underreporting as it was reported by more individuals than the test detected.

Conclusion

We conjecture that the reasonably good validity of the drug use responses might have been the result of the parent study being about a health issue other than drug use, and that interviewing was conducted outside households in mobile units. These findings buttress the value of conducting methodological trials to identify procedures which yield valid responses of drug use.

1. Introduction

During the last 20 years a growing number of countries have begun to monitor substance use and its consequences in the general population. This trend is not only evident in the US and in developed countries (Kokkevi, Fotiou, & Richardson, 2007; Farrel et al., 2003; Ramsay & Percy, 1997; Kraus, Augustin, Kunz-Ebrecht, & Orth, 2007), but also in countries in the developing world, such as in Latin America (Galduroz, Noto, Nappo, & Carlini, 2005; Cherpitel et al., 2007; Vicente et al., 2006). The development of standardized diagnostic instruments (Robins et al., 1985; Robins et al., 1988) may have also contributed to the increase in community studies as diagnostic criteria of substance use disorders can now be measured in general populations and the results compared across countries and cultures (Rubio-Stipec, Bravo, & Canino, 1991).

Monitoring substance use and substance use disorders in general populations faces a number of measurement and methodological challenges. Principally among these, current measures rely almost exclusively on respondents' own admissions of substance use. With the advent of toxicological tests, a body of evidence has grown about the extent to which substance use responses provided in surveys agree with test results (Magura & Kang, 1996; Brener, Billy, & Grady, 2003). The published studies tend to converge on substantial underreporting (Chen, Fang, Shyu, & Lin, 2006; Chermack et al., 2000; Vitale, van de Mheen, van de Wiel, & Garretsen, 2006; Tassiopoulos et al., 2006), strong effects of varying interview modes (Aquilino, 1992), and substantial variability depending on the population being studied and the study setting (Magura & Kang, 1996; Solbergsdottir, Bjornsson, Gudmundsson, Tyrfingsson, & Kristinsson, 2004; Downey, Helmus, & Schuster, 2000; Kedzior, Badcock, & Martin-Iverson, 2006; Abnet et al., 2004; Kim & Hill, 2003; Yacoubian & Urbach, 2002; Lu, Taylor, & Riley, 2001). However, the majority of data to date has been derived from specialized sub-populations such as visits to emergency rooms, arrestees and substance users entering treatment. With some notable exceptions, epidemiological surveys of the general population have not employed biological testing.

Among the first household studies to directly compare substance use responses with toxicological tests, two separate studies conducted in Chicago, Illinois (Fendrich, Johnson, Sudman, Wislar, & Spiehler, 1999) and in Puerto Rico (Colón, Robles, & Sahai, 2001) during 1997 found considerable underreporting. The sensitivity of substance use responses reported by the Chicago study was 18.0% for cocaine use and 30.8% for heroin use. We also found considerable underreporting in the Puerto Rico study with a sensitivity of cocaine use reports of 7.1%, and a sensitivity of heroin reports of 33.3%. Fendrich and colleagues conducted another household survey during 2001 in Chicago (Fendrich & Johnson, 2005) utilizing audio computer assisted self-interviews – ACASI, a method which has been shown to increase substance use reports (Newman et al., 2002). The second Chicago study employed both saliva and urine tests and examined the validity of cocaine and marihuana reports. The sensitivity of cocaine use reports was low (22.9%), but that of the marihuana reports was found to be substantially higher (69.7%). Thus, the available evidence, albeit scarce, suggests that the validity of drug use reports in general population surveys varies by type of substance and is generally low, with substantial underreporting of substance use. Several reports have called attention to the dangers of basing drug policies on data of uncertain quality (Council, 2001).

Biological measures can be useful to examine the validity of respondents' reports of substance use but, given the available technology, cannot be considered adequate measures with which to replace respondents' reports. Biological measures can only detect recent use and cannot detect quantity or frequency of use (Wolff et al., 1999). Furthermore, testing methods vary with respect to their time periods of detection (Dolan, Rouen, & Kimber, 2004), and they might also vary in the capacity of detecting different substances (Fendrich, Johnson, Wislar, & Hubbell, 2004). Given the high public health importance of monitoring substance use trends in general populations, there is a continuing need to find ways to improve the validity of respondents' own reports of their substance use.

In a household survey designed to examine the epidemiology of viral hepatitis (A, B, and C), HIV and herpes simplex type 2 in the general population of Puerto Rico, we employed a number of procedures believed to increase the validity of drug use reports and to increase confidentiality and privacy. We combined telling potential study participants of toxicological verification of drug use prior to the interview, ACASI self-interviewing, and interviewing in a mobile examination unit as opposed to the traditional method of interviewing in participants' households. Due to funding constraints, toxicological testing had to be discontinued after one-third of the study sample had been recruited. This unexpected event provided the opportunity of examining the effects on reports of drug use of participants knowing their responses would be examined against toxicological tests. This report presents the results of examining the validity of respondents' substance use reports when compared to the results of urinalysis tests. Specifically, the study was aimed at answering three questions: 1) Did toxicological testing have an effect on reports of drug use?; 2) Did use rates vary by measurement mode (i.e., reported versus urinalysis)?; and, 3) What was the specificity and sensitivity of drug use reporting when assessed against urinalysis?

2. Methods

2.1. Sample Design

The parent study consisted of a cross-sectional survey of the non-institutionalized population of Puerto Rico aged 21–64 years old. The island territory was stratified on the basis of population density and AIDS incidence rates attributed to injection drug users. Four strata were derived corresponding to municipalities with above and below median rates of population density and AIDS incidence rates.

Household segments (consecutive households within a census block) within each stratum were selected in three stages. In a first stage, census blocks were sorted by median age and median housing value and a systematic random selection was made with probability proportional to the number of households. The second selection stage consisted of a random selection of one block within each census block group. In a third stage, one segment of approximately 20–30 consecutive households was randomly selected from each block. A total of 108 segments were selected comprising 3487 occupied households.

2.2. Recruitment, Collection of Biological Specimens, Interviewing, and Study Sample

Enumerators were able to contact residents in 3386 households (97.1%), and 2123 of these households had at least one adult aged 21 to 64 years old. One eligible adult was randomly selected from each household and invited to participate. Consenting participants were given an appointment to visit the mobile examination unit located in the vicinity of their homes. Of the selected residents, 1654 (77.9%) consented to participate.

At the mobile examination unit, a formal informed consent procedure was conducted which provided participants with information of all study procedures, including toxicology tests to detect drug use. Upon formally consenting, participants were offered pretest counseling for risk reduction of hepatitis C, hepatitis B, HIV, and herpes simplex type 2. Participants completed a face-to-face interview to gather socio-demographic characteristics and other topics of interest to the parent study. Then, participants were asked to complete a self-administered questionnaire using an ACASI system implemented using QDS (Nova Research Co., Washington D.C.). After interviewing, participants were counseled about the tests to detect viral infections and about the meaning of the test results and asked to provide 15-ml of venous blood drawn by certified phlebotomists. Finally, each participant was asked to provide a urine sample. Participants received a $50 compensation for their time effort. All study procedures were reviewed and approved by the Institutional Review Board of the University of Puerto Rico Medical Sciences Campus. Field work was conducted between June 2005 and February 2008.

During field work, and after one-third of the study sample had been recruited, funding for the parent project was reduced and the scope of the project had to be proportionately reduced. As a result, collection of urine samples and urinalyses were discontinued. While urine samples were being collected, 532 participants were recruited. Of these, 523 provided suitable urine samples (98.3%); four subjects refused to provide urines, four provided inadequate amounts (< 10 ml), and one subject was handicapped and was not asked to provide a urine sample.

2.3. Measurements

The face-to face interview covered socio-demographic and socio-economic characteristics. The ACASI system ascertained use of marihuana, hashish, THC, powdered cocaine, crack cocaine, heroin, heroin and cocaine mixed together, other opiates, sedatives/tranquilizers, amphetamines/stimulants, hallucinogens, inhalants, and steroids. Participants were asked about their use of each drug during the last 3 days (to correspond to the urinalyses' period of detection), last year, and at some point in their life. The ACASI interview lasted on average 34 minutes. For each recall period, a positive response to marihuana, hashish, or THC use was defined as a report of marihuana use. Cocaine use was defined as a positive response to powdered cocaine, crack cocaine, or heroin and cocaine mixed together. Heroin use was defined as a positive response to heroin, heroin and cocaine mixed together, or other opiates.

Urinalyses were conducted using cannabinoid, cocaine metabolite (benzoylecgonine), and opiate enzyme immunoassays (Lin-Zhi International, Inc., Sunnyvale, CA). Samples exceeding the assay cutoff values (cannabinoid - 50 ng/mL; benzoylecgonine - 300 ng/mL; opiate - 300 ng/mL) were confirmed by gas chromatography/mass spectometry.

2.4. Statistical Analyses

Study subjects with and without toxicological tests were compared in their socio-demographic characteristics and in their rates of reported drug use. Bivariate comparisons were evaluated using chi-square distribution for tests of independence. Multivariate logistic regression models (one for each drug type and recall period) were fitted to compare the rates of reported drug use after controlling for socio-demographic characteristics. Paired differences between the proportion of subjects indicating use of each drug on the ACASI system and the proportion testing positive on the toxicological tests were evaluated using the McNemar exact test for paired proportions. Finally, the validity of drug use reports was evaluated against the urinalysis results by calculating specificity, sensitivity, and positive and negative predictive values. Specificity and positive and negative predictive values were calculated only for reports of recent use (last 3 days) to correspond with the urinalysis' period of detection which, according to the manufacturer of the bioassays, were 1–10 days for marihuana, 2–4 for cocaine, and 2–3 for heroin.

3. Results

Table 1 compares the socio-demographic characteristics of the study participants recruited during the period when urinalysis was being conducted with those of the participants recruited after urinalysis was discontinued. Both groups had similar gender, age, and marital status distributions (p = .464, p = .911, and p = .526, respectively). Study participants recruited during the period when urinalysis was being conducted had a significantly higher educational level than those recruited after urinalysis was discontinued (more than high school: 39.1% vs. 31.8%, p = .001). The differences between the two groups in employment status and place of residence were borderline significant. Among participants with toxicology, 65.6% were employed, and among participants without toxicology, 60.7% were employed (p = .054). The proportion of participants residing in a metropolitan area was 90.8% among participants with toxicology and 93.5% among participants without toxicology (p = .050).

Table 1.

Socio-demographic characteristics of study participants recruited during the period when urinalysis was being conducted compared to those recruited after urinalysis was discontinued.

	Recruitment Period
	While Urinalysis Was Conducted (n = 532) %	After Urinalysis Was Discontinued (n = 1122) %	p-value
Gender
Male	42.3	44.2
Female	57.7	55.8	.464
Age in years
21–29	21.6	22.5
30–49	50.6	50.5	.911
50–64	27.8	27.0
Education
< High school	19.7	27.8
High school	41.2	40.4	.001
> High school	39.1	31.8
Marital statusa
Unmarried	40.6	42.3
Married	59.4	57.7	.526
Employment status
Not employed	34.4	39.3
Employed	65.6	60.7	.054
Place of residenceb
Non-metropolitan area	9.2	6.5
Metropolitan area	90.8	93.5	.050

^aUnmarried includes individuals who were single, divorced, separated or widowed, and who were not living with a partner. Married includes individuals who were married or living with a partner.

^bAccording to the US Census 2000 definition of `statistical metropolitan area.'

Table 2 compares reports of drug use of study participants recruited during the period when urinalysis was being conducted with participants recruited after urinalysis was discontinued. None of the comparisons reached statistical significance (all p-values were > .200), and neither was there a discernible pattern of one group consistently reporting higher rates than the other group. Prevalence differences were of modest magnitude varying from 0.2% (heroin use last 3 days: 1.5% vs. 1.3%) to 1.4% (heroin use ever: from 5.6% to 7.0%). Nine multivariate logistic regression models (one for each drug type and recall period) were fitted to compare the rates of reported drug use after controlling for the socio-demographic characteristics shown on Table 1. In none of the models did reported rates of drug use vary significantly (p > .05) between study participants recruited during the period when urinalysis was being conducted and participants recruited after urinalysis was discontinued (analyses not shown).

Table 2.

Reports of drug use of study participants recruited during the period when urinalysis was being conducted compared to those recruited after urinalysis was discontinued.

	Recruitment Period
	While Urinalysis Was Conducted (n = 532) %	After Urinalysis Was Discontinued (n = 1122) %	p-value
Marihuanaa
Last 3 days	4.1	5.4	.289
Last year	12.0	12.6	.757
Ever	29.7	29.1	.788
Cocaineb
Last 3 days	3.4	2.5	.305
Last year	7.5	6.9	.627
Ever	14.9	14.9	.985
Heroinc
Last 3 days	1.5	1.3	.671
Last year	3.6	3.0	.559
Ever	5.6	7.0	.313

^aIncludes reports of use of marihuana, hashish, and THC.

^bIncludes reports of use of `speedballs' (cocaine and heroin mixed together), snorted cocaine, and smoked crack cocaine.

^cIncludes reports of use of `speedballs' (cocaine and heroin mixed together), heroin, and other opiates.

All subsequent analyses excluded those study participants who were recruited after urinalysis had been discontinued. Among those who did provide urine samples, reported rates of drug use and rates derived from urinalysis were calculated and are shown on Table 3. The report rates of use during the 3 days prior to the interview were compared to those detected with urinalysis. Reports of use during more remote periods (i.e., last year, ever) were not compared to the urinalysis results because of lack of correspondence of the reporting period with the detection period of urinalysis. Rates of reported drug use during the 3 days prior to the interview did not differ significantly from the rates derived with the urinalysis results. Marihuana use (last 3 days) was reported by 3.8% of participants, and urinalysis detected marihuana use among 3.8% of the participants (p > .999). The rate of reported cocaine use (last 3 days) was 3.4% and urinalysis detected 4.0% (p = .453). The heroin use rate calculated from reports was 1.5% and that calculated from urinalysis was 1.0% (p = .508).

Table 3.

Rates of drug use derived from participants' reports and from urinalysis (n = 523).

Substance Type	Reported		Detected with Urinalysis
	n	%	n	%	p-valuea
Marihuanab
Last 3 days	20	3.8	20	3.8	>.999
Last year	62	11.9
Ever	155	29.6
Cocainec
Last 3 days	18	3.4	21	4.0	.453
Last year	39	7.5
Ever	77	14.7
Heroind
Last 3 days	8	1.5	5	1.0	.508
Last year	19	3.6
Ever	30	5.7

^aP value of McNemar's exact test for paired proportions, calculated only with reports of last 3 days to correspond with urinalysis' period of detection.

^bIncludes reports of use of marihuana, hashish, and THC.

^cIncludes reports of use of `speedballs' (cocaine and heroin mixed together), snorted cocaine, and smoked crack cocaine.

^dIncludes reports of use of `speedballs' (cocaine and heroin mixed together), heroin, and other opiates.

Table 4 shows the estimates of specificity, sensitivity, and positive and negative predictive values of drug use reports using urinalysis as the `gold standard.' Specificity and positive and negative predictive values were calculated only for the most recent reports (last 3 days) to correspond with the urinalysis' period of detection. Specificity estimates were high for the reports of the three drugs (99.2% for marihuana, 99.6% for cocaine, and 98.8% for heroin). Sensitivity estimates varied by type of drug and by reporting period. Among drug types, sensitivity was highest for reports of marihuana use during the last 3 days (80.0%), somewhat lower for cocaine reports (76.2%), and lowest for heroin reports (40.0%). Across the three drug types, sensitivity estimates increased with the lengthening of the report periods (i.e., last year, ever). The sensitivity of marihuana reports increased from 80.0% for reports of use during the last 3 days to 95.0% for reports of use during the last year and for reports of lifetime use. Cocaine reports increased in sensitivity from 76.2% for reports of use during the last 3 days to 90.5% for reports of use during the last year and to 95.2% for reports of lifetime use. Sensitivity of heroin use reports increased from 40.0% for reports of use during the last 3 days to 60.0% for reports of use during the last year and for reports of lifetime use. Positive predictive values were 80.0% for reports of marihuana use during the last 3 days, 88.9% for reports of cocaine use, and dropped to 25.0% for reports of heroin use. Negative predictive values were 99% or higher for reports of all three drugs.

Table 4.

Validity estimates of drug use reports.

	Urinalysis Results
Reported	Positive	Negative	Specificitya	Sensitivity	Positive Predictive Value	Negative Predictive Value
Marihuanab
Last 3 days
Yes	16	4	99.2	80.0	80.0	99.2
No	4	499
Last year
Yes	19	43		95.0
No	1	460
Ever
Yes	19	136		95.0
No	1	367
Cocainec
Last 3 days
Yes	16	2	99.6	76.2	88.9	99.0
No	5	500
Last year
Yes	19	20		90.5
No	2	482
Ever
Yes	20	57		95.2
No	1	445
Heroind
Last 3 days
Yes	2	6	98.8	40.0	25.0	99.4
No	3	512
Last year
Yes	3	16		60.0
No	2	502
Ever
Yes	3	27		60.0
No	2	491

^aSpecificity, positive and negative predictive values not calculated for last year and ever use because they exceeded the tests' period of detection.

^bIncludes reports of use of marihuana, hashish, and THC.

^cIncludes reports of use of `speedballs' (cocaine and heroin mixed together), snorted cocaine, and smoked crack cocaine.

^dIncludes reports of use of `speedballs' (cocaine and heroin mixed together), heroin, and other opiates.

The urinalysis' 3-day period of detection meant that some respondents ended up being asked about the drug consumption behavior that had occurred during weekends while other respondents were asked about their behaviors over workdays. Drug consumption behaviors among recreational drug users might vary substantially from weekends to workdays. We examined the data for evidence that this variation across days of the week might have affected our validity estimates. Study subjects were grouped into two groups according to the weekday in which they were interviewed (i.e., Saturdays through Tuesdays versus Wednesdays through Fridays). Sensitivity rates were computed for each group and compared. No significant differences were found, nor was there a pattern of consistently higher or lower sensitivity rates in one group or the other (data not shown).

4. Discussion

Asking participants to provide urine samples for toxicological tests does not seem to have had an effect on reports of drug use. Participants who were told of toxicological verification prior to interviewing were equally likely to report drug use than participants who were not told of toxicological verification because toxicology tests had been discontinued. Rates of drug use reports were statistically equivalent across the three drugs examined and the three recall periods. Given that the two groups differed in other characteristics, the comparison of drug use reports were multivariately adjusted and still failed to show evidence of a significant difference. These findings suggest that introducing biological verification of drug use does not modify noticeably participants' self-reports. If so, the validity estimates of studies employing toxicological tests can be considered applicable to the great majority of other household studies which do not regularly collect biological specimens. To our knowledge, this question has not been analytically pursued before.

Compared against the results of urinalysis, participant reports of marihuana and cocaine use were found to be reasonably valid. The rates of recent drug use (last 3 days) calculated from participants reports were not significantly different to the rates calculated from the results of urinalysis. Moreover, the sensitivity of reports of recent marihuana and cocaine use were higher than 75%, increased to 90% or more for reports of use during more remote periods (i.e., last year, lifetime), and positive predictive values were 80% or more.

These results differ notably from previous studies of the validity of drug reports in household surveys. This study differed from prior studies mainly in that it was focused on a health issue – viral infections – and not on drug use, and in that interviewing was not conducted inside households but in a mobile examination unit. Individuals told that they have been selected to participate in a study examining drug use behaviors might feel some apprehension that they have been somehow selected because drug use is suspected. Thus, our study focus might have helped to ease apprehensions about the selection criteria. There is some evidence indicating that drug use self-reports obtained from men who have sex with men in community studies focused on HIV transmission are generally valid (Fendrich, Mackesy-Amiti, & Johnson, 2008).

Interviewing outside households might have also provided an additional incentive to honest reporting by offering participants greater privacy and confidence in the confidentiality of their responses. The context of the mobile unit might have also reinforced the impression of a study focused on health and not drug use. We have failed to identify studies examining the effects of interviewing outside households. However, lack of privacy and bystanders' presence are ubiquitous in household surveys and have been found to affect responses (Aquilino, 1997; Aquilino, Wright, & Supple, 2000).

Heroin use reports were found to be of substantially lower validity than the validity of marihuana and cocaine reports. The lower estimate of validity of heroin use reports does not seem to have been caused by underreporting of heroin use. Among the three drugs examined, heroin was the only drug where the last 3 day rate of reported use was higher than the rate derived from urinalysis (1.5% vs. 1.0%, respectively). Moreover, heroin use reports had a lower positive predictive value than reports of use of either marihuana or cocaine (25% versus ≥80% for both marihuana and cocaine). In fact, heroin was the only drug examined which was reported by more individuals than the test detected (i.e., reported by eight individuals but only detected by toxicology in two cases). Heroin overreporting in this study might have been the result of an emerging epidemic of Xylazine injection in Puerto Rico. Xylazine is a veterinarian analgesic that is being increasingly used to replace heroin in drug copping areas in Puerto Rico. A recent study (Rodríguez et al., 2008) analyzed 370 used syringes exchanged in syringe exchange programs in Puerto Rico and found presence of Xylazine in 37.6% of the syringes. More importantly, the authors reported that Xylazine was detected in 22.0% of the syringes returned by 59 individuals claiming to have injected heroin and not Xylazine. Thus, there might be a number of drug users who believed they had used heroin but instead had used Xylazine.

The findings of this study need to be interpreted with caution due to some of its limitations. The study was not designed to formally examine the factors contributing to valid drug use reports. Participants were not randomly assigned to presence or absence of toxicology. Moreover, toxicology tests had to be discontinued and the sample tested for toxicology, and from which the validity estimates were derived, cannot be considered a representative sample of the Puerto Rico household population. Given that the subjects not asked to undergo toxicology tests were surveyed later in the process, other “historical” factors may have contributed to the findings. In the absence of a true experiment, the validity of the results of this study is somewhat limited. Furthermore, this study used toxicology test as a gold standard against which to gauge the validity of self-reports. Yet toxicology tests have limitations of their own (Dolan, Rouen, & Kimber, 2004; Fendrich, Johnson, Wislar, & Hubbell, 2004). It might prove fruitful in the future to examine ways in which different sources of drug consumption information can be combined to render valid data.

Albeit with limitations, we believe this study contributes to the small body of studies which have examined the validity of drug use reports in general populations. This study found drug use reports of marihuana and cocaine to be of reasonable validity. Given that the validity literature shows that variations in study design, interview modes, and study settings have large effects, we have conjectured that the validity of the drug use reports might have been affected by the fact that the parent study was about a health issue (viral infections) other than drug use, and that interviewing was conducted outside households in mobile examination units. Monitoring substance use and abuse in general populations is of growing public health importance. We believe the findings of this study buttress the need to conduct methodological trials testing the effects of varying survey components to identify procedures which yield valid responses that can be compared across countries and cultures.