Discrepancies in Estimates of Prevalence and Correlates of Substance Use and Disorders Between Two National Surveys

Abstract

Aim:

To assess the degree to which methodological differences might influence estimates of prevalence and correlates of substance use and disorders by comparing results from two recent surveys administered to nationally representative U.S. samples.

Methods:

Post-hoc comparison of data from the 2002 National Survey on Drug Use and Health (NSDUH) to data from the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) administered in 2001-02.

Results:

Prevalence estimates for all substance use outcomes were higher in the NSDUH than in the NESARC; ratios of NSDUH to NESARC prevalences ranged from 2.1 to 5.7 for illegal drug use outcomes. In the NSDUH, past-year substance use disorder (SUD) prevalence estimates were higher for cocaine and heroin, but were similar to NESARC estimates for alcohol, marijuana, and hallucinogens. However, prevalence estimates for past-year SUD conditional on past-year use were substantially lower in the NSDUH for marijuana, hallucinogens, and cocaine. Associations among drug and SUD outcomes were substantially higher in the NESARC. Total SUD prevalence did not differ between surveys, but estimates for Blacks and Hispanics were higher in the NSDUH.

Conclusion:

There are a number of methodological variables that might have contributed to such discrepancies; among plausible candidates are factors related to privacy and anonymity, which may have resulted in higher use estimates in the NSDUH, and differences in SUD diagnostic instrumentation, which may have resulted in higher SUD prevalence among past-year substance users in the NESARC.

1. Introduction

The National Survey on Drug use and Health (NSDUH) is among the primary sources of information about prevalence, correlates, and trends in substance use and abuse in the United States (1). This annual survey collects data from a nationally representative sample on overall use patterns, correlates, and problems resulting from use, including DSM-IV symptoms of abuse and dependence. The recently completed National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) is another key source of information on substance abuse in the U.S. Drawing from the tradition of psychiatric epidemiology, the NESARC utilized an in-depth diagnostic interview and focused on diagnoses of psychiatric and substance use disorders as primary outcomes (2). Both the 2002 NSDUH and the first wave of the NESARC, conducted in 2001-02, generated prevalence estimates for lifetime and past-year substance use, and past-year substance use disorders (SUDs). The aim of this paper is to compare results across these widely used and widely cited national surveys. No such comparison has been published to date.

Differences in survey methodology can influence prevalence estimates (3-6), and so it might seem unreasonable to expect precise agreement between surveys and difficult to draw conclusions about areas of disagreement. Yet both of these surveys employed state-of-the-art methods and studied nationally representative samples of non-institutionalized civilians. While precise agreement may not be expected, areas of substantial disagreement should be brought to attention, particularly when they have substantive implications. Furthermore, few studies have examined the degree to which risk factor estimates and apparent comorbidity might vary as a result of divergent survey methods. Precise estimates of such parameters are crucial for prevention efforts and service planning, but very large samples would be required for such comparisons in the context of methodological experiments. The availability of these two large surveys of nationally representative samples presents an opportunity to address these issues.

Our objectives for the current study were to conduct cross-survey comparisons of the following: 1.) Estimates of prevalence for lifetime alcohol and drug use, 12-month alcohol and drug use, and 12-month alcohol and drug use disorder; 2.) estimates of association between drug and/or alcohol outcomes; and, 3.) estimates of association between alcohol or drug outcomes and demographic factors. We aimed to identify areas where substantial differences in estimates existed between the surveys and to generate hypotheses about potential sources for such discrepancies, recognizing that only controlled field experiments can unambiguously identify variables that contribute to differences in estimates.

Any number of methodological differences could influence prevalence estimates for substance use and/or SUD; some of the more salient differences between the NESARC and the NSDUH include privacy/anonymity considerations, response rate, and diagnostic instrumentation. It is generally accepted that under-reporting can be a serious problem for socially proscribed or illegal behaviors such as drug and alcohol abuse, and that underreporting can be alleviated by greater privacy and anonymity in the interview setting (7-13). This consideration might be expected to yield higher prevalence estimates in the NSDUH, which utilized computerized self-administration methods (ACASI) to collect sensitive data. Moreover, data collection was anonymous for the NSDUH; that is, participants' names were not recorded or linked with their answers. Related to these points, the NSDUH is conducted by a private firm (RTI, international) whereas the NESARC was conducted by the U.S. Census Bureau; this factor may also influence respondents' perception of confidentiality. The NESARC had a slightly higher net response rate than the NSDUH (81% vs. 75%), a factor that would generally be expected to increase substance use estimates, as difficult-to-recruit respondents have been shown to have higher rates of SUD (14). Moreover, differences in diagnostic instrumentation could influence substance use disorder prevalence estimates.

2. Methods

2.1 Overview of NSDUH and NESARC

The target population for both surveys was the non-institutionalized, civilian population of the United States. The NESARC sampled adults (ages 18 and over); the NSDUH interviewed individuals aged 12 and over. The sampling universes for both surveys included group quarters, i.e., facilities such as college dormitories, group homes, and other residences. Thorough descriptions of sampling schemes and other survey procedures are available elsewhere (2, 15, 16); brief summaries are provided below. For both surveys, ethical review and approval of all procedures was conducted by appropriate agencies and informed consent was obtained from all adult participants (ages 18 and up).

2.2 Sampling Schemes

Both surveys utilized multistage sampling designs with sociodemographic stratification. Post-stratification weights were applied to ensure that the samples were representative of the U.S. population. While sampling differed in a number of ways, weighting would be expected to compensate for any sampling differences.

For the NESARC, the first stage of sampling involved selection of primary sampling units (PSUs) approximating the U.S. Census Current Population Survey county-based PSUs. The second stage of sampling consisted of dwelling unit selection. Oversampling of Black and Hispanic residences was accomplished by utilizing Census Supplementary Survey data. The final stage of sampling consisted of selecting one person per housing unit (2). For the NSDUH, PSUs comprised approximately 500,000 area segments (groups of adjacent census blocks). The first stage of sampling involved selection of 8 such segments from each of 900 geographic “field interviewer” (FI) regions. The frames for the second stage of sampling consisted of lists of all dwelling units within segment boundaries. Samples of dwelling units were selected from these lists. Individuals were selected from rosters obtained by dwelling unit visits. Adolescents and young adults were oversampled, with one-third of the sample in each of three age groups: 12-18, 19-25, and 26+. Sampling rates also varied by state, as target sample sizes were fixed at 3600 in the 8 large states and 900 in the remaining states and D.C. (15).

2.3 Measures

Both surveys queried alcohol and drug use on a lifetime basis. The surveys employed different response options for recency of use, but both allowed for determination of past-year use. The NSDUH interview includes separate sections for each substance, and probes past-year DSM-IV diagnostic criteria. The primary assessment included in the NESARC was the Alcohol Use Disorder and Associated Disabilities Interview Schedule-DSM-IV version (AUDADIS-IV; 17, 18), which covers DSM-IV substance use disorders for past 12-month and life time frames. In contrast to the NSDUH, the NESARC asked about all drugs (except alcohol) in the same module, using a read list of substances to aid in the response of each question.

2.4 Sample Preparation and Data Analysis

The NSDUH sample originally consisted of 68,126 subjects. Prior to public release, about 21% of the sample was randomly removed to protect anonymity, leaving N=54,079. Prior to analysis, subjects aged 12-18 (n=17,709) were removed from the NSDUH sample to make it comparable to the NESARC adult sample; the final sample sizes were 36,370 for the NSDUH and 43,093 for the NESARC. All statistical analyses were conducted with weighted data using the SUDAAN statistical software package (Research Triangle Institute, 2004). Variance estimation utilized a Taylor linearization method appropriate for the multistage design of the surveys. Prevalence estimates for each survey were calculated for past-year and lifetime use of alcohol, marijuana, hallucinogens, cocaine, and heroin, and for past-year SUD diagnoses for these substances. SUD diagnoses combine abuse and dependence; i.e., a positive diagnosis of either results in a positive SUD diagnosis. To compare estimates across surveys, ratios of NSDUH to NESARC prevalences were calculated.

Bivariate logistic regression was used to compute odds-ratios between pairs of outcomes and between demographic variables and outcomes. For calculation of odds-ratios describing association between outcomes, an “other illegal drug” category was created for drugs other than marijuana to avoid high uncertainties associated with low prevalence estimates. Likewise, all drug use outcomes and all SUD outcomes were collapsed in the demographic correlate analyses. Between-sample comparisons were conducted using two-sample Z tests.

Results

3.1 Sociodemographics

After weighting, demographic composition of both samples reflected the target population in terms of gender, age, race/ethnicity, and population density. By design, each survey has variable sampling rates across population groups, hence the unweighted sample compositions differ considerably. The NSDUH sample was 11.6% Black, 11.9% Hispanic and 6.1% other non-White race/ethnicity. The NESARC sample was 19.1% Black, 19.3% Hispanic and 4.7% other race/ethnicity. In the NSDUH, 55.3% of participants were under age 29, compared to 20.1% in the NESARC.

3.2 Substance Use Prevalences

Comparisons of lifetime and past-year substance use prevalences across the two surveys are reported in the top two sections of Table 1. For all substance use outcomes, prevalences were significantly higher in the NSDUH. For illegal drug use, NSDUH : NESARC prevalence ratios ranged from 2.1 for lifetime marijuana use, to 5.4, for past-year heroin use. For 12-month and lifetime alcohol use, between-survey differences were significant, though smaller than those for drug use. Differences in past-year use might result from the fact that lifetime non-users were not queried about past-year use. Thus, prevalences of 12-month substance use conditional on lifetime use were compared; there were no significant differences for alcohol, hallucinogens, or heroin. However, the conditional prevalence of past-year marijuana use and past-year cocaine use were both significantly higher in the NSDUH (24.4% vs. 19.9% for marijuana; ratio=1.2; 95% CI: 1.2-1.3, p<0.001; and 15.3% vs. 9.2% for cocaine, ratio=1.7, 95% CI: 1.4-2.0, p<0.001).

Table 1.

Prevalence Estimates for Substance Use and SUD.

	NSDUH (N=36,370)		NESARC (N=43,093)		Comparison
Outcome	%	(95% CI)	%	(95% CI)	Ratio‡	(95% CI)	p
Lifetime Use
Alcohol	87.7	(87.0 - 88.3)	82.7	(81.5 - 83.9)	1.1	(1.0 - 1.1)	<0.001
Marijuana	42.8	(41.9 - 43.7)	20.5	(19.5 - 21.5)	2.1	(2.0 - 2.2)	<0.001
Hallucinogens	15.7	(15.1 - 16.3)	5.8	(5.4 - 6.2)	2.7	(2.5 - 2.9)	<0.001
Cocaine	16.0	(15.4 - 16.6)	6.1	(5.7 - 6.6)	2.6	(2.4 - 2.8)	<0.001
Heroin	1.7	(1.5 - 2.0)	0.33	(0.33 - 0.39)	5.2	(3.6 - 8.5)	<0.001
Past-Year Use
Alcohol	69.6	(68.8 - 70.5)	65.4	(64.3 - 66.6)	1.1	(1.0 - 1.1)	<0.001
Marijuana	10.4	(10.0 - 10.9)	4.1	(3.8 - 4.4)	2.6	(2.3 - 2.8)	<0.001
Hallucinogens	1.7	(1.6 - 1.8)	0.6	(0.5 - 0.7)	3.0	(2.5 - 3.6)	<0.001
Cocaine	2.4	(2.2 - 2.7)	0.6	(0.5 - 0.7)	4.3	(3.4 - 5.7)	<0.001
Heroin	0.17	(0.1 - 0.2)	0.03	(0.01 - 0.05)	5.4	(2.1 - Inf.)	<0.001
Past-Year Disorder
Alcohol	7.9	(7.5 - 8.3)	8.5	(8.0 - 8.9)	0.9	(0.9 - 1.0)	0.089
Marijuana	1.6	(1.4 - 1.7)	1.5	(1.3 - 1.6)	1.1	(0.9 - 1.2)	0.322
Hallucinogens	0.14	(0.10 - 0.18)	0.14	(0.10 - 0.19)	1.0	(0.7 - 1.6)	1.000
Cocaine	0.6	(0.5 - 0.8)	0.27	(0.21 - 0.33)	2.4	(1.7 - 3.5)	<0.001
Heroin	0.11	(0.07 - 0.16)	0.02	(0.00 - 0.05)	4.6	(1.5 - Inf.)	0.002

^‡Ratio of NSDUH prevalence to NESARC prevalence.

Bold indicates prevalences that are significantly (p≤0.002) higher than prevalence from compared survey.

3.3 SUD Prevalences

Past-year SUD prevalences are compared in the bottom section of Table 1. No significant differences were seen for alcohol, marijuana, or hallucinogens, but large differences were observed for cocaine and heroin; NSDUH estimates are 2.4 and 4.6 times higher than NESARC estimates, respectively. For past-year SUD prevalence, conditional on past-year use, NSDUH estimates were significantly lower than NESARC estimates for alcohol, marijuana, hallucinogens, and cocaine (all p <0.001, ratio=0.9, 0.4, 0.3 and 0.5, respectively).

3.4 Association Between Outcomes

Odds ratios between pairs of past-year substance use outcomes, pairs of lifetime substance use outcomes, and pairs of SUDs are listed in Table 2 for both surveys; in addition, the ratios of NESARC : NSDUH odds ratios (the interaction odds ratio) are listed. No significant between-survey differences were observed for associations of alcohol use with illegal drug use for either time frame. For associations between use of marijuana and other illegal drugs, ORs were 2.3 and 3.0 times higher in the NESARC for lifetime and past-year outcomes, respectively. Odds-ratios describing covariation between pairs of SUD outcomes were 1.4 to 4.7 times higher in the NESARC. In summary, apparent comorbidity and poly-drug use are substantially more prevalent in the NESARC.

Table 2.

Odds Ratios for Association Between Pairs of Substance Use or SUD Outcomes

	NSDUH (N=36,370)		NESARC (N=43,093)		Comparison
Outcome	OR	(95% CI)	OR	(95% CI)	Ratio‡	(95% CI)	p
Lifetime Use
Alcohol - Marijuana	27.4	(20.8 - 36.0)	19.9	(15.7 - 25.2)	0.7	(0.5 - 1.0)	0.087
Alcohol - Other Ill. Drug	24.3	(14.6 - 40.4)	16.9	(12.1 - 23.6)	0.7	(0.4 - 1.3)	0.242
Marijuana -Other Ill. Drug	62.2	(51.1 - 75.6)	145.5	(121.9 - 173.5)	2.3	(1.8 - 3.0)	<0.001
Past Year Use
Alcohol - Marijuana	10.4	(7.9 - 13.7)	10.2	(7.7 - 13.4)	1.0	(0.7 - 1.5)	0.909
Alcohol - Other Ill. Drug	13.6	(9.2 - 20.1)	25.8	(10.7 - 62.3)	1.9	(0.7 - 5.0)	0.191
Marijuana -Other Ill. Drug	42.5	(35.6 - 50.7)	126.5	(96.1 - 166.4)	3.0	(2.1 - 4.1)	<0.001
Past-Year Disorder
Alcohol - Marijuana	11.9	(9.8 - 14.5)	16.3	(13.1 - 20.2)	1.4	(1.0 - 1.8)	0.037
Alcohol - Other Ill. Drug	16.8	(12.0 - 23.4)	47.9	(29.4 - 78.3)	2.9	(1.6 - 5.2)	<0.001
Marijuana -Other Ill. Drug	26.0	(18.0 - 37.8)	122.7	(81.3 - 185.2)	4.7	(2.7 - 8.2)	<0.001

^‡Ratio of odds ratios, or interaction odds ratio (NESARC : NSDUH).

Bold indicates odds ratios that are significantly (p<0.05) higher than odds ratio from compared survey.

3.5 Sociodemographic Correlates

Odds ratios describing the relation between demographic variables (sex, age, race/ethnicity, and population density) and lifetime drug use, past-year drug use, and past-year SUD were compared across surveys. There were a number of small differences of nominal significance (0.01<p<0.05); we list here only statistically significant results where odds ratios differed by more than one-third (interaction odds ratio lower than 0.75 or greater than 1.33). All such differences involved SUD outcomes: in the NSDUH, Blacks were at higher odds for SUD than Whites (OR=1.2), whereas in the NESARC they were at lower odds (OR=0.8; interaction OR=1.4, p<0.001). Similar results were obtained for Hispanics, compared to Whites for SUD risk (OR=1.2 in the NSDUH, OR=0.9 in the NESARC, interaction OR=1.4, p=0.009). Total SUD prevalence was nearly identical in the surveys (9.1% in NESARC; 95% CI: 8.5-9.5; 9.0% in NSDUH; 95% CI: 8.6-9.5); hence the NSDUH estimates considerably higher SUD prevalence for Blacks and Hispanics than does the NESARC.

4. Discussion

4.1 Summary

Estimates of illegal drug use prevalence were 2.1 to 5.4 times higher in the NSDUH than in the NESARC. Relative differences in alcohol use prevalences were smaller but still considerable; the NSDUH reported more drinkers on both past-year and lifetime bases. Differences in SUD prevalences were observed for cocaine and heroin use disorders, with NSDUH prevalences being higher. However, SUD prevalence conditional upon past-year use was higher in the NESARC for several outcomes. Apparent poly-drug use and comorbid SUDs were several-fold more prevalent in the NESARC. Odds for SUD among minorities were approximately 40% higher in the NSDUH than in the NESARC. Given that these surveys were conducted at approximately the same time, with large samples and similar target populations, these findings raise serious concerns about the accuracy of substance use estimates as well as association and risk-factor estimates.

4.2 Possible Methodological Contributions to Discrepancies

In the Introduction, we suggested that privacy and anonymity considerations, diagnostic instrumentation, and response rates were among the salient differences between the NESARC and the NSDUH. Other sources of variation that might contribute to differences in estimates between surveys using different methodologies include coverage rate, data weighting, survey context, and differences in question text, survey structure, and response format.

Privacy and anonymity considerations would generally be expected to result in higher prevalence estimates for the NSDUH over the NESARC. Though designers of both surveys went to great lengths to protect respondent identity and assure respondents of the confidentiality of their data, anonymous conditions and the use of computerized self-administration methods (ACASI) to collect sensitive data would be expected to result in fewer cases of deliberate under-reporting of socially undesirable behaviors. In contrast, the NESARC used face-to-face interviews and, as the first of a two-wave study, collected identifying information. To our knowledge, no formal research on the effect of interviewer affiliation on response to sensitive questions has been conducted, but one might speculate that respondents would be less forthcoming in surveys administered by government agencies (the Census Bureau, in the NESARC) than in those administered by private firms. The stronger associations among drug use outcomes and SUDs in the NESARC are consistent with these expectations: individuals who refuse to disclose substance use or symptoms would be likely to do so across multiple outcomes, leading to higher apparent co-occurrence. The higher apparent odds of SUD among minorities in the NSDUH also supports a role for privacy and anonymity. Methodological experiments have demonstrated higher rates of drug use non-disclosure for minorities, which may be alleviated by greater perceived anonymity in survey conditions (7, 11, 19, 20).

Differences in diagnostic instrumentation are likely to contribute to SUD prevalence discrepancies. There were fewer discrepancies in overall SUD prevalence, compared to drug use estimates, but among subjects to whom SUD modules were administered (i.e., past-year users) substantially higher prevalences were observed in the NESARC. One possible explanation for this phenomenon is that the NESARC-AUDADIS is a more sensitive instrument than the NSDUH interview. While the latter reflects a direct operationalization of DSM-IV criteria, the AUDADIS utilizes a more thorough probing of SUD criteria, often asking multiple questions per criterion. Perhaps the NSDUH detected more cases of substance use because of anonymity and greater privacy associated with self-administration, while a more extensive probing of symptoms led to more sensitive detection of SUD among those who admitted use in the NESARC.

Difference in response rates between the surveys was moderate: 75% for the 2002 NSDUH and 81% for the NESARC. While the higher NESARC response rates might be expected to increase prevalence estimates of substance use (14), any such effect in this comparison is clearly outweighed by other factors.

Survey coverage, or the accuracy with which the targeted population is represented by the sampling frame, is likely to be similar between the NSDUH and NESARC. This is because they had identical target populations and utilized household-based sampling. In both cases, sources of undercoverage would be failure to identify household units in the sampling frame, or failure to identify individuals within households (2, 21). Differences in sampling and response rates for individual demographic groups should be compensated for by sample weighting. Hence, there is no obvious indication that sampling or coverage differences could have resulted in different substance use prevalences, but such effects cannot be ruled out.

Question text, response format, and survey structure might also influence response tendencies, leading to differences in prevalence estimates. For the results that were compared here, questions about lifetime substance use were worded most similarly, yet there were very large differences in lifetime use estimates. Differences in interview context may constitute another source of variability. The NESARC substance abuse modules are administered in the context of a comprehensive survey of mental health, whereas the NSDUH is specifically focused on substance use. It is plausible that such a difference could influence response tendencies, but it is difficult to speculate about the direction in which such effects would influence the results.

4.3 Strengths and Limitations

Undoubtedly, the principal limitation of this analysis is that it utilizes two surveys whose methods differed in a number of ways. We cannot unambiguously determine which differences account for the very large discrepancies observed here. This study has a number of strengths that stem from the use of these large, national databases. First, the samples are large enough to gain precise estimates of both prevalences and correlates from the two samples, and therefore, we were able to detect differences in associations among outcomes and in prevalences stratified by demographic groups; such statistical power would not be available in smaller methodological experiments. Additionally, the use of nationally representative samples minimizes the possibility that inter-survey disagreement might be attributable to particular characteristics of local samples.

4.4 Conclusion

Though differential rates of reporting in different survey conditions are well documented, the discrepancies in prevalence estimates shown here are exceptionally large. Moreover, our analyses have demonstrated differences in risk factor, poly-substance use, and comorbidity estimates. Ideally, controlled, single-variable experiments should be conducted to determine the effects of particular methodological variables on substance use, SUD prevalences, and their correlates. Variables of interest include mode of administration (self-administration versus interviewer administration), the collection of identifiers versus anonymous survey administration, interviewer affiliation, and interview context.

Regardless of the relative contribution of particular methodological differences, the discrepancies shown here are substantial and affect not only prevalence estimates but apparent comorbidity and demographic distribution of substance use problems as well. An accurate picture of the epidemiology of substance use, abuse, and dependence in the United States may well depend on a better understanding of the relation between survey methods and estimates.