Validity of Brief Screening Instrument for Adolescent Tobacco, Alcohol, and Drug Use

Abstract

BACKGROUND AND OBJECTIVE:

The National Institute on Alcohol Abuse and Alcoholism developed an alcohol screening instrument for youth based on epidemiologic data. This study examines the concurrent validity of this instrument, expanded to include tobacco and drugs, among pediatric patients, as well as the acceptability of its self-administration on an iPad.

METHODS:

Five hundred and twenty-five patients (54.5% female; 92.8% African American) aged 12 to 17 completed the Brief Screener for Tobacco, Alcohol, and other Drugs (BSTAD) via interviewer-administration or self-administration using an iPad. Diagnostic and Statistical Manual, Fifth Edition substance use disorders (SUDs) were identified using a modified Composite International Diagnostic Interview-2 Substance Abuse Module. Receiver operating characteristic curves, sensitivities, and specificities were obtained to determine optimal cut points on the BSTAD in relation to SUDs.

RESULTS:

One hundred fifty-nine (30.3%) adolescents reported past-year use of ≥1 substances on the BSTAD: 113 (21.5%) used alcohol, 84 (16.0%) used marijuana, and 50 (9.5%) used tobacco. Optimal cut points for past-year frequency of use items on the BSTAD to identify SUDs were ≥6 days of tobacco use (sensitivity = 0.95; specificity = 0.97); ≥2 days of alcohol use (sensitivity = 0.96; specificity = 0.85); and ≥2 days of marijuana use (sensitivity = 0.80; specificity = 0.93). iPad self-administration was preferred over interviewer administration (z = 5.8; P < .001).

CONCLUSIONS:

The BSTAD is a promising screening tool for identifying problematic tobacco, alcohol, and marijuana use in pediatric settings. Even low frequency of substance use among adolescents may indicate need for intervention.

What’s Known on This Subject:

The widely disseminated National Institute on Alcohol Abuse and Alcoholism screening tool for adolescent alcohol use was developed based on epidemiologic data. It has not been validated in a clinical sample and does not screen for tobacco or drug use.

What This Study Adds:

This study found that a measure that expanded the National Institute on Alcohol Abuse and Alcoholism adolescent alcohol use tool to include tobacco and drugs was sensitive and specific for identifying substance use disorders in a pediatric clinic patient population.

Substance use among US adolescents remains highly prevalent. In 2012, past-month use of alcohol, tobacco, and other drugs (ATOD) was 12.9%, 8.6%, and 9.5%, respectively, for youth 12 to 17 years of age.¹ Adolescent ATOD use is associated with mental^2–4 and physical⁵ health problems, poor school performance,^6,7 violence,^8,9 juvenile justice system involvement,¹⁰ and increased risk of developing an addictive disorder in adulthood.^11,12

Primary care visits present an opportunity to identify and intervene with substance-using youth, because most youth in the United States access the health care system,¹³ and preventing substance abuse among adolescents is a national public health priority (www.healthypeople.gov). Despite recommendations by the World Health Organization¹⁴ and American Academy of Pediatrics^15–17 that all adolescents receive screening for ATOD use, most adolescent medicine providers do not follow these best practice guidelines.^18–21

Existing instruments for screening adolescents’ ATOD use have notable limitations. The CRAFFT, a brief screening instrument recommended by the American Academy of Pediatrics,¹⁷ has a number of strengths, including its brevity and good psychometric properties,^22–24 but it does not screen for tobacco use, provide information on frequency of use, or discriminate between drug and alcohol use. Similarly, the Alcohol Use Disorders Identification Test²⁵ does not screen for tobacco or drug use. The 17-item substance-use scale of the Problem Oriented Screening Instrument for Teenagers²⁶ may be too lengthy for routine use in a busy pediatric practice.²⁴

To assist pediatricians in identifying adolescents at risk for alcohol-related problems, the National Institute on Alcohol Abuse and Alcoholism (NIAAA) developed a brief screening instrument²⁷ that asks patients about their frequency of drinking in the past year and that of their friends. The questions and clinical cut points against alcohol use disorders were established based on national epidemiologic survey data.²⁸ NIAAA’s screening instrument has been widely disseminated to practitioners across the United States. However, it has yet to be validated in a clinical sample.²⁷

The current study provides the first evaluation of the NIAAA instrument and its extension to tobacco and drug use among pediatric patients. This study seeks to (1) examine the concurrent validity of the self-report frequency of use item in NIAAA’s instrument that has been expanded to include drug and tobacco use against measures with known psychometric properties; (2) determine the utility of the expanded instrument as a brief assessment measure for tobacco, alcohol, and drug use; and (3) examine the acceptability of its self-administration on an iPad.

Methods

Participants

The study was conducted at 3 sites of a federally qualified health center in Baltimore, Maryland. Adolescents awaiting primary care appointments were enrolled from June 2012 through February 2013. Inclusion criteria were (1) 12 to 17 years of age and (2) willingness to provide informed assent. The study was approved by Friends Research Institute’s institutional review board (IRB). To protect participants’ confidentiality, no identifying information was collected, and a waiver of the requirement of written assent was obtained from the IRB. A waiver of parental consent was also approved by the IRB in accordance with the Office of Human Research Protection regulations (under 45 CFR 46.116[d]).

Brief Screener for Tobacco, Alcohol, and Other Drugs (BSTAD)

For the development of the BSTAD, the NIAAA instrument was expanded to include questions about tobacco and drug use and also the frequency of ATOD use during the past 30, 90, and 365 days (see Fig 1). Screening questions inquiring about any use in the past year were first asked for the 3 substance-use domains (tobacco, alcohol, and drugs) following NIAAA’s convention in which 12- to 14-year-olds were asked the questions about friends’ use first as a less-threatening way to approach the topic of substance use, followed by personal use questions, with the order reversed for adolescents ages 15 to 17 (and 14-year-olds who were in high school). Participants who endorsed personal use in any domain were asked additional questions to gauge frequency of use during the past 30, 90, and 365 days.

FIGURE 1.

The BSTAD. Note: consistent with the NIAAA instrument, if respondent is aged 12 to 14, friends questions are asked first; if aged 15 to 17 (or 14-year-olds in high school), personal-use questions are asked first.

Measures

In addition to the BSTAD, the following measures were administered.

Modified Composite International Diagnostic Interview

Items from the Composite International Diagnostic Interview—2 Substance Abuse Module (CIDI-2 SAM),^29–31 an instrument used to assess SUDs, were used as the criterion measure for identifying SUD as defined by the Diagnostic and Statistical Manual, Fifth Edition (DSM-5).³² The CIDI-2 SAM contains an item on craving, which is included as a symptom in meeting criteria for an SUD under DSM-5. Questions corresponding to the DSM-5 criteria were asked for tobacco, alcohol, and 9 categories of drugs in which the adolescent reported past-year use. Consistent with DSM-5 diagnostic thresholds, participants were classified as having an SUD if they reported 2 or more of 11 criteria for a given substance.

Usability and Acceptability

Participants were asked to rate their agreement (on a Likert-type scale of 1 = strongly disagree to 5 = strongly agree) with a series of items gauging their comfort with the screening questions and the administration format. Examples of questions included “These questions were easy to understand”; “I was comfortable answering these questions about my alcohol, tobacco, and drug use”; “I was able to remember the number of days I used alcohol, tobacco, or drugs in the past year”; and “I would prefer to answer these questions myself on an iPad instead of having a person ask me.”

Procedures

Adolescents in the appropriate age range were identified by clinic staff. A research assistant (RA) approached adolescents (and parents, if present) in the waiting area before their medical visit and asked if they would like to hear about the study. After their appointment, the RA obtained informed assent in a private room using an IRB-approved information sheet, after which parents were asked to return to the waiting area, and study measures were administered to participants.

Participants first completed the BSTAD, which was administered by the RA for the first half of the study sample (n = 262) and self-administered using a touchscreen tablet (iPad) for the second half (n = 263). For adolescents using the iPad, the RA was available to provide assistance with reading and to resolve technical problems. For all participants, the RA subsequently administered the usability/acceptability questions, followed by the modified CIDI-2 SAM items and 3 other measures not reported on in the current study. Participants received a $20 Subway gift card for participation.

Statistical Analysis

The analysis focused on adolescents’ use of alcohol, tobacco, and marijuana only because use of other substances was rare in this sample. To be consistent with DSM-5 diagnoses, which are based on symptoms occurring over the past year, data analysis focused on frequency of substance use in the past year. Peer substance use was not examined in the current study.

Receiver operating characteristic (ROC) analyses were used to establish cut points for number of days of use of tobacco, alcohol, and marijuana against the “gold standard” of DSM-5 SUD for each substance. ROC curves plot the sensitivity of a test against 1 minus the test’s specificity, creating a useful visual depiction of a test’s performance across the range of possible cut points.

Optimal cut points in relation to DSM-5 criteria for each substance were established by visual inspection of the ROC curves and examination of areas under the curve (AUC), sensitivities, and specificities. Additionally, participants’ usability/acceptability ratings were compared for the subsample that self-administered the screening on the iPad with the subsample that completed the interviewer-administered screening using the Mann-Whitney U test.

Results

Sample Characteristics

Of 584 adolescents approached by research staff, 54 (9.2%) refused participation. The target sample of 525 participants was reached by enrolling 530 participants who completed the study, because data from 5 participants were not transmitted to the Web-based data system due to technical problems. The sample of 525 adolescents was 54.5% female and 92.8% African American (see Table 1). Regarding age, 50.9% was aged 12 to 14 years and 49.1% was aged 15 to 17 years.

TABLE 1.

Participant Characteristics

Variable	Total Sample (N = 525)
	n (%)
Gender
Male	239 (45.5)
Female	286 (54.5)
Race
African American	487 (92.8)
White	4 (0.8)
Other race	34 (6.5)
Age group
12–14 y old	267 (50.9)
15–17 y old	258 (49.1)
School enrollment statusa
Middle school	196 (37.3)
High school	315 (60.0)
Not enrolled/other	13 (2.5)
Mode of BSTAD administration
Interviewer	262 (49.9)
iPad	263 (50.1)
Substance use, past y
Tobacco	50 (9.5)
Alcohol	113 (21.5)
Marijuana	84 (16.0)

^aData missing for 1 participant.

Cross-Check for Consistency in Responding

A cross-check of the data was conducted to compare 30-, 90-, and 365-day responses for each substance for each individual case. Four cases (0.76% of total sample) were found to have minor inconsistencies in responses (3 were completed on iPad; 1 completed by interviewer).

Substance Use

Of 525 participants, 159 (30.3%) reported use of ≥1 substances (alcohol, tobacco, other drugs) during the past year on the BSTAD, with 113 (21.5%) adolescents reporting alcohol use, 84 (16.0%) reporting marijuana use, and 50 (9.5%) reporting tobacco use. Sixteen (3.0%) participants reported using ≥1 illicit drugs other than marijuana in the past year: 9 (1.7%) reported misuse of prescription opioids; 7 (1.3%) reported misuse of over-the-counter medications; and 2 (0.4%) reported misuse of prescription sedatives. Use of cocaine/crack, amphetamines, and misuse of prescription stimulants was reported by 1 participant each (0.2%). No participants reported using heroin, hallucinogens, or inhalants during the past year.

Evaluation of Concurrent Validity

Table 2 shows the number (%) of participants meeting or exceeding the established cut point for each substance as determined by examination of the ROC curves (see Fig 2 for the smoothed ROC curves and their respective confidence bands), as well as the AUC, sensitivity, specificity, and 95% confidence intervals in relation to DSM-5 SUD for each substance at the identified cut points.

TABLE 2.

ROC AUC, Sensitivity, and Specificity for the Past-Year Frequency of Use Items on the BSTAD in Relation to DSM-5 SUD (N = 525)

	Cut Point on BSTADa	Met/Exceeded Cut Point, n (%)	Met DSM-5 Criteria for SUD, n (%)	AUC	Sensitivity (95% CI)	Specificity (95% CI)
Tobacco	≥6 d	37 (7.0)	21 (4.0)	0.96	0.95 (0.81–1.00)	0.97 (0.95–0.98)
Alcohol	≥2 d	98 (18.7)	24 (4.5)	0.90	0.96 (0.83–1.00)	0.85 (0.82–0.88)
Marijuana	≥2 d	77 (14.7)	56 (10.7)	0.87	0.80 (0.69–0.89)	0.93 (0.91–0.95)

DSM-5 SUD requires meeting ≥2 of 11 possible criteria. Sensitivity refers to the proportion of adolescents meeting DSM-5 criteria for SUD who were identified as meeting or exceeding the cut point on the BSTAD. Specificity refers to the proportion of adolescents not meeting DSM-5 criteria for SUD who were identified as falling below the cut point on the BSTAD. CI = confidence interval.

^aFor past-year use.

FIGURE 2.

ROC curves and confidence interval bands for tobacco, alcohol, and marijuana for past-year use frequency of use items on the BSTAD in relation to DSM-5 SUD (N = 525).

Tobacco

ROC analysis showed that the optimal cut point for tobacco use on the BSTAD in relation to DSM-5 SUD was ≥6 days in the past year. The AUC for tobacco use was 0.96. Its sensitivity was 0.95, indicating that using a frequency of use cut point of ≥6 days in the past year correctly identified 95% of adolescents who met DSM-5 criteria. Specificity was 0.97 for tobacco use, indicating that, at this cut point, the single frequency of use item correctly identified 97% of adolescents who did not meet DSM-5 criteria.

Alcohol

ROC analysis showed the optimal cut point on the BSTAD for frequency of alcohol use in identifying DSM-5 alcohol use disorder to be ≥2 days of use in the past year. AUC and sensitivity values were 0.90 and 0.96, respectively, and its specificity was 0.85.

Marijuana

Similar to alcohol use, ROC analysis showed the optimal cut point for frequency of marijuana use on the BSTAD to be ≥2 days of use in the past year. AUC and sensitivity values were 0.87 and 0.80, respectively, and specificity was 0.93.

BSTAD Usability by Mode of Administration (Interviewer vs iPad)

There were no significant differences based on mode of administration in comprehension, comfort, ability to recall past-year frequency of use, or willingness to answer similar questions during a future medical visit based on reported level of agreement with the following statements (all statements are shown in Table 3): “These questions were easy to understand”; “I was comfortable answering these questions about my alcohol, tobacco, and drug use”; “I was able to remember the number of days I used alcohol, tobacco, or drugs in the past year”; and “I would be willing to answer questions like these at my doctor’s office every year,” respectively (all Ps > .05). For the item that pertains to answering questions in the doctor’s office, findings should be interpreted cautiously because participants may have perceived that item as assessing whether they would answer questions as part of a confidential research study versus assessing whether they would share substance use information with their providers.

TABLE 3.

Usability and Acceptability Items (N = 525)

Item	Strongly agree or agree, %
These questions were easy to understand.	99.6
I was comfortable answering these questions about my alcohol, tobacco, and drug use.	93.9
I was able to remember the number of days I used alcohol, tobacco, or drugs in the past year. (n = 159)a	61.0
I would be willing to answer questions like these at my doctor’s office every year.	89.8
The iPad touch screen was easy to use. (n = 263)b	99.6
I would prefer that a person ask me these questions in the doctor’s office instead of answering them myself on the iPad. (n = 263)b	20.9
I would prefer to answer these questions myself on an iPad instead of having a person ask me. (n = 262)c	42.3

^aOnly participants who reported alcohol, tobacco, or drug use in the past year are included in percentage.

^bOnly participants who self-administered BSTAD questions on iPad are included in this percentage.

^cOnly participants who were asked BSTAD questions by interviewer are included in this percentage.

Participants in each subsample based on mode of administration were asked if they would have preferred to answer questions using the other mode, and iPad self-administration showed an advantage as the preferred mode (z = 5.8; P < .001). Only 20.9% of participants who completed the screening on the iPad agreed or strongly agreed that they would have preferred that an interviewer had asked the questions. Conversely, among those who completed the screening with the interviewer, 42.3% agreed or strongly agreed that they would have preferred to self-administer the screening on an iPad.

Discussion

The NIAAA’s adolescent alcohol screening tool was empirically developed from epidemiologic data²⁸ and has been widely disseminated. The current study is the first to examine the performance of the self-reported frequency of use item from this tool in a pediatric patient sample and the first formal extension of this tool to include drug and tobacco use.

This study furthers the effort to develop and test a simple, brief screen to rapidly triage pediatric patients by risk level. Single-item substance use screeners have been tested and recommended for adult primary care populations and have been found to have low cut points in terms of number of days of use in the past year.³³ Because onset of substance use typically occurs during the adolescent years, patients in this age group are a critical target for screening and intervention. The overall performance metrics for the frequency of use items (including AUC, sensitivity, specificity) compared with the gold standard of meeting DSM-5 criteria for tobacco, alcohol, and marijuana SUD were favorable.

Clinicians need a substance use screening measure that is brief, reliable, and practical.²⁴ The past-year frequency of use items on the BSTAD constitute a common question for tobacco, alcohol, and drugs. This question generally offers high sensitivity and specificity and meets the need for brevity and reliability for each of the 3 substances most commonly used by adolescents. There is no need to score the instrument, and it is simple to recall the cutoffs for these 3 substances. Because many adolescent treatment providers may find it important to screen for any use of tobacco or other substances (not just their use disorders), it should be noted that this information is also provided by the BSTAD.

The cutoffs in the current study of 2 days for either alcohol or marijuana and 6 days for tobacco use in the past year support the frequency of use screening question as a reliable, rapid, and simple strategy for pediatricians to triage patients into low-risk and higher-risk groups. However, it is important to note that although the measure may be an excellent screening tool, the low cut points limit the ability of frequency of use on its own to act as a nuanced measure of the degree of problem severity. For patients who screen positive on the BSTAD for any of these substances, further inquiry regarding problems associated with use should be pursued, as is the case with any screening measure. Minimally, for all patients screening positive, a brief discussion/intervention by the practitioner is warranted.

Brief screening tools that offer the flexibility of patient- or provider-administration provide flexibility in pediatric practices and permit patient choice. In the current study, self-administration of the BSTAD on an iPad was feasible and well received by participants. Self-administration would benefit a busy pediatric practice because data could potentially be entered directly into the electronic medical record and immediately reviewed by the provider. Another potential advantage of self-administration is that there is some evidence that this approach yields more accurate and reliable responses from adolescents compared with administration by an interviewer for sensitive topics.^34,35

Rates of substance use in the past year reported by participants were 9.5% for cigarettes or tobacco products, 21.5% for alcohol, and 16.0% for marijuana. These rates differ somewhat from national rates for 12- to 17-year-olds in 2012, which were 15.2%, 26.3%, and 13.5% for past-year use of tobacco, alcohol, and marijuana, respectively.¹ However, substance use prevalence in our largely African American sample is more in line with national data for African American adolescents’ rates of past year tobacco (9.7%), alcohol (20.8%), and marijuana (13.5%) use. Rates of illicit substance use other than marijuana were found to be low in this sample, which is generally consistent with rates found in national data. A notable exception is that rates of prescription drug misuse are higher nationally than what was found in our sample.¹ Thus, a degree of caution is warranted when generalizing our findings to other groups of adolescents.

The recently released DSM-5 criteria for SUD collapses 2 separate disorders from DSM-IV (labeled substance abuse and substance dependence) into a single category of SUD. The use of the new DSM-5 criteria can be considered a strength of the study, although the CIDI-2 has not been formally validated against the DSM-5. However, the CIDI has been validated for the DSM-IV,^36–38 which includes all of the same criteria in the DSM-5, with 2 exceptions (a craving criterion was added to the DSM-5 and the legal problem criterion from the DSM-IV was eliminated for the DSM-5). A limitation of the study is that it was conducted in a single city, with a largely African American population. Further replication in other localities with diverse patient populations is warranted. Although it would have been useful to determine whether optimal cut points differed by gender or age group, the prevalence of DSM-5 SUDs in these subsamples was too low to permit such analysis. Examining cut points for these subsamples should be a focus of future research. Additionally, data from all assessments administered in this study were self-report, and, therefore, substance use may have been underreported. Future studies should include the collection of biological samples to test for the presence of substances that can serve as validity information beyond self-report, although their relatively brief window limits their utility in that regard. Finally, the administration order (self- vs interviewer-administered) was not randomized, nor was the order of administration of the instruments.

Conclusions

This study provides promising evidence supporting the validity and utility of using past-year frequency of use as a quick and accurate screen for adolescents’ problematic tobacco, alcohol, and drug use. Given the association between substance use and depression, suicide, violence, fatal car accidents, academic problems, and the development of SUDs^2,6,8,12,39 and the potential effectiveness of brief interventions and treatment of substance-involved adolescents,^40,41 it is of considerable importance that pediatricians, family physicians, and other health care providers screen adolescents for substance use. The use of the BSTAD or similar instruments can be an important step in identifying substance use and successfully intervening in the lives of adolescent patients.

Glossary

ATOD

alcohol, tobacco, and other drugs

AUC

area under the curve

BSTAD

Brief Screener for Tobacco, Alcohol, and other Drugs

CIDI-2 SAM

Composite International Diagnostic Interview-2 Substance Abuse Module

DSM-5

Diagnostic and Statistical Manual, Fifth Edition

IRB

institutional review board

NIAAA

National Institute on Alcohol Abuse and Alcoholism

RA

research assistant

ROC

receiver operating characteristic

SUD

substance use disorder