Test-retest reliability of a self-administered Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) in primary care patients

Abstract

The time required to conduct drug and alcohol screening has been a major barrier to its implementation in mainstream healthcare settings. Because patient self-administered tools are potentially more efficient, we translated the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) into an audio guided computer assisted self interview (ACASI) format. This study reports on the test-retest reliability of the ACASI ASSIST in an adult primary care population. Adult primary care patients completed the ACASI ASSIST, in English or Spanish, twice within a 1–4 week period. Among the 101 participants, there were no significant differences between test administrations in detecting moderate to high risk use for tobacco, alcohol, or any other drug class. Substance risk scores from the two administrations had excellent concordance (90–98%) and high correlation (ICC 0.90–0.97) for tobacco, alcohol, and drugs. The ACASI ASSIST has good test-retest reliability, and warrants additional study to evaluate its validity for detecting unhealthy substance use.

1. INTRODUCTION

Substance abuse leads to more death and disability than any other preventable health condition, yet only a minority (~11%) of those with drug and alcohol use disorders receive specialized treatment (McGinnis & Foege, 1993; Mokdad, Marks, Stroup, & Gerberding, 2004; Robert Wood Johnson Foundation and Schneider Institute for Health Policy, 2001; Substance Abuse and Mental Health Services Administration (SAMHSA), 2012). Primary healthcare providers often constitute the only health system contacts for this population, particularly for those who have not yet developed severe drug use disorders (Babor et al., 2007; Solberg, Maciosek, & Edwards, 2008; Whitlock, Polen, Green, Orleans, & Klein, 2004). There is therefore intense interest in identifying and addressing substance use in general healthcare settings, but significant challenges exist to integrating this practice into regular medical care.

Screening and brief intervention (SBI) describes an approach that seeks to identify and provide interventions for problematic substance use in mainstream healthcare settings (Madras et al., 2009; McCance-Katz & Satterfield, 2012; Substance Abuse and Mental Health Services Administration (SAMHSA), 1997, 2011). It involves streamlined screening and assessment, followed by brief interventions that can be carried out by medical providers in the course of a regular office visit. A major impediment to wide-scale implementation of SBI for drugs other than alcohol and tobacco is lack of a paradigm for routine and efficient screening and assessment for drug use in general medical settings, (Babor et al., 2007; Madras et al., 2009)and it often goes undetected as a result (D’Amico, Paddock, Burnam, & Kung, 2005; Friedmann, McCullough, & Saitz, 2001; R. Saitz, Mulvey, Plough, & Samet, 1997).

A number of screening and assessment approaches have been developed to identify unhealthy substance use.. Following the tobacco Clinical Practice Guideline, many practices systematically screen for tobacco use at each visit, using strategies such as inclusion of tobacco use in the vital signs, or documentation of tobacco use status in the electronic health record (Fiore et al., 2008). The AUDIT and AUDIT-C are widely recommended for alcohol screening and assessment, and have been implemented as part of routine primary care in large health systems including the Veterans Administration (Bradley K., 2013; National Institute on Alcohol Abuse and Alcoholism (NIAAA), 2007). Screening and assessment tools for other drugs, including a single-item drug screen and a 10-item version of the DAST, have been developed and validated in primary care populations (Smith, Schmidt, Allensworth-Davies, & Saitz, 2010; Yudko, Lozhkina, & Fouts, 2007), but have yet to be widely adopted in practice. In busy medical practice settings, identification of drug and alcohol use could be facilitated by having a unified screening paradigm that integrates tobacco, alcohol, and drugs in a brief screen that generates clinically relevant results.

One such instrument is the “Alcohol, Smoking and Substance Involvement Screening Test (ASSIST),” a validated structured interview that was developed by the World Health Organization for use in general healthcare settings (Center for Integrated Health Solutions, 2012; Humeniuk, 2008). However, the ASSIST has proven difficult to incorporate into routine care (Babor et al., 2007; National Institute on Drug Abuse (NIDA), 2010), in part because it takes approximately 5–15 minutes of face-to-face interaction with the patient to administer, has complex skip patterns, and bases its calculation of risk on a scoring system that must be computed by the interviewer. Yet the ASSIST has some clear advantages for use in SBI programs, in comparison to briefer assessment instruments. Specifically, it includes tobacco and alcohol alongside other drugs; gives a substance specific risk score (low, moderate, high) that can be used to guide the clinical intervention; and screens for injection drug use (Humeniuk, 2008; Humeniuk et al., 2012; McNeely J., 2013; Mdege & Lang, 2011). The ASSIST identifies lifetime as well as current use, which may be clinically relevant for preventive care such as identification of individuals at risk for developing health sequelae of prior use (e.g. hepatitis C, chronic obstructive pulmonary disease (COPD)), or for identifying individuals who are at risk of relapse and may need closer monitoring than those with no history of substance use. These features of the ASSIST have the potential to streamline the substance use history and assessment of the primary care provider, and maximize use of the provider’s time for delivering clinical interventions.

To increase the potential feasibility of using the ASSIST in medical settings, we adapted the previously validated interviewer-administered ASSIST to a patient self-administered format using audio guided computer assisted self interview (ACASI) technology. ACASI instruments make it possible to support even patients with limited reading ability in completing self-administered questionnaires because all relevant text is read aloud in real time, and response options are clearly indicated using symbols. An ACASI version of the ASSIST could be completed in the waiting room, with results conveyed to the provider at the start of the visit. Such an innovation could reduce barriers to implementing routine screening and assessment for substance use in the primary care setting, by providing an instrument that more efficiently fits into clinical workflows (Tai, 2012). A self-administered screening approach also has the potential to reduce the stigma associated with disclosure of substance use in a face-to-face interview, and could potentially make patients feel more comfortable and respond more honestly (Kim, Dubowitz, Hudson-Martin, & Lane, 2008; Tourangeau & Smith, 1996; Wight et al., 2000). Unlike interviewer-administered screening tools which rely on individuals delivering the items exactly as written (Bradley et al., 2011), the self-administered approach delivers validated screening instruments consistently and with high fidelity, even in the context of routine clinical care.

Computerized self interview approaches, particularly ACASI, have proven sensitive for detecting stigmatized behaviors, have comparable validity to traditional interview formats, are easily adapted to multiple languages, and can be integrated into medical settings (Bertollo, Alexander, Shinn, & Aybar, 2007; Butler et al., 2001; Harris et al., 2012; Murphy, Bijur, Rosenbloom, Bernstein, & Gallagher, 2013; Rogers et al., 2005; Satre, Wolfe, Eisendrath, & Weisner, 2008; Schackman et al., 2009). Notably, an ACASI version of the AUDIT was shown to be feasible, acceptable to patients, and equally good as an interviewer- or ‘pencil and paper’-administered version at detecting problem drinking among English speaking patients (Butler SF, 2003; Chan-Pensley, 1999; Neumann et al., 2004). This suggests that an ACASI ASSIST could be a viable option in general healthcare settings, but with the exception of one small study in an English-speaking college student population, the validity of this approach has not been tested (Spear, 2009).

As a preliminary step in a line of research investigating the feasibility and validity of an ACASI ASSIST for substance use screening and assessment in primary care, we examined its test-retest reliability. Assessment of test-retest reliability is an important early step in instrument development, but has sometimes been overlooked in the development of substance use screening tools (Reinert & Allen, 2002; Selin, 2003; Sobell, Kwan, & Sobell, 1995). Test-retest reliability measures the consistency of an instrument’s measurements, and is usually undertaken prior to studies of diagnostic accuracy (sensitivity, specificity). Test-retest reliability was studied by the WHO ASSIST Working Group for the interviewer-administered ASSIST, as part of the process of developing that instrument (Ali et al., 2002). In that international study, a total of 236 participants recruited from primary care and substance abuse treatment settings completed the ASSIST twice in a 1–3 day period. The ASSIST items demonstrated good test-retest reliability, with average kappa coefficients across the ASSIST items for each substance class ranging from 0.61 to 0.78. This earlier work informs the present study, which examines test-retest reliability of the ACASI ASSIST in a U.S. adult primary care population.

2. MATERIALS AND METHODS

2.1. Participants and recruitment

The study was conducted in the adult primary care medicine clinic of a large municipal hospital in New York City. The first enrollment period was April-August 2011, during which time 85 participants were enrolled. Because the rate of follow-up was lower than anticipated, to increase the sample size for this test-retest reliability analysis we added a second enrollment period, January-April 2012, during which 61 participants were enrolled. While the parent study was broader in scope and included additional study procedures, the test-retest reliability of the ACASI ASSIST was the primary focus, and is the only aspect of the study presented here. All recruitment was from the clinic waiting area. Individuals were approached consecutively, screened for eligibility, and offered participation in the study regardless of their demeanor. There was no advertisement, and all recruitment was done by three research assistants (RAs) during the initial enrollment period, and by one RA during the second enrollment period.

Eligible individuals were required to be current clinic patients, age 18–65. In the initial enrollment period we oversampled patients anticipated to have greater difficulty using the ACASI instrument by recruiting a pre-set number of participants who met one of the following additional criteria: less than high school education, 50–65 years old, or primary language Spanish. In the second enrollment period, only individuals fluent in English were eligible, and a purposeful sampling approach was used to achieve approximately equal numbers of male and female participants.

Potential participants received a written information sheet, and verbal consent for participation was obtained prior to any study assessments. All participants received a modest cash incentive plus a round-trip transit card for participating, and were offered referral information for tobacco and alcohol/drug treatment services. The institutional review board of [name blinded] reviewed and approved of all study procedures.

2.2. Study Instrument: ACASI ASSIST

The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) was developed by the World Health Organization (WHO) and a global consortium of substance abuse researchers as a screening tool for substance use and addiction, applicable to primary care patients in diverse populations (Humeniuk, 2008). It has been used for both clinical screening and for research purposes, and has been translated and validated by WHO in multiple languages. The ASSIST provides a substance specific risk stratification that can guide clinical interventions; typically monitoring and prevention for low risk, brief intervention for moderate risk, and treatment or treatment referral for high risk use. The ASSIST was developed and adapted over a five-year period by WHO, leading up to the ASSIST V3.0, which was the basis for the ACASI ASSIST used in our study.

ASSIST V3.0 is a brief structured interview covering nine substances (tobacco, alcohol, cannabis, cocaine, stimulants, inhalants, sedatives, hallucinogens, opioids), that assesses lifetime use, current use, consequences of use, and failure to stop or cut down. The first item in the ASSIST asks about lifetime use for each substance. Individuals who screen negative for lifetime use of all substances receive no further questions. Those who screen positive for lifetime use go on to complete the subsequent items pertaining to the substances for which they reported lifetime use. For example, an individual who reported lifetime use of marijuana only would receive the follow-up questions only with reference to marijuana. Item 2 of the ASSIST assesses current (past 3 months) use, while items 4–7 assess various problems related to substance use, and item 8 asks about any lifetime or current injection drug use. Individuals who report lifetime use but no current use of a substance are still administered items 6–7 for that substance, as well as item 8 on injection drug use. Item 6 asks about whether close contacts have ever expressed concern about the individual’s use, while item 7 asks if they have ever tried and failed to cut down on their use of that substance.

The ASSIST instrument used in our study maintained all of the structural features of the WHO ASSIST V3.0, but was adapted to include two additional substances: prescription opioids and prescription stimulants. This change is consistent with the inclusion of these substance classes in the NIDA-modified ASSIST (National Institute on Drug Abuse (NIDA), 2010). The survey preamble, questions, and format were otherwise unchanged from the ASSIST V3.0. We translated the modified ASSIST to audio computer assisted self interview (ACASI) format, in English and in Spanish, using QDS Software (Nova Research Co.). This ‘ACASI ASSIST’ was administered on touch-screen tablet computers, with headphones. Participants were guided by voice instruction and written text on the computer screen in the same way that the ASSIST would be administered by an interviewer who was reading the instrument verbatim, maintaining identical language and skip patterns to those specified in the WHO ASSIST V3.0. The preamble and ASSIST items were delivered in their entirety, and written text on the computer screen was identical to the words of the voice instruction.

2.3. Study Procedures

Assessments were conducted in either English or Spanish, depending on the preferred language of the participant. At the first visit (Time 1), participants met with the RA in a private room to complete the ACASI ASSIST. For participants recruited during the first enrollment period, the ACASI ASSIST was followed by the interviewer administered (IA) ASSIST at Time 1. Participants recruited during the second enrollment period did not receive the IA ASSIST, but did receive a 4-item self- substance use screening questionnaire prior to the ACASI ASSIST. The 4-item screener asked about past 12 months use of tobacco, alcohol, illicit drugs, and prescription drugs, and was administered in this study to provide an assessment of this instrument’s test-retest reliability (only results from the ACASI ASSIST are presented here). Demographic information was collected at the end of the first visit, and all participants were scheduled to return within 1–3 weeks for a second visit (Time 2). Those who returned for the Time 2 visit slightly outside of the designated window (1 individual at 6 days, 10 individuals at 21–27 days) were still included. At the second visit, the ACASI ASSIST was administered, and the IA ASSIST was not administered.

The RA waited outside the room while participants completed the ACASI ASSIST, to assure privacy and encourage independent completion of the questionnaire. Participants who needed assistance had the option of clicking a ‘help’ button that provided additional explanation of the item they were answering, and they were able to call the RA into the room to request technical assistance. During the first enrollment period, requests for assistance using the ACASI were tracked at Time 1 by the RA, who filled out a form indicating whether the participant requested assistance, and of what type (i.e. help using the computer, comprehending the questions, or other assistance). Time for each ACASI ASSIST interview was recorded automatically by the computer.

2.4. Measures

Prevalence

Prevalence of lifetime use and current (past 3 months) use was based on responses to ACASI ASSIST Questions 1 and 2, respectively.

Risk scores

The ASSIST global scores and substance specific involvement scores (SSIS) were calculated using standard ASSIST methodology (Humeniuk et al., 2008). These scores included the two substance categories (prescription opioids and stimulants) that we added to the original ASSIST V3.0. The global score represents the sum of all responses to ASSIST Question 1–8, and has a potential range of 0 to 498. Following the standard approach to scoring the ASSIST, the SSIS is the sum of responses to ASSIST Questions 2–7, for each substance, and has a potential range of 0 to 39. Because the ASSIST instrument specifies that individuals who report no current use of a substance should still receive follow-up questions about whether a ‘friend or relative or anyone else ever expressed concern’ about that their use (Question 6), or if they ‘ever tried and failed’ to reduce their use (Question 7), the SSIS may be greater than zero even for individuals who are not current users of that substance. ASSIST scores were further aggregated into a summary ‘drug’ category that includes all substances other than tobacco and alcohol (and includes misuse of prescription medications). The summary ‘drug score’ was calculated from the sum of individual SSIS across all drugs, and has a potential range of 0 to 390.

Risk level

WHO-recommended cutoffs were used to determine level of risk (low, moderate, or high) from the SSIS for each substance. For alcohol, a score of 0–10 constitutes low risk, 11–26 moderate risk, and 27+ high risk. For tobacco and all other drugs, a score of 0–3 constitutes low risk, 4–26 moderate risk, and 27+ high risk. Low risk individuals are not considered to require intervention for their substance use, whereas for those with moderate or high risk an intervention is indicated (Humeniuk, 2008). In our analyses, we collapsed the moderate and high risk levels into a single ‘moderate-high risk’ category. The choice was made to combine moderate and high risk users because high risk substance use was relatively infrequent in this study population, and we felt it was most relevant to focus on distinguishing between individuals whose substance use requires clinical intervention (due to moderate or high risk use) versus those who do not require intervention (due to low risk use). Additional tables showing the concordance of risk scores for moderate risk and high risk use alone are included in the Appendix (Tables 3a and 3b). Level of risk for the aggregate ‘drugs’ category was based on the highest risk level for any substance included in that category. For example, an individual with low risk marijuana use and moderate risk cocaine use would be classified as ‘moderate-high risk’ in the drugs category.

2.5. Statistical Analysis

The analysis was restricted to individuals who completed both study visits. One participant who skipped all ASSIST items at both visits was excluded from the analysis. To characterize the study population, we used descriptive statistics (means and proportions) to summarize participants’ demographic characteristics, and to describe the prevalence of substance use based on responses given on the ACASI ASSIST at Time 1 or Time 2. Comparing participants recruited during the first versus the second enrollment period, we did not detect any statistically significant (P<.05) difference in ASSIST scores or rates of moderate-high risk use at Time 1, for tobacco, alcohol, or other drugs. Based on this finding, we combined the two groups for the test-retest reliability analysis.

Analysis of agreement for classification of risk level

To evaluate test-retest reliability, we began by examining the level of agreement in classifying individuals as low versus moderate-high risk users, for all 12 substance classes queried by the ACASI ASSIST. We first estimated the prevalence of low risk versus moderate-high risk use for each substance class at Time 1 and Time 2, then calculated the proportion of individuals who had either concordant risk levels (low risk at both time points or moderate-high risk at both time points), an increase in risk level (from low risk at Time 1 to moderate-high risk at Time 2), or a decrease in risk level (from moderate-high risk at Time 1 to low risk at Time 2). To test the degree of concordance between Time 1 and Time 2 results, we examined the proportion of concordant results for individuals having low risk versus moderate-high risk use using McNemar tests, with P-values >.05 considered indicative of no significant change in score between Time 1 and Time 2. We additionally examined the correlation of results at Time 1 and Time 2, for low versus moderate-high risk use, using Cohen’s Kappa. Kappa coefficients were computed for all substance classes having prevalence greater than 20% in the study population. They were not calculated for lower prevalence substance classes because the dependence of Kappa on prevalence can compromise its interpretation in conditions of markedly high or low prevalence (Spitznagel & Helzer, 1985; Thompson & Walter, 1988). The Kappa coefficients were interpreted using the standard guidelines of 0.20 or less indicating poor agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement, and 0.81 to 1.0 almost perfect or perfect agreement (Landis & Koch, 1977).

While only the analysis of agreement for the combined category of moderate-high risk substance use is shown in the manuscript, the same analytic steps were applied to two separate analyses of moderate risk and of high risk use only. Results of those analyses are included as an appendix (Tables 3a and 3b). The same analysis was applied to the three pre-specified subgroups of individuals who were anticipated, based on prior studies, to have greater difficulty using the ACASI instrument (Butler et al., 2001; Reichmann et al., 2010; Satre et al., 2008). These subgroups were: those having less than high school education; those 50 years old or greater; and those whose primary language was Spanish.

Analysis of agreement for ASSIST scores

In a second approach to our analysis of agreement, we examined correlation between the ACASI ASSIST scores generated at Time 1 and Time 2, using Intraclass Correlation Coefficients (ICCs). We calculated ICCs, assessing agreement for repeated ACASI ASSIST measurements, for the following scores, all of which are continuous variables: a) global score, b) SSIS for each substance, and c) a summary ‘drug score,’ which included all substances other than tobacco and alcohol. We additionally computed the ICC for the global score limited to the items that comprise the WHO ASSIST V3.0, and thus did not include the prescription stimulant and prescription opioid items. The ICC calculations used a single measurement, absolute agreement definition, two-way mixed model. ICCs were interpreted using recommended guidelines of Cicchetti for reliability of clinical instruments: less than 0.40 indicates poor agreement, 0.40 to 0.59 fair agreement, 0.60–0.74 good agreement, and 0.75 to 1.00 excellent agreement (Cicchetti, 1994). We considered using the approach of Bland and Altman as an additional evaluation of agreement, but were unable to apply it here because its interpretation requires that distribution of differences between the measures be normally distributed, which was not the case for all of our difference scores (Bland & Altman, 1999). All analyses were conducted in IBM SPSS Statistics 20.

3. RESULTS

3.1. Characteristics of the study population

Of the 146 individuals who completed assessments at Time 1, 101 (69%) returned and completed assessments for Time 2. Individuals from the initial enrollment period had a lower rate of follow up visits than those from the second enrollment period (55% vs. 89%). Those who failed to return at Time 2 did not differ significantly in prevalence of current drug or alcohol use, age, sex, education, or income, but were more likely to be foreign born, Hispanic, and to speak Spanish as their primary language. The mean time that elapsed between Time 1 and Time 2 assessments was 12 days (SD=6, range 6–27 days), and 72% of participants returned within 14 days. The average time required to complete the ACASI ASSIST on the first administration was 5.4 minutes (range 1.5 – 17.7 minutes). Data on requests for assistance in completing the ACASI ASSIST was captured for the first 35 participants. Among this group, 5 (14%) requested some type of assistance (3 asked for help using the computer, 2 for help with comprehension of the questions). After removing from the analysis the single individual who skipped all items of the ASSIST, responses were 100% complete for all participants.

Characteristics of the 101 subjects who completed two visits and were included in this analysis are presented in Table 1. Thirty-six percent of participants were born outside the United States, and 26% had a primary language other than English. Thirty percent of participants did not complete high school, while 22% had a high school diploma or GED.

Table 1.

Demographic characteristics of the 101 participants

Characteristic	N (%)
Age (years)
Mean, SD	46, SD 10
Median	47
Range	19–64
Interquartile range	14
Gender
Female	50 (49.5)
Male	51 (50.5)
Hispanic	35 (34.7)
Race
Black/African American	50 (49.5)
White/Caucasian	18 (17.8)
Other	33 (32.7)
Foreign Born	36 (35.6)
Primary language
English	75 (74.3)
Spanish	23 (22.8)
Other	3 (3.0)
Education
Less than HS	30 (29.7)
HS grad or GED	22 (21.8)
Some college or trade school	34 (33.7)
College or graduate degree	13 (12.8)
Other	1 (1.0)
Don’t know/Refused	1 (1.0)
Income
<$5,000	39 (38.6)
$5,000 to $14,999	23 (22.8)
$15,000 to 49,999	23 (22.8)
$50,000 or greater	5 (5.0)
Don’t know/Refused	11 (10.9)
Employment
Employed full-time	23 (22.8)
Employed part-time or occasional work	17 (16.8)
Unemployed	34 (33.7)
Disability	20 (19.8)
Student/Other	6 (5.9)
Don’t know/Refused	1 (1.0)

3.2 Self-reported prevalence of substance use and risk level

Prevalence of lifetime and past 3 months substance use, based on responses to questions 1 and 2 of the ACASI ASSIST, is shown in Table 2. As indicated in the table, ‘drugs’ refers to all substances other than alcohol and tobacco. Fifty-nine percent of participants reported lifetime use of tobacco, while 83% reported lifetime use of alcohol, and 74% reported lifetime use of at least one other drug. Rates of current use were 32% for tobacco, 67% for alcohol, and 42% for drugs. In the drug category, marijuana was the substance with highest prevalence of lifetime use, while cocaine was the substance with highest prevalence of current use. The majority (59%) of current drug users also reported current use of tobacco, and 38% reported current use of alcohol at least weekly. For the 33 individuals who reported no current alcohol use, 27% were current users of at least one drug (data not shown).

Table 2.

Prevalence and risk level by substance at Time 1, based on ACASI ASSIST responses (N=101)

Substance category	Lifetime Use N (%)	Current Use N (%)	Low Risk N (%)	Moderate to High Risk N (%)
Tobacco	60 (59.4)	32 (32.0)	66 (65.3)	35 (34.6)
Alcohol	83 (82.2)	68 (67.3)	76 (75.2)	25 (24.8)
Drugs	75 (74.3)	42 (41.6)	53 (52.5)	48 (47.5)
Marijuana	51 (50.5)	18 (17.8)	77 (76.2)	24 (23.8)
Cocaine	40 (39.6)	19 (18.8)	73 (72.3)	28 (27.7)
Prescription Opioids	31 (30.7)	10 (10.0)	85 (84.2)	16 (15.8)
Sedatives	29 (28.7)	15 (15.0)	88 (87.1)	13 (12.9)
Heroin	22 (21.8)	10 (9.9)	84 (83.2)	17 (16.8)
Hallucinogens	21 (20.8)	2 (2.0)	91 (90.1)	10 (9.9)
Prescription Stimulants	14 (13.9)	5 (5.0)	97 (96.0)	4 (3.9)
Methamphetamine	9 (8.9)	3 (3.0)	96 (95.0)	5 (5.0)
Inhalants	8 (7.9)	2 (2.0)	96 (95.0)	5 (5.0)
Other	8 (7.9)	5 (5.0)	94 (93.1)	7 (6.9)

Risk level was specified using the standard ASSIST cutoffs designating low risk (not requiring clinical intervention) and moderate-high risk (requiring clinical intervention). For each substance queried, between 65% and 96% of participants had ASSIST scores indicating low risk use, while 4–28% had moderate-high risk use (Table 2). Some individuals with ASSIST scores indicating moderate-high risk use did not have current (past 3 months) use. For example, although 24 individuals had moderate-high risk use of marijuana, only 18 reported current marijuana use.

3.3. Concordance of risk level classification at Time 1 and Time 2

The first measure of test-retest reliability was concordance in classification of risk level between Time 1 and Time 2. Overall, 93% of individuals had a concordant risk level at Time 1 and Time 2, meaning that they either scored ‘low risk’ for all substances at both administrations of the ACASI ASSIST, or scored ‘moderate-high risk’ for at least one substance at both administrations. Two individuals (2%) had an increased risk at Time 2, while 5 (5%) had a decreased risk at Time 2. Table 3 shows change in risk category by substance. For example, in the summary drug category, 53 individuals had a low risk score for each drug on the ACASI ASSIST at Time 1, and 56 individuals had a low risk score for each drug at Time 2, while 48 individuals had a moderate-high risk score for at least one drug at Time 1, and 45 at Time 2. There were no statistically significant changes (P<0.05) in risk level between Time 1 and Time 2, based on McNemar’s test, for the summary drug category, or for any individual substance. Kappa coefficients were calculated for those substance classes having greater than 20% prevalence of moderate-high risk use in the study population, and indicated substantial to almost perfect agreement between Time 1 and Time 2. All Kappa coefficients were significant at the P<0.001 level. For each of the 3 prespecified subgroups (age 50–65 (n=38); less than high school education (n=30); or Spanish as a primary language (n=18)), we also failed to detect any statistically significant changes in risk level (data not shown).

Table 3.

Concordance of Time 1 and Time 2 results for classification of low risk versus moderate-high risk use (N=101).

	Individuals in risk category N (%)		Concordant risk levela N (%)	Increased risk levelb N	Decreased risk levelc N	McNemar’s test P-valued	Kappae
Substance Use Variable	Time 1	Time 2
Tobacco			94 (93.1)	5	2	0.453	0.850
Low risk	66 (65.3)	63 (62.4)
Mod OR high risk	35 (34.7)	38 (37.6)
Alcohol			95 (94.1)	2	4	0.687	0.836
Low risk	76 (75.2)	78 (77.2)
Mod OR high risk	25 (24.8)	23 (22.8)
Drugs (summary category)f			94 (93.1)	2	5	0.453	0.861
Low risk	53 (52.5)	56 (55.4)
Mod OR high risk	48 (47.5)	45 (44.6)
Marijuana			92 (91.1)	6	3	0.508	0.764
Low risk	77 (76.2)	74 (73.3)
Mod OR high risk	24 (23.8)	27 (26.7)
Cocaine			95 (94.1)	4	2	0.687	0.855
Low risk	73 (72.3)	71 (70.3)
Mod OR high risk	28 (27.7)	30 (29.7)
Prescription opioids			92 (91.1)	4	5	1.000	--
Low risk	85 (84.2)	86 (85.1)
Mod OR high risk	16 (15.8)	15 (14.9)
Sedatives			91 (90.1)	4	6	0.754	--
Low risk	88 (87.1)	90 (89.1)
Mod OR high risk	13 (12.9)	11 (10.9)
Hallucinogens			96 (95.0)	0	5	0.063	--
Low risk	91 (90.1)	96 (95.0)
Mod OR high risk	10 (9.9)	5 (5.0)
Heroin			96 (95.0)	1	4	0.375	--
Low risk	84 (83.2)	87 (86.1)
Mod OR high risk	17 (16.8)	14 (13.9)
Prescription stimulants			99 (98.0)	1	1	1.000	--
Low risk	97 (96.0)	97 (96.0)
Mod OR high risk	4 (4.0)	4 (4.0)
Inhalants			96 (95.0)	2	3	1.000	--
Low risk	96 (95.0)	97 (96.0)
Mod OR high risk	5 (5.0)	4 (4.0)
Methamphetamine			97 (96.0)	3	1	0.625	--
Low risk	96 (95.0)	94 (93.1)
Mod OR high risk	5 (5.0)	7 (6.9)
Other Drug			95 (94.1)	2	4	0.687	--
Low risk	94 (93.1)	96 (95.0)
Mod OR high risk	7 (6.9)	5 (5.0)

^aConcordant risk level = remained in same risk level between T1 and T2

^bIncreased risk level = went from low to moderate-high risk

^cDecreased risk level = went from moderate-high to low risk

^dP-value for McNemar’s test of correlated proportions, comparing the proportion of individuals having low risk versus moderate-high risk scores at T1 and T2. A non-significant P-value implies that there was no significant change in scores for the study population between T1 and T2.

^eCohen’s Kappa was calculated for all substances having 20% or greater prevalence of moderate-high risk use in the study population. All kappa values shown were statistically significant at the P<.001 level.

^fRisk level is based on highest risk score for any substance in this category

3.4. Correlation of ASSIST scores at Time 1 and Time 2

We examined correlation of ASSIST scores at Time 1 and Time 2 for the ASSIST global score (sum across all substances and all items), and for tobacco, alcohol and drug scores. As shown in Table 4, we observed excellent agreement between Time 1 and Time 2 scores for each of these substance categories using the ICC: global score ICC=0.968; tobacco score ICC=0.900; alcohol score ICC=0.904; and drug score ICC=0.969. The ICC for global score was similar with and without inclusion of the prescription stimulant and prescription opioid items (ICC=0.965 without these items, and ICC=0.968 with these items included). All correlations were significant at the P<0.001 level.

Table 4.

Correlation of ASSIST global, alcohol, and drug scores at Time 1 and Time 2, for the 101 subjects

	Time 1	Time 2	ICCT1T2*
Global score			0.968
Mean ± SD	47 ± 69	47 ± 71
Range	0–492	0–498
Tobacco score			0.900
Mean ± SD	6 ± 10	6 ± 10
Range	0–31	0–31
Alcohol score			0.904
Mean ± SD	8 ± 11	8 ± 10
Range	0–39	0–39
Drugs score			0.969
Mean ± SD	21 ± 50	21 ± 50
Range	0–390	0–390

^*ICC calculations used a single measurement, absolute agreement definition, two-way mixed model. All were significant at the P<0.001 level.

4. DISCUSSION

The ACASI ASSIST demonstrated overall good test-retest reliability and required on average only five minutes for primary care patients to complete. These findings are particularly important to consider in the context of our study population, which included a high proportion of individuals who might be anticipated to have difficulty with a self-administered computerized screening tool. Many of our participants were born outside the US, had Spanish as their primary language, or had low levels of formal education. Based on prior studies, these populations may be anticipated to have more difficulty with a computer self-administered instrument (Butler et al., 2001; Reichmann et al., 2010; Satre et al., 2008). We nonetheless found overall good test-retest reliability among participants, including among our subgroups of Spanish speakers, those with less than high school education, and individuals over 50 years old. We believe that the use of ACASI technology, together with a touchscreen computer, was important to making this self-administered instrument feasible in our study population.

We examined two types of test-retest reliability; concordance, and correlation. Prior to our study, only correlation (using Cohen’s Kappa or Pearson’s r) had been used to assess test-retest reliability of the ASSIST (Ali et al., 2002; Humeniuk, 2008). Because concordance assesses change in the classification of risk, which is what drives the subsequent clinical intervention, it is arguably the more relevant measure for this type of screening test (Bland & Altman, 1986). In the clinical setting it is more important to know, for example, whether the instrument is reliable in distinguishing between low risk use of cocaine (requiring little or no clinical intervention) and moderate-high risk use of cocaine (requiring brief intervention and potential treatment referral), than it is to know whether the substance specific risk score changed from a score of 5 to a score of 7.

We observed good to excellent correlation between ASSIST scores at the two administrations using the kappa statistic for low versus moderate-high risk use and the ICC for ASSIST scores, but were limited by being able to examine correlation only in the substance classes that had sufficient prevalence to reliably compute these statistics. Unfortunately, our measures of correlation for the test-retest reliability of the ACASI ASSIST cannot be directly compared to those derived for the interviewer-administered WHO ASSIST in an earlier test-retest reliability study, because that study used an earlier version of the ASSIST instrument (Ali et al., 2002).

The prevalence of substance use in our primary care population is higher than rates in the general population (SAMHSA, 2012). While rates of moderate-high risk use were much lower than rates of any use or low risk use in all substance classes, we still found that nearly half of our population had moderate-high risk use of drugs, indicating that they should receive a clinical intervention. This prevalence of moderate-high risk use was higher than that estimated in an earlier prevalence study using an interviewer-administered ASSIST in the same primary care clinic, perhaps due to differences in the recruitment strategy, sample, and mode of administration of the ASSIST (Lee, Delbanco, Wu, & Gourevitch, 2011). Nonetheless, high prevalence in our study was comparable to that found in other safety net adult primary care settings (Madras et al., 2009; R. Saitz et al., 2010; Smith et al., 2010). In the validation study of a single item screening question for drugs (Smith, 2010), for example, 35% of the primary care study population screened positive for past year drug use, while in our study 42% reported current drug use. We also found that rates of tobacco use were much higher among participants in our study than for the general NYC population (32% versus 14%) (New York City Department of Health and Mental Hygiene (DOHMH), 2011).

Notably, a screening approach that focused on alcohol use alone would have failed to identify a significant proportion of risk in our population. Among low-risk alcohol users, one-third had moderate-high risk drug use, which is likely to have gone undetected with an alcohol screening tool. These findings indicate that substance use screening in a safety net primary care population may identify a significant burden of clinically relevant use, and that using a screening tool that integrates screening for alcohol and drugs may be necessary to identify the patient population having risky substance use behaviors.

4.1. Limitations of the study

Our study has several limitations. Despite the overall high prevalence of substance use, given the small sample size we had limited ability to analyze results for some specific drug classes, including hallucinogens, prescription stimulants, methamphetamine, and inhalants, use of each of which was reported by fewer than 10 individuals. Our sample also included relatively few individuals with high-risk substance use, which limited our ability to look at the ACASI ASSIST’s test-retest reliability among those with the most severe substance use. Low prevalence is likely to be a limitation for any study conducted in a general primary care population. However, the drawbacks to testing an instrument in a relatively low prevalence population should be weighed against the benefits of testing an instrument in the population for which it is intended to be used – in this case, a general adult primary care population.

Generalizability of our findings is limited by having conducted the study at a single adult primary care clinic site. Our sample reflects the characteristics of a diverse urban safety net clinic population. While this may impact the generalizability of our findings to other clinical settings, it also represents the type of clinical population that may be a higher priority for substance use screening, due to relatively high prevalence of drug and alcohol use. Testing the ACASI ASSIST in this context could thus be interpreted as a strength of our approach.

Self-administered tools such as the ACASI ASSIST are likely to be less acceptable to individuals who have greater difficulty completing them, and this could have influenced our results through loss to follow-up. Those who failed to return at Time 2 may have been less comfortable using the ACASI ASSIST because they were Spanish speaking, although it was delivered in Spanish. A prior study of computer self-administered substance use screening for alcohol in primary care indicated poorer performance in Spanish versus English speakers, even when administered in the participant’s first language (Butler SF, 2003), although more recent clinical experience with this modality has demonstrated good feasibility (Murphy et al., 2013; Schackman et al., 2009). Though we did not detect significant differences in results for our Spanish speaking subgroup, further testing in this population may be indicated. Individuals with less than high school education levels also had lower rates of follow-up at Time 2 compared to the study population as a whole, though the difference in follow-up rates did not reach the level of statistical significance. It is possible that those with lower education level were less comfortable with the ACASI ASSIST, and for that reason failed to return. In the future, qualitative interviews could contribute to our understanding of how acceptable the ACASI ASSIST is for individuals with lower education or literacy levels.

In a test-retest study of an instrument that measures an impermanent characteristic such as substance use, it is possible that differences between Time 1 and Time 2 results reflect actual changes in use, which we were not able to evaluate in the present study. We chose a brief timeframe in order to reduce the likelihood that individuals would have significant changes in their actual substance use. Some have postulated that research assessments alone can produce changes in self-reported substance use (Bernstein, Bernstein, & Heeren, 2010; Humeniuk et al., 2012), and it is possible that reactivity to the research assessments at Time 1 could have influenced substance use behavior and reporting at Time 2. We tried to minimize this source of bias by limiting the assessments given at Time 1, but individuals enrolled during the first enrollment period did complete the interviewer-administered ASSIST (following their completion of the ACASI ASSIST) at Time 1. Yet even among these individuals, the ACASI ASSIST demonstrated good test-retest reliability. Given that the instrument relies on self-reported responses, there is also a risk that responses could be biased by individuals who remembered how they answered to items on the first administration, and tried to give the same answer on the second administration.

4.2. Conclusions

Our study indicates that a self-administered ACASI ASSIST has good test-retest reliability in a primary care population. The ACASI ASSIST has the potential to be a useful tool for conducting screening and a clinically-oriented structured assessment for unhealthy substance use in primary care. However, before recommending broad implementation of the ACASI ASSIST, its validity, especially as regards its sensitivity and specificity in detecting unhealthy substance use, must first be evaluated. This will require additional studies that are able to compare the ACASI ASSIST to reference standard measures.

Generally speaking, computer self-administered tools have promise for facilitating substance use screening, assessment, and even treatment interventions in time-pressured medical settings (Harris et al., 2012; Lotfipour et al., 2012; Murphy et al., 2013; Proctor & Hoffmann, 2012; Vaca, Winn, Anderson, Kim, & Arcila, 2010). Brief screeners such as the single-item drug screening question (Smith et al., 2010), may be adequate for identifying individuals with unhealthy drug use, but require further assessment to determine what drugs are being used, and at what level of risk, in order to guide the clinical intervention. The ASSIST has the advantage of providing this information, as well as integrating the screening and assessment process for tobacco, alcohol, and other drugs. Yet the interviewer-administered ASSIST is not feasible in most primary care environments because it requires an interviewer who can devote up to 15 minutes to its administration. A computer self-administered assessment could be completed on a tablet computer or kiosk in the waiting area, or even at home via an internet portal, prior to the medical visit, and have its results incorporated into the electronic health record at the point of care. A potential disadvantage of using a self-administered tool is that some patients may feel more comfortable responding to questions posed by an interviewer, and self-administration eliminates the possibility of developing of rapport with the interviewer during the screening process. However, this may be counterbalanced by the possibility that a self-administered instrument can encourage reporting of stigmatized behavior (Kim et al., 2008; Tourangeau & Smith, 1996; Wight et al., 2000) and has high fidelity, since it is not influenced by factors such as tone of voice, paraphrasing, or editorializing on the part of the person administering it. While self-administered questionnaires run the risk of being difficult for individuals with limited reading ability, ACASI technology has the potential to make these assessments feasible in diverse populations. This approach warrants further study in the context of implementing substance use screening in primary care settings.