Validation of a substance and alcohol use assessment instrument among orphans and vulnerable children in Zambia using Audio Computer Assisted Self-Interviewing (ACASI)

Abstract

Background

Substance and alcohol misuse is a global problem that increases the risk of HIV infection. This is a concern among orphans and vulnerable children (OVC) in sub-Saharan Africa who may have elevated substance use rates. The Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST) is a reliable and valid instrument of substance use among adults in primary care high-income settings. This study examined psychometric properties of the ASSIST among OVC in Zambia using Audio Computer Assisted Self-Interviewing (ACASI).

Methods

Baseline data from an ongoing randomized trial of interventions to reduce HIV risk behaviors were analyzed. The analysis included 502 OVC ages 13-17 living in low-income, high-density neighborhoods in Lusaka, Zambia. Internal consistency of the ASSIST was assessed and discriminant validity was measured using items from the Youth Self Report as criterion variables.

Results

Internal reliability was strong with a Cronbach's alpha of ≥0.80 for each of the specific substance scales and total substance involvement. For all substances except tobacco and sedatives, discriminant validity was demonstrated in distinguishing between low risk use and moderate use. Sensitivity and specificity analysis indicated adequate area under the curve across substance types (AUC range: 0.68 – 0.80). Discrimination between moderate and high risk was demonstrated for alcohol and total substance involvement.

Conclusions

ASSIST administered via ACASI is a reliable instrument and an appropriate tool for distinguishing between low and hazardous substance use among adolescent OVC populations in sub-Saharan Africa. Additional examination is warranted to determine its ability to measure gradations of severity within hazardous use.

1. INTRODUCTION

The burden associated with substance and alcohol use disorders is highest among adolescent and young adult populations (Degenhardt et al., 2013; Whiteford et al., 2013). There are few studies on substance use among adolescents from low and middle-income countries (LMIC), however, particularly in sub-Saharan Africa (UNODC, 2012). Recent reports suggest that rates of illicit substance use and hazardous drinking are high and may be increasing in Africa (Kalichman et al., 2007; Shield et al., 2013; UNODC, 2012; World Health Organization, 2004). Substance and alcohol use among orphans and vulnerable children (OVC; youth who are either single or double orphans and/or are at disproportionately high risk of poor outcomes compared to their peers) is of particular concern in Africa due to the link between substance use, including non-injection substances, and HIV risk, worse disease course for HIV, and other negative childhood outcomes (Kalichman et al., 2007; Morojele et al., 2006; Shuper et al., 2010; UNODC, 2012). Male orphans, in particular, may have an increased risk for substance abuse (Pufall et al., 2014). To accurately estimate the prevalence of substance and alcohol use disorders among youth and OVC populations in LMIC, screen for these problems in primary care centers, and evaluate interventions designed to manage and treat these problems, there is a need for locally validated instruments. There has been a call for increased psychometric testing of substance use instruments in these settings (Moodley et al., 2012), though few studies have attempted to validate substance use assessment instruments in sub-Saharan Africa.

The Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST) was originally developed by the World Health Organization (WHO) in 2001 as a tool for substance and alcohol use screening in primary care settings (WHO ASSIST Working Group, 2002). The ASSIST is unique among assessment instruments of substance use in that it includes items relevant for lifetime and past three month-use, degree of and associated problems with use, and the ability to detect high risk use and acute intoxication for several types of substances (Humeniuk et al., 2010). The original version was found to be feasible and have adequate test-retest reliability among adult populations in several geographic locations (WHO ASSIST Working Group, 2002). Among adults in both high (USA, UK, Australia) and low- and middle-income (Brazil, India, Thailand, Zimbabwe) settings, concurrent validity was indicated for the ASSIST by significant correlations with other substance use instruments including the Alcohol Use Disorders Identification Test (AUDIT) and the MINI International Neuropsychiatric Interview (MINI-Plus). Discriminant validity was indicated through its ability to differentiate use, abuse, and dependence as defined by independent clinical evaluation and MINI diagnoses (Humeniuk et al., 2008). Additional published reports of psychometric testing for the ASSIST in high and upper-middle income settings among adult populations reported good criterion validity of the ASSIST and high internal consistency (Henrique et al., 2004; Khan et al., 2012, 2011; Newcombe et al., 2005; Rubio Valldolid et al., 2014). One U.S.-based study with adults compared a version of the ASSIST administered through an Audio Computer Assisted Self-Interviewing (ACASI) system with a traditional interviewer method and found excellent concordance across the two modalities in ability to detect moderate to high risk use (McNeely et al., 2015).

Two recent studies, both conducted in high-income countries, were the first to validate the ASSIST with youth populations. Gryczynski et al. (2014) found good internal consistency for the tobacco (α=.87), alcohol (α=.72), and cannabis (α=.88) scales and concurrent validity with the CRAFFT screening tool, but the ASSIST was only able to discriminate between levels of severity for cannabis. Nichols et al. (2014), using an ACASI-based ASSIST, found high concordance of ASSIST-reported marijuana and tobacco with toxicology reports. To our knowledge, no substance use assessment instrument has been validated with a youth population classified as OVC in a low-income setting, and there are currently no published reports of the utility of the ASSIST using ACASI with this population.

The current study aims to evaluate the validity and reliability of the ASSIST (version 3.1) among a sample of OVC adolescents in Lusaka, Zambia administered via ACASI. The psychometric properties to be tested include internal reliability and discriminant validity.

2. METHODS

2.1 Participants and Procedure

The current study is a secondary analysis of data from the baseline visit of an ongoing randomized controlled trial in Lusaka, Zambia. The purpose of the trial is to test the effectiveness of two different interventions to reduce HIV risk behaviors and mental health symptoms among OVC adolescents. Participants enrolled between March, 2014 and December, 2015 were included in this analysis.

Participants were recruited from 16 low-income, high density neighborhoods (referred to as “compounds”) in Lusaka, the capital city of Zambia.. We first conducted general community meetings in these neighborhoods, in which research staff presented an overview of the study and informed community members of dates and times when assessments were taking place at the local parish (or other community site). Our primary method of recruitment was through collaboration with home-based care workers (HBCWs) operating in these 16 communities throughout Lusaka who were part of pre-existing community outreach efforts. The HBCWs had extensive connections throughout the communities, and their role was to visit families regularly to link them to services (e.g., nutrition, education). We trained 68 HBCWs and asked them to identify OVC (13-17 years old) under their care or in their communities who the HBCWs believed may have behaviors that increase HIV infection risk, , such as sexual risk behaviors and substance use. The HBCWs informed potentially eligible families of the study using a brief recruitment script. Families (the OVC and one of his/her caregivers) who were interested in participating in the study were referred to a study field site, typically a local parish located in the family's compound, on a specific day and time to meet with research staff, hear details about the study, provide informed consent, and complete a screening and baseline interview.

We also attempted to recruit through referral—by other outreach organizations in the communities, and by adolescent participants or their caregivers who referred neighbors, friends, or family members to the study who they believed would benefit from the services being offered.

After providing informed consent, both the adolescent and his/her caregiver completed a brief screener. If behaviors that increase the risk of HIV (e.g., sexual risk behaviors) were indicated on the screener through either self- or caregiver-report, and if the adolescent met PEPFAR criteria for an OVC (Office of the U.S. Global AIDS Coordinator, 2006), he/she was enrolled into the study and completed a larger baseline assessment. Data for the current analysis were obtained from those baseline assessments. All eligible adolescents (as of December, 2015) were included in this analysis. The study received ethical approval from the Johns Hopkins Bloomberg School of Public Health Institutional Review Board and the University of Zambia ethical review board.

2.2 Measures

Adolescents who met eligibility criteria were asked to complete the full study assessment, which was administered using a laptop-based ACASI developed by Tufts University (Tufts University School of Medicine, 2014). A systematic review by Langhaug and colleagues (2010) of 26 studies on sexual behavior in sub-Saharan Africa indicated that study participants were more likely to answer sensitive questions accurately when the interview was administered via ACASI compared to a face-to-face interviewer and that ACASI was feasible in LMIC settings. ACASI allows respondents to complete the questionnaire using a laptop computer and headphones, reading on their own and hearing a recorded voice speak the questions and response options. Participants were provided a brief training on how to use the laptops. Research staff were stationed nearby to answer questions and provide assistance, if necessary.

To build ACASI, the full interview was first translated from English to two local languages, Nyanja and Bemba, and then back-translated to English to ensure accuracy. Nearly all adolescents in Lusaka speak at least one of these three languages and the ability to do so was an inclusion criterion. All questions were then reviewed by community groups of adolescents from our target population for conceptual understanding and translation accuracy. For the audio component of ACASI, research staff in Zambia recorded the full interview in each of the three study languages, English, Bemba, and Nyanja. Study participants were able to complete the interview in the language of their choice.

The comprehensive baseline assessment included the following tools: 1) OVC Well-being Scale (Catholic Relief Services, 2008), 2) the World Aids Foundation Questionnaire (Sikkema et al., 2005), 3) the Child PTSD Symptom Scale (Foa et al., 2001), 4) a locally developed functioning instrument, 5) items assessing symptoms of traumatic grief, 6) a social support scale, 7) The Euroqol five-dimensional questionnaire on physical health status (Burström et al., 2014), 8) the ASSIST (Humeniuk et al., 2010), and 9) the Youth Self Report (YSR; Achenbach, 1991). This paper reports on the responses to the ASSIST and the YSR sections of the interview.

2.2.1 Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST), version 3.1

The ASSIST, version 3.1 (Humeniuk et al., 2010) consists of eight items asking study participants about their use and degree of involvement with: tobacco, alcohol, inhalants, cannabis, cocaine, amphetamines, sedatives, hallucinogens, injection drug use, and other substances. Although typically included in the ASSIST, opioids were not included in the present investigation. A summary of the ASSIST questions and response options is provided in Table 1.

Table 1.

Questions and response options included in the ASSIST, version 3.1

Question	Response options and associated score
1) In your life, which of the following substances have you used (non-medical use only)?	Yes (1) /No (0)
a. Item asked for: tobacco, alcohol, inhalants, cannabis, cocaine, amphetamines, sedatives, hallucinogens, other substance
2) In past three months, how often have you used the substances mentioned in Q1?	Never (0), once or twice (2), monthly (3), weekly (4), daily or almost daily (6)
a. Item asked for each substance endorsed in Q1.
3) During past three months, how often have you had an urge or strong desire to use?	Never (0), once or twice (3), monthly (4), weekly (5), daily or almost daily (6)
a. Item asked for each substance used in past three months
4) During past three months, how often has your use led to health, social, legal, or financial problems?	Never (0), once or twice (4), monthly (5), weekly (6), daily or almost daily (7)
a. Item asked for each substance used in past three months
5) During past three months, how often have you failed to do what was expected of you because of use?	Never (0), once or twice (5), monthly (6), weekly (7), daily or almost daily (8)
a. Item asked for each substance used in past three months
6) Has a friend or relative or anyone else ever expressed concern about your use?	No, never (0), Yes, in the past three months (6), Yes, but not in the past three months (3)
a. Item asked for each substance endorsed in Q1
7) Have you ever tried to cut down on using but failed?	No, never (0), Yes, in the past three months (6), Yes, but not in the past three months (3)
a. Item asked for each substance endorsed in Q1
8) Have you ever used any drug by injection?	No, never, Yes, in the past three months, Yes, but not in the past three months (no scoring for Q8)
a. Item asked for all participants who endorsed any substance in Q1

Specific substance involvement (SSI) score calculated from questions 2 through 7 and total substance involvement (TSI) score calculated from questions 1 through 8 for all substances

Following ASSIST scoring guidelines, among those who reported using a substance, a specific substance involvement score (SSI) was calculated for that substance by summing response options to ASSIST questions two through seven (SSI is not calculated for injection drug use). A total substance involvement (TSI) score was calculated by summing responses to questions one through eight for all substances.

2.2.2 Youth Self Report (YSR)

The YSR is a standardized 112-item assessment tool of depression and anxiety symptoms, school-based problems, anger and aggression, somatic complaints, and behavioral and school-based problems (Achenbach, 1991). Participants are asked to respond to each symptom or behavior in the YSR by indicating whether it was “not true” (0) about themselves, “somewhat or sometimes true” (1), or “very or often true” (2) within the past four weeks. It is the most widely used multidimensional measurement tool of child an adolescent behavioral health in the world and has been translated and tested among diverse populations, including adolescents in sub-Saharan Africa (e.g., de Groot et al., 1996; Ivanova et al., 2007; Klasen et al., 2010; O'Keefe et al., 2006 for a complete list of studies see: http://www.aseba.org/research/research.html). Exploratory and confirmatory factor analyses conducted with the YSR have indicated an eight-syndrome factor structure with the following descriptors: 1) anxious/depressed; 2) withdrawn/depressed; 3) somatic complaints; 4) social problems; 5) thought problems; 6) attention problems; 7) rule-breaking behavior; and 8) aggressive behavior (Achenbach and Rescorla, 2001; Achenbach, 1991). Iyanova et al. (2007) found that this 8-factor model was a good fit to data from 23 countries, including populations similar to our study in low- and middle-income countries.

The rule-breaking behavior sub-scale includes three items on substance use: “in the past 4 weeks, I have smoked tobacco,” “in the past 4 weeks, I drank alcohol without my parent's approval,” and “in the past 4 weeks, I have used drugs for nonmedical purposes.” In the present study, we were interested in assessing the ability of the ASSIST to accurately measure substance involvement specifically, not externalizing or ‘rule-breaking’ behaviors more generally. We also did not aim to specifically evaluate the sub-scales of the YSR. Therefore, only these three items from the YSR were used for analysis, serving in the role of criterion variables in discriminant validity testing (see Data Analysis section below). In a study of adolescents in the U.S., Branstetter and colleagues (2011) found that the substance use item from the YSR was highly correlated with a validated adolescent substance use measure, the Drug Involvement Scale for Adolescents (DISA; Eggert et al., 1996). In that study, the YSR item had a strong correlation with both the composite score from the DISA (r=.65, p<.001) and with negative outcomes following substance use (r=.58, p<.001).

2.3 Data Analysis

The planned sample size for the ongoing parent randomized trial is 750 (including accounting for loss to follow-up). Based on a type 1 and type 2 error rate of .05 and .20, respectively, this sample size provides statistical power to detect a risk ratio of 0.70 or a minimum difference of 11% between the study arms in the proportion of adolescents who report using a condom at their last sexual encounter. For the present psychometric study, we included all adolescents enrolled in the trial by December, 2015 (n=502). Based on our previous psychometric studies in Zambia with mental and behavioral health assessment instruments (Michalopoulos et al., 2015; Murray et al., 2011), and other studies evaluating the ASSIST with adolescent populations (Gryczynski et al., 2014; Nichols et al., 2014), we believed this sample size was adequate to detect moderate mean ASSIST score differences between those with and without substance use problems.

Cronbach's alpha coefficient, a measure of internal reliability (Cicchetti, 1994; Nunnally and Bernstein, 1994), was calculated for each specific substance scale (SSI) as well as for the total substance scale (TSI).

Discriminant validity of the ASSIST was examined comparing the extent to which the SSI score was differentiated across levels of a criterion variable known to measure substance use (i.e., the YSR questions listed above; Cicchetti, 1994). For each substance, we calculated mean SSI scores for each YSR response level (“not true”, “somewhat true”, and “very true”) on the corresponding YSR item: the tobacco YSR question for the tobacco SSI score, the alcohol YSR question for the alcohol SSI score, and the drugs for nonmedical purposes YSR question for all other SSI scores.

For each substance, we conducted a Kruskal-Wallis one-way analysis of variance test to determine if the SSI means differed significantly across the three YSR response levels. We also used a Mann Whitney U-test to determine which, if any, among the three pairwise comparisons were statistically significant. For these pairwise tests, an adjusted p value of 0.017 was used as the significance level to account for three multiple comparisons. The Kruskal-Wallis and Mann Whitney tests were used in place of the more traditional ANOVA and t tests because of non-normally distributed ASSIST data.

Finally, in order to describe the utility of the ASSIST among adolescent OVC populations in Zambia, we conducted receiver operating characteristic (ROC) analyses to test the performance of the scale by estimating area under the curve, sensitivity, and specificity, and associated cut-off scores for this population. These discriminative tests were conducted between low risk (“not true” on YSR) and moderate risk (“somewhat true” on YSR) use; and between moderate risk and high risk (“very true” on YSR) use. All analyses were conducted using Stata, version 13 (StataCorp, 2013).

3. RESULTS

A total of 502 adolescents ages 13-17 participated in baseline assessments between March, 2014 and December, 2015, including 230 males (45.8%) and 272 females (54.2%) (Table 2). Participants were, on average, 15 years old and the majority (75.1%) reported currently attending school. Over 70% of the sample was at least a single orphan (i.e., had a mother or a father who had died), and 24% reported that both parents had died. One hundred twenty-two participants (24.3%) reported HIV positive status and 36.5% reported having some kind of disability (i.e., visual or audial impairment or physical disability).

Table 2.

Characteristics of adolescent study sample (n=502)

	N (%)
Age, mean (SD)	15 (1.4)
Male	230 (45.8)
Currently in school	377 (75.1)
Days absent from school (past 6 months)
0 days	131 (34.8)
1-4 days	134 (34.5)
5-10 days	40 (10.6)
11 or more days	29 (7.7)
Not attended school in past 6 months	43 (11.4)
Primary caretaker
Mother	215 (43.0)
Father	46 (9.2)
Stepparent	25 (5.0)
Grandparent	116 (23.2)
Sibling	21 (4.2)
Other relative	21 (4.2)
Other person (non-relative)	25 (5.0)
No one	31 (6.2)
Orphan status
Non-orphan	138 (27.5)
Single orphan (father alive)	58 (11.6)
Single orphan (mother alive)	185 (36.9)
Double orphan	120 (24.0)
HIV status
HIV positive	122 (24.3)
HIV negative	329 (65.9)
Don't know	11 (2.2)
Biological parent HIV status
One or both HIV positive	151 (31.9)
Neither HIV positive	79 (85.87)
Don't know	11 (2.2)
Have a disability	183 (36.5)

Table 3 summarizes the number of participants who ever used each substance, the mean ASSIST SSI scores among those who ever used each substance, and internal consistency for each SSI score as measured by Cronbach's alpha coefficient. Overall, 54% of participants reported ever using any substance. Most commonly reported were alcohol (31.3%), inhalants (23;5%), tobacco (21.5%) and cannabis (19.1%). The least commonly reported substances were sedatives (11.2%), followed by amphetamines (13.0%), and injection drugs (13.3%).

Table 3.

Number who ever used, mean substance involvement scores, and internal reliability (n=502)

Substance	N (%) ever used	Mean (SD)	Cronbach's alphac
Tobacco	108 (21.5)	24.1 (18.0)	.80
Alcohol	157 (31.3)	15.0 (10.9)	.85
Inhalants	118 (23.5)	13.7 (10.5)	.84
Cannabis	96 (19.1)	16.3 (12.0)	.89
Cocaine	77 (15.3)	14.8 (11.0)	.85
Amphetamines	63 (13.0)	14.3 (10.9)	.86
Sedatives	56 (11.2)	15.1 (11.5)	.87
Hallucinogens	69 (13.8)	14.2 (10.9)	.84
Other substance	87 (17.3)	13.8 (10.7)	.83
Injection drug use	48 (13.3)	-	-
Total substance involvement	270 (53.8)	33.9 (61.8)	.98

Means and internal consistency calculated from ASSIST questions two through seven and only include participants who ever reported using the substance except for total substance involvement score, which is all questions 1-8 and includes all participants.

The highest SSI mean scores were for tobacco (24.1), sedatives (15.1), and cannabis (16.3). The lowest SSI mean scores were for other substance (13.8), inhalants (13.7), and hallucinogens (14.2) Cronbach's alpha coefficients were very good across all substance types and the TSI with a value of 0.80 or above for each.

Table 4 displays the results of the discriminant validity tests. For all substance types except for tobacco and sedatives there was a statistically significant difference in mean SSI scores across the levels of the YSR criterion. Pairwise tests presented in Table 5 suggested that difference in mean scores were significant only in the comparison of low risk (“not true” level of YSR) vs. moderate risk (“somewhat true” level of YSR) use except for alcohol and the TSI for which low vs. moderate risk and moderate vs. high risk (“very true” level of YSR) use were both statistically significant. Discrimination between low and moderate risk use was adequate for alcohol (AUC=0.71, sensitivity=69%, specificity=69%), inhalants (AUC=0.80, sensitivity=75%, specificity=75%), cannabis (AUC=0.76, sensitivity=65%, specificity=75%), cocaine (AUC=0.81, sensitivity=75%, specificity=72%), amphetamines (AUC=0.72, sensitivity=67%, specificity=71%), hallucinogens (AUC=0.77, sensitivity=73%, specificity=80%), and other substance (AUC=0.78, sensitivity=65%, specificity=72%). Discrimination between moderate and high risk use was poor (AUC<0.70) for all substances.

Table 4.

Discriminant validity testing of substance involvement scores using Youth Self Report criterion questions

	Youth Self Report Responsea
	Not true	Somewhat true	Very true
	ASSIST score mean (SD)			K-W χ2b
Tobacco (n=108)c	20.1 (16.9)	24.0 (18.9)	29.4 (17.8)	4.9
Alcohol (n=157)d	8.6 (9.5)	16.0 (10.3)	21.8 (8.6)	38.3*
Inhalants (n=118)	11.2 (9.2)	21.5 (8.6)	16.8 (12.5)	14.6*
Cannabis (n=96)	12.8 (10.6)	22.6 (10.7)	20 (13.6)	12.1*
Cocaine (n=77)	10.0 (9.9)	20.5 (7.5)	19.2 (11.7)	15.7*
Amphetamines (n=63)	9.6 (9.7)	18.1 (10.8)	18.2 (10.4)	10.1*
Sedatives (n=56)	11.5 (10.6)	18.1 (10.3)	17.4 (13.0)	3.7
Hallucinogens (n=69)	9.9 (9.7)	19.7 (9.7)	18.2 (11.4)	12.2*
Other substance (n=87)	9.6 (9.4)	19.9 (9.4)	17.7 (10.8)	15.9*
Total substance involvement (n=502)	12.2 (33.9)	39.8 (54.2)	76.7 (87.7)	95.4*

^*P<.05

^aYouth Self Report criterion question for all substances except tobacco and alcohol: In the past 4 weeks, I have used drugs for nonmedical purposes

^bKruskal-Wallis test of one-way analysis of ranks for non-normal data.

^cTobacco Youth Self Report criterion question: In the past 4 weeks, I have smoked tobacco

^dAlcohol: Youth Self Report criterion question: In the past 4 weeks, I drank alcohol without my parent's permission

Table 5.

Discriminative validity results of the ASSIST using receiver operating characteristic analysis and pairwise Mann Whitney U-tests

Substance	AUC (95% CI)	ROC Sens (%)	ROC Spec (%)	% Correctly classified	ASSIST cut-off	Z a
Tobacco
Low (n=44) vs. moderate risk use (n=30)	0.56 (0.43, 0.70)	56.7	56.8	56.8	24	−0.9
Moderate (n=30) vs. high risk use (n=34)	0.57 (0.43, 0.51)	55.9	50.0	53.1	26	−1.0
Alcohol
Low (n=56) vs. moderate risk use (n=58)	0.71 (0.61, 0.80)	69.0	69.0	69.0	11	−3.9*
Moderate (n=58) vs. high risk use (n=43)	0.67 (0.56, 0.77)	60.5	50.0	54.5	19	−2.9*
Inhalants
Low (n=89) vs. moderate risk use (n=23)	0.80 (0.67, 0.92)	75.0	74.7	74.7	19	−3.7*
Moderate (n=16) vs. high risk use (n=23)	0.40 (0.22, 0.58)	43.5	43.8	43.6	22	1.04
Cannabis
Low (n=55) vs. moderate risk use (n=17)	0.76 (0.61, 0.90)	64.7	74.6	72.2	21	−3.2*
moderate (n=17) vs. high risk use (n=23)	0.46 (0.28, 0.65)	47.8	41.2	45.0	25	0.4
Cocaine
Low (n=39) vs. moderate risk use (n=16)	0.81 (0.69, 0.93)	75.0	71.8	72.7	18	−3.6*
moderate (n=16) vs. high risk use (n=21)	0.44 (0.25, 0.64)	42.9	43.8	43.2	21	0.6
Amphetamines
Low (n=28) vs. moderate risk use (n=15)	0.72 (0.56, 0.89)	66.7	71.4	69.8	15	−2.4*
moderate (n=15) vs. high risk use (n=20)	0.49 (0.29, 0.69)	45.0	46.7	45.7	20	0.1
Sedatives
Low (n=23) vs. moderate risk use (n=16)	0.68 (0.49, 0.86)	68.8	69.6	69.2	16	−1.8
Moderate (n=16) vs. high risk use (n=17)	0.50 (0.29, 0.71)	47.1	50.0	48.5	21	0.0
Hallucinogens
Low (n=35) vs. moderate risk use (n=15)	0.77 (0.62, 0.92)	73.3	80.0	78.0	20	−3.1*
Moderate (n=15) vs. severe use (n=18)	0.45 (0.24, 0.65)	38.9	40.0	39.4	21	0.5
Other substance
Low (n=46) vs. moderate risk use (n=17)	0.78 (0.65, 0.91)	64.7	71.7	69.8	15	−3.4*
Moderate (n=17) vs. high risk use (n=23)	0.42 (0.24, 0.61)	39.1	41.2	40.0	21	0.8
Total substance involvement
Low (n=271) vs. moderate risk use (n=106)	0.69 (0.63, 0.75)	67.9	62.7	64.2	1	−6.3*
Moderate (n=106) vs. high risk use (n=123)	0.62 (0.55, 0.69)	58.5	59.4	59.0	24	−3.2*

^*P<.017

AUC: Area under the curve; ROC: Receiver operating characteristics; Sens: Sensitivity; Spec: Specificity

Low use corresponds to “not true” on Youth Self Report; Moderate use corresponds to “somewhat true” on Youth Self Report; “Severe use” corresponds to “very true” on Youth Self Report.

^aMann-Whitney U test (z statistic) for pairwise comparisons is analogous to a t-test for non-normally distributed data. For all pairwise comparisons, Bonferonni correction was used to calculate P values.

We conducted a sensitivity analysis to assess any differences across participants based on how they were recruited into the study. There were no statistically significant differences in mean ASSIST scores or across the three YSR criterion variables based on recruitment method.

4. DISCUSSION

This study investigated the reliability and validity of the ASSIST as delivered through ACASI in identifying substance involvement among an OVC population in Lusaka, Zambia. The three primary findings of the study were: 1) the percentage of adolescents reporting substance use was very high with 54% reporting lifetime use of at least one substance type; 2) all substance specific scales and the total substance involvement scale displayed good to excellent internal reliability; and 3) discriminant validity was indicated for most substance types in differentiating between low and moderate risk substance involvement but not for any substances in moderate vs. high risk involvement.

Substance use involvement was high in this study, particularly for alcohol use, inhalants, cocaine, and injection drugs. Reported use of tobacco and alcohol and mean SSI scores for tobacco, alcohol, and cannabis were higher in this sample than in a validity study of the ASSIST among a sample of youth in an urban health center the U.S. (Gryczynski et al., 2014), but reported use was lower than a second U.S.-based ASSIST validity study, which included a sample of HIV-infected youth (Nichols et al., 2014). The high number of adolescents reporting use in our study is a concern given studies suggesting that there is a strong association between substance use and HIV risk as well as other adverse consequences among youth (Kalichman et al., 2007; Morojele et al., 2006; Shuper et al., 2010; UNODC, 2012). Of particular concern: our study found that over 13% of adolescents had used injection drugs, 46% of whom had used in the past three months. Injection drug use can cause direct transmission of HIV through sharing of needles, and use of other substances increases odds of infection by leading to risky sexual behaviors and occurrence of transactional sex (Mataure et al., 2002; Meghdadpour et al., 2012; Palen et al., 2006). A recent study conducted in South Africa among orphans found that perceived vulnerability to HIV was associated with significantly increased odds of substance and alcohol use (Meghdadpour et al., 2012). Our investigation similarly suggests that OVC in Zambia are at significant risk for substance involvement and the adverse consequences associated with it, such as HIV, and highlights the importance of validating assessment tools of substance use for this population.

We found that the ASSIST had excellent internal reliability among this OVC sample. Cronbach's alpha coefficients were at or above 0.80 for all SSI scales and the TSI scale. These results are in line with previous ASSIST validation studies. In an investigation of the ASSIST with adolescents, Gryczynski et al. (2014) reported high internal consistency for past year substance users of 0.78, 0.68, and 0.81, respectively, for tobacco, alcohol, and cannabis. Studies of the ASSIST with adult populations also found high internal reliability, with alpha coefficients ranging from 0.74 to 0.95 (Humeniuk et al., 2008; Khan et al., 2011; Rubio Valldolid et al., 2014).

The results of our discriminant validity analysis suggest that the ASSIST has good ability to differentiate between low risk of hazardous substance use and moderate risk across all substances except tobacco and sedatives. Additionally, for alcohol and the TSI, the ASSIST accurately discriminated between moderate and high risk use. For all other substances, however, there was no statistical difference in mean scores between moderate and high risk use. In one of the only other validation studies of adolescents, Gryczynski et al. (2014) found ASSIST SSI scores for tobacco, alcohol, and cannabis (no other substances were included in that study) were significantly higher among those meeting DSM-V criteria for substance use disorder compared to those not meeting those criteria, however, the ASSIST was able to adequately distinguish severity of use only for cannabis (Gryczynski et al., 2014). Taken together with our findings, this suggests that the ASSIST may be very useful in identifying hazardous use from non-hazardous use among youth populations, but its ability to differentiate levels of severity within hazardous use may be limited. This is in contrast to studies with adult populations in which the ASSIST displayed the ability to distinguish between severity of use, including between use, abuse, and dependence (Humeniuk et al., 2008; Khan et al., 2011; Newcombe et al., 2005; Rubio Valldolid et al., 2014).

Although the clinical cut-off for moderate alcohol risk in our study matched the ASSIST recommended cut-off score exactly (score of 11 or higher), the optimal clinical cut-points differentiating low and moderate risk in our study were otherwise very high relative to the currently recommended ASSIST cut-off scores (Humeniuk et al., 2010). The recommended score for all other substances for moderate risk is 4; our cut-off scores ranged from 15 to 21 across substances, which are in many cases closer to the ASSIST recommended cut-off for high risk (27 or above). The cut-off scores in our study are especially high given that the ASSIST recommended scores are for adults. The validity study by Gryczynski et al (2014) with youth found an optimal cut-off score of ≥ 2 for alcohol, cannabis, and tobacco. The high cut-offs among our study population may indicate an overall greater risk for and/or severity of substance use problems than the general population. Therefore, those who scored above the cut-off in our study may benefit from more intensive intervention. Additionally, given the relatively high SSI means in the “low” risk group compared to other studies, these individuals may also benefit from services, potentially taking the form of brief interventions.

4.1 Limitations

There were several limitations of our study to consider. Our study interview did not include several of the criterion assessments (e.g., AUDIT, MINI) used in other ASSIST validation studies, limiting our ability to directly compare our results with them. Although the YSR contains items on substance use, it is not a “gold standard” substance use criterion. Replication of our findings with substance and alcohol specific assessment tools would be useful, particularly in testing the ability of the ASSIST in discriminating between moderate and high risk of use. Second, although we believe the use of the ACASI-administered version of the ASSIST was likely to result in more truthful responses from participants based on prior research (Langhaug et al., 2009; Mensch et al., 2003; Nichols et al., 2014), we did not have an interviewer-administered comparison group to test that hypothesis in our study. Third, we presented “optimal” cut-points for this study, in which sensitivity and specificity were maximized. There may be cases in screening primary care centers where greater sensitivity (resulting in fewer false negatives) would be more advantageous. Finally, it is important to note that this study tested the adult version of the ASSIST. There has been a draft version of the ASSIST for youth under development (although not published), which will potentially include revised cut-off score recommendations, revised items, and wording (Gryczynski et al., 2014; Humeniuk et al., 2011).

4.2 Conclusion

To our knowledge, this was one of the first validation studies of a substance use assessment instrument among OVC in sub-Saharan Africa. Our results suggest that the ASSIST, a tool originally developed for use among adult populations in primary care settings, is a reliable and valid instrument for identifying hazardous substance and alcohol use among OVC youth as implemented with an ACASI system. In high-income countries, ACASI-administered ASSIST appears to be a valid and feasible method for assessing substance use in primary care clinic settings (McNeely, 2015). Although our study found that the ASSIST is a valid tool in Zambia, it was tested in a field-based research setting. Additional investigation is necessary to determine if this method is currently feasible in LMIC primary care settings, given that most primary care clinics do not have electronic health record systems or available resources for laptops or tablets. Future studies should include more refined substance and alcohol use criteria, which will be useful in providing information on the utility of the ASSIST in distinguishing between gradations of hazardous use. The findings of high use suggest that more work is needed on treatment and prevention of substance use in adolescent populations in lower resource settings.

Highlights.

• Prevalence of substance and alcohol use was high among OVC in Zambia

• The ASSIST scale displayed excellent internal reliability for all substance types

• ASSIST had strong ability to discriminate between low and hazardous substance use

• ASSIST had poor ability to discriminate between moderate and severe substance use