Evaluation of the AC-OK mental health and substance abuse screening measure in an international sample of latino immigrants

Abstract

Background

Early detection and intervention in primary care is integral to behavioral health. Valid, practical screening assessments are scarce, particularly for non-English speaking populations. We address this need by evaluating the psychometric properties of the Spanish version of the AC-OK Screen for Co-occurring Disorders for first and second generation immigrant Latinos in Massachusetts, USA, and Madrid and Barcelona, Spain.

Methods

567 Latino participants were recruited in waiting areas of clinics or by referral from Primary Care, Mental Health, Substance Use, and HIV Treatment Clinics, as well as Community Agencies. We use confirmatory factor analyses to evaluate the factor structure, correlation analysis to examine concurrent and discriminant validity, and receiver operating curves (ROC) to determine the ability of the AC-OK to approximate a composite of established instruments designed to measure depression, generalized anxiety, post-traumatic stress disorder symptoms, alcohol, and drugs (including benzodiazepines) as external criteria.

Results

The original two factor structure was replicated in samples of Latino respondents in the US and Spain. Correlations with other measures followed the expected pattern. In both the US and Spain, ROC analyses suggested that the AC-OK scale was an adequate approximation to other specific measures of mental health (ROC=.90) and substance abuse problems (ROC=.83).

Conclusions

The Spanish version of the AC-OK Screen has good to excellent psychometric properties in both its subscales. These findings are robust across sites, gender, and type of clinic. We recommend its use for clinical research and for routine screening at treatment centers.

1. Introduction

Co-occurring disorders of addiction and mental health, referred to as dual diagnosis, are common among healthcare service users (Sacks et al., 2013). Approximately 8.9 million U.S. adults have co-occurring addiction and mental health disorders (Cherry and Dillon, 2012). Yet only 7.4% of them receive treatment for both conditions (Sacks et al., 2013). The prevalence of co-occurring substance use and mental health problems (sub-diagnostic, defined as drug or alcohol use and elevated mental health symptoms which interfere with functioning and/or result in social, work and/or legal difficulty) is substantially higher (Fein et al., 2007; Grant et al., 2004; Rohde et al., 2001; Saitz et al., 2010). If unrecognized, the individual and societal cost of these co-occurring conditions can be physically and economically damaging (Burns and Teesson, 2002; Kushner et al., 2000; Lasser et al., 2000). In the United States, 7.9 million adults have co-occurring mental health and substance use disorders (Hedden, 2015). We also know that nearly one third of people with any mental illness and approximately one-half of people with severe mental illness additionally experience substance abuse (National Alliance on Mental Illness, 2013). At a rate of 5.8%, Latinos have the second highest rate of lifetime co-occurring psychiatric illness and substance abuse disorders after Whites (Mericle et al., 2012). For Spain, to our knowledge, no specific prevalence rates of co-occurring disorders among Latinos have been published, but a recent report by the European Monitoring Center for Drugs and Drug Addiction (EMCDDA) lists prevalence rates of co-occurring disorders found across several studies. The prevalence of co-occurring disorders was found to be 21% in a general population study, and ranged from 13 to 59% among drug users in treatment and 18 to 67% among drug users not in treatment (Torrens et al., 2015).

People who have a dual diagnosis tend to enter treatment struggling with suicidal ideation, and are more likely to attempt suicide and to die from their suicide attempts (National Alliance on Mental Illness, 2013). They are also at increased risk of impulsive and violent behaviors and are therefore slightly more likely to be involved in the criminal justice system, and have more problems with substance abuse than others entering treatment solely for addiction (Cherry et al., 2008; National Alliance on Mental Illness, 2013).

Although Latino adults have one-fourth the risk of dual diagnoses compared to the general U.S. population (Vega et al., 2009), Latino immigrants show an increased risk of developing mental illness, substance use, and co-morbid psychiatric illness and substance use after immigration to the U.S (Borges et al., 2011; Breslau et al., 2011; Jiménez-Castro et al., 2010). Furthermore, U.S. nativity seems to increase the likelihood of reporting a dual diagnosis among Latinos (Vega et al., 2009). There are also significant racial/ethnic disparities in accessing behavioral health treatment among people with co-occurring disorders, with Whites more likely to receive and be referred to treatment than other racial/ethnic groups, including Latinos (Priester et al., 2016). Latino immigrants in both the United States and Spain also cite significant barriers to accessing behavioral health care including being unfamiliar with available services and wanting to resolve the problem by themselves and location-specific issues of cost and linguistic barriers (Falgas et al., 2017).

While both Spain and the United States offer competitive employment and better wages, there is variation in the composition of Latino immigrants that are more likely to immigrate to the U.S. versus Spain (Connor and Massey, 2011). Because of its geographic proximity and thus lower costs of immigration, it is more common for Latino immigrants from Mexico, Central America, and the Caribbean to move to the U.S. (Connor and Massey, 2011). In fact, 53% of all U.S. migrants are Latino (Acosta and De la Cruz, 2011). Data from the 2011 U.S. Census indicates that Mexicans account for more than half (55%) of the foreign born Latinos, followed by El Salvador and Cuba in the Caribbean (Acosta and De la Cruz, 2011). The nature of migration and immigration to the U.S. may also vary by country of origin (Torres and Wallace, 2013). For Mexican immigrants or Puerto Rican migrants, involuntary or unplanned migration may be more likely to result from economic circumstances or family obligation (Ellis et al., 1996), while for Cubans and other Central and South Americans, involuntary immigration may be more likely to result from political conflict or a combination of political and economic reasons (Cislo et al., 2010). In contrast, Latino immigrants from South America—more distant from the U.S.—may be more likely to move to Spain because of a comparatively easier process of social integration (Connor and Massey, 2011). Out of all the countries in the European Union(Padilla and Peixoto, 2007), Spain has the largest number of Latino migrants, with Latinos from the South, Central America and the Caribbean representing 28% of all migrants (Padilla and Peixoto, 2007). Demographic data from 2014 shows that in Spain, the largest numbers of Latin American immigrants are from Ecuador, followed by Colombia and Bolivia (Arroyo-Perez et al., 2014). Altogether, both countries have different composition of Latinos, but the shared magnitude of the migratory populations suggests there is a need to address certain challenges, in our case in terms of screening for dual disorders. Having available a Spanish language instrument with good psychometric properties will assuredly prove beneficial for all Spanish speaking populations.

The principal barriers to universal screening for dual disorders are the lack of availability of trained staff and of reliable and valid screening instruments (Cherry et al., 2008). These barriers are exacerbated when aiming to screen non-English speaking populations (Martinez, 2010). The requirement to allocate trained staff to screening can be facilitated if the screening measure is simple and easy to understand and complete. However, there are few such screening measures for dual diagnosis with demonstrated adequate psychometric properties.

Our search for available screening instruments identified that instruments designed for a comprehensive dual diagnosis assessment are extensive and therefore time consuming, and most entail specialized clinical skills and extensive training to administer such as the Composite International Diagnostic Interview (CIDI) (World Health Organization, 1993) or the Comprehensive Addictions and Psychological Evaluation (CAAPE) (Hoffmann, 2000). Other instruments focus only on one disorder (mental health or substance abuse) such as the Fagerström Test for Nicotine Dependence (FTND) (Heatherton et al., 1991) or the Patient Health Questionnaire 9 (PHQ-9) (Kroenke et al., 2001) for depression. And still others such as the CAGE-AID (Brown and Rounds, 1994), designed to identify people with a mental health disorder that have a substance use problem, only collect information on lifetime and not current use. As a result, we chose the AC-OK Screen (AC - Andrew Cherry and OK - Oklahoma) (Cherry et al., 2008) for the evaluation of co-occurring disorders because it was easy to interpret, did not require much training to administer and had good psychometric properties in English-speaking populations (Cherry and Dillon, 2012). It was designed to cast a wide net to be a useful screener in mental health and substance abuse treatment agencies. It seeks to identify people that may have co-occurring problems (to have high sensitivity), and to identify people who do not need a comprehensive co-occurring disorder assessment because they lack a dual disorder (specificity).

This paper evaluates the Spanish translation and adaptation of the AC-OK for use with Latino migrants in Spain and the United States. Psychometric properties are compared to the original English version. Further, it analyzes receiver operating curves for predicting mental disorder and substance abuse using various cut-off points on the AC-OK, with a range of concurrent screener measures as the validity criterion.

2. Methods

2.1 Setting and Study Sample-Screener

We recruited participants through direct contact in mental health, substance use, primary care, and HIV clinic waiting rooms, as well as community agencies in Boston, Massachusetts (MA), and Madrid and Barcelona, Spain. We also expanded our efforts to include community-based organizations (i.e., organizations providing social or cultural services for immigrants) serving a diverse population of Latinos. Recruitment activities were conducted between July 2013 and August 2014. Approval was obtained from the institutional review boards of participating institutions.

2.2 Research Procedures

We trained research staff to conduct the research protocol using standardized interviewing techniques and conducted mock interviews before entering the field. To ensure consistency across sites quality control procedures were implemented and included a quality check of the first 3 cases of all new interviewers and a continued randomized 15% of additional cases was implemented. Once certified, interviewers recruited patients in waiting areas of clinics, by referral in community groups, or by telephone screening of people in clinics who had consented to learn about the study. Participants were usually screened in the clinic, in a private setting. In cases where some interviews or portions of interviews were unable to be completed in person, they were later completed over the phone. Participation in this study did not interfere with the treatment received.

After obtaining informed consent, research staff conducted a 1.5 hour interview that included a series of instruments designed mental health disorders, use of substances and HIV risk behaviors, as well as socio-demographics, cultural, contextual and social factors. This interview was part of an assessment protocol for The International Latino Research Partnership (ILRP), a project funded by the National Institute on Drug Abuse to conduct cross-national comparative research in the U.S. and Spain. Its goals were to assess and improve Latino migrants’ behavioral health needs as part of the delivery of a targeted clinical intervention. Participants were eligible is they were 18 years old or older, and self-identified as being first or second generation Latino (from any Spanish-speaking Caribbean, Central or South American country). Interviews were audio recorded and administered using a tablet with CommCare technology (Chatfield et al., 2013). Participants were compensated with a $40/30€ gift card.

2.3 Measures

2.3.1 AC-OK Screen

The English version of the AC-OK (Cherry and Dillon, 2012) includes 15 items drawn from well-established assessment questionnaires in mental health and substance abuse. Nine items make up the mental health subscale and six items make up the substance abuse subscale. It has good reliability (Mental Health Screen [α = .79]; Substance Abuse Screen [α =.89]), good convergent validity, excellent sensitivity and sufficient specificity to be highly useful in screening for dual disorders in behavioral health settings (Cherry and Dillon, 2012).

We translated and adapted the AC-OK Screen to Spanish using established methodology (Matías-Carrelo et al., 2003). Our goal was to achieve a linguistic version for use at all sites and with semantic, content and technical equivalence to the original. We convened a Multinational Bilingual Committee (MNBC) with representatives from each site who met in person to discuss the equivalence of the measure post translation. The MNBC made the final decisions regarding each item through consensus between the members. The complete survey for the study, including additional instruments, was piloted to ensure necessary adjustments. The final AC-OK screen includes some items in the English version that were slightly modified from their original version to accommodate conceptual equivalence. In all cases the items remain comparable, if not identical to the original items. Items in English and Spanish, as well as means and standard deviations for each item by site, are available as supplementary materials¹. Item means ranged from .13 to .71 in Massachusetts and from .14 to .80 in Spain, with item 1 receiving the highest endorsement in both sites.

2.3.2 Instruments used for Specific Disorders

We chose a series of Spanish standardized measures for their psychometric properties, brevity and because many were translated and/or adapted for use among Spanish-speaking populations (Alegria et al., 2004; Canino et al., 2008). Depression was assessed using the Patient Health Questionnaire (PHQ-9), with nine criteria upon which a major depressive disorder diagnosis is based using DSM-IV criteria (Kroenke et al., 2001). Anxiety was assessed using the General Anxiety Disorder 7-item screener (GAD-7), a brief clinical measure for generalized anxiety (Spitzer et al., 2006). To assess for trauma, we used the Brief Trauma Questionnaire, a short self-report measure that examines respondents’ experiences with potentially traumatic events (Schnurr et al., 1999), as well as the Post-Traumatic Stress Disorder Checklist, a self-report measure of the 17 DSM-IV symptoms of PTSD (Blanchard et al., 1996). For Substance Abuse, we used the Alcohol Use Disorders Identification Test (AUDIT), a World Health Organization (WHO) screener for excessive drinking (Babor et al., 2001); the Drug Abuse Screening Test (DAST), a 10-item, yes/no self-report instrument designed for clinical screening of substance use (Yudko et al., 2007); and a selection of 10 items from the Benzodiazepine Dependence Questionnaire (BDEPQ), designed to measure misuse/dependence to benzodiazepine tranquilizers, sedatives and hypnotics (Baillie and Mattick, 1996). See Table 1.

Table 1.

Instruments for Specific Disorders

Depression: Patient Health Questionnaire 9 (PHQ-9)19	Nine criteria upon which depression diagnosis is based, using DSM-IV criteria for major depressive disorder. Internal consistency was .79 for Latinos and .80 for non-Latino whites. Spanish version has good agreement with independent mental health professional Dx (k = 0.74; overall accuracy, 88%; sensitivity, 87%; specificity, 88%).
Generalized Anxiety Disorder: 7 item scale (GAD-7)20	Brief clinical measure for the assessment of GAD; In Spanish a cut-off point of 10 showed adequate values of sensitivity (86.8%) and specificity (93.4%); AUC statistically significant [AUC = 0.957–0.985; p < 0.001]; Cronbach’s alpha=(0.94).
Trauma: PTSD (PCL-C) Checklist22	PTSD Checklist (PCL): item self-report measure of the 17 DSM-IV symptoms of PTSD. In test of Spanish version, no bias observed at level of composite PCL-C scale score.28
Alcohol Use Disorders Identification Test (AUDIT)23	Screening tool developed by the World Health Organization to identify the preliminary signs of hazardous drinking and mild dependence. Using test developmental sample, sensitivities for the AUDIT were in the mid 0.90’s and never below the high 0.80’s. Specificities across countries and across criteria averaged in the 0.80’s.29
DAST-1024	10-item, yes/no self-report instrument designed for clinical screening of substance use. Sensitivity ranged from 95% to 41%, and specificity ranged from 68% to 99%, using cutoff scores from 1/2 to 3/4.[30] The lowest sensitivity value of 41% was achieved with psychiatric patients in India based on discharge diagnosis.24
BDEPQ Benzodiazepine Dependence Questionnaire25	30 item self-report questionnaire designed to measure dependence on benzodiazepine tranquilizers, sedatives and hypnotics. Items cover all aspects of the dependence syndrome with the exception of withdrawal symptoms. Each item is rated on a four point Likert scale referring to experiences in the last month. In predictive validity analyses, ROC curves indicate that the AUC for the BDEPQ was.74 with a standard error of .07.

We imputed DAST scores for a subset of patients (less than 19% missing, n=109), who did not receive the full DAST module due to a skip pattern in the questionnaire that was administered in both sites when people said they did not use drugs in the last 30 days. Using the MI procedure in Stata (StataCorp LP, 2011) we created twenty complete datasets, and imputed missing values using a chained equations approach(Allison, 2002). Our analyses are based on one randomly selected dataset out of these twenty imputed datasets.

We also included questions to record race and ethnicity, education, age, gender, economic status, time living in U.S./Spain, number of visits to the home country, and language proficiency. Clinic type included whether the participant was recruited from a mental health, substance use, primary care or HIV clinic, or from a community agency site.

2.4 Statistical Analyses

We used Confirmatory Factor Analysis (CFA) to test model fit of a two factor model (Cherry and Dillon, 2012). We carried out the CFA analysis using Mplus (Version 7.3) with weighted least squares means and variance adjusted (WLSMV) method. Model fit was assessed using the comparative fit index (CFI), Tucker-Lewis index (TLI) and Root Mean Square Error of Approximation (RMSEA). For CFI and TLI indices, excellent model fit is suggested by values greater than or equal to .95 while for RMSEA, excellent model fit is suggested by values less than or equal to 0.06 (Brown, 2006).

We examined concurrent validity by calculating correlations between the two factors in the AC-OK and various criterion measures of MH and SA. We defined the union of three screening instruments for mental health (PHQ-9, GAD-7 and the PCL) as our comparison criteria to the AC-OK MH subscale, and of three screening instruments for substance use (AUDIT, DAST and BDEPQ) to compare to the AC-OK SA subscale. The aggregate of MH symptoms was defined as: being positive on scores of 10 or more on the PHQ-9, 10 or more the GAD-7; or 30 or more on the PCL. The aggregate of SA symptoms was defined as: being positive on scores of 8 or more on the AUDIT, 3 or more on the DAST or 7 or more on the Benzodiazepine Questionnaire. Published cut-offs scores were used (Allen et al., 1997; Babor et al., 2001; Baillie and Mattick, 1996; Blanchard et al., 1996; Kroenke et al., 2001; Orlando and Marshall, 2002; Spitzer et al., 2006; Yudko et al., 2007).

To compare the sensitivity and specificity of the brief AC-OK with regard to these inclusive criteria and to consider the implications of predicting need for MH and/or SA services, we calculated receiver operating curves (ROC) using various cutpoints on the AC-OK. The ROC analysis plots a curve of the sensitivity and 1 minus specificity of different scores on the AC-OK measured against external criteria (Murphy et al., 1987; Rey et al., 1992). The area under the curve (AUC) provides a measure of the magnitude of the association between the AC-OK and the external criterion described above. The points plotted on the curve also provided information about the trade-off of sensitivity to specificity for possible cutoff scores for each subscale of AC-OK. ROC analyses were conducted for the aggregate MH and SU criteria and also for each measure individually.

3. Results

3.1 Confirmatory Factor Analysis (CFA)

The two-factor model proposed by the developers of the AC-OK was found to fit the data well. Items loading in the MH subscale ranged from .49 to .93 and items loading in the SA subscale had loadings from .84 to .97. The goodness of fit statistics for this two-factor model were CFI=0.99, TFI=0.99 and RMSEA=0.05 in MA and CFI=0.98, TFI= 0.97 and RMSEA=0.06 in Spain. The CFA model allowed the factors to be correlated. The estimate of the correlation was 0.75 in MA and 0.73 in Spain. See Table 3.

Table 3.

Confirmatory Factor Analysis (CFA) using weighted least squares means and variance adjusted (WLSMV) method for the Spanish AC-OK Screen (N=567)

	Massachusetts		Spain
Items-Scale	Factor 1	Factor 2	Factor 1	Factor 2
OK1-MH		0.93		0.85
OK2-MH		0.81		0.88
OK3-SA	0.94		0.93
OK4-SA	0.97		0.89
OK5-SA	0.96		0.96
OK6-SA	0.97		0.91
OK7-SA	0.93		0.84
OK8-SA	0.94		0.94
OK9-MH		0.76		0.85
OK10-MH		0.81		0.81
OK11-MH		0.75		0.76
OK12-MH		0.74		0.73
OK13-MH		0.76		0.68
OK14-MH		0.68		0.49
OK15-MH		0.86		0.76

Note: In MA, CFI=0.99, TFI=0.99 & RMSEA=0.05. In Spain CFI=0.98, TFI= 0.97 & RMSEA=0.06.

A total of 567 participants completed the study. Our U.S. sample had the following composition: Central America (40%), US-native born to Latino parents (22%), Puerto Rico (16%), the Caribbean (11%), South America (10%) and Spain (1%). In contrast, the majority of the participants in Spain were of South American origin (86% in Madrid and 80% in Barcelona), followed by Caribbean (12%) in Madrid and Central America (10%) in Barcelona. In Spain 100% of the sample was foreign born and in the US 78% were foreign born.

In Massachusetts (MA), participants were older (mean age 44, SD=13.5 vs. mean age 40, SD=11.4 Spain), with a predominance of females in both sites (N=351, 62%), and a wide age range (18 to 82 years). The two samples were comparable in economic status, race, citizenship, depression scores, generalized anxiety score, and benzodiazepine use. Samples varied in alcohol abuse (AUDIT), with a significant higher percentage of positive endorsement to harmful or hazardous drinking in Spain as compared to MA. In MA, the trend was for more severe levels of PTSD symptoms. No statistical difference by site was observed in the composite MH Criteria, while Spain had a significantly greater percentage of participants’ classified as positive for the composite SA Criteria. See Table 2.

Table 2.

Demographic and Clinical Variables - Massachusetts vs. Spain

	Massachusetts n=194		Spain n=373		P-value
	N	%	N	%
Age
18–34	51	26.29	116	31.1	0.00
35–49	66	34.02	185	49.6
50+	77	39.69	72	19.3
Gender
Female	132	68.04	219	58.71	0.03
Male	62	31.96	154	41.29
Economic Status
Live very well or comfortable	57	29.38	111	29.76	0.93
Live check-to-check or poor	137	70.62	262	70.24
Citizenship
Citizen	114	58.76	228	61.13	0.58
Non-citizen	80	41.24	145	38.87
Race
White	45	23.2	90	24.13	0.8
Non-white	149	76.8	283	75.87
Clinic Type
Primary Care	140	72.16	221	59.25	0.00
Mental Health	29	14.95	50	13.4
Substance Use	15	7.73	52	13.94
HIV	9	4.64	38	10.19
Community Site	1	0.52	12	3.22
Depression (PHQ-9)
Negative	118	60.82	234	62.73	0.66
Positive(PHQ-9>=10)	76	39.18	139	37.27
Generalized Anxiety (GAD-7)
Negative	145	74.74	290	77.75	0.42
Positive(GAD-7>=10)	49	25.26	83	22.25
PTSD (PCL)
Negative	91	46.91	205	54.96	0.07
Positive (PCL>=30)	103	53.09	168	45.04
Composite Mental Health Criteria (PHQ-9, GAD-7, PCL)
Negative	83	42.78	171	45.84	0.49
Positive	111	57.22	202	54.16
Alcohol Use (AUDIT)
Negative	169	87.11	261	69.97	0.00
Positive(AUDIT>=8)	25	12.89	112	30.03
Drug Abuse (DAST)
Negative	175	90.21	323	86.60	0.21
Positive(DAST>=3)	19	9.79	50	13.40
Benzodiazepines (BDEPQ)
Negative	160	82.47	313	83.91	0.66
Positive(BDEPQ>=7)	34	17.53	60	16.09
Composite Substance Abuse Criteria (DAST, AUDIT, BDEPQ)
Negative	139	71.65	225	60.32	0.01
Positive	55	28.35	148	39.68

3.2 Reliability

Internal consistency reliability values were estimated using Cronbach’s alpha coefficient. In Massachusetts, alpha values were .90 (total scale), .84 Mental Health, and .92 Substance Abuse. In Spain, alpha values were .88 (total scale), .80 Mental Health, and .89 Substance Abuse. All values exceed the minimum reliability standard of .70, indicating good internal consistency (Nunnally and Bernstein, 1994).

3.3 Concurrent Validity

The correlations between the two AC-OK subscales and criterion measures are presented in Table 4. As expected, higher correlations were observed within mental health instruments (PHQ-9, GAD-7 and the PCL) and the AC-OK Mental Health (MH) subscale than those correlations observed between the MH subscale and substance use instruments (AUDIT, DAST and BDEPQ). Correlations of the AC-OK MH subscale and the PHQ-9, GAD-7 and the PCL ranged from .64 to .80 across both sites and from .26 to .57 for the AUDIT, DAST and BDEPQ. The correlation between benzodiazepine and the AC-OK MH was large in both sites, .56 (MA) and .57 (Spain).

Table 4.

Concurrent and Discriminant Validity: correlation analysis between the AC-OK measures and various criterion measures.

	AC-OK SA	AC-OK MH	PHQ-9	GAD-7	PCL	AUDIT	DAST	BDEPQ
AC-OK SA	1.00	0.56	0.39	0.36	0.42	0.54	0.62	0.32
AC-OK MH	0.56	1.00	0.77	0.74	0.80	0.26	0.34	0.56
PHQ-9	0.45	0.73	1.00	0.83	0.86	0.22	0.24	0.60
GAD-7	0.43	0.64	0.82	1.00	0.83	0.24	0.21	0.53
PCL	0.46	0.70	0.74	0.69	1.00	0.26	0.28	0.62
AUDIT	0.70	0.41	0.37	0.34	0.34	1.00	0.39	0.29
DAST	0.61	0.43	0.33	0.30	0.34	0.38	1.00	0.36
BDEPQ	0.44	0.57	0.52	0.46	0.54	0.25	0.45	1.00

Note: The top triangle reports correlations in Massachusetts whereas the bottom one reports correlation in Spain. All correlation coefficients are significant at p<0.001.

Correlations between the AC-OK Substance Abuse (SA) and substance use instruments ranged from .32 to .70. Generally, the correlations were medium (i.e., > .50) to large (i.e., > .80) except for the AC-OK SA and the Benzodiazepine questionnaire (.32 in MA, .44 in Spain). The Benzodiazepine questionnaire correlated more highly in both sites with the AC-OK MH subscale than with the AC-OK SA subscale, indicative of its use in the treatment of depression. This can also be observed in that the BDEPQ’s highest correlations are with the PHQ-9 and PCL. Correlations across domains between the AC-OK SA and the PHQ-9, GAD-7 and the PCL were low (.36 to .46) across both sites. The correlation between the two AC-OK subscales (MH and SA) was identical (r=.56) in Spain and MA.

3.4 ROC Curve Predictive Validity Analyses-Sensitivity and Specificity

The top panel of Figure 1 shows the results for AC-OK-MH, relative to the criterion defined by the combination of three standard measures (PHQ-9, GAD-7, PCL). The curves show strong associations between the brief AC-OK-MH screen and the composite scales. This is reflected in the AUC = 0.92 (CI: 0.91, 0.93) in Massachusetts and AUC=.89 (0.88, 0.90) in Spain. Each plot shows 11 points connecting the line segments. The point on the top right side of both plots corresponds to a screening rule that defines everyone to be a case (Screen if AC-OK-MH ≥0). Clearly the sensitivity is 1.0 and the specificity is 0 for this rule. As one moves to the left, the next point corresponds to (AC-OK-MH ≥ 1), and the sensitivity in both sites remains excellent, but the specificity improves. One can see the trade off in the plots – as the screening threshold is increased, the sensitivity decreases while the specificity increases. Depending on the application of the screen, one might choose different cutpoints. For example, a study that plans further assessment, might choose a cutpoint of (AC-OK-MH ≥ 2), which corresponds to sensitivity/specificity values of 0.93/0.65 in Massachusetts and 0.96/0.61 in Spain.

Figure 1.

ROC Curves

The bottom panel of Figure 1 shows the results for AC-OK-SA relative to the criterion defined by the combination of three standard measures (AUDIT, DAST, BDEPQ). The correspondence of the AC-OK-SA to the criterion is weaker than for the mental health screen, but the screen is still quite informative AUC = 0.80 (95%CI 0.73, 0.87) in MA and AUC=0.84 (95%CI 0.80, 0.88) in Spain. In this domain, the criteria screen is seen to be more inclusive than the AC-OK-SA. When the screening rule is set to AC-OK-SA≥ 1, the sensitivity drops below .90 in Spain, and drops below .80 in Massachusetts. The sensitivity/specificity values for a cutpoint of 1 are 0.73/0.83 in MA and 0.80/0.77 in Spain. The sensitivity/specificity values for the next cutpoint of 2 are 0.58/0.90 in MA and 0.72/0.90 in Spain. If one wanted to keep sensitivity as high as possible in both sites, a cut-point of 1 would be advised, but this assumes that further assessment would be used to eliminate the false positives.

4. Discussion

This study investigates the psychometric properties of the AC-OK quick screen for co-occurring disorders translated and adapted into Spanish, and tested with immigrant Latino populations. When comparing our results to the original English version (Cherry and Dillon, 2012), our results replicate the originally observed two factor model as the best fit to the data with moderate to high loadings (.49 – .97), and with items grouped in the same two factor structure: Mental Health and Substance Abuse. Our results for internal consistency are good and are also comparable to the original English sample. When we combine both sites, alpha is .82 (compared to α=.92 English) for MH and .90 (compared to α=.80 English) for SA. The developers also reported excellent sensitivity with the AC-OK correctly identifying 96% of those later found to have a co-occurring disorder using the ASI-Psyc (an Addiction Severity Index subscale for psychiatric problems), and correctly identifying 90.5% of those found to have a co-occurring disorder using the CAR-sa (Client Assessment Record substance use). The English AC-OK specificity was reported as fair, suggesting that approximately 50% of the individuals identified as having a potential co-occurring disorder, as actually having the disorder. While using other methods (Receiver Operating Curves), not reported by the developers in the English version, we also found this to be true. The accuracy of the AC-OK to discriminate between those with a MH disorder and those without in our study was ROC auc=.90 (combined sample), which is considered to be excellent. Sensitivity was almost perfect (almost 1) for both cutpoints, while specificity is only recommended for a cutpoint of 2 or more. In terms of predicting Substance Abuse, the ROC auc=.83 (combined sample) had a lower correspondence to the criteria, with a sensitivity inferior to the MH estimate, but with a higher specificity.

Our results based on analyzing each measure individually for Substance Abuse, pinpointed the Benzodiazepine questionnaire as the culprit for our low sensitivities. Similar to what was observed in our correlation analyses, the dispensing of benzodiazepines for the treatment of depression and/or anxiety might be confounding its status as a clear-cut substance abuse criteria measure. BDEPQ users in our sample had significant higher levels of depression, anxiety, PTSD, drug abuse, alcohol abuse and trauma exposure than non-users. Although the number of participants with a cutoff classification of severe level in the BDEPQ is comparable for both sites (17.53% MA, 16.09% SP p≤ 0.7), differences emerged upon further examination. For a cutpoint of 2 or more, the sensitivities in MA were considerably lower than in Spain (0.47 vs. 0.68) (See supplemental material²). BDEPQ users in MA, compared to those in Spain, were significantly older, poorer, and had citizenship status. A limitation to our study is that we did not measure physical health conditions or chronic pain problems, which might have ancillary implications that explain the differences observed. Another is possible dissimilarities in prescription medication dispensing practices between both countries. Patients in MA might be given benzodiazepines to treat a wider range of problems than in Spain.

We decided to use a cutoff point of two or more positive responses in each of the two AC-OK subscales for participants to qualify for a larger intervention trial (the ILRP project). Using this definition, the AC-OK identified 28% of our immigrant Latinos as potentially having a co-occurring disorder. The number of eligible individuals was found to be significantly higher (p≤ .03) in Spain (31%), than in Massachusetts (23%), possibly because the Spanish sample had more non-primary care participants. We also explored how effective the AC-OK was at identifying positive cases compared to two single disorder instruments such as the PHQ-9 (depression) and the Audit (alcohol use). Our results show that the AC-OK identified 160 positive cases out of 173 cases positive to all three instruments, or 92% of potential cases. The use of both the PHQ-9 and Audit identified 13 positive cases that were not picked up by the AC-OK and a remaining 67 cases that were, in all 80 cases or 46% out of the total potential cases. In other words, the AC-OK identified twice as many cases as using both the PHQ-9 and Audit.

Among the limitations to our study are that our sample is not representative of the entire population of Latino immigrants in either Country. Additionally, our Boston site lacks the national representation of Mexicans and Cubans. Even so, the similarities found for our confirmatory factor analyses, concurrent and predictive validity across sites makes this a valuable and relevant instrument for screening Latinos, both immigrant and non-immigrant. Another limitation is that we used a sample of convenience already seeking services. The differences that might be found in terms of psychometric properties of the instrument between our sample and other populations not privy to the health care system in either country might be the cause of investigation for a future study.

In general, the ability to identify patients with dual diagnosis and to ensure early detection and treatment is a shared public health goal in any language and for any population, across countries and continents. Given that unmet need for treatment among those with a mental health disorder is greatest in traditionally underserved groups, such as racial and ethnic minorities (Wang et al., 2005), having a screening instrument for dual diagnosis that is valid and reliable, tackles a significant health challenge. In our case, given that migrant populations have an exacerbated need to overcome multiple barriers that can be categorized from individual to systemic in order to receive treatment services, it is of substantial relevance and has the potential to impact an existing health disparity.

Our work addresses at least one of the barriers to identifying and treating substance abuse and mental health problems among Latino immigrants: language. The Spanish AC-OK Screen demonstrates good to excellent psychometric properties in both its subscales, and these findings are robust across sites, gender, and type of clinic. It is also a brief, cost-effective and practical dual disorder screening instrument for primary and specialty care with Latino immigrant populations. We recommend its use, both for clinical research, and for routine screening at treatment centers.

Highlights.

• International sample provides data to evaluate psychometrics of Spanish AC-OK measure

• Addresses language as a barrier to treating Latino immigrants

• Spanish AC-OK is effective in detecting co-occurring disorders in preventive care