Inter-rater reliability and concurrent validity of DSM-IV opioid dependence in a Hmong isolate using the Thai version of the Semi-Structured Assessment for Drug Dependence and Alcoholism (SSADDA)

Abstract

Because isolated populations offer relative genetic and environmental homogeneity, they are important resources for mapping genes for complex traits. Reliable and valid phenotypic characterization of the disease in the population studied is essential. We examined diagnostic reliability and concurrent validity of DSM-IV opioid dependence (OD) in a Hmong population in Thailand with historically high rates of opium (and heroin) use.

578 Thai-speaking Hmong individuals were assessed for lifetime OD, using Thai versions of both the Semi-Structured Assessment for Drug Dependence and Alcoholism (Thai SSADDA) and the Mini-Neuropsychiatric Interview (Thai MINI; adapted for lifetime diagnoses). In a subsample of 123 individuals, two raters interviewed each subject independently within a 2-week period. Chance-corrected agreement on the OD diagnosis was assessed between raters and instruments.

Results showed excellent agreement for the DSM-IV diagnosis of OD both for the SSADDA (κ=0.97) and MINI: (κ=1.00) and between the SSADDA and MINI (κ=0.97).

Consistent with reliability assessments of English versions, our data demonstrate high reliability for Thai versions of the SSADDA and MINI in the diagnosis of OD. The high concordance between instruments supports the concurrent validity of the diagnosis.

Either interview provides reliable, valid OD diagnoses in Thai-speaking Hmong individuals.

1. Introduction

Opioid dependence (OD) is a common disorder in many regions worldwide that causes considerable social, medical and psychiatric disability. Current data demonstrate the importance of genetic factors in its etiology (Kendler, Jacobson, Prescott, & Neale, 2003; Tsuang et al., 1996), and there is a strong expectation of genetic heterogeneity and complex inheritance. Human research has identified several promising candidate risk loci contributing to OD in nonisolates (Gelernter et al., 2006). However, advances in methods for the genetic analysis of complex traits, including recent successes in identifying genes for common disorders in genetically and environmentally isolated populations has led to a global interest in the study of such specialized groups, which pose several theoretical advantages for gene mapping (Peltonen, Palotie, & Lange, 2000).

The opium poppy is indigenous to Northern Thailand. Historically, high rates of OD have been observed among the country’s ethnic hill tribe populations living in that region (Westermeyer, 1977). Such groups are likely to have greater genetic, environmental, and phenotypic homogeneity than the Thai population taken as a whole, and this provides potential advantages for gene identification efforts, including their relative geographic and cultural isolation, the reportedly high lifetime prevalence rates of OD (estimated as ≥ 10% as compared to ~0.1–0.2% in the U.S.), the presence of apparent historical bottlenecks (i.e., evolutionary events that produce a reduction in population size and in turn lead to increased genetic homogeneity), cultural prohibitions (albeit currently weakening) against marrying outside of the population, large and intact pedigrees, and the relative absence (e.g., in comparison to U.S. populations) of other drugs of abuse.

As part of an international collaborative effort to assess the feasibility of a comprehensive, community-wide genetic linkage study of OD in one such village (with a Hmong population), we sought to develop methods for the reliable phenotypic characterization of OD in Thai-speaking populations. We chose to adapt the Semi-Structured Assessment for Drug Dependence and Alcoholism (SSADDA)(Pierucci-Lagha et al., 2007; Pierucci-Lagha et al., 2005) based on its specialized development for genetic studies of substance use disorders, its excellent psychometric properties, and our prior extensive experience with the instrument in the assessment of opioid dependence (Gelernter et al., 2006) and other disorders (Kalayasiri et al., 2005). The current study reports on the diagnostic reliability and concurrent validity of DSM-IV opioid dependence (OD) using the opioid dependence section of the newly translated instrument (the Thai SSADDA) as well as a previously validated Thai version of the Mini International Neuropsychiatric Interview (Thai MINI)(Sheehan et al., 1998), newly adapted by us for lifetime diagnosis in the proposed study population.

2. Methods

2.1. Overview

The study population consisted of 578 Thai-speaking individuals from a Hmong hill tribe village in Northern Thailand who were interviewed as part of a genetic linkage study of opioid dependence (OD). All subjects underwent diagnostic interviews using Thai versions of two clinician-administered rating scales adapted for the current study. Given prior estimates of reliability of the English instrument (Pierucci-Lagha et al., 2005), predictions of power, and issues of resources and time, 123 individuals among those initially enrolled participated in an inter-rater reliability substudy (a more conservative estimate of reliability that test-retest), comprised of two separate interviews conducted within a two-week period by different nurse interviewers who received specific training in both instruments.

2.2. Assessments

The primary phenotypic instrument, the Semi-Structured Assessment for Drug Dependence and Alcoholism (Thai SSADDA), was derived from the English version of the instrument, which has been shown to yield reliable diagnostic criteria and diagnoses (Pierucci-Lagha et al., 2007; Pierucci-Lagha et al., 2005). Translation of the parent instrument into Thai was conducted by two Thai psychiatrists (RK and AS) who are fluent in English and Thai, in collaboration with two addiction genetics psychiatrists with substantial experience with the SSADDA (RM and JG; fluent in English only), incorporating culturally relevant forms (e.g., opium, in addition to heroin), names, and units for all drug categories (relying on the expertise of Thai study physicians and the input of local villagers). A dually literate, non-physician, lay translator independently back translated the initial draft. Thai and U.S. study physicians (RK, AS, RM and JG) reviewed inconsistencies in the back translated and original English versions and amended the original version through consensus to create the final instrument. The current study employed three sections of the Thai SSADDA: Demographic Information (Section A), Medical History (Section B), and Opioid Dependence (Section G).

In addition to the Thai SSADDA, the current study employed a previously translated and validated Thai language version of the Mini-International Neuropsychiatric Interview (Thai MINI) as a secondary phenotypic assessment (Kittirattanapaiboon & Khamwongpin, 2005). The Thai MINI was modified for the current study to assess lifetime (rather than current) neuropsychiatric disorders, including substance dependence, since lifetime (trait) rather than current (state) diagnoses are of the most importance for studies of genetic vulnerability. All sections of the Thai MINI were used, although we report here only on data pertaining to the evaluation of OD.

Interviews were conducted in a predetermined fixed order, consisting of the Demographics and Medical History sections of the Thai SSADDA, followed by the Thai MINI, Lifetime Version, and ending with the OD section of the Thai SSADDA. Interview assignments were balanced across the 11 interviewers, such that all potential interviewer pairs were used. Completed assessments were subjected to a strict quality assurance protocol in which: 1) interviews were first edited for completeness and accuracy by the primary nurse interviewers, 2) edited interviews were then cross-edited by an independent pair of trained interviewers, 3) cross-edited interviews were reviewed and a finalized interview produced by the primary interviewers, and 4) finalized interviews were submitted for quality assurance (QA) review by a trained, SSADDA-certified, Thai study physician (RK) who had no a priori knowledge of subjects or their diagnostic status (i.e., error checks were to confirm procedural accuracy in completion of the interview forms, not diagnostic accuracy). Upon completion of the QA review, interviews were scored using standard algorithms to determine the presence of a DSM-IV diagnosis of OD.

2.3. Interviewers

Phenotypic assessments using these instruments were conducted by a team of 11 psychiatric nurses with an average of greater than 12 years of hospital-based (in Chiang Mai), psychiatric experience under the longstanding clinical supervision and training of a third Thai study physician (KS). Interviewers had prior experience with the Thai MINI, but underwent additional specialized training in the modified lifetime version, as well as full formal training in the relevant sections of the Thai SSADDA (conducted by RK, RM and JG). Subsequent to training, and as part of a process of formal certification prior to the study, each nurse clinician conducted a series of 10 patient assessments (8 training and 2 certification) of established clinical cases (diagnosed by a senior psychiatrist, KS), consisting of non-study subjects with and without OD. All training interviews were reviewed individually for procedural accuracy by a SSADDA-certified Thai study physician (RK) and results of training interviews were reviewed as a group with all 11 nurse clinicians, Thai study physicians (RK and KS), and U.S. investigators (RM and JG). Completion of the initial training course, 8 training interviews, a review session and group discussion of findings from the training interviews, and subsequent completion of two diagnostically accurate final interviews constituted criteria for certification. Upon certification, nurse interviewers were allowed to conduct in-person interviews with village members deemed eligible for participation in the genetic linkage study. Our QA and training procedures derive from those developed for the English instrument.

2.4. Subjects

Subjects were ascertained by a semi-systematic approach, designed to enhance both the collection of families enriched for OD and the statistical power of reliability estimates. Interviewers traveled to the village where all subjects lived (study village) and interviewed subjects in their homes and/or a private community setting. We employed a purposive screening/enrollment strategy. Eligible individuals endorsing a personal history of opioid use/dependence and opioid use by first-, second-, or third-degree relatives were prioritized for enrollment in both the total and inter-rater reliability subsamples (per an independent, non-interviewer, research assistant). Study inclusion and exclusion criteria were 1) age ≥ 18 years, 2) verbal fluency in the Thai language, 3) Hmong heritage, 4) residence in the study village, 5) ability to understand the study requirements and give informed consent, and 6) at least two first-, second-, or third-degree family members with suspected opioid use. Fourteen of 15 families approached agreed to participate, and aside from children <18 years old, fewer than 10% of those approached were excluded (typically based on considerations of Thai fluency among the elder generations). Eligible individuals underwent a formal process of informed consent, during which the study requirements, potential risks and benefits, and alternatives to participation were explained. All procedures were approved by the institutional review boards of the respective institutions in the United States and Thailand, including the Human Investigations Committee at Yale University School of Medicine, the Ethics Committee at the Faculty of Medicine, Chulalongkorn University, and the Ethics Committee of the Thai Ministry of Public Health. Enrollment of subjects was conducted with the prior approval of the local community leaders, village elders, and family heads.

2.5. Analyses

We assessed within-instrument inter-rater reliability in the subsample that was interviewed twice with the SSADDA (N=123). Individual DSM-IV OD criteria were also assessed with respect to their inter-rater reliability. We also evaluated the OD diagnostic concordance between instruments (SSADDA vs. MINI) in all 578 participants. Given the anticipated prevalence of the trait in the study population (>10%), all statistical analyses were conducted in SAS using the κ statistic for chance-corrected agreement (AGREE option in PROC FREQ) (Cohen, 1988). Reliability and concordance were categorized according to previously suggested criteria (i.e., <0.40 as poor; 0.40–0.59 as fair; 0.60–0.74 as good; and 0.75–1.00 as excellent) (Cicchetti & Sparrow, 1981).

3. Results

A summary of subject demographics is presented in Table 1.

Table 1.

Demographic information for inter-rater reliability (N=123) and total (N=578) study samples

Variable		(N=123)		(N=578)
		N	%	N	%
Gender	Male	68	55.3	313	54.2
	Female	55	44.7	265	45.9
Race	Hmong	123	100	576	99.7
	Thai	0	0	1	0.2
	Other hill tribe	0	0	1	0.2
Adopted	No	123	100	575	99.5
	Yes	0	0	3	0.5
Twin or other multiple	No	121	98.4	575	99.5
	Yes	2	1.6	3	0.5
Marital Status	Married	90	73.2	415	71.8
	Widowed	4	3.2	15	2.6
	Separated	2	1.6	3	0.5
	Divorced	6	4.9	17	2.9
	Never Married	21	17.7	128	22.2
Currently employed	No	23	18.7	117	20.2
	Yes	100	81.3	461	79.8
Working full-time (among employed)	No	9	9.8	45	9.8
	Yes	90	90.2	415	90.2
		Mean +SD	N	Mean +SD	N
# Children	Including no children	4.1 ± 3.3	116	3.6 ± 3.2	534
	Excluding no children	4.9 ± 3.0	96	4.7 ± 3.0	420
Education (years)		4.6 ± 5.0	123	5.3 ± 5.3	578
# of months employed in the past year	9.4 ± 4.8	123	9.3 ± 4.8	578
Age	38.2 ± 14.8	123	36.0 ± 15.0	578

As predicted, prevalence rates of opioid dependence were high in our samples: 16% and 42% in the total and inter-rater samples, respectively. Thai versions of the MINI (κ=1.00; Table 2) and SSADDA (κ=0.97; Table 3) showed excellent inter-rater reliability for the categorical DSM-IV diagnosis of OD. Similarly, there was excellent agreement for individual OD criteria (κ=0.79–0.97) (Table 3). Finally, diagnostic assessments of OD between instruments (i.e., Thai SSADDA vs. Thai MINI) were highly concordant (κ=0.97)(Table 4).

Table 2.

Inter-rater agreement for MINI-based Opioid Dependence Diagnosis and Criteria (N=123)

Variable	Interview #1	Interview #2		% Agreement	Kappa
		No	Yes
Opioid Dependence (DSM-IV)	No	71	0	100.0	1.00
	Yes	0	52

Table 3.

Inter-rater agreement for SSADDA-based Opioid Dependence Diagnosis and Criteria (N=123)

Variable	Interview #1	Interview #2		% Agreement	Kappa
		No	Yes
Opioid Dependence (DSM-IV)	No	70	1	98.4	0.97
	Yes	1	51
1) Tolerance	No	86	4	91.9	0.79
	Yes	6	27
2) Withdrawal	No	75	4	92.7	0.84
	Yes	5	39
3) Taking the substance in larger amounts or over a longer period than intended	No	71	1	96.7	0.93
	Yes	3	48
4) Persistent desire or unsuccessful efforts to cut down or control substance use	No	78	4	95.1	0.89
	Yes	2	39
5) Great deal of time spent in obtaining, using, or recovering from substance use	No	80	3	95.1	0.89
	Yes	3	37
6) Giving up or reducing important social, occupational, or recreational activities	No	70	1	98.4	0.97
	Yes	1	51
7) Continued substance use despite persistent or recurrent physical or psychological problems.	No	81	4	93.5	0.85
	Yes	4	34

Table 4.

Inter-rater agreement between MINI and SSADDA OD Diagnosis (N=578)

		MINI Opiate Dependence		% Agreement	Kappa
		No	Yes
SSADDA Opioid Dependence (DSM-IV)	No	485	3	99.3	0.97
	Yes	1	89

4. Discussion

The current study evaluated the inter-rater reliability and diagnostic concordance of Thai versions of the SSADDA and MINI developed for genetic studies of OD in Northern Thailand. In brief, the study confirmed anecdotal reports of historically high levels of lifetime OD among villagers and demonstrated excellent inter-rater reliability for Thai versions of both the MINI (K=1.00) and the SSADDA (K=0.97) as assessed by Cohen’s κ (Cohen, 1988). We also found excellent agreement between instruments (κ =0.97), consistent with the concurrent validity of the diagnosis and the utility of either instrument to assess the phenotype of OD in our Hmong study population.

The reliability of the Thai version of the SSADDA is consistent with prior studies showing excellent test-retest (κ=0.94) and inter-rater (κ=0.91) reliability of the English version of the instrument in U.S. populations (Pierucci-Lagha et al., 2007; Pierucci-Lagha et al., 2005). Although caution is warranted in directly comparing κ values across studies (as reliability estimates can be population dependent), the high level of reliability of the Thai version is consistent with that of the original version. Individual DSM-IV symptom criteria showed similarly high levels of chance-corrected agreement (i.e., Thai: 0.79–0.97; English: 0.56–0.90, as reported previously).

In addition to the instrument’s psychometric properties, several additional factors may have contributed to the high levels of reliability observed in the current study. Although the SSADDA was developed for use by interviewers without formal clinical training and has been shown to be reliable when administered by this group, our study used a highly skilled team of trained psychiatric nurses with considerable prior experience in the evaluation and treatment of clinical populations (including members of the specific ethnic group under study). Therefore, reliability may be lower in a less skilled set of interviewers, particularly those without experience interacting with Hmong individuals. As such, future evaluation of the Thai instrument’s reliability in non-clinically trained research personnel and in other ethnic groups in Thailand is warranted.

The ascertainment/enrollment strategy could also have contributed to the high degree of inter-rater reliability. By prioritizing families where OD was highly prevalent and increasing the number of OD cases in our inter-rater subsample, we may have inadvertently selected more severe (and perhaps less ambiguously affected) cases. The effectiveness of our approach, which was designed to maximize the yield of the genetic linkage study and balance the composition of the inter-rater subsample, is evident in the considerably higher rates of OD in those initially studied (i.e., approximately 42% of 123) as compared to the total sample (i.e., approximately 16% of 578). That being said, we did not find evidence of significant differences in severity (as assessed by DSM-IV symptom count) between those OD subjects enrolled initially (N=52 of 123, 5.7±1.4) as compared to subsequently (N=38 of 455, 5.8±1.4; t(88)=0.11, p=0.91). Neither can we exclude the possibility that characteristics of OD in the specific population we studied contributed to a more homogeneous phenotypic presentation. Importantly, the high (but not extreme) rates of OD in our inter-rater reliability sample would not have contributed to inflated estimates of κ, but rather more reliable ones (i.e., as κ is less robust in instances where the base rate of the trait is either very high or very low; e.g., >95% or <5%) (Spitznagel & Helzer, 1985).

We cannot rule out the possibility that the concurrent administration of the MINI and the SSADDA in the same setting, and/or the fixed (i.e., non-randomized) order of presentation, might have systematically influenced our findings. For example, prior administration of the MINI might have “primed” participants for subsequent recall of information solicited by the SSADDA, thereby improving the latter’s reliability.

The very high reliability of the Thai version of the MINI is noteworthy in the current study. Several other factors might also be relevant to the MINI’s high reliability in the current study, including adaptation of the instrument for lifetime as compared to current dependence, prior experience of the interview team with the Thai MINI, and/or the evaluation of the instrument in a primary substance-dependent population (e.g., in contrast to assessments of substance dependence in primary psychiatric populations, as was initially the case for the instrument) (Sheehan et al., 1998).

Several other limitations of our study are worth noting. We did not conduct an assessment of the test-retest (i.e., within-rater) reliability of instruments. Ordinarily, reliability would be greater, not less, in such a design (and we chose to conduct the more conservative comparison for this reason). Similarly, we cannot comment directly on the generalizability of the instrument in other populations, including non-isolated Thai-speaking populations, and/or the suitability of other sections of the instrument in the current (or other) population(s). However, reliable use of the English SSADDA in outbred U.S. samples suggests that this would not be a major concern (Pierucci-Lagha et al., 2007; Pierucci-Lagha et al., 2005). Moreover, we have recently used the complete Thai version of the SSADDA in a primary Thai drug-treatment cohort as part of a genetic study of methamphetamine dependence and replicated vulnerability factors (e.g., dependence severity) for stimulant-induced psychosis found previously by our group in the U.S. with the English version (Kalayasiri et al., 2005; Kalayasiri, Mutirangura, Verachai, Gelernter, & Malison, 2009). While the diagnostic concordance between the SSADDA and the MINI in our study supports the concurrent validity of the SSADDA, we note that other approaches to the assessment of concurrent validity were not conducted (e.g., other structured interviews, such as the Structured Clinical Interview for DSM-IV or SCID, and/or clinical interview by a primary psychiatrist). In addition, other forms of validity (e.g., discriminant, construct) were not assessed in the current study and are required to support more fully the validity of the diagnosis obtained with the SSADDA. Finally, the current study does not address the instruments’ diagnostic reliability in establishing OD in primary psychiatric populations, where dually diagnosed individuals might present additional challenges (e.g., intermediate or more variable trait severity and/or overlapping symptoms from multiple disorders).

In summary, our data demonstrate high reliability for Thai versions of the SSADDA and the MINI for a lifetime diagnosis of OD. High inter-instrument concordance supports the concurrent validity of the diagnosis in our study population, and as such, both interviews appear to provide reliable, valid OD diagnoses in Thai-speaking Hmong individuals.

Research Highlights.

• The Thai SSADDA validly assesses opioid dependence in a Hmong isolate

• The interview has excellent inter-rater reliability in diagnosing opiate dependence

Concurrent validity of the diagnosis in the Hmong is supported