Reliability and validity of the Severity of Dependence Scale for detecting cannabis dependence in frequent cannabis users

Abstract

The Severity of Dependence Scale (SDS) measures with five items the degree of psychological dependence on several illicit drugs, including cannabis. Its psychometric properties have not yet been examined in young adult frequent cannabis users, an eminently high‐risk group for cannabis dependence.

Internal consistency and criterion validity of the SDS were investigated within an enriched community based sample of 577 Dutch frequent (≥ three days per week in the past 12 months) cannabis users between 18–30 years. Criterion validity was tested against the Composite International Diagnostic Interview (CIDI) 3.0 DSM‐IV diagnosis cannabis dependence, and psychometric properties were assessed separately for males and females and for ethnic subgroups.

Principal component analysis showed that all items of the scale loaded on a single factor and reliability of the SDS total score was good (Cronbach's α = 0.70). However, criterion validity against the CIDI diagnosis cannabis dependence was low: area under curve (AUC) was 0.68 (95% confidence interval: 0.64–0.73) and at the optimal differentiating cut‐off (SDS ≥ 4), sensitivity was 61.3% and specificity 63.5%. Results were similar for subgroups on gender and ethnicity.

While internal consistency of the SDS is good, its use as a screener to differentiate between dependence and non‐dependence within populations of young adult frequent cannabis users is not recommended. Copyright © 2013 John Wiley & Sons, Ltd.

Introduction

The Severity of Dependence Scale (SDS) is a five item self report instrument that measures psychological dependence experienced by users of different types of illicit drugs, including cannabis (Gossop et al., 1995). While it was constructed as a quantitative, dimensional measure of cannabis dependence, interest in this brief instrument and its validity as a dichotomous diagnostic screener for epidemiological studies has grown (Klempová, 2011). Psychometric properties of the cannabis version of the SDS have been reported to be satisfactory in samples of adults and adolescents from the general population, and in clinical samples (schizophrenia spectrum disorder). (Hides et al., 2007; Kedzior et al., 2006; Martin et al., 2006; Piontek et al., 2008; Steiner et al., 2008; Swift et al., 1998). However, the SDS has not yet been tested in young adult frequent and heavy cannabis users. This is an important high risk group for dependence since prevalence of frequent use is highest among young adults (e.g. European Monitoring Centre for Drugs and Drug Addiction (EMCDDA), 2010) and frequency of use is highly correlated to cannabis dependence (Coffey et al., 2002; Grant and Pickering, 1998). Nonetheless, only a minority of all frequent cannabis users become dependent, with estimates ranging from 20% to 50% among (nearly) daily users (Coffey et al., 2002; EMCDDA, 2009). Therefore, the SDS could be very useful as a brief instrument to differentiate between frequent users with and without cannabis dependence, either for its use as a quick screening tool for indicated prevention or treatment or for scientific research among frequent users.

The current study aimed to investigate the reliability and validity of the SDS in this high risk population set against a dichotomous diagnosis of cannabis dependence. Furthermore, differences in psychometric properties between males and females were investigated, since Steiner et al. (2008) found a higher optimal SDS cutoff for males than for females. In addition, these analyses were performed for ethnic subgroups (Western versus non‐Western).

Method

Procedures and participants

Data were collected as part of a prospective study on predictors and course of frequent cannabis use and dependence in an enriched community cohort of 600 young adult frequent cannabis users (CanDep). For a detailed description of the design and assessment procedures, the reader is referred (Van der Pol et al., 2011; Liebregts et al., 2011). In short, cannabis users between 18–30 years who used cannabis on three or more days per week for at least a year were recruited from “coffee shops”. Most Dutch cannabis users buy their cannabis at these cafe‐like outlets, where the sale, use and possession of small amounts of cannabis is legally tolerated under certain conditions. In addition to the recruitment in coffee shops, snowball sampling was used: all participants were asked to bring in a maximum of three frequent cannabis users from their social network. Although there is no consensus on the definition of frequent cannabis use, Coffey et al. (2002) found that cannabis users who used on 3–4 days were more like (near) daily users than like less frequent users. Therefore, we used the criterion of at least three days per week for the past 12 months as our definition of frequent use.

Before enrolment into the baseline wave (September 2008–April 2009), participants were screened for eligibility, including administration of the SDS. When participants gave permission, an interview appointment was made for the full face‐to‐face interview, usually within two weeks. The screening data on the SDS were incomplete for 23 (3.8%) of the participants, and these were therefore excluded from the analysis. Thus, the final sample included 577 respondents.

Measures

The SDS is a five item screening questionnaire with item scores ranging from zero to three using the following answering categories (Gossop et al., 1995): items 1–4: 0 = never/almost never, 1 = sometimes, 2 = often, and 3 = always/nearly always; item 5: 0 = not difficult, 1 = quite difficult, 2 = very difficult, and 3 = impossible) and a maximum total score of 15. The items read as follows:

During the past year …

1. Did you think your use of cannabis was out of control?

2. Did the prospect of missing a dose of cannabis make you anxious or worried?

3. Did you worry about your use of cannabis?

4. Did you wish you could stop the use of cannabis?

5. How difficult did you find it to stop, or go without cannabis?

The SDS has shown to have good internal consistency, a one‐dimensional factor structure and good sensitivity and specificity for the detection of cannabis dependence with cutoff scores ranging between two and four found in various populations (Piontek et al., 2008).

Psychometric properties of the SDS were assessed using the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition (DSM‐IV) diagnosis of cannabis dependence as the “gold standard”, measured with the fully structured Composite International Diagnostic Interview (CIDI) version 3.0. The CIDI 3.0 consists of 18 items to assess whether at least three out of seven DSM‐IV cannabis dependence symptoms were present in the preceding 12 months.

Moreover, cannabis abuse is more prevalent in frequent cannabis users compared to the general population. Moreover, some authors have suggested from item response theory (IRT) analyses that the four abuse and seven dependence criteria are best described to reflect a single underlying disorder (Compton et al., 2009; Piontek et al., 2011). Therefore, psychometric properties of the SDS were also assessed with DSM‐IV cannabis use disorder (CUD; either a CIDI dependence or abuse diagnosis) as the external criterion. The CIDI 3.0 combined CUD diagnosis is based on the dependence diagnose and six additional items to assess whether at least one out of four DSM‐IV cannabis abuse symptoms were present in the preceding 12 months.

Analysis

The content validity of the SDS in the current study was assessed using principal component analysis, to establish the number of underlying dimensions. Cronbach's α was used as a measure of internal consistency. Concurrent validity of the SDS was assessed in all frequent users as the correlation between the total SDS score and the number of endorsed DSM‐IV cannabis dependence criteria (irrespective of the dependence diagnosis), using Spearman's rho correlation coefficient. Since the SDS was originally a dimensional scale, the number of positive criteria was used as a proxy of dependence severity. Finally, criterion validity of the SDS was assessed with the 12 month CIDI DSM‐IV diagnosis of cannabis dependence as the external criterion using receiver operating characteristic (ROC) analysis to determine sensitivity, specificity and optimal cutoff values. ROC‐AUC values of ≥ 0.90 are considered excellent, 0.80–0.90 good, 0.70–0.80 fair and < 0.70 poor (Metz, 1978). The most applicable cutoff should be chosen depending on the context of screening: a higher cutoff results in a lower sensitivity and a higher specificity. Thus, when correctly ruling out dependence is more important than correctly diagnosing dependence, the higher cutoff is favoured (and vice versa). However, the cutoff with the least errors, thus the maximum sum of sensitivity and specificity, is often identified as “the optimal discriminating cutoff score”. Analyses were also performed on the total SDS sum score using CUD as the external criterion, and for subgroups based on gender and ethnicity.

Results

Sample characteristics

Seventy‐nine percent of the sample was male and the mean age was 22.1 years [standard deviation (SD) = 3.1]. Most respondents were either student (43.8%) or employed (41.2%). Nearly a third (32.8%) was attending or had completed higher professional education, and 27.9% was of non‐Western origin. On average 31.5% used cannabis daily for the past 12 months, 36.7% nearly daily and 31.7% on 3–4 days per week. The mean number of joints consumed on a typical cannabis using day was 3.4 (SD = 2.2).

The criteria for a 12‐month CIDI diagnosis of cannabis dependence (with or without abuse) and abuse (without dependence) were fulfilled by 41.6% and 12.5%, respectively. Lifetime prevalences were 54.6% and 26.0%, respectively. The mean SDS score was 3.8 (SD = 2.5). Negative endorsements (“never or almost never”) varied between the different SDS items: (1) “control” 60.3%, (2) “dose” 72.6%, (3) “worry” 29.3%, (4) “stop wish” 29.3% and (5) “stop difficulty” 25.5%.

Psychometric properties of the SDS

Unrotated principal component analysis showed a single factor structure with an eigenvalue of 2.27, which accounted for 45.5% of the total variation in the scores. Factor loading were 0.75, 0.61, 0.72, 0.63 and 0.64, respectively. Cronbach's α of the SDS total score was 0.70.

Spearman's rho correlation between SDS total score and the number of positive dependence criteria was 0.48 (p < 0.001). Using the dichotomous outcome indicating presence or absence of a 12‐month DSM‐IV cannabis dependence diagnosis in the ROC analyses resulted in an area under the curve (AUC) of 0.68 [95% confidence interval (CI): 0.64–0.73]. At the optimal differentiating cutoff (SDS ≥ 4), sensitivity was 61.3%, specificity 63.5%, positive predictive value 54.4%, and negative predictive value 69.7% (Table 1). Table 1 also shows that the AUC for the presence of CUD was very similar, with a largely overlapping 95%CI (0.64; 95%CI: 0.60–0.69).

Table 1.

Criterion validity of the SDS at several cutoff scores, when discriminating between frequent cannabis users with and without the CIDI DSM‐IV diagnosis: (1) cannabis dependence, (2) cannabis abuse or dependence

	(%)	≥2	≥3	≥4	≥5	≥6	≥7	≥8
1. Cannabis dependence versus non dependence	Sensitivity	93.3	79.6	61.3	45.8	35.8	22.5	16.3
AUC = 0.68 (0.64–0.73)	Specificity	23.1	42.7	63.5	78.9	87.5	93.2	95.0
Prevalence = 41.6%	ppv	46.4	49.7	54.4	60.8	67.2	70.1	69.6
	npv	83.0	74.6	69.7	67.2	65.7	62.8	61.4
2. Cannabis use disorders versus non disorders	Sensitivity	89.7	73.7	55.1	40.4	31.4	19.2	14.4
AUC = 0.64 (0.60–0.69)	Specificity	23.4	41.9	63.0	79.2	88.7	93.6	95.8
Prevalence = 54.1%	ppv	58.0	59.9	63.7	69.6	76.6	77.9	80.4
	npv	66.0	57.5	54.4	53.0	52.3	49.6	48.8

In italic typeface the optimal differentiating cutoff score with the highest sum of sensitivity and specificity.

Note: ppv, positive predictive value; npv, negative predictive value.

Prevalence of cannabis dependence did not differ between males and females (p = 0.73), the same was true for the AUCs (p = 0.91). However, Table 2 shows that the shape of the distribution differed, with somewhat different optimal differentiating cutoffs (estimates shown in bold typeface) and a relatively low sensitivity and specificity at a cutoff of ≥ 4 in females. Finally, Western and non‐Western frequent cannabis users did not differ in prevalence of cannabis dependence and AUCs (p = 0.19 and p = 0.40, respectively).

Table 2.

Criterion validity of the SDS at several cutoff scores, when discriminating between frequent cannabis users with and without the CIDI DSM‐IV diagnosis cannabis dependence for subgroups based on gender and ethnicity

(%)		≥2	≥3	≥4	≥5	≥6	≥7	≥8
Male	Sensitivity	93.6	78.7	62.2	46.8	36.2	20.7	15.4
AUC = 0.68 (0.63–0.73)	Specificity	21.3	42.9	64.9	77.6	86.2	92.9	94.8
Prevalence = 41.2%	ppv	45.5	49.2	55.5	59.5	64.8	67.2	67.4
	npv	82.6	74.2	71.0	67.5	65.8	62.6	61.5
Female	Sensitivity	92.3	82.7	57.7	42.3	34.6	28.8	19.2
AUC = 0.69 (0.59–0.78)	Specificity	30.4	42.0	58.0	84.1	92.8	94.2	95.7
Prevalence = 43.0%	ppv	50.0	51.8	50.8	66.7	78.3	78.9	76.9
	npv	84.0	76.3	64.5	65.9	65.3	63.7	61.1
Western	Sensitivity	93.3	78.3	59.4	43.9	32.2	17.2	14.4
AUC = 0.67 (0.62–0.73)	Specificity	21.2	43.2	64.4	80.1	89.4	93.6	94.9
Prevalence = 43.3%	ppv	47.5	51.3	56.0	62.7	69.9	67.4	68.4
	npv	80.6	72.3	67.6	65.2	63.4	59.7	59.3
Non‐Western	Sensitivity	93.3	83.3	66.7	51.7	46.7	38.3	21.7
AUC = 0.72 (0.63–0.80)	Specificity	27.7	41.6	61.4	76.2	83.2	92.1	95.0
Prevalence = 37.3%	ppv	43.4	45.9	50.6	56.4	62.2	74.2	72.2
	npv	87.5	80.8	75.6	72.6	72.4	71.5	67.1

In italic typeface the cutoff score with the highest sum of sensitivity and specificity.

Note: ppv, positive predictive value; npv, negative predictive value.

Post hoc analyses

Different recruitment strategies (coffee shops versus chain referral) and different ways to assess the SDS (face‐to‐face mainly in coffee shop participants versus telephone mainly in chain referrals) may have influenced the performance of the SDS. Therefore, a post hoc analysis comparing the performance of the SDS for different recruitment methods was conducted. This analysis revealed that criterion validity was significantly lower in respondents recruited through chain referral (AUC = 0.65, 95%CI: 0.59–0.70; at cutoff ≥ 4: sensitivity = 53.0%, specificity = 64.2%) than in those directly recruited from coffee shops (AUC = 0.77, 95%CI: 0.71–0.84; at cutoff ≥ 4: sensitivity = 83.1%, specificity = 62.2%). In chain referrals, the prevalence of cannabis dependence was higher (44.5% compared to 35.3% in coffee shops, p = 0.04), but frequency of cannabis use was lower (22.1 compared to 23.7 days in the past 28 days, p = 0.002).

Discussion

In a population of young adult frequent cannabis users the SDS showed good content validity (one dimension, Cronbach's α = 0.70), moderate concurrent validity (correlation between SDS total score and the number of DSM‐IV criteria: Spearman's rho = 0.48) but a disappointing criterion validity with CIDI DSM‐IV diagnosis of cannabis dependence (AUC = 0.68). The optimal differentiating cutoff was established at a SDS total score ≥ 4. No significant differences in SDS performance parameters were observed for CUD as external criterion, between male and female cannabis users and between users from different ethnic background.

Reliability was somewhat lower than in previous studies (0.72–0.83), whereas the AUC with CIDI DSM‐IV as the external criterion was much lower than in previous studies (0.85–0.90) in a variety of populations with lower as well as similar cannabis dependence prevalence rates (Piontek et al., 2008). The optimal cutoff score was comparable or slightly higher than in previous studies (≥ 2–4) (Piontek et al., 2008). While the reported optimal cutoff minimize the errors, the intended use of the instrument should guide the decision for the most appropriate cutoff. For screening purposes, when all probable cases need to be identified, the cutoff should be decreased. However, to avoid over classifying, the cutoff should be increased. It should be noted however, that the CIDI may not be as accurate “gold standard” as a clinician rated diagnosis such as the Structured Clinical Interview for DSM Disorders (SCID). Still, it is a commonly used and validated instrument which is practical in its use since it is designed to be used by trained lay interviewers (Ustun et al., 1997).

Different explanations are possible for the mediocre performance of the SDS in the current study. First, some argue that higher prevalences are associated with higher sensitivity and lower specificity (Brenner and Gefeller, 1997). Even though this may hold in the comparison of the results of the present study compared to general population studies, it does not for the results of the studies among long‐term users (Swift et al., 1998), adolescents (Martin et al., 2006), and in a clinical sample (Hides et al., 2007) with very similar prevalence rates of DSM‐III‐R or DSM‐IV cannabis dependence based on the CIDI (also reviewed in Piontek et al., 2008). Moreover, the performance of the SDS was not significantly better for cannabis dependence (prevalence 41.6%; AUC = 0.68) than for CUD (prevalence 54.1%; AUC = 0.64) as the external criterion. Second, the performance of the SDS may be influenced by the different recruitment strategies that were used in the current study. The poorer screening performance in chain referral compared to coffee shop participants may be explained either by a difference in assessment (since chain referrals were usually contacted and screened by telephone instead of face‐to‐face) or by population differences (chain referrals had a higher prevalence of cannabis dependence and used cannabis less frequently). Given our conclusion regarding the effect of the prevalence on SDS performance, it seems more likely that the low validity of the SDS in the current study is partly a result of the assessment by telephone (instead of face‐to‐face). However, more specific studies on this issue are needed to draw final conclusions. Finally, a hint of a specific cultural factor that could explain the disappointing performance of the SDS is given by the relatively low factor loading of item two in the factor analyses. Although the present study cannot test this theory, it could be argued that anxiousness or worries about missing a dose of cannabis would be less common in the Netherlands, since cannabis is relatively easily accessible through coffee shops.

In conclusion, the relatively low sensitivity and specificity presented in this paper indicate that the SDS does not correlate well with the CIDI DSM‐VI diagnosis in frequent cannabis users. Therefore, use of the SDS as a screener to differentiate between dependence and non‐dependence is not recommended within populations of young adult frequent and heavy cannabis users.

Declaration of interest statement

The authors have no conflicts of interest.