Derivation and assessment of the opioid use disorder severity scale: prediction of health, psychological and social adjustment problems

Abstract

Background:

Severity of substance use disorder (SUD) is typically evaluated by tabulating the number of symptoms. The resulting estimate of disorder severity is, however, biased due to intercorrelations among symptoms and their unequal salience.

Objective.

Employing item response theory (IRT) methodology, opioid use disorder symptoms were calibrated to derive the Opioid Use Disorder Severity Scale (OUDSS) and assess its predictive ability in men and women separately.

Methods:

A two-parameter IRT model was utilized to derive the OUDSS from DSM-IV symptoms recorded on the Structured Clinical Interview for DSM-IV (SCID) in 438 men and 429 women who reported at least one lifetime opioid consumption event. The predictive ability of the OUDSS was evaluated using the 10 health, psychological, and social adjustment domains of the revised Drug Use Screening Inventory (DUSI-R) assessed 2 years later.

Results:

The OUDSS score predicted the severity of problems in all 10 DUSI-R domains in men and women. The OUDSS also predicted the DUSI-R diagnostic cutoff score of overall problem density score in men and women (OR = 2.21 and OR = 4.83, respectively). Withdrawal was the most frequently endorsed symptom in this sample of opioid users. The other symptoms’ frequencies, while somewhat lower than withdrawal’s, did not differ from it substantially, indicating a similar severity threshold.

Conclusions:

The OUDSS enables dimensional measurement of opioid use severity on an interval scale. The OUDSS and DUSI-R together can identify problem areas requiring prevention or treatment.

Diagnosis of medical diseases is a multifaceted process blending observation of morphological and physiological signs of disease during the physical examination with the interpretation of symptoms reported by the patient, review of information contained in the medical history of the patient and family, and indicators of pathology revealed by specialized laboratory tests. Nevertheless, substance use disorder (SUD) diagnosis and measurement of its severity are often limited to tabulating symptoms. Symptom counts, however, have shortcomings that threaten validity, including strong correlations between the symptoms (i.e., item dependency) and the symptoms’ unequal relationship with disorder severity. The resulting scale is thus also ordinal rather than interval, limiting analytic approaches. Measurement accuracy has important practical ramifications, considering that treatment placement (e.g., inpatient vs. outpatient) and intervention tactics are based largely on a judgment about disorder severity (1).

One approach that may help surmount measurement difficulties is item response theory (IRT), a psychometric test theory that relates the performance of an examinee (or symptom endorsement) to a latent trait (severity) that the test (or the diagnostic criteria) is intended to measure. Instruments developed using IRT methods have the advantage of simultaneously accounting for the characteristics of the individual and the properties of each item used in a scale. Applied to diagnostic criteria, the IRT parameters allow taking into account that different symptoms have different frequencies of endorsement and different ability to discriminate between levels of severity, which are mapped on an interval scale.

It has been demonstrated that substance use disorder symptoms are indicators of a single trait (unidimensional) (2–4), a condition for unidimensional IRT application. This finding is consistent with the concept of liability to SUD, a latent continuous trait underlying variation in the risk for the disorder in the population (5–12). Accordingly, IRT has been applied to OUD diagnostic criteria (13). The symptoms in that study, however, were limited to DSM-IV opioid dependence, thus potentially censoring the limited range of severity measurement.

This study had two aims: First, IRT methodology was employed to derive the Opioid Use Disorder Severity Scale (OUDSS) that could extend to the subdiagnostic segment of the opioid-using population (i.e., those who have not developed the number of symptoms requisite for a positive OUD diagnosis), using the DSM-IV symptoms of both abuse and dependence. The resulting score could thus inform urgency for prevention as well as treatment placement. Second, in view of manifold deleterious consequences of hazardous opioid use, including fatal overdose (14), trauma (15), infection (16–20), criminal justice involvement (21–23), and serious psychiatric illness (24,25), it was determined whether the OUDSS score predicts health, psychological, and social adjustment problems 2 years later and detects individuals who require intervention in these areas.

Methods

Participants

The sample consisted of 438 men and 429 women (average age 41.1 and 38.7 years, respectively) who reported at least one lifetime opioid consumption event. They were recruited using advertisement, public service announcements, random digit telephone calls, and posters displayed in addiction treatment facilities. Within this sample of opioid users, 79 men and 39 women qualified for a DSM-IV (26) diagnosis of opioid use disorder (abuse or dependence). An IQ of 80 or higher, good health, and no lifetime psychosis were required to be included in the study.

Men and women were analyzed separately, because sex differences in addiction have been reported with respect to etiology, natural history, psychiatric comorbidity, and medical consequences (27–29). IQ was in the average range in the men (M = 106) and women (M = 104). Family socioeconomic status based on Hollingshead’s four-factor index corresponded to the middle class (men: M = 40.2; women: M = 40.8). African-Americans comprised 21% and 18.4% of the samples of men and women, respectively. The rest of the sample were European-Americans.

Instrumentation

Structured clinical interview for DSM-IV (SCID) (26).

The Opioid Use Disorder Severity Scale (OUDSS) was derived from the symptoms recorded in the SCID. This interview, administered by trained master-level clinical associates, records 1) tolerance, 2) withdrawal, 3) consumption of increasing amounts or for a longer period than intended (abbreviated as “larger/longer”), 4) desire or unsuccessful efforts to lower consumption (“quit/control”), 5) extensive effort or time devoted to obtaining or consuming opioids (“time spent”), 6) reduced social, occupational and recreation activities (“activities given up”), 7) persisting consumption despite physical or psychosocial problems (“physical/psychological”), 8) failure to fulfill major role obligations (“neglect roles”), 9) persisting consumption in the face of hazard (“hazardous use”), 10) legal problems, and 11) continued use despite or recurrent social or interpersonal problems (“social/interpersonal”). Each symptom was coded as present (“1”) or absent (“0”).

Drug use screening inventory (DUSI-R) (30)

The DUSI-R’s psychometric properties (31,32) and utility in clinical (32,33) settings have been previously discussed. The revised self-administered DUSI-R quantifies severity of 1) substance use, 2) mental health problems, 3) medical problems, 4) behavior dysregulation, 5) school performance/adjustment problems (where applicable), 6) family maladjustment, 7) difficulties in peer relationships, 8) social skills deficits, 9) work maladjustment (where applicable) and 10) leisure/recreation problems. The problem density score for each of these scales, ranging from 0% to 100%, is computed as the number of endorsed problems divided by the number of items multiplied by 100. The same procedure is used to calculate the overall problem density (OPD) score based on the entire set of 149 items. An OPD score exceeding 20% indicates the need for intervention (33).

In addition, the DUSI-R’s Lie scale, consisting of 10 items, detects bias toward under-reporting problems. A score of seven or higher reflecting denial of high probability behaviors (e.g., “I have never told a lie”) disqualified the person from the study.

Procedure

This project was approved by the Institutional Review Board of the University of Pittsburgh. After a complete description of the study was provided to the participants, written informed consent was obtained. All of the study participants were informed that the findings from this research were protected by a Certificate of Confidentiality issued to Center for Education and Drug Abuse Research (CEDAR) by the National Institute on Drug Abuse. Next, the assessments were individually conducted in sound-attenuated rooms by trained Master-level examiners. Before discharging the participant from the laboratory, the questionnaires were checked to ensure data completeness. Lastly, the participant was debriefed and compensated for their time and incidental expenses at the rate of $10/hour.

Statistical analysis

Mplus (34) was employed to conduct exploratory factor analysis (EFA) for binary variables (presence/absence of OUD symptoms) using weighted least squares with mean and variance correction (WLSMV) in a two-group (men and women) analysis to assess the dimensionality of the data (35).

To derive the Opioid Use Disorder Severity Scale (OUDSS), IRT analysis was conducted. The relationship between the latent trait (severity) and the probability of item (symptom) endorsement in IRT is described by the item characteristic function (IRF). In the widely-used two-parameter models, the frequency of endorsement corresponds to the item difficulty (location) parameter (b; the severity scale value at which the probability of a symptom’s endorsement exceed 0.5) and the item’s ability to distinguish between different levels of severity corresponds to the item discrimination parameter (a proportional to the slope of IRF at the point b on the severity scale). Mplus implements IRT analysis within the framework of confirmatory factor analysis (CFA), and provides an alternative parameterization of the unidimensional factor model to output the discrimination and location item parameter estimates for a two-parameter-logistic IRT model (36), relating a particular symptom to the severity of the disorder (37).

Finally, differential item functioning (DIF) analysis was conducted using latent class variable modeling (38) for examining measurement invariance between men and women (differences in performance across the entire range of the severity scale). First, in the restricted two-sample model (M0), all item thresholds and all item factor loadings for respective OUD symptoms are set equal across sexes. The mean and variance of the OUDSS latent variable are set to 0 and 1, respectively, in males, but set free in females. In the second step, 22 (2 × 11 symptoms) models are created from M0 by consecutively setting free an individual threshold and an individual factor loading as a single parameter in the female group. Because M0 is nested in all 22 models, the change in likelihood ratio χ² statistics with one degree of freedom is used to compare nested models and decide whether a free threshold or a factor loading is identical across groups. The Benjamini-Hochberg false discovery rate (FDR) procedure (39) was then applied to adjust the 22 p-values to determine in which of the comparisons, if any, the models are to be rejected. If none of the 22 models is rejected, measurement invariance is confirmed. Otherwise, if particular models are rejected, there are particular non-invariant parameters that demonstrate DIF, and measurement invariance does not hold.

Variables-centered and person-centered analyses examined whether the OUDSS has practical utility. First, simple linear regression analysis was conducted to assess the association between OUDSS and DUSI-R scales. Logistic regression analysis was performed to evaluate the accuracy of the IRT-based OUDSS for identifying men and women who evince clinically significant health, behavior, and social adjustment problem 2 years later, using the DUSI-R OPD score’s diagnostic threshold (33). The unidimensional CFA model used for OUDSS derivation was extended to include the DUSI-R subscales as outcome measures predicted by the OUDSS, with error terms correlated to account for sources of shared variance other than opioid use severity. In estimating the regression coefficients of the DUSI-R subscales, the factor loadings of the OUD symptoms were fixed to their values estimated under the original CFA model. Finally, receiver operating characteristic (ROC) analysis was performed, comparing the predictive validity of the OUDSS and the OUD symptom counts.

Results

All the item-total correlations of the OUD symptoms (Table 1) are high in both sexes. Alpha coefficients, documenting coherence among the symptoms, are 0.96 in men and 0.95 in women. In exploratory factor analysis, all the symptoms have high loadings on the first factor, which accounts for 73% and 80% of overall variance in men and women, respectively. These results clearly demonstrate a unidimensional structure of OUD symptoms (40). The good fit of one-factor model in both men and women is also supported by the subsequent confirmatory factor analysis (χ² = 50.75, df = 44, p = .23, RMSEA = .019, CFI = .99, TLI = .99; and χ² = 38.55, df = 44, p = .70, RMSEA<.001, CFI = .99, TLI = .99, respectively).

Table 1.

Endorsement rates, item-total correlations, and factor loadings of OUD symptoms.

	Men				Women
OUD Symptoms	Total Sample (n, %, p*) (n = 438)	Subsample with OUD diagnosis (n, %, p**) (n = 79)	ρ (n = 438)	λ (n = 438)	Total Sample (n, %) (n = 429)	Subsample with OUD diagnosis (n, %) (n = 39)	ρ (n = 429)	λ (n = 429)
Tolerance	56, 12.8 (.001)	55, 69.6 (.406)	.85	.88	31, 7.2	30, 76.9	.91	.93
Withdrawal	66, 15.1 (.002)	66, 83.5 (.605)	.91	.93	35, 83.2	34, 87.2	.94	.95
Larger amounts/longer time	61, 13.9 (.001)	59, 74.7 (.970)	.87	.90	29, 6.8	29, 74.4	.91	.93
Desire/efforts to quit/control	57, 13.0 (.001)	56, 70.9 (.920)	.86	.89	28, 6.5	28, 71.8	.91	.92
Time spent to obtain, use, or recover	63, 14.4 (.003)	62, 78.5 (.254)	.91	.93	34, 7.9	34, 87.2	.93	.94
Activities given up	45, 10.3 (.064)	44, 55.7 (.050)	.82	.86	29, 6.8	29, 74.4	.93	.94
Physical/psychological problem	30, 6.8 (.361)	29, 36.7 (.042)	.64	.69	23, 5.4	22, 56.4	.80	.83
Neglect roles	36, 8.2 (.014)	35, 44.3 (.940)	.73	.77	18, 4.2	17, 43.6	.65	.70
Hazardous use	38, 8.7 (.040)	38, 48.1 (.396)	.73	.75	22, 5.1	22, 56.4	.78	.82
Legal problems	60, 13.7 (.001)	60, 75.9 (.850)	.86	.89	30, 7.0	29, 74.4	.86	.89
Social/interpersonal problems	58, 13.2 (.008)	56, 70.9 (.190)	.86	.89	33, 7.7	32, 82.1	.92	.94

Notes: OUD, opioid use disorder; ρ, symptom-total score correlations; λ, factor loadings;

^*p-values for comparison of male versus female endorsement rates in total samples;

^**p-values for comparisons of men and women who qualify for OUD diagnosis.

The results of IRT analyses are depicted in Figure 1. As can be seen, withdrawal is the most frequently endorsed symptom in both sexes (Table 2), thus manifest at relatively low disorder severity. Only 2.7% of men and 0.9% of women in the sample had any other OUD symptoms in the absence of withdrawal. Other symptoms, however, have similar endorsement rates (e.g., “time spent,” “legal problems”) in the narrow low range of the Opioid Use Disorder Severity Scale. It should also be noted that withdrawal or any other symptoms are virtually never observed among individuals in this sample who do not qualify for an OUD diagnosis (withdrawal: males, 0%; females, 0.3%). At the high end of severity, women most frequently endorse “use despite a persistent physical or psychological problem” (“physical/psychological”), whereas men endorse neglect of role obligations.

Figure 1.

Item response functions (IRF) of DSM-IV OUD abuse and dependence symptoms in men and women. IRF shows the association between the probability of endorsing a particular symptom and Opioid Use Disorder Severity Scale. Probability of each diagnostic symptom in relation to severity of opioid use disorder is reported. Withdrawal is the most frequently endorsed symptom in the sample of opioid users. The other symptoms do not differ from withdrawal substantially, indicating a similar severity threshold. A two-parameter logistic item response theory model is used to derive the Opioid Use Disorder Severity Scale.

Table 2.

Association between IRT-based OUDSS score and problems measured by the DUSI-R and its diagnostic threshold 2 years later.

DUSI Domains	Standardized Regression Coefficients		Logistic Regression Analysis
	Men	Women	Men	Women
	β (p-value)	β (p-value)	OR (p-value); (95% CI)	OR (p-value); (95% CI)
Substance Use	.29 (<.001)	.58 (<.001)	2.01 (.001); (1.36, 2.95)	8.88 (<.001); (4.49, 17.55)
Psychiatric	.13 (.018)	.27 (<.001)	1.40 (.08); (.96, 2.03)	3.78 (<.001); (1.81,7.90)
Health	.18 (.002)	.21 (<.001)	1.55 (.06); (.99, 2.43)	1.90 (.15); (.79, 4.56)
Behavior	.16 (.003)	.36 (<.001)	1.81 (.003); (1.23, 2.67)	2.77 (.002); (1.44, 5.32)
School	.14 (.011)	.21 (<.001)	1.71 (.006); (1.17, 2.49)	2.22 (.010); (1.22, 4.07)
Family	.17 (.003)	.26 (<.001)	1.61 (.013); (1.11, 2.35)	3.77 (<.001); (1.85, 7.72)
Peers	.25 (<.001)	.32 (<.001)	1.85 (.002); (1.26, 2.71)	4.24 (<.001); (2.23, 8.04)
Social Skill	.19 (<.001)	.56 (<.001)	1.74 (.028); (1.06, 2.85)	3.25 (<.001); (1.68, 6.31)
Work	.32 (<.001)	.43 (<.001)	2.18 (.001); (1.43, 3.22)	7.70 (<.001); (3.98, 14.91)
Leisure/Recreation	.21 (<.001)	.28 (<.001)	1.73 (.006); (1.17, 2.58)	2.65 (.008); (1.29, 5.42)
Overall Problem Density Score	.27 (<.001)	.45 (<.001)	2.21 (.001); (1.50, 3.25)	4.83 (<.001); (2.47, 9.45)

Upon showing the homogeneity of the single-factor structure for both sexes, measurement invariance approach was used to test DIF. The restricted M0 model had a likelihood ratio χ² statistic of 238.56, df = 4044, p = 1.00. While the mean and variance of OUDSS in males are fixed at 0 and 1, respectively, the mean and variance in females are 0.42 and 0.56, respectively. Twenty-two nested models were then fitted and compared with the M0 model. The analyses suggest that measurement invariance does not hold across sexes for two symptoms, “quit/control” and “physical/psychological,” indicating that men and women cannot be combined for OUDSS derivation. Item thresholds and item loadings for these items significantly differ (p < .001 for both parameters, and p = .003 for item threshold and p = .006 for item loading, respectively). Therefore, sex-specific OUDSS was derived.

Figure 2a,b presents the DUSI-R profiles of the men and women categorized according to whether they are subdiagnostic or qualify for opioid use disorder. In the subdiagnostic segment of this opioid-using sample, men scored higher than women on scales measuring substance use consequences (t = 7.04, p < .001), social skills deficit (t = 9.30, p < .001), work maladjustment (t = 5.44, p < .001), peer relationship problems (t = 8.08, p < .001), and the overall problem density score (t = 3.81, p < .001), whereas women scored higher than men on health problems (t = −2.67, p = .008). The severity of substance use consequences (t = 3.51, p < .001) and health problems (t = 2.67, p = .008) was significantly higher in women than in men in the affected (three or more DSM-IV OUD symptoms) sample.

Figure 2.

DUSI-R profiles of men and women (a) who are not qualified for a DSM-IV OUD diagnosis, subdiagnostic and (b) qualified for opioid use disorder. Asterisks denote significant differences between men and women at .01 level. In subdiagnostic group, men scored higher than women on substance use consequences, social skills deficit, work maladjustment, peer relationship problems, whereas women scored higher than men on health problems. In the affected sample, substance use consequences and health problems are significantly higher in women than in men.

As can be seen in Table 2, simple linear regression analysis shows that the OUDSS and all the DUSI-R scales are significantly correlated in men and women. Logistic regression analysis also indicates that the OUDSS predicts the DUSI-R diagnostic threshold at 2 years later with the exception of psychiatric problems in men and health problems in both men and women.

The results of the extended CFA with the DUSI-R scales modeling the relationships between the IRT-based OUDSS score at baseline and the DUSI-R scores 2 years later show that in both men and women, the OUDSS predicted all DUSI-R domains: substance use (β = .49, p < .001; β = 1.74, p < .001), psychiatric problems (β = .22, p = .018; β = .71, p < .001), behavioral problems (β = .27, p = .004; β = .95, p < .001), health problems (β = .27, p = .004; β = .55, p < .001), school performance (β = .24, p = .011; β = .58, p < .001), family maladjustment (β = .28, p = .003; β = .69, p < .001), social competence (β = .31, p < .001; β = .89, p < .001), work maladjustment (β = .55, p < .001; β = 1.57, p < .001), peer relationships (β = .42, p < .001; β = 1.29, p < .001), and leisure/recreation (β = .35, p < .001; β = .73, p < .001). The logistic regression analysis documents a very strong association between the OUDSS and DSM-IV OUD diagnosis in men and women (OR = 167.79, p < .001, and OR = 334.89, p < .001).

For comparison, a linear regression and logistic regression were used to predict the DUSI-R overall problem density score (OPD) and its diagnostic threshold by the symptom count. In men and women, the symptom count predicted the DUSI-R OPD (β = .29, p < .001; β = .43, p < .001) and diagnostic threshold of the OPD score (OR = 1.19, p < .001; OR = 1.32, p < .001). The area under the curve, sensitivity, and specificity of ROC were similar between the OUDSS (men: area under the curve = .59, sensitivity = .60, specificity = .57; women: area under the curve = .59, sensitivity = .60, specificity = .59) and the symptom count (men: area under the curve = .59, sensitivity = .60, specificity = .58; women: area under the curve = .59, specificity = .60, specificity = .58).

Discussion

This study confirms previous findings showing that the DSM-IV diagnostic criteria for opioid use disorder (OUD) are indicators of a unidimensional trait (4,5,41), and extends prior research by incorporating opioid abuse criteria as indicators of disorder severity. Although DSM-5 criteria do not exactly match DSM-IV, the differences between the respective symptom sets are small (the absence of legal problems in the former and of craving in the latter). As the symptoms are indicators rather than causes of the latent trait, the absence of an indicator does not substantially affect the meaning/content of the trait, consistent with the trait invariance property of IRT (e.g., 37).

The study also replicates prior research (5) in that withdrawal is the most frequently reported symptom, thus corresponding to the relatively low end of OUD severity in men and women. These findings indicate that withdrawal may drive the emergence of the other symptoms (42). Notably, however, withdrawal and other OUD symptoms were almost never observed among individuals who do not qualify for an OUD diagnosis in this sample of opioid users, corresponding to the high discriminating ability of the symptoms and, on the other hand, their inability to measure severity below the diagnostic threshold.

It is noteworthy that withdrawal and tolerance are not by themselves diagnostic criteria in the DSM-5 taxonomy if the opioid is prescribed by a physician. This is consistent with understanding that these physiological symptoms, developing in the course of therapy, are not necessarily followed by the compulsive pursuit of a drug, the essence of addiction (43), while frequently playing a role in its initiation if it does occur. In effect, withdrawal and tolerance, as causes of other OUD symptoms that are seldom observed in the absence of the physiological symptoms, may be redundant in the item set used for severity measurement. This possibility will be explored in future work.

Most symptoms in both sexes cluster in the relatively low range of OUD severity. However, differential expression of OUD symptoms in men and women points to the need to tailor treatment placement and intervention tactics to each sex, especially in light of the distinctive configuration of problems measured by the DUSI-R 2 years later. In addition, supporting the above conclusions pertaining to withdrawal and tolerance, this finding suggests the need to expand the symptom set to better cover the low severity (subdiagnostic) range. Such a modification would allow for more nuanced measurement that is particularly beneficial for individuals who are proximal to the diagnostic threshold. Opioid-using men who are subdiagnostic for OUD (less than three DSM-IV OUD symptoms) (Figure 2a) report more severe problems on DUSI-R scales measuring substance use effects, social skill deficits, work maladjustment, and peer relationship problems, whereas in the affected population the scores on the DUSI-R scales measuring substance use consequences and health problems are more severe in women (Figure 2b). This finding corresponds to the expectation for multifactorial disorders that greater severity will be observed in the sex that is affected less frequently (44).

While the predictive validity of the OUDSS is virtually identical to that of the symptom count, the IRT-derived instrument has substantial advantages over the sum of items, in addition to the benefits incurred from using an interval rather than ordinal scale. For instance, using the IRT-derived instrument allows a substantial shortening of the administration time. Importantly, lengthy questionnaires not only detract from treatment delivery but may also incur an unacceptable cost. Fixed length tests may not even be the optimum method of measurement. As discussed by Weiss (45), some items in fixed length instruments may contribute to error because measurement precision declines at both the high and low level of the trait. Computer adaptive testing (CAT), afforded by IRT-derived tests, provides a solution to these problems. It mitigates measurement error while maximizing efficiency since only the items pertinent to accurately measuring trait level are administered. Moreover, the cost is minimal because scoring the responses is conducted automatically and immediately after completion of the questionnaire. Privacy is also ensured because there is no record of the person’s responses on paper and access to the information is protected by a password. These advantages have led to the adoption of the CAT format in research to evaluate mental health and psychopathology (46).

Two limitations of this study warrant caution in interpreting the results. First, selective attrition may have influenced the results, although analysis of this potential validity threat did not reveal a systematic bias as the variables in this study show no difference between retained and attrited participants (10,32–32). Second, the OUD criteria align with the DSM-IV taxonomy because data collection began before the advent of the DSM-5. However, SUD diagnoses formulated according to the DSM-IV and DSM-5 taxonomies have strong concordance (47). As noted above, from a methodological perspective, it is likely that replicating this study with DSM-5 criteria for IRT analysis would not substantially alter the results.

In conclusion, applying IRT methodology to OUD symptoms enables quantification of OUD severity on an interval scale. The OUDSS score forecasts drug-related problems 2 years later. Thus, self-administering the OUDSS and DUSI-R would expeditiously yield three important pieces of information (1): presence/absence of OUD (2), a severity score on the continuum encompassing the subdiagnostic and affected segments of the opioid-using population, and (3) problem areas requiring prevention or treatment. Lastly, the finding that withdrawal and other symptoms correspond to the low end of OUD severity in both sexes underscores the importance of extending the severity measurement scale using other indicators.