Abstract
Introduction:
Substance use disorders (SUDs) are underdiagnosed in healthcare settings. The Substance Use Symptom Checklist (SUSC) is a practical, patient-report questionnaire that has been used to assess SUD symptoms based on Diagnostic and Statistical Manual of Mental Disorders-5th edition (DSM-5) criteria. This study evaluates the test-retest reliability of SUSCs completed in primary and mental health care settings.
Methods:
We identified 1194 patients who completed two SUSCs 1–21 days apart as part of routine care after reporting daily cannabis use and/or any past-year other drug use on behavioral health screens. Test-retest reliability of SUSC scores was evaluated within the full sample, subsamples who completed both checklists in primary care (n=451) or mental health clinics (n=512) where SUSC implementation differed, and subgroups defined by sex, insurance status, age, and substance use reported on behavioral health screens.
Results:
In the full sample, test-retest reliability was high for indices reflecting the number of SUD symptoms endorsed (ICC=0.75, 95% CI:0.72–0.77) and DSM-5 SUD severity (kappa=0.72, 95% CI:0.69–0.75). These reliability estimates were higher in primary care (ICC=0.81, 95% CI:0.77–0.84; kappa=0.79, 95% CI:0.75–0.82, respectively) than in mental health clinics (ICC=0.74, 95% CI:0.70–0.78; kappa=0.73, 95% CI:0.68–0.77). Reliability differed by age and substance use reported on behavioral health screens, but not by sex or insurance status.
Conclusions:
The SUSC has good-to-excellent test-retest reliability when completed as part of routine primary or mental health care. Symptom checklists can reliably measure symptoms consistent with DSM-5 SUD criteria, which may aid SUD-related care in primary care and mental health settings.
Keywords: substance use disorder, assessment, measurement-based care, primary care, symptom checklist
INTRODUCTION
Substance use and substance use disorders (SUDs) contribute significantly to disability and early mortality globally (Degenhardt et al., 2018; Rehm and Shield, 2019). In the US, 22.9% of people age 18 or older report past-month use of drugs other than alcohol and tobacco and 8.9% meet past-year criteria for a substance use disorder (SUD) other than alcohol and tobacco use disorders (Substance Abuse and Mental Health Services Administration [SAMHSA], 2023). Most people with SUDs do not seek specialty treatment (SAMHSA, 2023); however SUDs can be diagnosed within primary care settings where counseling from integrated behavioral health specialists and medications for some SUDs may be offered.
DSM-5 SUD symptoms can potentially be assessed by using practical, patient-report questionnaires after patients report high risk substance use on behavioral health screens. In contrast to other forms of assessment (e.g., drug toxicology screens), patient-report questionnaires can provide information about SUD symptoms to help providers diagnose SUDs and determine the severity of SUDs when symptoms are present. Because SUD symptoms constitute negative consequences from substance use, patient-report symptom questionnaires may also help patients recognize adverse consequences of substance use, help providers engage in discussions about SUD-related harms and treatment options, and highlight SUD treatment as a potential pathway for recovery and reduced substance use-related harm.
In 2015, Kaiser Permanente (KP) Washington began to implement screening of all adult primary care patients for past-year cannabis and other drug use across primary care settings (Bobb et al., 2017; Glass et al., 2018; Lapham et al., 2017; Lee et al., 2023; Richards et al., 2019; Yeung et al., 2020). Patients who screened positive for past-year daily cannabis use or any past-year use of other illegal drugs or prescription medications for non-medical reasons were typically prompted to complete assessments of the 11 DSM-5 SUD criteria using the Substance Use Symptom Checklist (SUSC) (Matson et al., 2023; Sayre et al., 2020) to aid SUD diagnosis and facilitate substance use related discussions and shared decision-making (Bradley et al., 2019; Sayre et al., 2020). Previous cross-sectional analyses supported the construct validity of SUSCs completed in routine care, with SUSCs discriminating SUD severity along a scaled, unidimensional continuum of severity consistent with DSM-5 conceptualizations of SUD (American Psychiatric Association, 2013) and having minimal differential item functioning across demographic subgroups (Matson et al., 2023). A previous study also showed that Alcohol Symptom Checklists, which measure DSM-5 alcohol use disorder symptoms, had high test-retest reliability when used in routine care (Hallgren et al., 2022). However, no studies have evaluated the test-retest reliability of the SUSC.
The current study evaluated the test-retest reliability of SUSCs completed in routine care. It also evaluated whether test-retest reliability varied between primary care versus mental health clinic settings and across other patient factors, including age, sex, substance use reported on behavioral health screens (daily cannabis and/or any other drug use) and insurance status. Using routine care data helped increase external validity and generalizability of the findings by evaluating the measure’s performance under real-world conditions (vs. confidential research settings) and by including patients who may not have opted to participate in SUD-focused research.
METHODS
Study Setting and Population
The current study used data from electronic health records (EHRs) from KP Washington. The KP Washington Health Research Institute’s Institutional Review Board approved the study procedures with waivers of consent and Health Insurance Portability and Accountability Act authorization.
Patients were included in the current study if they (a) had ≥1 visit to a KP Washington primary care setting between March 1, 2015 and February 29, 2020, (b) completed two SUSCs 1–21 days apart, (c) reported daily cannabis use and/or any past-year other drug use the same day each SUSC was completed (described below), and (d) were at least 18 years old when the SUSCs were completed. The 21-day test-retest window was selected to reflect a period in which past-year SUD symptoms were not expected to change significantly and to match previous test-retest reliability studies (Hallgren et al., 2022; Hasin et al., 2020). For patients with more than one pair of SUSCs completed 1–21 days apart, a single pair of SUSCs was retained by selecting the pair that was completed over the shortest time interval.
Measures
Cannabis and Other Drug Use Screens
KP Washington’s annual behavioral health screens include two questions about past-year substance use (Matson et al., 2022; Sayre et al., 2020). They include (1) the validated Single-Item Screen—Cannabis (Matson et al., 2022), which asks about frequency of past-year cannabis use, and (2) the Single-Item Screen—Other Drug which was adapted from a validated screen (Smith et al., 2010) and asks about frequency of past-year use of illegal drugs or prescription medications for non-medical reasons, excluding cannabis. The decision to use these two questions was informed by input from the health delivery system and the legal status of cannabis in Washington state. Response options for both items are on a 5-point Likert scale ranging from “never” to “daily or almost daily” (Babor et al., 2001). Both questions were included in the behavioral health screen completed annually at primary care visits and the mental health monitoring tool completed prior to most mental health visits. Patients who reported “daily or almost daily” cannabis use or any past-year other drug use on behavioral health screens were asked to complete a SUSC. This targeted SUD symptom assessment approach was developed in partnership with the health delivery system to limit burden to patients and providers and to focus assessment toward patients with the highest risk. The annual behavioral screen included separate questions to screen for high-risk alcohol consumption. Patients reporting high-risk alcohol consumption were typically asked to complete a separate Alcohol Symptom Checklist that assessed DSM-5 alcohol use disorder symptoms. The psychometric performance of Alcohol Symptom Checklists is reported elsewhere (Hallgren et al., 2022, 2021).
Substance Use Symptom Checklist (SUSC)
The SUSC (available in supplement) (Matson et al., 2023; Sayre et al., 2020) is an 11-item self-report questionnaire on which patients indicate whether they have experienced each of the 11 DSM-5 SUD criteria within the past year, a timeframe consistent with DSM-5. Total scores reflect summed symptom counts (0–11) that may assist with determining if an SUD is present (if ≥2 symptoms are verified by providers and determined to be recurrent, per DSM-5) and, when SUD is present, the severity of the SUD (i.e., 2–3 criteria reflecting mild SUD; 4–5 moderate SUD; 6–11 severe SUD). The checklist is available in 17 languages and typically takes 1–2 minutes to complete.
Questions on the SUSC are not phrased to ask about a specific substance, so the same form can be used for all patients reporting daily cannabis and/or any other past-year drug use, regardless of the specific substance(s) they report using. Questions were not tailored to specific substances to mitigate workflow burden (e.g., multiple versions of checklists would need to be stocked), assessment fatigue (i.e., substance-specific assessments could become very lengthy for patients who use multiple substances), and potential difficulties in accurately attributing symptoms to specific substances (i.e., it is often difficult to tell which substance(s) caused which symptom(s) for patients who use multiple substances). The checklist includes a section where patients could indicate which substance(s) they used; however, this information was often missing and therefore was not utilized for the current analysis.
Procedures for implementing the SUSC differed in primary care and mental health clinics. In primary care clinics, medical assistants received training in procedures for standard administration of the SUSC with practice facilitation, and this training was later built into standard staff onboarding (Lee et al., 2023). After patients reported daily cannabis and/or any other past-year drug use on behavioral health screens, primary care medical assistants were prompted by the EHR to give patients a paper form with the SUSC. Patients completed the checklist in writing, medical assistants entered patient responses into the EHR, and results were available to the patient’s primary care provider during the visit.
In mental health clinics, implementation was left up to sites, with no standard training procedures for staff. When patients in mental health clinics reported daily cannabis and/or any other drug use on the mental health monitoring tool, an EHR prompt requested mental health clinicians to administer the SUSC. However, the procedures for administering the SUSC were less standardized in specialty mental health settings (e.g., paper checklists were not always routinely stocked and mental health clinicians could have administered the checklist in ad hoc ways, including by reading the measure aloud which can introduce variability and measurement error; Williams et al., 2015).
Descriptive measures and covariates
Patient age, sex, race, and ethnicity were obtained from the EHR. Medicaid and Medicare insurance coverage status were obtained from enrollment records. SUD diagnoses by healthcare providers within the two years prior to completing the SUSC were identified using data from insurance claims and the EHR, and included visit diagnoses and diagnoses that were active on problem lists within the prior two years.
Analytic Approach
Test-retest reliability analyses were modeled after previous studies that evaluated the test-retest reliability of confidential, clinician-administered diagnostic research interviews (Hasin et al., 2020) and the test-retest reliability of Alcohol Symptom Checklists (Hallgren et al., 2022). Test-retest reliability coefficients were estimated for four composite measures derived from the number of SUD symptoms reported on the checklist, including (1) total scores (i.e., number of SUD symptoms reported on the checklist, 0–11), (2) an ordered categorical measure reflecting SUD severity (i.e., total scores consistent with mild, moderate, severe, or no SUD), (3) a binary indicator of symptoms consistent with SUD (2+ symptoms) versus no SUD (0–1 symptoms), and (4) a binary indicator of symptoms consistent with moderate or severe SUD (4+ symptoms) versus no SUD or mild SUD (0–3 symptoms). Test-retest reliability estimates were computed using one-way single-measures agreement intraclass correlation coefficients (ICCs; McGraw and Wong, 1996) for total scores, weighted kappa (Cohen, 1968) for the ordered categorical measure, and kappa (Cohen, 1960) for the binary indicators. Test-retest reliabilities were estimated for the 11 SUSC items (reflecting individual SUD criteria) using kappa. Reliability coefficients were interpreted using established rules of thumb (Cicchetti, 1994) for excellent (0.75–1.00), good (0.60–0.74), fair (0.40–0.59), and poor reliability (≤0.39).
Test-retest reliability coefficients (and 95% CIs) were estimated for the full analytic sample, then within stratified subsamples of patients who completed both SUSCs in primary care settings (primary care clinic subsample) or both SUSCs in mental health settings (mental health clinic subsample). We hypothesized that test-retest reliability would be higher in the primary care clinic subsample, due to standardized procedures for administering SUSCs in primary care, compared to the mental health clinic subsample where procedures for administering SUSCs were less standardized (Hallgren et al., 2022). Differences in reliability between the primary care versus mental health clinics were tested using parametric bootstrapping with 10,000 resampled coefficient estimates.
Additional analyses tested whether test-retest reliability differed based on patient age, sex, substances endorsed on behavioral health screens preceding the first SUSC (i.e., daily cannabis use only, other past-year drug use only, or both daily cannabis and past-year other drug use) and Medicaid and Medicare insurance coverage. For these analyses, a dichotomous code was created to reflect that the first (T1) and second (T2) SUSCs agreed (agreement=1) or disagreed (agreement=0) in reflecting symptoms consistent with moderate/severe SUD (4–11 symptoms) or both checklists reflecting mild/no SUD (0–3 symptoms). We selected these categories due to people with moderate/severe SUD more often being recommended SUD treatment (e.g., with medications or specialty care), and those with no SUD or mild SUD more often being offered briefer interventions or no intervention. Small subgroup sizes precluded evaluating differences in reliability across race and ethnicity (e.g., n<20 for many subgroups). Log-likelihood ratio chi-square tests from logistic regression models were used to test for differences in the odds of discordance across the levels of each covariate.
RESULTS
Patient Characteristics
1194 patients met inclusion criteria for the study sample, of whom 1064 (89%) reported only daily cannabis use on T1 screens, 73 (6%) reported only other past-year drug use, and 57 (5%) reported both daily cannabis and other past-year drug use. The primary analytic sample and primary care and mental health clinic subsamples are described in Table 1. Supplemental analyses show how the study sample differed from a broader sample of 20,560 adult patients who completed a SUSC during the same period but did not complete two SUSC’s 1–21 days apart; the study sample reported greater SUD severity on SUSCs and more often had previous SUD diagnoses documented by providers (Supplemental Table 1). In the study sample, 794 patients (66%) had only one eligible pair of SUSCs and 400 patients (34%) had two or more eligible pairs.
Table 1.
Descriptive Statistics for Full Sample and for Primary Care and Mental Health Subsamples
| Full sample (N = 1194) | Primary care subsample (n = 451) | Mental health subsample (n = 512) | P-value of difference between subsamples | |||||
|---|---|---|---|---|---|---|---|---|
|
| ||||||||
| n | (%) | n | (%) | n | (%) | |||
|
|
||||||||
| Age (y) | 18–24 | 298 | (25.0%) | 104 | (23.1%) | 125 | (24.4%) | 0.90 |
| 25–44 | 586 | (49.1%) | 222 | (49.2%) | 253 | (49.4%) | ||
| 45–64 | 250 | (20.9%) | 99 | (22.0%) | 109 | (21.3%) | ||
| 65+ | 60 | (5.0%) | 26 | (5.8%) | 25 | (4.9%) | ||
| Sex | Female | 696 | (58.3%) | 252 | (55.9%) | 321 | (62.7%) | 0.04 |
| Male | 498 | (41.7%) | 199 | (44.1%) | 191 | (37.3%) | ||
| Race | Asian or Asian American | 26 | (2.2%) | 14 | (3.1%) | 12 | (2.3%) | 0.001 |
| Black or African American | 44 | (3.7%) | 26 | (5.8%) | 8 | (1.6%) | ||
| Native American or Alaskan Native | 15 | (1.3%) | 10 | (2.2%) | 3 | (0.6%) | ||
| Native Hawaiian or Pacific Islander | 11 | (0.9%) | 3 | (0.7%) | 4 | (0.8%) | ||
| White | 949 | (79.5%) | 332 | (73.6%) | 427 | (83.4%) | ||
| More than one race | 64 | (5.4%) | 30 | (6.7%) | 23 | (4.5%) | ||
| Other race | 42 | (3.5%) | 19 | (4.2%) | 14 | (2.7%) | ||
| Unknown race | 43 | (3.6%) | 17 | (3.8%) | 21 | (4.1%) | ||
| Ethnicity | Hispanic | 85 | (7.1%) | 42 | (9.3%) | 21 | (4.1%) | 0.005 |
| Not Hispanic | 1062 | (88.9%) | 391 | (86.7%) | 467 | (91.2%) | ||
| Unknown ethnicity | 47 | (3.9%) | 18 | (4.0%) | 24 | (4.7%) | ||
| Insurance | Medicaid | 134 | (11.2%) | 50 | (11.1%) | 58 | (11.3%) | 0.99 |
| Medicare | 122 | (10.2%) | 38 | (8.4%) | 62 | (12.1%) | 0.08 | |
| Substance use reported on behavioral health screens (T1) | Cannabis only | 1064 | (89.1%) | 387 | (85.8%) | 472 | (92.2%) | 0.006 |
| Other drug only | 73 | (6.1%) | 35 | (7.8%) | 20 | (3.9%) | ||
| Both cannabis and other drug | 57 | (4.8%) | 29 | (6.4%) | 20 | (3.9%) | ||
| SUD Severity (T1) | No SUD (0 or 1 symptoms) | 708 | (59.3%) | 240 | (53.2%) | 353 | (68.9%) | < 0.001 |
| Mild SUD (2 or 3 symptoms) | 185 | (15.5%) | 79 | (17.5%) | 60 | (11.7%) | ||
| Moderate SUD (4 or 5 symptoms) | 113 | (9.5%) | 38 | (8.4%) | 45 | (8.8%) | ||
| Severe SUD (6+ symptoms) | 188 | (15.7%) | 94 | (20.8%) | 54 | (10.5%) | ||
| SUD symptom counts (T1), M (SD) | 2.24 | (3.09) | 2.69 | (3.29) | 1.63 | (2.72) | < 0.001 | |
| SUDs diagnosed by healthcare provider in past two years | Alcohol use disorder | 120 | (10.1%) | 40 | (8.9%) | 63 | (12.3%) | 0.11 |
| Cannabis use disorder | 211 | (17.7%) | 43 | (9.5%) | 137 | (26.8%) | < 0.001 | |
| Cocaine or other stimulant use disorder | 40 | (3.4%) | 18 | (4.0%) | 16 | (3.1%) | 0.58 | |
| Opioid use disorder | 70 | (5.9%) | 35 | (7.8%) | 24 | (4.7%) | 0.06 | |
| Other drug use disorder | 65 | (5.4%) | 24 | (5.3%) | 32 | (6.2%) | 0.63 | |
| Any substance use disorder | 335 | (28.1%) | 99 | (22.0%) | 186 | (36.3%) | < 0.001 | |
| Two or more of any SUD categories above | 93 | (7.8%) | 32 | (7.1%) | 47 | (9.2%) | 0.29 | |
Note. P-values reflect differences between the primary care only and mental health only subgroups and were computed using chi-square tests. Patients in the primary care and mental health subsamples completed both Substance Use Symptom Checklists in a primary care or specialty mental health setting, respectively. Bold font is used to indicate p-values < .05.
Full Sample Test-Retest Reliability
Test-retest reliability estimates are reported in Table 2. In the full sample, test-retest reliability was excellent for the number of SUD symptoms endorsed (ICC=0.75), good for the categorical SUD severity designation (weighted kappa=0.72), fair for the binary indicator of any SUD (kappa=0.59), and good for the binary indicator of moderate/severe SUD (kappa=0.68). Reliability coefficients ranged from 0.52–0.61 for the 11 individual SUSC items (Table 2). There were significant but small decreases in the number of symptoms reported at T2 compared to T1 (mean differences≤0.25 symptoms, d≤0.12; Supplemental Table 2).
Table 2.
Test-Retest Reliability Coefficients (and 95% CIs) for the Substance Use Symptom Checklist
| Test-retest reliability (95% CI) | ||||
|---|---|---|---|---|
|
| ||||
| Full-scale measures | Full sample (N = 1194) | Primary care subsample (n = 451) | Mental health subsample (n = 512) | P-value of difference between subsamples |
|
| ||||
| Number of SUD symptoms | 0.75 (0.72, 0.77) | 0.81 (0.77, 0.84) | 0.74 (0.70, 0.78) | 0.01 |
| DSM-5 SUD severity | 0.72 (0.69, 0.75) | 0.79 (0.75, 0.82) | 0.73 (0.68, 0.77) | 0.02 |
| Any SUD (mild, moderate, or severe vs. no SUD) | 0.59 (0.54, 0.64) | 0.64 (0.57, 0.71) | 0.64 (0.57, 0.72) | 0.99 |
| SUD moderate or severe (vs. no SUD or mild SUD) | 0.68 (0.63, 0.73) | 0.76 (0.7, 0.83) | 0.67 (0.58, 0.75) | 0.07 |
|
| ||||
| Item-level responses | ||||
|
| ||||
| 1. Tolerance | 0.54 (0.48, 0.59) | 0.60 (0.52, 0.68) | 0.57 (0.48, 0.65) | |
| 2. Withdrawal | 0.59 (0.53, 0.65) | 0.63 (0.54, 0.72) | 0.55 (0.43, 0.66) | |
| 3. Larger/longer | 0.56 (0.50, 0.62) | 0.62 (0.54, 0.71) | 0.56 (0.46, 0.67) | |
| 4. Quit/control | 0.59 (0.53, 0.65) | 0.66 (0.58, 0.75) | 0.57 (0.47, 0.67) | |
| 5. Time spent | 0.54 (0.46, 0.61) | 0.63 (0.53, 0.73) | 0.50 (0.36, 0.65) | |
| 6. Physical/psychological problems | 0.60 (0.55, 0.65) | 0.62 (0.54, 0.70) | 0.69 (0.61, 0.78) | |
| 7. Neglect roles | 0.59 (0.52, 0.66) | 0.72 (0.62, 0.81) | 0.47 (0.33, 0.60) | |
| 8. Hazardous use | 0.61 (0.54, 0.68) | 0.65 (0.55, 0.74) | 0.65 (0.54, 0.77) | |
| 9. Social/interpersonal problems | 0.56 (0.50, 0.62) | 0.65 (0.57, 0.74) | 0.57 (0.47, 0.66) | |
| 10. Craving | 0.56 (0.51, 0.62) | 0.63 (0.55, 0.71) | 0.58 (0.48, 0.68) | |
| 11. Activities given up | 0.52 (0.45, 0.60) | 0.57 (0.47, 0.67) | 0.47 (0.34, 0.61) | |
Note. Test-retest reliability coefficients were estimated using a one-way single-measures agreement intraclass correlation coefficient (ICC) for number of SUD symptoms, weighted kappa for DSM-5 SUD severity, and kappa for all binary indicators (any SUD, SUD moderate or severe, and item-level responses). Bold font is used to indicate measures where there were significant differences in test-retest reliability between primary care and mental health clinic subsamples (p < .05).
Primary Care and Mental Health Subsamples
451 patients (38%) completed both SUSCs in primary care (primary care subsample) and 512 (43%) completed both in mental health clinics (mental health subsample). The remaining 231 (19%) were completed in different settings (e.g., one in primary care, one in a mental health clinic or other specialty setting) and were not included in setting-based analyses. Compared to the primary care subsample, the mental health clinic subsample had a slightly but significantly higher proportion of patients who were female, white, non-Hispanic, reported daily cannabis only on behavioral health screens, reported symptoms consistent with no SUD on the SUSC (0–1 symptoms), and diagnosed with cannabis use disorder by a healthcare provider in the past two years (Table 1).
In the primary care subsample, test-retest reliability was excellent for the number of SUD symptoms endorsed (ICC=0.81), excellent for the categorical measure reflecting DSM-5 SUD severity (weighted kappa=0.79), good for the binary indicator of symptoms consistent with any SUD (kappa=0.64), and excellent for the binary indicator of symptoms consistent with moderate/severe SUD (kappa=0.76). Coefficients ranged from 0.57 to 0.72 for the 11 individual SUD criteria (Table 2).
In the mental health clinic subsample, test-retest reliability was good for the number of SUD symptoms endorsed (ICC=0.74), good for the categorical measure reflecting DSM-5 SUD severity (weighted kappa=0.73), good for the binary indicator of symptoms consistent with any SUD (kappa=0.64), and good for the binary indicator of symptoms consistent with moderate/severe SUD (kappa=0.67). Reliability coefficients ranged from 0.47 to 0.69 for the 11 individual SUD criteria (Table 2). Parametric bootstrapping analyses indicated that test-retest reliability coefficients for the number of SUD symptoms endorsed and the categorical measure reflecting DSM-5 SUD severity were significantly higher in the primary care subsample than in the mental health clinic subsample, but there were not significant differences for the two binary indicators of symptoms consistent with any SUD and symptoms consistent with moderate/severe SUD (Table 2).
Test-Retest Reliability of Symptoms Consistent with Moderate/Severe SUD (vs. Mild/No SUD) Across Patient Subgroups
Table 3 provides results from analyses testing whether test-retest reliability for the binary indicator of symptoms consistent with moderate/severe SUD (vs. mild/no SUD) differed across other patient subgroups, stratified within primary care and mental health clinic subsamples to eliminate setting as a potential confounder (i.e., given the setting-based differences reported above).
Table 3.
Test-Retest Reliability of Substance Use Symptom Checklists Across Subgroups.
| Primary care subsample (n = 451) | Mental health subsample (n = 512) | ||||
|---|---|---|---|---|---|
|
|
|||||
| Measure | Subgroup | Kappa (95% CI) | χ2 (p-value) | Kappa (95% CI) | χ2 (p-value) |
|
| |||||
| Age (y) | 18–24 | 0.80 (0.68,0.92) | 10.19 (0.02) | 0.70 (0.57,0.83) | 11.84 (0.01) |
| 25–44 | 0.69 (0.58,0.80) | 0.60 (0.47,0.73) | |||
| 45–64 | 0.85 (0.72,0.98) | 0.65 (0.29,1.00) | |||
| 65+ | 1.00 (1.00,1.00) | 0.78 (0.36,1.00) | |||
| Sex | Female | 0.76 (0.66,0.86) | 0.95 (0.33) | 0.67 (0.56,0.78) | 0.17 (0.68) |
| Male | 0.76 (0.66,0.85) | 0.67 (0.53,0.80) | |||
| Substance use reported on behavioral health screens (T1) | Cannabis only | 0.71 (0.63,0.80) | 0.61 (0.74) | 0.67 (0.58,0.77) | 9.17 (0.01) |
| Other drug only | 0.68 (0.34,1.00) | 0.59 (0.23,0.94) | |||
| Both cannabis and other drug | 0.71 (0.46,0.96) | 0.37 (−0.01,0.75) | |||
| Medicaid insurance coverage | No | 0.75 (0.67,0.82) | 0.91 (0.34) | 0.66 (0.57,0.75) | 0.10 (0.75) |
| Yes | 0.87 (0.73,1.00) | 0.73 (0.51,0.95) | |||
| Medicare insurance coverage | No | 0.75 (0.68,0.82) | 3.06 (0.08) | 0.66 (0.57,0.75) | 0.97 (0.32) |
| Yes | 0.91 (0.73,1.00) | 0.68 (0.38,0.97) | |||
Note: Chi-square values and p-values reflect are used to indicate whether there were significant differences in test-retest reliability across subgroups, within either the primary care or mental health clinic subsamples, and were obtained using log-likelihood values from logistic regression models. Bold font is used to indicate measures where there were significant differences in test-retest reliability across subgroups (p < .05).
In the primary care clinic subsample, test-retest reliability differed by age group (lowest reliability for ages 25–44, highest for ages 65+) but did not differ by sex, substances reported on behavioral health screens, or Medicaid or Medicare insurance. In the mental health clinic subsample, test-retest reliability differed by age group (lowest reliability for ages 25–44, highest for ages 65+) and substance use reported on behavioral health screens (lowest for patients reporting both daily cannabis and any other drug use on behavioral health screens), but did not differ by sex, Medicaid, or Medicare insurance coverage.
DISCUSSION
The current study found that a practical, self-report SUSC had good-to-excellent test-retest reliability when completed in routine care by patients reporting daily cannabis use or any other drug use over the past year on behavioral health screens.
For patients who completed both SUSCs in primary care, where procedures for administering the SUSC were most standardized, test-retest reliability estimates for symptom counts were in the excellent range (ICC=0.81). The reliability estimates were somewhat lower than those obtained by Hasin et al. (2020), who used gold-standard semi-structured diagnostic interviews administered by highly trained and supervised research clinicians in a non-clinical context (e.g., ICC=0.90, 0.91, 0.97, and 0.86 for past-year DSM-5 symptoms of cannabis, cocaine, heroin, and opioid use disorders, respectively). Our findings suggest that in primary care where it may not be practical for clinicians to administer diagnostic interviews (Bradley et al., 2019), SUSCs that invite patients to self-report symptoms consistent with DSM-5 SUD can have high test-retest reliability when they are completed as patient-report questionnaires.
In the current study, test-retest reliability was significantly lower (but still “good”; Cicchetti, 1994) for checklists completed in specialty mental health settings (ICC=0.74 for symptom counts) compared to those completed in primary care settings (ICC=0.81). Although this difference was relatively small, it aligns with our hypothesis and previous findings with Alcohol Symptom Checklists (Hallgren et al., 2022) and may be due to differences in how checklists were implemented in mental health clinics (i.e., limited training, non-standardized procedures for administering SUSCs) versus primary care clinics (standard implementation with practice facilitation, training, and greater standardization using patient self-report on paper). There may be additional reasons for this difference. For example, although we anticipated that past-year SUD symptoms would be stable over a 1–21 day period for most patients, it is possible that some patients, particularly those receiving care in mental health clinics, experienced legitimate changes in symptoms over the 1–21 day period. In some cases, patients who had decreased substance use between the two SUSCs may have reported their recent experiences of SUD symptoms. It is possible that in some cases, clinicians asked patients to report on their symptoms during the intermediate period, even though the checklist asks them to report past-year symptoms. Further research is needed to understand how patients and clinicians may utilize SUD symptom measures with shorter timeframes for clinical monitoring (National Council for Behavioral Health, 2018).
In both the primary care and mental health clinic subsamples, we found lower test-retest reliability in patients aged 25–44, and within the mental health clinic subsample we found lower test-retest reliability in patients reporting both daily cannabis and other drug use within the past year on behavioral health screens. These differences were not hypothesized and reasons for them are speculative. For example, patients in these subgroups could have more often been actively addressing their substance use or receiving SUD treatment, which could have resulted in meaningful changes in SUD symptom reporting that would have been modeled as error variance in this study. Alternatively, patients with daily cannabis and other drug use could have had greater difficulty reliably reporting whether certain SUD symptoms were present, for example, if some substances (e.g., cannabis) were used to mitigate SUD symptoms caused by other substances (e.g., withdrawal from other drugs; Reiman, 2009). Future studies should further evaluate ways to efficiently and reliably measure symptoms that may be attributable to different substances. Patients in mental health settings who used cannabis and other substances may have more often had other health concerns (e.g., pain, mental health concerns) that could have created difficulty distinguishing whether some symptoms were due to substances versus another medical or mental health condition (SAMHSA, 2020).
The good-to-excellent test-retest reliability suggests that the SUSC can provide reliable information to support healthcare providers in assessing DSM-5 SUD symptoms and the severity of SUD, which in turn could aid diagnosis. As used in the KP Washington health system, the 11-item SUSC may operate best as follow-up assessment that can provide more detailed information about DSM-5 SUD symptoms after high-risk substance use is detected on briefer population-based screens (e.g., single-item substance use screens). Although several substance use screening and assessment protocols are available (e.g., Brown and Rounds, 1995; McNeely et al., 2016; Pautrat et al., 2022; Smith et al., 2010), the SUSC is unique in providing information that specifically invites patients to self-report symptoms that map onto each of the 11 DSM-5 criteria required to support diagnosis of SUD and determine its severity. Existing structured interviews are reliable (Hasin et al., 2020) but were largely designed for research and are impractical for most routine care settings. Additionally, collecting patient-report measures of DSM-5 SUD symptoms may help providers engage patients in discussions about concerns patients have about their substance use and facilitate discussions about treatment.
Limitations and Strengths
This study has noteworthy limitations. By using existing clinical data, we were unable to know why patients in our sample completed two SUSCs 1–21 days apart. It is therefore likely there was selection bias in the study sample that we were unable to measure. For example, some patients in the study sample may have been undergoing clinical monitoring of their SUD symptoms. The sample was limited by being predominantly white and non-Hispanic, and sample sizes for racial and ethnic minority subgroups were insufficient to examine test-retest reliability within these subgroups. Additional efforts to evaluate test-retest reliability of SUSCs among patients of color is warranted. Most of the sample (89%) was included for daily cannabis use without other drug use, and the results predominantly reflect test-retest reliability for SUSCs among people who report daily cannabis use, with more limited representation of people who use other drugs and no representation of people who reported less than daily cannabis use without other drug use. The study sample would not be expected to represent some patient populations who are engaged with SUD treatments (e.g., patients receiving office-based opioid use disorder treatment). Patients may not have accurately recognized SUD symptoms (e.g., cannabis withdrawal) and could have reported fewer SUD symptoms at both time points than they would have reported in confidential research assessments due to SUSCs being shared with medical providers and entered into the EHR. The behavioral health screening questions did not differentiate which specific substances were used besides cannabis and many patients did not report the substance(s) they thought caused their symptoms; thus, we could not evaluate subgroups defined by specific drug classes (e.g., opioids, stimulants). Providers who use the SUSC may therefore need to ask patients about substances used and which substance(s) symptoms are attributable to.
Our study also had several strengths. Test-retest reliability was evaluated in routine-care conditions, which provides high external validity for the performance of SUSCs that are administered as part of clinical care, shared with medical providers, and entered into the EHR. Using existing clinical data mitigated potential sampling bias that could result if the study had recruited participants who were willing to engage in SUD-related research. The 1–21 day test-retest window reflected a period in which past-year SUD symptoms have a low likelihood of substantively changing, minimizing potential confounding that could be attributable to actual changes in SUD symptoms. The large sample size allowed us to obtain reasonably precise reliability estimates and to test for setting- and patient-level factors that may increase or decrease test-retest reliability. The high test-retest reliability observed here strengthens recent findings supporting the construct validity of the SUSC (Matson et al., 2023) and complements findings supporting the construct validity and test-retest reliability of Alcohol Symptom Checklists (Hallgren et al., 2022, 2021).
Conclusion
The SUSC has high test-retest reliability when assessing symptoms consistent with DSM-5 SUD criteria in routine primary care and mental health care settings among patients who report daily cannabis use and/or past-year other drug use. Reliable assessment of these symptoms using a practical, patient-report measure could aid providers in primary care and mental health care settings with diagnosing DSM-5 SUDs and engaging patients in discussion about the SUD symptoms they self-report.
Supplementary Material
Highlights.
1194 patients reported DSM-5 SUD symptoms on Substance Use Symptom Checklists (SUSCs)
SUSCs had high test-retest reliability across settings and demographic subgroups
Reliability was higher in primary care, where assessment processes were standardized
SUSCs provide test-retest reliable information about symptoms consistent with DSM-5 SUD
Acknowledgements:
Research reported in this publication was supported by the National Institute on Drug Abuse (NIDA) of the National Institutes of Health (NIH) under award number UG1DA040314 (CTN-0113) and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) under award number R33AA028073. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH, NIDA, or NIAAA.
The authors wish to thank Megan Addis for administrative support provided on this project.
Footnotes
Declaration of Interest
All authors declare no conflicts of interest.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
REFERENCES
- American Psychiatric Association, 2013. Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub. [Google Scholar]
- Babor TF, Higgins-Biddle JC, Saunders JB, Monteiro MG, World Health Organization, 2001. AUDIT: the alcohol use disorders identification test: guidelines for use in primary health care. [Google Scholar]
- Bobb JF, Lee AK, Lapham GT, Oliver M, Ludman E, Achtmeyer C, Parrish R, Caldeiro RM, Lozano P, Richards JE, Bradley KA, 2017. Evaluation of a Pilot Implementation to Integrate Alcohol-Related Care within Primary Care. Int J Environ Res Public Health 14. 10.3390/ijerph14091030 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bradley KA, Caldeiro RM, Hallgren KA, Kivlahan DR, 2019. Making measurement-based care for addictions a reality in primary care. Addiction 114, 1355–1356. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brown RL, Rounds LA, 1995. Conjoint screening questionnaires for alcohol and other drug abuse: criterion validity in a primary care practice. Wisconsin medical journal. [PubMed] [Google Scholar]
- Cicchetti DV, 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment 6, 284–290. 10.1037/1040-3590.6.4.284 [DOI] [Google Scholar]
- Cohen J, 1968. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 213. [DOI] [PubMed] [Google Scholar]
- Cohen J, 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 37–46. 10.1177/001316446002000104 [DOI] [Google Scholar]
- Degenhardt L, Charlson F, Ferrari A, Santomauro D, Erskine H, Mantilla-Herrara A, Whiteford H, Leung J, Naghavi M, 2018. The global burden of disease attributable to alcohol and drug use in 195 countries and territories, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Psychiatry 5, 987–1012. 10.1016/S2215-0366(18)30337-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Glass JE, Bobb JF, Lee AK, Richards JE, Lapham GT, Ludman E, Achtmeyer C, Caldeiro RM, Parrish R, Williams EC, Lozano P, Bradley KA, 2018. Study protocol: a cluster-randomized trial implementing Sustained Patient-centered Alcohol-related Care (SPARC trial). Implement Sci 13, 108. 10.1186/s13012-018-0795-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hallgren KA, Matson TE, Oliver M, Caldeiro RM, Kivlahan DR, Bradley KA, 2022. Practical assessment of DSM-5 alcohol use disorder criteria in routine care: High test-retest reliability of an Alcohol Symptom Checklist. Alcohol Clin Exp Res 46, 458–467. 10.1111/acer.14778 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hallgren KA, Matson TE, Oliver M, Witkiewitz K, Bobb JF, Lee AK, Caldeiro RM, Kivlahan D, Bradley KA, 2021. Practical Assessment of Alcohol Use Disorder in Routine Primary Care: Performance of an Alcohol Symptom Checklist. J GEN INTERN MED. 10.1007/s11606-021-07038-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hasin D, Shmulewitz D, Stohl M, Greenstein E, Roncone S, Aharonovich E, Wall M, 2020. Test-retest reliability of DSM-5 substance disorder measures as assessed with the PRISM-5, a clinician-administered diagnostic interview. Drug and Alcohol Dependence 216, 108294. 10.1016/j.drugalcdep.2020.108294 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lapham GT, Lee AK, Caldeiro RM, McCarty D, Browne KC, Walker DD, Kivlahan DR, Bradley KA, 2017. Frequency of cannabis use among primary care patients in Washington State. The Journal of the American Board of Family Medicine 30, 795–805. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lee AK, Bobb JF, Richards JE, Achtmeyer CE, Ludman E, Oliver M, Caldeiro RM, Parrish R, Lozano PM, Lapham GT, 2023. Integrating alcohol-related prevention and treatment into primary care: a cluster randomized implementation trial. JAMA Internal Medicine 183, 319–328. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Marsden J, Tai B, Ali R, Hu L, Rush AJ, Volkow N, 2019. Measurement-based care using DSM-5 for opioid use disorder: can we make opioid medication treatment more effective? Addiction 114, 1346–1353. 10.1111/add.14546 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Matson TE, Hallgren KA, Lapham GT, Oliver MM, Wang X, Williams EC, Bradley KA, 2023. Psychometric performance of a Substance Use Symptom Checklist to help clinicians assess substance use disorders in primary care. JAMA Network Open 6, e2316283–e2316283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Matson TE, Lapham GT, Bobb JF, Oliver M, Hallgren KA, Williams EC, Bradley KA, 2022. Validity of the Single-Item Screen–Cannabis (SIS-C) for Cannabis Use Disorder Screening in Routine Care. JAMA Network Open 5, e2239772. 10.1001/jamanetworkopen.2022.39772 [DOI] [PMC free article] [PubMed] [Google Scholar]
- McGraw KO, Wong SP, 1996. Forming inferences about some intraclass correlation coefficients. Psychological methods 1, 30. [Google Scholar]
- McNeely J, Wu L-T, Subramaniam G, Sharma G, Cathers LA, Svikis D, Sleiter L, Russell L, Nordeck C, Sharma A, O’Grady KE, Bouk LB, Cushing C, King J, Wahle A, Schwartz RP, 2016. Performance of the Tobacco, Alcohol, Prescription medication, and other Substance use (TAPS) Tool for substance use screening in primary care patients. Ann Intern Med 165, 690–699. 10.7326/M16-0317 [DOI] [PMC free article] [PubMed] [Google Scholar]
- National Council for Behavioral Health, 2018. Implementing Care for Alcohol & Other Drug Use in Medical Setting: An Extension of SBIRT. [Google Scholar]
- Pautrat M, Pierre Lebeau J, Laporte C, 2022. Identifying available addictive disorder screening tests validated in primary care: A systematic review. Addictive Behaviors 126, 107180. 10.1016/j.addbeh.2021.107180 [DOI] [PubMed] [Google Scholar]
- Rehm J, Shield KD, 2019. Global Burden of Disease and the Impact of Mental and Addictive Disorders. Current Psychiatry Reports 21, 10. 10.1007/s11920-019-0997-0 [DOI] [PubMed] [Google Scholar]
- Reiman A, 2009. Cannabis as a substitute for alcohol and other drugs. Harm Reduct J 6, 35. 10.1186/1477-7517-6-35 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Richards JE, Bobb JF, Lee AK, Lapham GT, Williams EC, Glass JE, Ludman EJ, Achtmeyer C, Caldeiro RM, Oliver M, 2019. Integration of screening, assessment, and treatment for cannabis and other drug use disorders in primary care: An evaluation in three pilot sites. Drug and alcohol dependence 201, 134–141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sayre M, Lapham GT, Lee AK, Oliver M, Bobb JF, Caldeiro RM, Bradley KA, 2020. Routine Assessment of Symptoms of Substance Use Disorders in Primary Care: Prevalence and Severity of Reported Symptoms. J Gen Intern Med 35, 1111–1119. 10.1007/s11606-020-05650-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Smith PC, Schmidt SM, Allensworth-Davies D, Saitz R, 2010. A single-question screening test for drug use in primary care. Archives of internal medicine 170, 1155–1160. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Substance Abuse and Mental Health Services Administration, 2023. Key substance use and mental health indicators in the United States: Results from the 2022 National Survey on Drug Use and Health. [Google Scholar]
- Substance Abuse and Mental Health Services Administration, 2020. Substance use disorder treatment for people with co-occurring disorders., Treatment Improvement Protocol (TIP) Series. Substance Abuse and Mental Health Services Administration, Rockville, MD. [Google Scholar]
- Williams EC, Achtmeyer CE, Thomas RM, Grossbard JR, Lapham GT, Chavez LJ, Ludman EJ, Berger D, Bradley KA, 2015. Factors Underlying Quality Problems with Alcohol Screening Prompted by a Clinical Reminder in Primary Care: A Multi-site Qualitative Study. J Gen Intern Med 30, 1125–1132. 10.1007/s11606-015-3248-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- Yeung K, Richards J, Goemer E, Lozano P, Lapham G, Williams E, Glass J, Lee A, Achtmeyer C, Caldeiro R, Parrish R, Bradley K, 2020. Costs of using evidence-based implementation strategies for behavioral health integration in a large primary care system. Health Services Research 55, 913–923. 10.1111/1475-6773.13592 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.