Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Oct 1.
Published in final edited form as: Drug Alcohol Depend. 2021 Aug 28;227:108980. doi: 10.1016/j.drugalcdep.2021.108980

Development and Validation of a Prediction Model for Opioid Use Disorder Among Youth

Nicole M Wagner 1,2, Ingrid A Binswanger 2,3,4, Susan M Shetterly 2, Deborah J Rinehart 4,5, Kris F Wain 2, Christian Hopfer 6, Jason M Glanz 2,7
PMCID: PMC8464513  NIHMSID: NIHMS1737514  PMID: 34482048

Abstract

Background

Youth are vulnerable to opioid use initiation and its complications. With growing rates of opioid overdose, strategies to identify youth at risk of opioid use disorder (OUD) to efficiently focus prevention interventions are needed. This study developed and validated a prediction model of OUD in youth aged 14–18 years.

Methods

The model was developed in a Colorado healthcare system (derivation site) using Cox proportional hazards regression analysis. Model predictors and outcomes were identified using electronic health record data. The model was externally validated in a separate Denver safety net health system (validation site). Youth were followed for up to 3.5 years. We evaluated internal and external validity using discrimination and calibration.

Results

The derivation cohort included 76,603 youth, of whom 108 developed an OUD diagnosis. The model contained 3 predictors (smoking status, mental health diagnosis, and non-opioid substance use or disorder) and demonstrated good calibration (p=0.90) and discrimination (bootstrap-corrected C-statistic = 0.76: 95% CI = 0.70, 0.82). Sensitivity and specificity were 57% and 84% respectively with a positive predictive value (PPV) of 0.49%. The validation cohort included 45,790 youth of whom, 74 developed an OUD diagnoses. The model demonstrated poorer calibration (p<0.001) but good discrimination (C-statistic = 0.89; 95% CI = 0.84, 0.95), sensitivity of 87.8% specificity of 68.6%, and PPV of 0.45%.

Conclusions

In two Colorado healthcare systems, the prediction model identified 57–88% of subsequent OUD diagnoses in youth. However, PPV <1% suggests universal prevention strategies for opioid use in youth may be the best health system approach.

Keywords: adolescent, youth, opioid use disorder, prognostic model, prediction model

1. Introduction

Based on self-reported symptoms and opioid use in 2019, an estimated 2.3% of adolescents misuse opioids and nearly 90,000 adolescents met criteria for opioid use disorder (OUD) (Substance Abuse and Mental Health Services Administration, 2020). The estimates of opioid misuse and OUD more than double as adolescents’ transition into young adults. The shift between opioid initiation and regular use can be rapid (Subramaniam and Stitzer, 2009), and early onset of opioid use increases the risk of negative health outcomes (McCabe et al., 2007). As the rates of opioid overdose continue to rise, and are exacerbated by the COVID-19 pandemic (Ahmed et al., 2021; Mattson et al., 2021), strategies to identify youth at high risk of OUD are needed to target preventive interventions at the right time.

Many youth have contact with health care providers in health systems for routine care such as vaccines, sports physicals, and medications (Leidner et al., 2021). Evidence suggests there may be multiple missed substance use prevention opportunities for youth in health systems (Krist et al., 2020). Less than half of primary care providers report screening youth for substance use using a validated measure, as recommended by the American Academy of Pediatrics (AAP) (Committee on Substance Abuse, 2016; Harris et al., 2012). Insufficient time is a commonly reported barrier to youth substance use screening in practice (Van Hook et al., 2007). Novel approaches to identifying youth at risk of OUD could supplement clinician screening at healthcare visits, facilitating the implementation of targeted preventive services.

Studies have identified factors associated with OUD in youth, including anxiety, depression, and attention deficit hyperactivity disorder, and co-occurring non-opioid substance use disorders (SUD) (Subramaniam and Stitzer, 2009; Subramaniam et al., 2009; Welsh et al., 2017). Current youth prediction models rely on complex survey measures, such as heritability scales, peer drug use, and problematic behavior scales, that are not readily available or easily collected in the healthcare setting (Miech et al., 2015; Tarter et al., 2020). Clinically useful prediction models have been developed to identify overdose and OUD in adults on chronic opioid therapy (Cochran et al., 2014; Glanz et al., 2018; Hylan et al., 2015). However, youth who experience overdoses are less likely to have been prescribed opioid analgesics than adults who experience overdoses (Chatterjee et al., 2019). Therefore, models developed for adults may not accurately predict poor outcomes among youth. We developed and validated a prediction model of OUD in youth aged 14 to 18 in two diverse healthcare systems.

2. Materials and methods

2.1. Study design, study sites, population, and data sources

We conducted a retrospective cohort study to develop, internally validate, and externally validate a prediction model for OUD among youth. The model was developed and internally validated among youth enrolled in a large, integrated health insurance plan and delivery system, Kaiser Permanente Colorado (KPCO; derivation site). KPCO members were enrolled in the health plan and receive health services, including pharmacy services, within the KPCO system. Demographic information and health care encounters, including diagnoses (International Classification of Diseases [ICD]-9 and ICD 10 codes), pharmacy dispensings, and drug use reported in the social history, were captured in the electronic health record (EHR). In rare instances, when services were unavailable within the KPCO system, claims data from the health plan provided information on services rendered. The model was externally validated in youth age 14–18 from the Denver Health and Hospital Authority (DH; validation site). DH is an integrated academic healthcare system that serves as the primary safety net for the City and County of Denver, Colorado. DH provides services to approximately 21% of the population in Denver County. This study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) (Moons et al., 2015).

Participants at both sites included high school aged youth, defined as age 14–18, enrolled in the respective healthcare systems between January 1, 2016 and June 30, 2019. The index date indicating the start of follow-up was the first date a person met the age criteria and had been enrolled for at least 6 months. As DH does not have enrollment data, we used empanelment to assess enrollment. Empanelment is defined as at least one primary care visit in the 18 months prior to the index date (Glanz et al., 2018; Hambidge et al., 2017). Eligible participants were excluded if they had an OUD diagnosis in the year prior to the index date. Predictors were assessed in the year prior to the index date. Patients were followed from the index date through the end of the study period (December 31, 2019) for the OUD outcome. Follow-up was censored at the time of an OUD diagnosis, disenrollment from the health plan, no longer meeting empanelment criteria, or end of the study period. This study was approved by the KPCO Institutional Review Board and Colorado Multiple Institutional Review Board.

2.2. Predictor Variables

Potential predictors included known and suspected correlates of OUD that could be ascertained in EHR data (Bohnert et al., 2011; Chatterjee et al., 2019; John et al., 2019; Lim et al., 2021; Lyons et al., 2019; Sharma et al., 2016; Subramaniam and Stitzer, 2009; U.S. Department of Health and Human Services (HHS) Office of the Surgeon General, 2016). Potential predictors included demographics; diagnoses or social history consistent with non-opioid, non-tobacco, substance use or a SUD (alcohol, cannabis, cocaine, other stimulants, sedatives, hypnotics, anxiolytics, hallucinogens, or other/mixed); diagnoses consistent with automobile crash or unintentional injury, mental health disorder (e.g., anxiety/adjustment, conduct, depressive, and eating disorders), or chronic pain (Centers for Disease Control and Prevention, 2018); opioid medication dispensings; tobacco use in the social history; and diagnoses or social history consistent with sexual or other behavioral risk indicators (e.g., pregnancy, hepatitis B, and hepatitis C). Demographic variables, including age, sex (male, female), and race or ethnicity were captured from the EHR on the index date. Race or ethnicity categories included white, non-white (Hispanic, Black, Asian, Native American or Alaska Native, Native Hawaiian or other Pacific Islander, or other), or unknown (missing). Diagnoses and risk behaviors were dichotomous variables with a yes value representing the presence of a diagnosis and a no value representing the absence of the diagnosis in at least six months up to the year prior to index date. The time frame used was consistent with prediction models for OUD and overdose using insured cohorts (Cochran et al., 2014; Glanz et al., 2018) and is further supported by evidence suggesting adolescent onset of regular opioid use occurs on average 6 months prior to meeting criteria for an OUD (Subramaniam and Stitzer, 2009).

To ensure adequate degrees of freedom, associated conditions were grouped into predictor variables for mental health conditions, non-opioid, non-tobacco substance use or SUD (Hereafter referred to as “substance use or SUD”), and sexual or other behavioral risk indicators. Tobacco use was often captured as part of standard visit intake. As such, tobacco use was measured using the most recent record of smoking status in the social history. Categories included current or former, nonsmoker, and unknown (missing data). (U.S. Department of Health and Human Services (HHS) Office of the Surgeon General, 2016)

2.3. Outcome Measure

Our primary outcome was incident OUD. OUD was defined as the presence of ICD-9 and ICD-10 diagnosis codes F11, 304.0, 304.7, 305.5 or a pharmacy dispensing for a buprenorphine-containing product, excluding patches used for pain. To avoid misclassification, pharmacy dispensing of oral naltrexone and extended release naltrexone were not included as they are often used to treat alcohol use disorder in these healthcare systems. OUD was captured as binary outcome where the outcome event date was the first date evidence of the OUD was captured in the EHR during the follow-up.

2.4. Analysis

In the cohort from the derivation site (KPCO), we used a Cox proportional hazards regression to assess the association between baseline predictor variables and incident OUD. First, we assessed frequencies and the distribution of the predictor variables to identify sparse data. The number of predictors in the model was restricted to 10 OUD diagnoses per degree of freedom to avoid overfitting the model (Collins et al., 2016). We therefore grouped medications and diagnoses into clinically meaningful categories. We then examined the bivariable associations between each predictor variable and OUD. We chose predictors for the multivariable model using both statistical and clinically meaningful significance. Time was estimated as the total number of days after the index date until OUD or other censoring event. The final model was identified using backwards selection (Moons et al., 2015). Using best subsets methods, we confirmed the backwards selection did not miss important comparably sized models. Proportional hazards assumptions were assessed using a global goodness-of-fit test.

2.4.1. Internal Validation

Internal validity of the final model was assessed in the derivation site using discrimination and calibration. Discrimination, or the ability of the model to distinguish between patients at high and low risk of developing an OUD, was assessed using Harrell bootstrap resampling, calculating the bootstrap-corrected concordance statistic (C-statistic) (Harrell, 2001). We further assessed discrimination using the Royston-Sauerbrei D Statistic, and the explained variation using the R2 statistic based on the D statistic (Royston and Sauerbrei, 2004). The D statistic assesses the separation between persons with high and low predicted risk. A D statistic of 1 suggests the model fails to discriminate between high and low risk individuals (Royston and Altman, 2013). Calibration, or how much agreement is found between observed outcomes and predicted outcomes, was assessed using the Greenwood-D’Agonstino-Nam test (Demler et al., 2015). The overall goal of the prediction model was to identify youth at higher risk of OUD to target preventive interventions for reduction in opioid initiation and negative outcomes. Preventive services are not always low intensity, indicating a need to minimize false positives (increase specificity). Therefore, to assess sensitivity, specificity, and positive predictive value of the model, we identified a cutoff using the receiver operating characteristics (ROC) curve. The ROC curve presents a range of possible cutoffs for sensitivity and specificity (Hanley and Mcneil, 1982). We used the cutoff in the top left corner of the curve, the Youden index. The Youden index represents the greatest value of separation between patients with and without an OUD, maximizing the sensitivity and specificity (Steyerberg, 2019). We graphically compared the high and low risk groups using plots of observed and predicted cumulative incidence over time (Steyerberg, 2019).

2.4.2. External Validation

We externally validated the model using an independent cohort of patients at DH by examining discrimination and calibration. Using the linear predictor coefficients and baseline survival in the derivation model, we assessed the predicted risk for youth OUD for individuals in the DH cohort. Next, we assessed discrimination by calculating the C-statistic, D Statistic, and R2 statistic. We applied the Youden index from the derivation model to assess sensitivity, specificity, and positive predictive value. We evaluated calibration using the Greenwood-D’Agostino-Nam test (Demler et al., 2015). All analyses were conducted using SAS version 9.4 (SAS Institute Inc., Cary, NC).

3. Results

3.1. Descriptive statistics at derivation and validation sites

We identified 76,627 youth at the derivation site. Of these, 24 (0.03%) were excluded due to an OUD diagnosis prior to the index date. Youth were just over 50% male and entered the cohort, on average, at age 15. Less than 2% of the population had indicators of non-opioid, non-tobacco substance use or SUD, less than 15% of the cohort had a mental health diagnosis, and approximately 5% received an opioid dispensing in the prior year (Table 1). During the follow-up period, 108 (0.14%) youth developed an OUD diagnosis, of which 49 (45%) were initially identified in the emergency department (ED) or inpatient setting, 39 (36%) in the ambulatory setting, and 15 (14%) in other settings (such as institutional stays). Only one OUD outcome was initially identified due to a buprenorphine dispensing. Average follow-up time was 1.9 years.

Table 1.

Baseline demographic, clinical and risk behavior characteristics at derivation and validation sites

Characteristica Derivation Site
(n=76,603)		 Validation Site
(n=45,790)		 p-valueb
Age, mean (SD) 15.4 (1.48) 15.4 (1.47) 0.87
Male, n (%) 38,903 (50.8) 22,198 (48.5) <0.001
Race/Ethnicity, n (%)
 Non-Hispanic White 34,310 (44.8) 5,896 (12.9) <0.001
 Non-Whitec 25,284 (33.0) 35,915 (78.4)
 Missing 17,009 (22.2) 3,979 (8.7)
Insurance coverage, n (%)
 Medicaid or state subsidized 17,399 (22.7) 29,767 (65.0) <0.001
 Commercial or other insurance 59,204 (77.3) 7,233 (15.8)
 Uninsured 0 (0) 8,790 (19.2)
Substance use or substance use disorderd, n (%) 1,231 (1.6) 3,478 (7.6) <0.001
Smoking status, n (%)
 Current/Former smoker 1,242 (1.6) 1314 (2.9) <0.001
 Nonsmoker 57,904 (75.6) 25,684 (56.1)
 Unknown 17,457 (22.8) 18,792 (41.0)
Mental health diagnosis, n (%) 11,340 (14.8) 12,971 (28.3) <0.001
Chronic pain diagnosis, n (%) 14,008 (18.3) 13,077 (28.6) <0.001
Automobile crash or unintentional injury, n (%) 4,491 (5.9) 3,831 (8.4) <0.001
Opioid dispensing, n (%) 3,969 (5.2) 2,446 (5.3) 0.22
Sexual or other behavioral risk indicators, n (%) 2,593 (3.4) 1,779 (3.9) <0.001
Open in a new tab
a)

Baseline demographic characteristics were captured on the index date into the cohort. Clinical and risk behavior characteristics were captured in the year prior to the index date unless otherwise noted.

b)

P values from chi-square tests except for age which used t-test.

c)

Non-white includes Hispanic, Black, Asian, Native American or Alaska Native, Native Hawaiian or other Pacific Islander, or other

Substance use or substance use disorder includes alcohol, cannabis, cocaine, other stimulants, sedatives, hypnotics, anxiolytics, hallucinogens, or other/mixed

At the validation site, 45,845 youth were identified. A total of 55 (0.12%) youth were excluded due to an OUD diagnosis prior to the index date. The derivation and validation cohorts were significantly different in all measured characteristics except age and opioid dispensings (Table 1). Notably, the validation site had a larger non-white and Medicaid or state subsidized population with a higher proportion of substance use or SUD. During the follow-up period, 74 (0.16%) youth received an OUD diagnosis. Average follow-up time was 1.6 years at the validation site.

3.2. Model development at the derivation site

The associations between predictor variables and OUD in the derivation site are shown in Table 2. Based on significant association and clinical significance, we selected the following 9 variables as potential predictors to include in the model: sex, smoking status (current or former, nonsmoker, and unknown), substance use or SUD, mental health diagnosis, chronic pain diagnosis, opioid dispensing, sexual and other behavioral risk indicators, and automobile crash or unintentional injury in the prior year (Table 3).

Table 2.

Bivariate association between predictors and opioid use disorder in 14–18 year olds in the derivation site

Characteristic OUD P valuea
No
(n=76,495)		 Yes
(n=108)
Age, mean years (SD) 15.4 (1.48) 16.3 (1.48) <0.0001
Race/ethnicity, n (%)b
 Non-Hispanic White, 34,245 (44.8%) 65 (60.2%) 0.006
 Hispanic or Non-Whitec 25,258 (33%) 26 (24.1%)
 Unknown 16992 (22.2%) 17 (15.7%)
Male vs Female, n (%) 38843 (50.8%) 60 (55.6%) 0.32
Smoking status
 Current or Former n (%) 1223 (1.6%) 19 (17.6%) <.0001
 Nonsmoker, n (%) 57838 (75.6%) 66 (61.1%)
 Unknown, n (%) 17434 (22.8%) 23 (21.3%)
Substance use or SUD, n (%)d 1209 (1.6%) 22 (20.4%) <.0001
Mental health diagnosis, n (%)e 11287 (14.8%) 53 (49.1%) <.0001
Chronic pain diagnosis, n (%) 13980 (18.3%) 28 (25.9%) 0.04
Opioid dispensing in prior year, n (%) 3958 (5.2%) 11 (10.2%) 0.02
Sexual or other behavioral risk indicators, n (%)f 2588 (3.4%) 5 (4.6%) 0.47
Automobile crash or unintentional injury, n (%)g 4475 (5.9%) 16 (14.8%) <.0001
Open in a new tab
a)

p-value for chi-square test except for age where p-value from t-test

b)

Age and race were not included in the final variable model

c)

Non-white includes Hispanic, Black, Asian, Native American or Alaska Native, Native Hawaiian or other Pacific Islander, or other

d)

Substance use or SUD excludes tobacco. Persons with history of opioid abuse excluded in study selection.

e)

Mental health condition included Affective/mood disorder, anxiety/adjustment disorder, autism spectrum disorder, conduct disorder, depression, dementia, eating disorder, bipolar/manic disorder, gender identity disorder, other (non-schizophrenic) psychotic disorder, personality disorder, psychotic disorder, post-traumatic stress disorder, schizophrenia spectrum disorder, suicidal ideation, potential self-harm, and definite self-harm

f)

Sexual or other behavioral risk indicators includes pregnancy, syphilis, gonorrhea, chlamydia, chancroid, trichomoniasis, herpes, hepatitis B, hepatitis C, pelvic inflammatory disease, human papillomavirus, condyloma acuminatum, granuloma inguinale, and salpingitis and oophoritis

g)

Includes diagnoses for any automobile accident including as a bystander and unintential injuries (e.g., accidents from sporting events, daily living, guns, and fire)

Table 3.

Cox regression analysis for the full and selected prediction models for opioid use disorder among youth in the derivation site

Full model Selected model
Characteristic Hazard Ratio (95% CI) B a coefficient Hazard Ratio (95% CI)
Male gender vs female 1.28 (0.87, 1.88)
Smoking
 Current/former vs nonsmoker 4.61 (2.21, 9.61) 1.53321 4.63 (2.23, 9.62)
 Unknown vs nonsmoker 1.95 (1.21, 3.16) 0.68374 1.98 (1.23, 3.20)
Substance use or SUD 5.44 (3.00, 9.85) 1.75541 5.79 (3.22, 10.39)
Mental health diagnosis 4.03 (2.66, 6.10) 1.41108 4.10 (2.72, 6.18)
Chronic pain diagnosis 1.06 (0.67, 1.66)
Opioid dispensing in prior year 1.12 (0.59, 2.14)
Teen Pregnancy or STD 0.86 (0.35, 2.14)
Car accident or unintentional injury 1.43 (0.81, 2.53)
Open in a new tab
a

S0(3.5)=0.998703 (3.5 year baseline survival)

The final model included three variables: smoking status, substance use or SUD, and mental health diagnosis (Table 3). All variables in the final model were clinically significant. The global goodness-of-fit (P = 0.12) demonstrated the proportional hazards assumption was met. The uncorrected c-statistic was 0.76 (95% CI 0.70–0.82), the bootstrap-corrected c-statistic was 0.76 (95% CI 0.70–0.82), and the D statistic was 1.55 (95% CI 1.24–1.77). The R2 explained 36.4% (30.0%, 42.8%) of the variation. We used the Youden index point in the ROC curve to classify high versus low risk. A total of 12,671 (16.5%) youth were classified as high risk, of whom 62 (0.49%) developed an OUD diagnosis. Sensitivity was 57.4% and specificity was 82.7% with a positive predictive value of 0.49%. The plot of the observed and predicted OUD risk over time (Figure 1a, derivation site) demonstrated good discrimination in the well-separated curves for the high and low risk groups (Table 4). The graph also demonstrated good agreement between observed and predicted risk over time, indicating good calibration. Greenwood-Nam-D’Agostino results were not statistically significant (P = 0.90) indicating a well calibrated model.

Figure 1.

Figure 1

Open in a new tab

Plots o:f observed and predicted risk at the derivation (a) and validations (b) sites at high and low risk based on the Youden index at the derivation site.

Table 4:

Observed cumulative risk of opioid use disorder (OUD) by high and low risk among youth in two health systems

Risk level Derivation site Validation Site
Individuals N (%) OUD N Cumulative OUD risk per 1000 person years (%) Individuals N (%) OUD (n) Cumulative OUD risk per 1000 person years (%)
Low 63,932 (83.5%) 46 0.39 31,349 (68.5%) 9 0.20
High 12,671 (16.5%) 62 2.48 14,441 (31.5%) 65 2.36
Open in a new tab

3.3. External validation

The selected model from the derivation site was used to predict OUD in youth age 14–18 in the validation site. The model demonstrated good discrimination, with a c-statistic of 0.89 (95% CI 0.84, 0.95), a D statistic of 1.77(95% CI 1.55, 1.98) and a R2 of 0.43% (0.36%, 0.49%). Using the Youden index from the derivation site, there were 27,600 (32%) youth classified into the high-risk group, of whom 65 (0.24%) developed an OUD diagnosis. The validation site’s sensitivity and specificity were 87.8% and 68.6%, respectively. The positive predictive value was 0.45%. The plots of the observed and predicted risk (Figure 1b, validation site) showed effective discrimination in the separation of high and low risk groups, but lower calibration (Table 4). The separation between the predicted and observed outcomes demonstrated over prediction of OUD. Lower calibration was also indicated by the statistically significant Greenwood-Nam D’Agostino test (p<0.001).

4. Discussion

We developed and validated a prediction model to identify youth at elevated risk of developing OUD within three and a half years. The model included three variables readily available within most EHRs, or quickly assessed at the point of care, including smoking status, mental health diagnosis, and substance use or SUD. The prediction model reduced the target population by at least 70% in both sites. This could be beneficial for costly interventions requiring repeated visits with highly trained professionals. However, despite the model’s ability to narrow the population size, OUD is a rare outcome and the positive predictive value of the final model was less than 1% at both sites. This suggests that a very small portion of the target population will develop an OUD diagnosis, and that the model will be associated with a significant number of false positives. This has implications if the model is used to label patients as at risk for OUD in the health record, since negative attitudes towards patients with SUD (Van Boekel et al., 2013) could lead to unintended negative consequences for their health care. If applied with caution to avoid stigma, the model could be used for interventions such as increased OUD screening frequency, overdose education, or naloxone distribution (McDonald and Strang, 2016). However, this model would miss 13–43% who develop an OUD, depending on the type of healthcare system. Particularly with a rare outcome (<0.2%), a low sensitivity may limit the value of the model for some interventions and health systems.

Predictors identified in the model were known risk factors for OUD and opioid overdose (Lyons et al., 2019; Sharma et al., 2016). Tobacco use was highly predictive of OUD, which is consistent with a prior prediction model identifying tobacco use as predictive of overdose risk in adults receiving long term opioid therapy (Glanz et al., 2018). While opioid medications have been associated with opioid misuse (Miech et al., 2015), this variable was not retained in the final model. This may reflect national survey results indicating youth with OUD often receive opioids from family or friends rather than the health care system (Hudgins et al., 2019). It may also reflect the national trends in reduction of opioid prescribing in youth (Gagne et al., 2019), as demonstrated by the low prevalence in opioid dispensings in both cohorts (approximately 5%).

This model was built for the youth population, using data readily available in the EHR, and evaluated in an external health system. The prediction model was internally valid and demonstrated good discrimination at both sites. However, calibration at the validation site was relatively poor. This limitation is likely due to the variability in population characteristics between sites (Table 1). A higher proportion of the population at the validation site had mental health and substance use risk factors which may partially explain why the model overpredicted OUD at the validation site. However, the sensitivity of the model was almost 90% suggesting the model could be used to identify a majority of youth who would become diagnosed with OUD at the validation site. Therefore, targeting a larger proportion of the population may be warranted.

This study had several limitations. The derivation and validation site were both located in Colorado and the populations may not be representative of other states. However, the percentage of OUD cases were similar between the study population and a national commercially insured cohort (0.14% and 0.2% respectively) (Hadland et al., 2017). The limitation of generalizability is also minimized by the diversity in populations across the derivation and validation sites differing significantly on nearly all baseline characteristics (see Table 1). For example, the population identifying as Non-Hispanic White was 45% in the derivation cohort and 13% in the validation site. In addition to generalizability, the predictors included in the model were restricted due to the sparse number of OUD outcomes and data limitations. While family history of opioid use is a known risk factor for substance use disorders (U.S. Department of Health and Human Services (HHS) Office of the Surgeon General, 2016), family history data was not consistently captured in the EHR at the derivation and validation sites and was therefore not included in the model. Similarly, substance use screening was not consistently captured in the EHR and could not be controlled for in the model. Lastly, we may not have captured some OUD outcomes because of lack of universal screening, patient underreporting, and limited patient use of the health system (Harris et al., 2012; Williams and Nowatzki, 2005). Substance use screening behaviors may be inconsistent, leading to limited OUD identification (Harris et al., 2012). Despite these limitations, almost 50% of OUD outcomes were identified in ED or inpatient settings, indicating some OUD cases missed in the ambulatory setting were still captured.

The prediction model identified a three-factor model to predict OUD in youth for use in health systems to administer low resource substance use prevention strategies or strategies that prevent deaths. The pathway from beginning opioid use to OUD can occur within months (Subramaniam and Stitzer, 2009). While using a prediction model provides an opportunity to begin prevention efforts early to avoid an opioid use disorder, the large number of false positives highlight the need for cautious use of the model. To further establish factors that predict an OUD diagnosis in youth and validate the generalizability of this model, assessment of the model in a large national sample is needed.

Funding

This work was supported by the National Institute on Drug Abuse Helping to End Addiction Long-term Initiative [grant number 3R01DA042059-04S2]; Dr. Hopfer was also supported by the National Institute on Drug Abuse [grant numbers 5K24DA032555-08, 5R01DA042755-04]; and Dr. Wagner was also supported by National Heart Lung and Blood Institute [grant number 5K12HL137862-04].

Abbreviations

OUD

Opioid Use Disorder

EHR

Electronic Health Record

KPCO

Kaiser Permanente Colorado

DH

Denver Health Hospital and Authority

AAP

American Academy of Pediatrics

TRIPOD

Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis

ROC

Receiver Operating Characteristics

Footnotes

Declaration of interest: none

References

  1. Ahmed F, Rossen L, Sutton P, 2021. Provisional drug overdose death counts. National Center for Health Statistics. [Google Scholar]
  2. Bohnert AS, Valenstein M, Bair MJ, Ganoczy D, McCarthy JF, Ilgen MA, Blow FC, 2011. Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA 305(13), 1315–1321. [DOI] [PubMed] [Google Scholar]
  3. Centers for Disease Control and Prevention, 2018. Quality Improvement and Care Coordination: Implementing the CDC Guideline for Prescribing Opioids for Chronic Pain. National Center for Injury Prevention and Control, Division of Unintentional Injury Prevention, Atlanta, GA. [Google Scholar]
  4. Chatterjee A, Larochelle MR, Xuan Z, Wang N, Bernson D, Silverstein M, Hadland SE, Land T, Samet JH, Walley AY, Bagley SM, 2019. Non-fatal opioid-related overdoses among adolescents in Massachusetts 2012–2014. Drug Alcohol Depend 194, 28–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Cochran BN, Flentje A, Heck NC, Van Den Bos J, Perlman D, Torres J, Valuck R, Carter J, 2014. Factors predicting development of opioid use disorders among individuals who receive an initial opioid prescription: mathematical modeling using a database of commercially-insured individuals. Drug and alcohol dependence 138, 202–208. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Collins GS, Ogundimu EO, Altman DG, 2016. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 35(2), 214–226. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Committee on Substance Abuse, 2016. Substance Use Screening, Brief Intervention, and Referral to Treatment. Pediatrics 138(1), e20161210. [DOI] [PubMed] [Google Scholar]
  8. Demler OV, Paynter NP, Cook NR, 2015. Tests of calibration and goodness-of-fit in the survival setting. Stat Med 34(10), 1659–1680. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Gagne JJ, He M, Bateman BT, 2019. Trends in opioid prescription in children and adolescents in a commercially insured population in the United States, 2004–2017. JAMA pediatrics 173(1), 98–99. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Glanz JM, Narwaney KJ, Mueller SR, Gardner EM, Calcaterra SL, Xu S, Breslin K, Binswanger IA, 2018. Prediction Model for Two-Year Risk of Opioid Overdose Among Patients Prescribed Chronic Opioid Therapy. Journal of General Internal Medicine 33(10), 1646–1653. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Hadland SE, Wharam JF, Schuster MA, Zhang F, Samet JH, Larochelle MR, 2017. Trends in Receipt of Buprenorphine and Naltrexone for Opioid Use Disorder Among Adolescents and Young Adults, 2001–2014. Jama Pediatrics 171(8), 747–755. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Hambidge SJ, Ross C, Shoup JA, Wain K, Narwaney K, Breslin K, Weintraub ES, McNeil MM, 2017. Integration of data from a safety net health care system into the Vaccine Safety Datalink. Vaccine 35(9), 1329–1334. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Hanley JA, Mcneil BJ, 1982. The Meaning and Use of the Area under a Receiver Operating Characteristic (Roc) Curve. Radiology 143(1), 29–36. [DOI] [PubMed] [Google Scholar]
  14. Harrell FE, 2001. Resampling, validating, describing, and simplifying the model, Regression modeling strategies. Springer, pp. 87–103. [Google Scholar]
  15. Harris SK, Herr-Zaya K, Weinstein Z, Whelton K, Perfas Fernando J., Castro-Donlan C, Straus J, Schoneman K, Botticelli M, Levy S, 2012. Results of a statewide survey of adolescent substance use screening rates and practices in primary care. Substance abuse 33(4), 321–326. [DOI] [PubMed] [Google Scholar]
  16. Hudgins JD, Porter JJ, Monuteaux MC, Bourgeois FT, 2019. Prescription opioid use and misuse among adolescents and young adults in the United States: A national survey study. PLoS medicine 16(11), e1002922. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Hylan TR, Von Korff M, Saunders K, Masters E, Palmer RE, Carrell D, Cronkite D, Mardekian J, Gross D, 2015. Automated Prediction of Risk for Problem Opioid Use in a Primary Care Setting. Journal of Pain 16(4), 380–387. [DOI] [PubMed] [Google Scholar]
  18. John WS, Zhu H, Mannelli P, Subramaniam GA, Schwartz RP, McNeely J, Wu LT, 2019. Prevalence and patterns of opioid misuse and opioid use disorder among primary care patients who use tobacco. Drug and Alcohol Dependence 194, 468–475. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Krist AH, Davidson KW, Mangione CM, Barry MJ, Cabana M, Caughey AB, Donahue K, Doubeni CA, Epling JW, Kubik M, 2020. Primary Care–Based Interventions to Prevent Illicit Drug Use in Children, Adolescents, and Young Adults: US Preventive Services Task Force Recommendation Statement. Jama 323(20), 2060–2066. [DOI] [PubMed] [Google Scholar]
  20. Leidner AJ, Tang Z, Tsai Y, 2021. Primary Care Use Among Commercially Insured Adolescents: Evidence From the 2018 Healthcare Effectiveness Data and Information Set. American Journal of Preventive Medicine 60(3), 411–414. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Lim JK, Earlywine JJ, Bagley SM, Marshall BDL, Hadland SE, 2021. Polysubstance Involvement in Opioid Overdose Deaths in Adolescents and Young Adults, 1999–2018. JAMA Pediatrics. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Lyons RM, Yule AM, Schiff D, Bagley SM, Wilens TE, 2019. Risk Factors for Drug Overdose in Young People: A Systematic Review of the Literature. J Child Adolesc Psychopharmacol 29(7), 487–497. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Mattson CL, Tanz LJ, Quinn K, Kariisa M, Patel P, Davis NL, 2021. Trends and geographic patterns in drug and synthetic opioid overdose deaths—United States, 2013–2019. Morbidity and Mortality Weekly Report 70(6), 202. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. McCabe SE, West BT, Morales M, Cranford JA, Boyd CJ, 2007. Does early onset of non-medical use of prescription drugs predict subsequent prescription drug abuse and dependence? Results from a national study. Addiction 102(12), 1920–1930. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. McDonald R, Strang J, 2016. Are take-home naloxone programmes effective? Systematic review utilizing application of the Bradford Hill criteria. Addiction 111(7), 1177–1187. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Miech R, Johnston L, O’Malley PM, Keyes KM, Heard K, 2015. Prescription Opioids in Adolescence and Future Opioid Misuse. Pediatrics 136(5), e1169–1177. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS, 2015. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162(1), W1–73. [DOI] [PubMed] [Google Scholar]
  28. Royston P, Altman DG, 2013. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol 13(1), 33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Royston P, Sauerbrei W, 2004. A new measure of prognostic separation in survival data. Stat Med 23(5), 723–748. [DOI] [PubMed] [Google Scholar]
  30. Sharma B, Bruner A, Barnett G, Fishman M, 2016. Opioid use disorders. Child and Adolescent Psychiatric Clinics 25(3), 473–487. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Steyerberg EW, 2019. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating, 2nd ed.Springer, Switzerland AG. [Google Scholar]
  32. Subramaniam GA, Stitzer MA, 2009. Clinical characteristics of treatment-seeking prescription opioid vs. heroin-using adolescents with opioid use disorder. Drug and Alcohol Dependence 101(1–2), 13–19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Subramaniam GA, Stitzer ML, Woody G, Fishman MJ, Kolodner K, 2009. Clinical characteristics of treatment-seeking adolescents with opioid versus cannabis/alcohol use disorders. Drug and Alcohol Dependence 99(1–3), 141–149. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Substance Abuse and Mental Health Services Administration, 2020. Key substance use and mental health indicators in the United States: Results from the 2019 National Survey on Drug Use and Health. Center for Behavioral Statistics and Quality, Substance Abuse and Mental Health Services Adminstration, Rockville, MD. [Google Scholar]
  35. Tarter RE, Kirisci L, Cochran G, Seybert A, Reynolds M, Vanyukov M, 2020. Forecasting Opioid Use Disorder at 25 Years of Age in 16-Year-Old Adolescents. Journal of Pediatrics 225, 207–213. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. U.S. Department of Health and Human Services (HHS) Office of the Surgeon General, 2016. Facing Addiction in America: The Surgeon General’s Report on Alcohol, Drugs, and Health HHS, Washington, DC. [PubMed] [Google Scholar]
  37. Van Boekel LC, Brouwers EP, Van Weeghel J, Garretsen HF, 2013. Stigma among health professionals towards patients with substance use disorders and its consequences for healthcare delivery: systematic review. Drug and alcohol dependence 131(1–2), 23–35. [DOI] [PubMed] [Google Scholar]
  38. Van Hook S, Harris SK, Brooks T, Carey P, Kossack R, Kulig J, Knight JR, New England Partnership for Substance Abuse Research, 2007. The “Six T’s”: barriers to screening teens for substance abuse in primary care. Journal of Adolescent Health 40(5), 456–461. [DOI] [PubMed] [Google Scholar]
  39. Welsh JW, Knight JR, Hou SS-Y, Malowney M, Schram P, Sherritt L, Boyd JW, 2017. Association between substance use diagnoses and psychiatric disorders in an adolescent and young adult clinic-based population. Journal of Adolescent Health 60(6), 648–652. [DOI] [PubMed] [Google Scholar]
  40. Williams RJ, Nowatzki N, 2005. Validity of adolescent self-report of substance use. Substance use & misuse 40(3), 299–311. [DOI] [PubMed] [Google Scholar]

RESOURCES