Prediction Model for Two-Year Risk of Opioid Overdose Among Patients Prescribed Chronic Opioid Therapy

Abstract

Background

Naloxone is a life-saving opioid antagonist. Chronic pain guidelines recommend that physicians co-prescribe naloxone to patients at high risk for opioid overdose. However, clinical tools to efficiently identify patients who could benefit from naloxone are lacking.

Objective

To develop and validate an overdose predictive model which could be used in primary care settings to assess the need for naloxone.

Design

Retrospective cohort.

Setting

Derivation site was an integrated health system in Colorado; validation site was a safety-net health system in Colorado.

Participants

We developed a predictive model in a cohort of 42,828 patients taking chronic opioid therapy and externally validated the model in 10,708 patients.

Main Measures

Potential predictors and outcomes (nonfatal pharmaceutical and heroin overdoses) were extracted from electronic health records. Fatal overdose outcomes were identified from state vital records. To match the approximate shelf-life of naloxone, we used Cox proportional hazards regression to model the 2-year risk of overdose. Calibration and discrimination were assessed.

Key Results

A five-variable predictive model showed good calibration and discrimination (bootstrap-corrected c-statistic = 0.73, 95% confidence interval [CI] 0.69–0.78) in the derivation site, with sensitivity of 66.1% and specificity of 66.6%. In the validation site, the model showed good discrimination (c-statistic = 0.75, 95% CI 0.70–0.80) and less than ideal calibration, with sensitivity and specificity of 82.2% and 49.5%, respectively.

Conclusions

Among patients on chronic opioid therapy, the predictive model identified 66–82% of all subsequent opioid overdoses. This model is an efficient screening tool to identify patients who could benefit from naloxone to prevent overdose deaths. Population differences across the two sites limited calibration in the validation site.

Electronic supplementary material

The online version of this article (10.1007/s11606-017-4288-3) contains supplementary material, which is available to authorized users.

INTRODUCTION

Overdoses from pharmaceutical opioids and heroin are a significant public health concern.^1,2 Fatalities due to pharmaceutical opioids increased nearly fourfold from 2000 to 2014, and recent data show that heroin overdose rates are rising.¹ Since opioids are prescribed in an estimated one in five ambulatory visits for pain, targeted overdose-prevention interventions are needed.³

Naloxone is an effective medication used by medical personnel to reverse opioid-induced respiratory depression. In the 1990s, community-based organizations began distributing naloxone to people using heroin for lay bystander administration.⁴ The Food and Drug Administration recently approved new delivery devices for bystander naloxone administration, and national efforts have sought to expand access to naloxone.^5–7 In 2016, the Centers for Disease Control and Prevention (CDC) issued guidelines encouraging providers to co-prescribe naloxone to patients receiving chronic opioid therapy at high risk for overdose.⁶ Many states have also passed laws to permit naloxone standing orders, allowing people to acquire naloxone from pharmacies without a prescription.⁷

Despite these efforts, naloxone uptake appears to be low.^8–10 Cost is one significant barrier to expanded access. The Narcan nasal spray is $150 for a two-dose package, while the cost of two doses of the Evzio auto-injector increased from $690 in 2014 to $4500 in 2016.⁸ In addition to cost barriers, patients receiving chronic opioid therapy may not actively seek naloxone, because they underestimate their risk for overdose, are unaware of the benefits of naloxone, or fear that requesting naloxone will erode trust with their provider.¹¹ Although clinicians can offer and prescribe naloxone to patients, they do not have the tools to efficiently identify patients at highest risk for overdose who would benefit from naloxone.¹²

Numerous studies have examined risk factors for opioid overdose using complex statistical models designed to establish causation.^13–19 In contrast, a prognostic model²⁰ to guide clinical decision-making should be based on a limited set of common diagnoses and characteristics that can be easily collected while a provider is seeing the patient. This implies that the model cannot rely on excessively detailed clinical information, nor can it include future changes impacting risk, such as new diagnoses, discontinuation of opioids, or dose changes.

The objective of this study was to build and validate a prognostic statistical model that could be used by primary care providers and health systems to objectively and efficiently identify patients who are appropriate candidates for naloxone prescriptions at the point of care. We developed the model to accurately predict the risk of fatal and nonfatal opioid or heroin overdose within 2 years, the approximate shelf-life of naloxone.²¹ To maximize generalizability, we aimed to externally validate the model in a health system serving a different patient population.

METHODS

Study Design, Study Sites, and Sources of Data

We developed and internally validated a prognostic model²⁰ in a cohort of patients taking chronic opioid therapy enrolled in the Kaiser Permanente Colorado (KPCO) health plan. KPCO is a group model integrated health system that provides healthcare to approximately 670,000 members. KPCO operates its own pharmacies, and most opioid prescriptions are dispensed to members in these pharmacies.

The model was externally validated in a separate cohort receiving care at the Denver Health and Hospital Authority (DH). DH is a safety-net system serving more than 139,000 ethnically diverse patients. Every DH primary care facility has its own pharmacy.

Demographic and clinical data on patients in both cohorts were extracted from electronic health records (EHR) and pharmacy databases. These data included diagnoses (International Classification of Diseases [ICD]-9 codes) in outpatient, emergency department, and inpatient settings, and prescriptions dispensed in outpatient pharmacies.

Death data (death date and all recorded causes of deaths using ICD-10 codes) were obtained through linkage with state vital records. Social security number, name (including partial names), and birth date were used to identify and confirm matches.

The study was approved by the KPCO Institutional Review Board and the Colorado Multiple Institutional Review Board. Study data are protected by a federal Certificate of Confidentiality.

Study Cohorts

In both sites, we identified patients receiving chronic opioid therapy. First, we identified individuals who received one or more opioid prescriptions between January 1, 2006, and December 31, 2014. Among these patients, we identified those with opioid prescriptions dispensed on at least three different dates within 90 days, irrespective of the number of different opioid prescriptions on a given day. These represented cohorts of patients with incident chronic opioid therapy during the 8-year observation period. Among the first three dispensing dates in the 90-day period, the third dispensing date represented the index date for each patient. Opioids were identified using National Drug Codes and prescription dispensing dates from automated pharmacy files. Long-acting and extended-release (LA/ER, including methadone for pain) and short-acting opioids were included, but methadone dispensed from addiction clinics and all buprenorphine-containing products were excluded, as these are largely prescribed for addiction treatment. Patients prescribed chronic opioid therapy who also received methadone or buprenorphine for addiction treatment during follow-up were not excluded. Since many patients are exposed to opioids intermittently prior to starting chronic opioid therapy, patients who had been exposed to opioids prior to the index date were not excluded.

Patients were excluded if they had a cancer diagnosis or a do-not-resuscitate order in the year prior to the index date, or if they were younger than 18 on the index date. In KPCO, patients were also excluded if they were not continuously enrolled 365 days prior to the index date or if they did not have pharmacy coverage. Since DH does not have enrollment, we excluded patients who did not meet the definition of empanelment used in prior studies: at least one primary care visit in the 18 months prior to the index date.^22,23

Patients were followed up to 2 years after the index date; follow-up was censored at the time of the opioid overdose, death from any cause, disenrollment from the health plan (KPCO) or more than 18 months since the last healthcare visit (DH), or September 30, 2015.

Opioid Exposure and Other Predictor Variables

We considered several known or suspected correlates of opioid overdose^13–19 as potential predictors, including demographics; medications dispensed—including opioid medications and their formulations (LA/ER)—tobacco use; and substance abuse/dependence, mental health, and chronic disease diagnoses (Table 1). Demographic variables were extracted from the EHR, and included age on index date, gender, race/ethnicity, and Medicaid coverage. Using automated pharmacy files, medication exposures included opioids, psychotropics, anticonvulsants, stimulants and muscle relaxants. For opioid exposures, we calculated the average daily morphine milligram equivalents (MME) for all opioid prescriptions on the index date or prior prescriptions whose days’ supply covered the index date.²⁴ Dose and formulation variables were based on active prescriptions covering the index date (e.g., a person with a long-acting opioid prescribed 6 days before the index date in a 30-day quantity was classified as being on an LA/ER formulation). These variables were not included in the model as time-varying covariates, because clinicians would not be privy to information about future changes in dose and formulation when employing a prognostic model. History of opioids dispensed prior to meeting the definition of chronic opioid therapy was examined in the model.

Table 1.

Factors Considered as Potential Predictors at the Derivation Site

Demographics

Age

Gender

Race/ethnicity

Medicaid coverage

Substance/tobacco use/abuse

Alcohol abuse

Drug abuse

Substance abuse/dependence (alcohol abuse and drug abuse)

Tobacco use

Tobacco abuse

Tobacco dependence

Mental health condition

Any mental health diagnosis

Depression

Psychosis

Panic attacks

Prescription medications

Psychotropic drugs

Benzodiazepines

Antidepressants

Anti-anxiety drugs

Antipsychotics

Sedatives/hypnotics

Anticonvulsants

Stimulants

Muscle relaxants

Opioid medications

Opioid dose, mean milligrams morphine equivalents

Long-acting or extended-release opioid

History of opioid dispensing prior to initiating chronic opioid therapy

Medical diagnosis

Pain

HIV

Hepatitis C virus

Chronic obstructive pulmonary disease

Asthma

Liver disease

Kidney disease

Obesity

Tobacco use was based on social history or a tobacco abuse/dependence diagnosis. Diagnoses such as hepatitis C infection were based on ICD–9 codes collected in routine clinical care based on patient history, laboratory results, and diagnostic procedures. Other than age and opioid dose and formulation, all variables were assessed in the 365 days prior to the index date.

Outcome: Opioid Overdose

Between January 1, 2006, and September 30, 2015, we identified fatal and nonfatal pharmaceutical opioid and heroin overdoses in emergency department and inpatient settings. We included unintentional, undetermined and suicidal overdoses, as all opioid-induced respiratory depression can be reversed with naloxone. Consistent with codes used or validated in prior overdose studies and injury surveillance guidelines, we used ICD-9 codes 965.0–965.02, 965.09, and E850.0–E850.2 for nonfatal overdoses; for fatal overdoses, we used ICD-10 underlying cause of death codes for drug poisoning (X40–X44, X60–X64, X85, Y10–Y14) and a contributing cause of death code indicating opioid involvement (T40.0–T40.4, T40.6).^{13,17,25–28} Overdoses were captured within 2 years of the index date, as the shelf-life of naloxone is approximately 2 years.²¹

Predictive Model Development and Internal Validity

In the derivation (KPCO) cohort, we used Cox proportional hazards regression to examine associations between baseline predictor variables and incident opioid overdose. Prior to building the model, we examined frequencies and distributions for potential predictor variables to identify missing values, sparse categories, and extreme values. If cause of death was unknown, the patient was excluded from the analysis, since some of these deaths may have been opioid-related.

To avoid overfitting the model, we restricted the number of potential predictors and required at least ten overdoses per degree of freedom. We first grouped medications and diagnoses into broad, clinically meaningful categories so that the model could be used in a typical primary care visit. We then examined the bivariable associations between potential predictors and overdose risk. Guided by statistical and clinical significance, we chose nine predictors for the multivariable model. With backward elimination, we sequentially removed variables with a P value >0.05 from the model. Backward elimination is the preferred choice for variable selection in predictive modeling.²⁰ Continuous variables were tested using simple parametric forms (a single linear term, low-order polynomials, logarithmic) and restricted cubic splines; categorical variables were modeled using indicator variables. We evaluated proportional hazards assumptions using Schoenfeld residual plots and a global goodness-of-fit test.

We evaluated the final model’s internal validity at the derivation site by examining discrimination and calibration.²⁹ Using the Harrell bootstrap resampling,³⁰ we calculated the bootstrap-corrected concordance statistic (c-statistic) to assess discrimination (the extent to which patients who were predicted to be at higher risk exhibit higher overdose rates relative to those predicted to be at lower risk). Calibration (the extent to which the predicted overdose risk agrees with the observed risk) was evaluated with the Greenwood-D’Agostino-Nam test.³¹ Discrimination and calibration were assessed graphically by dividing the cohort into tertiles based on the predicted risk, and then plotting the mean predicted cumulative risk with the Kaplan-Meier observed cumulative risk for each tertile.³² Widely separated Kaplan-Meier cumulative risk curves indicate better discrimination; agreement between predicted and observed risk over the follow-up indicate better calibration.³³ In addition, we calculated the D-statistic to further assess discrimination and the R² statistic on the basis of the D-statistic to quantify explained variation.^{33, 34, 35}

To improve the model’s clinical utility, the regression coefficients were translated into a point-based risk score, with more points indicating higher overdose risk.³⁶

External Validation

The external validity of the model was tested in an independent cohort of DH patients (validation cohort) by examining discrimination and calibration.³³ We first calculated the 2-year predicted overdose risk for patients in the validation cohort using coefficients for the linear predictor and baseline survival from the derivation model. We then calculated the c-statistic to assess discrimination. The validation cohort was divided into tertiles based on the predicted risk from the derivation model, and discrimination and calibration were assessed graphically. Calibration was further assessed with the Greenwood-D’Agostino-Nam test.³¹ The D-statistic and R² statistic were also calculated.^{33, 34, 35}

All analyses were conducted using R statistical software (R Foundation for Statistical Computing, Vienna, Austria) and SAS version 9.4 (SAS Institute Inc., Cary, NC).

RESULTS

Derivation and Internal Validation of Risk Model

At the derivation site, we identified 58,162 adult patients who received chronic opioid therapy and who met enrollment and pharmacy coverage criteria between January 1, 2006, and December 31, 2014. Of these, 15,270 (26.3%) were excluded because they had a diagnosis of cancer or a do-not-resuscitate order. Another 64 (0.1%) patients were excluded because they died of unknown causes, some of which may have been opioid-related. In the cohort of 42,828 patients, there were 110 incident nonfatal opioid overdoses, of which five were from heroin, and 11 fatal opioid overdoses. Among the 42,828 patients, 36 (0.08%) had a prior overdose in the year before the index date. The 2-year Kaplan-Meier overdose risk was 0.33% (95% confidence interval [CI] 0.28–0.40). Baseline characteristics and opioid prescription types covering the index date are displayed in Table 2 and eTable 1 (online), respectively. Overall, the average number of opioid prescriptions over the follow-up was 7.88 (standard deviation [SD] = 9.97). In the year prior to the index date, 40.6% the of cohort had received prior intermittent opioid prescriptions. Among those who received prior prescriptions, the mean number of prescriptions was 2.28 (SD = 1.78).

Table 2.

Baseline Demographic and Clinical Characteristics of the Study Cohorts at the Derivation and Validation Sites

Characteristic	Derivation site cohort (n = 42,828)	Validation site cohort (n = 10,708)
Mean age, years (SD)*	54.9 (17.6)	48.2 (13.1)
Women, no. (%)*	24,784 (57.9)	6031 (56.3)
Race, no. (%)*
White	30,170 (70.4)	3130 (29.2)
Black	1869 (4.4)	2182 (20.4)
Hispanic	5270 (12.3)	5057 (47.2)
Unknown	3596 (8.4)	143 (1.3)
Other	1923 (4.5)	196 (1.8)
Medicaid coverage, no. (%)*	1372 (3.2)	3524 (32.9)
Mental health diagnosis, no. (%)*	13,338 (31.1)	4440 (41.5)
Psychotropic prescription, no. (%)	21,809 (50.9)	5548 (51.8)
Substance abuse/dependence, no. (%)*	2711 (6.3)	2236 (20.9)
Tobacco use or tobacco abuse/dependence, no. (%)*	9937 (23.2)	5614 (52.4)
History of opioid dispensing in the year prior to initiating chronic opioid therapy, no. (%)*	17,383 (40.6)	3635 (33.9)
Long-acting or extended-release opioid, no. (%)*	1374 (3.2)	586 (5.5)
Opioid dose, mean milligrams morphine equivalents (SD)*	61.8 (59.7)	44.8 (56.0)
Medical diagnoses, no. (%)
Pain	35,603 (83.1)	8956 (83.6)
HIV*	108 (0.3)	195 (1.8)
Hepatitis C virus*	529 (1.2)	1064 (9.9)
Chronic obstructive pulmonary disease*	5008 (11.7)	833 (7.8)
Asthma	5007 (11.7)	1289 (12.0)
Liver disease*	1880 (4.4)	1379 (12.9)
Kidney disease*	4846 (11.3)	787 (7.3)
Obesity*†	14,897 (34.8)	4415 (41.2)
Utilization in year prior to index date‡
Primary care visits, mean no. (SD)	3.8 (3.0)	4.1 (3.2)
Pharmacy visits, mean no. (SD)	15.0 (9.9)	11.9 (9.9)

*p < 0.05

^†10.9% missing at the derivation site cohort and 8.2% missing at the validation site

^‡Utilization data provided for descriptive purposes only

The following nine variables were selected as potential overdose predictors: age; daily opioid dose; LA/ER opioid prescription; psychotropic prescription; history of opioid prescription; tobacco use; and mental health, substance abuse/dependence, and hepatitis C diagnoses (Table 3). Age showed a U-shaped relationship with overdose risk, with the lowest risk among individuals 55–65 years of age. Thus, a quadratic term for age was included to improve model fit. The final model included five of these variables (age; age-squared; LA/ER formulation; tobacco use; and mental health and substance abuse/dependence diagnoses, Table 3). Schoenfeld residual plots and the global goodness-of-fit test (P = 0.46) indicated that the proportional hazards assumption was met.

Table 3.

Unadjusted and Adjusted (final) Cox Regression Models for Predicting 2-Year Overdose Risk for Patients Prescribed Chronic Opioid Therapy at the Derivation Site

Characteristic	Unadjusted model	Adjusted model*
	Hazard ratio (95 CI†)	β coefficient	Hazard ratio (95 CI†)
Age (per year)	0.93 (0.88–0.98) ‡	−0.06915	0.93 (0.89–0.98)
Age-squared	1.00 (1.00–1.00) ‡	0.0005626	1.00 (1.00–1.00)
Mental health diagnosis§	4.18 (2.88–6.07)	1.22076	3.39 (2.32–4.96)
Psychotropic prescription	2.82 (1.88–4.25)
Substance abuse/dependence diagnosis||	6.01 (4.03–8.96)	1.24387	3.47 (2.25–5.36)
Tobacco use or tobacco abuse/dependence diagnosis¶	2.31 (1.60–3.32)	0.42788	1.53 (1.03–2.28)
History of opioid prescriptions in the year prior to initiating chronic opioid therapy	1.43 (1.00–2.05)
Long-acting or extended-release opioid formulation	2.47 (1.25–4.87)	0.68552	1.99 (1.00–3.93)
Daily opioid dose (per 10 mg morphine equivalents)**	1.01 (0.99–1.03)
Hepatitis C diagnosis††	2.82 (1.04–7.63)

*S₀(2) = 0.99768 (2-year baseline survival)

^†CI, confidence interval

^‡Age and age-squared included together in the unadjusted model

^§Identified using ICD-9 codes 295, 296, 297, 298, 300, 301, 309, 311

^||Included drug and alcohol abuse and dependence. Identified using ICD-9 codes 292, 304, 305.2, 305.3, 305.4, 305.5, 305.6, 305.7, 305.8, 305.9, V65.42, 291, 303, 303.0, 303.9, 305.0, 357.5, 425.5, 535.3, 571.0, 571.1, 571.2, 571.3, 980, V11.3

^¶Based on information in the electronic social history table or ICD-9 code 305.1

**Calculated for each prescription by multiplying the quantity of each prescription by the strength of the prescription and then multiplying by a conversion factor. MMEs were then added together for all the prescriptions on the index date

^††Identified using ICD-9 codes 070.41, 070.44, 070.51, 070.54, 070.7, 070.70, 070.71

The model effectively discriminated between high- and low-risk patients (Table 4), with an uncorrected c-statistic of 0.75 (95% CI 0.70–0.79) and a bootstrap-corrected c-statistic of 0.73 (95% CI 0.69–0.78). The D-statistic was 1.56 (95% CI 1.28–1.85), and the R² was 36.8% (95% CI 28.04–44.85%). Patients were divided into tertiles according to their predicted overdose risk, and observed and predicted risks were graphed over 2 years (Fig. 1, derivation cohort). The graph demonstrates good discrimination, with the curve for the highest risk tertile well separated from the two lower risk tertiles. The graph also demonstrates effective calibration, since observed and predicted risks show good agreement over time. The Greenwood-D’Agostino-Nam test (P = 0.22) indicated that the model was well-calibrated. When risk was dichotomized into high (third tertile) versus lower (first and second tertiles) risk, the model had sensitivity of 66.1% and specificity of 66.6%. The positive predictive value for the model was 0.56%.

Table 4.

Observed Cumulative Risk of Opioid Overdose at 2 Years, by Tertiles of Predicted Risk from the Derivation Site Model

Tertile	Derivation site			Validation site
	Individuals (n)	Overdoses (n)	Two-year cumulative overdose risk (%)	Individuals (n)*	Overdoses (n)	Two-year cumulative overdose risk (%)
1	14,299	15	0.12	1923	6	0.33
2	14,189	26	0.22	3336	15	0.48
3	14,340	80	0.68	5449	97	1.88

*The predicted risk for the cohort at the derivation site was categorized by tertiles, and the same risk thresholds were then applied to the validation site. Because the risk was higher at the validation site, the proportion of patients was not equal across the tertiles, and there were more patients in higher-risk tertiles

Figure 1.

Plots of observed and predicted 2-year risk of opioid overdose at the derivation and validation sites, by tertiles of predicted risk from the derivation site model.

External Validation of Risk Model

At the validation site, we identified a cohort of 12,876 adult patients receiving chronic opioid therapy who met empanelment criteria between January 1, 2006, and December 31, 2014. Of these, 2168 (16.8%) were excluded because they had a diagnosis of cancer or a do-not-resuscitate order. In the cohort of 10,708 patients, there were 65 incident nonfatal overdoses, 15 of which were from heroin, and 53 fatal overdoses, nine of which were from heroin. Among 10,708 patients, 23 (0.2%) had an overdose in the year prior to the index date. The 2-year Kaplan-Meier overdose risk was 1.2% (95% CI 0.97–1.4). The derivation and validation cohorts differed significantly by most baseline characteristics (Table 2). Opioid prescriptions are shown in eTable 1 (online). Utilization rates over the follow-up were similar between sites (eTable 2, online).

When the derivation model was used to predict overdose at the validation site, the c-statistic was 0.75 (95% CI 0.70–0.80), indicating good discrimination. Model calibration, in contrast, was relatively poor. The Greenwood-D’Agostino-Nam test was statistically significant (P < 0.001); Figure 1 (validation cohort) demonstrates that the model under-predicted overdose. When the third tertile was considered high-risk (vs. lower-risk), the model’s sensitivity and specificity were 82.2% and 49.5%, respectively. The positive predictive value was 1.8%. The D-statistic was 1.53 (95% CI 1.24–1.82), and the R² was 35.8% (95% CI 26.83–44.09%).

The prognostic index was converted to a points-based scale (Table 5) to allow providers to classify patients as low- (≤ 85), medium- (86–103), or high-risk (≥ 104) by summing the points based on patient characteristics. The cut points are based on risk tertiles. As an example, a 50-year-old (84 points for age) with a diagnosis of a substance use disorder (22 points) in the last year would have a score of 106, indicating high 2-year opioid overdose risk.

Table 5.

Point-Based System* to Calculate 2-Year Risk of Opioid Overdose Based on Model Coefficients

Characteristic	Number of points
Age (years)
18–25	98
26–30	94
31–35	91
36–40	88
41–45	85
46–50	84
51–55	83
56–60	82
61–65	83
66–70	84
71–75	84
76–80	86
81–85	87
86–90	89
91–95	92
96–100	96
Mental health diagnosis	22
Substance abuse/dependence diagnosis	22
Tobacco use or tobacco abuse/dependence diagnosis	8
Long-acting or extended-release opioid formulation	12

*Scores of summed points, based on patient characteristics, indicate low (≤ 85), medium (86–103) or high risk (≥104)

DISCUSSION

We developed a prognostic model to identify patients on chronic opioid therapy at high risk for overdose. The model comprised five variables that were readily available from the EHR in two diverse health systems. Although calibration was less than ideal in the validation cohort, the model demonstrated good internal validity and discriminated between patients with and without overdose in both health systems. At the point of care, providers could use the model to risk-stratify patients for naloxone prescribing.

Since naloxone is a safe and effective medication that prevents overdose death, universal prescribing to all patients on chronic opioid therapy could have a significant public health impact. At present, however, evidence supporting the effectiveness of such practice is limited. In addition, due to rising costs,⁸ universal prescribing may not be feasible in many health systems. Further, clinicians must inform patients of their risk, describe how to recognize an overdose, explain how to use naloxone, and write the prescription. Prior studies suggest these steps may be a barrier to universal naloxone prescribing given brief primary care visits and competing preventive and chronic care needs,¹² such as developing alternative pain management strategies and reducing exposure to opioids.

Our results corroborate the findings of prior overdose risk studies among patients prescribed chronic opioid therapy.^13,15 The annual overdose incidence rate in our derivation cohort was approximately 167.4/100,000 person-years, and patients taking LA/ER formulations had a significantly higher overdose risk than patients taking short-acting opioids. Moreover, only 3% of patients were initiated on treatment with LA/ER formulations, reflecting CDC prescribing guidelines recommending that physicians start with short-acting opioids.⁶

Current national policies and guidelines for opioid prescribing practices highlight dose as an important indicator of opioid overdose risk.^6,13 In our study, however, opioid dose did not remain in the final model. Dose is likely most predictive of overdose events shortly after a prescription.¹³ Since our model was designed to predict overdose events for up to 2 years after a prescription, dose may have fluctuated significantly over the follow-up, rendering certain stable variables more predictive of longer-term overdoses. In our model, mental health diagnoses, substance use disorders, LA/ER opioid formulations, and tobacco use were stronger overdose predictors. These results suggest that focusing primarily on dose may not be the best approach for targeting preventive clinical interventions which confer benefit over longer periods.

Prescription or illicit opioid use may lead to overdose for a range of reasons, including prescribing error, suicide, addiction, or risk behavior. While separate models could be developed and tested for each of these pathways to overdose, it is not practical to employ multiple overdose risk models in primary care. Since naloxone is effective regardless of the cause or intent of the opioid poisoning, we employed a broad overdose outcome definition.

In contrast to prior risk models,^37,38 we built a parsimonious model based on information readily available in the EHR and externally validated the model. While discrimination was good at the external site, the model’s calibration was less than ideal. This limitation can be attributed to the significant case mix differences between the two cohorts (Table 2). The baseline overdose rate was significantly lower in the derivation cohort, and may in part explain why the model under-predicted overdose risk in the validation cohort. In addition, the ratio of nonfatal to fatal overdose was considerably higher in the derivation cohort than in the validation cohort, suggesting that nonfatal overdoses may have been under-ascertained in the validation cohort. While these factors may have negatively affected the model’s overall performance, the model was also associated with sensitivity of greater than 80% in the validation cohort. This latter result suggests that a large majority of the DH patients at risk for overdose would have been targeted for naloxone prescription, which is appropriate given its favorable risk–benefit profile.³⁹

As described above, the intent of the model was to risk-stratify patients prescribed chronic opioid therapy into groups that would be indicated for naloxone. In addition to guiding providers, the model could be used by health systems to target wide-scale naloxone interventions. However, it may not be advisable to use the model for applications with substantially different risk–benefit profiles. For instance, due to its miscalibration, the model should not be used directly by patients to determine their individual overdose risk. In addition, because of the low positive predictive value of the model, most patients indicated for naloxone will not experience an overdose. Therefore, physicians should not use our model to, for example, decide whom to taper off opioid therapy, which may have unintended adverse effects. For such decisions, physicians should consider other factors that may affect the risk–benefit ratio of tapering, including pain, risk of withdrawal, and risk of transitioning to heroin use.^40–45

Amidst the current opioid epidemic, we developed a five-variable model using readily available EHR data to identify high-risk patients who would benefit from naloxone to prevent deaths. While the model needs further evaluation, it was able to predict 66–82% of opioid overdoses in two disparate patient populations.

Compliance with Ethical Standards

Conflict of Interest

Dr. Ingrid Binswanger is currently employed by the Colorado Permanente Medical Group and receives royalties from UpToDate but does not have any additional conflict of interest. All remaining authors declare that they do not have a conflict of interest.