Assessing opioid overdose risk: A review of clinical prediction models utilizing patient-level data

Abstract

Drug, and specifically opioid-related, overdoses remain a major public health problem in the United States. Multiple studies have examined individual risk factors associated with overdose risk, but research developing clinical risk prediction tools for overdose has only emerged in the last few years. We conducted a comprehensive review of the literature on patient-level factors associated with opioid-related overdose risk, with an emphasis on clinical risk prediction models for opioid-related overdose in the United States. Studies that developed and/or validated clinical prediction models were closely reviewed and evaluated to determine the state of the field. We identified 12 studies that reported risk prediction models for opioid-related overdose risk. Published models were developed from a variety of data sources, including Veterans Health Administration data, Medicare data, commercial insurance data, and statewide linked datasets. Studies reported model performance using measures of discrimination, usually at good-to-excellent levels, though they did not always assess calibration. C-statistics were better for models that included clinical predictors (c-statistics: 0.75-0.95) compared to models without them (c-statistics: 0.69-0.82). External validation of models was rare, and we found no studies evaluating implementation of models or risk prediction tools into clinical practice. A common feature of these models was a high rate of false positives, largely because opioid-related overdose is rare in the general population. Thus, efforts to implement prediction models into practice should take into account that published models overestimate overdose risk for many low-risk patients. Future prediction models assessing overdose risk should employ external validation and address model calibration. In order to translate findings from prediction models into clinical public health benefit, future studies should focus on developing clinical prediction tools based on prediction models, implementing these tools into clinical practice, and evaluating the impact of these models on treatment decisions, patient outcomes, and, ultimately, opioid overdose rates.

Introduction

High rates of drug-related overdoses constitute a major public health crisis and account for considerable morbidity and mortality in the United States. The rate of opioid-related overdose increased by 9.7% from 2018 to 2019 (1), and 70% of the 70,000 overdose deaths in 2017 involved opioids (2). The current opioid-overdose crisis has been characterized as consisting of three distinct “waves.” As opioid prescribing began to take off in the 1990s due to efforts to treat chronic pain more aggressively, overdose deaths due to prescription opioids also increased; these increases in prescribing plateaued around 2012 (3, 4). As the first wave crested, a second wave of heroin-related fatal overdoses emerged in 2010 and rose rapidly, matching rates of overdoses involving prescription opioids by 2016 (5). The recent influx of synthetic opioids, especially illicitly manufactured fentanyl, into the United States’ drug supply has resulted in a third wave of opioid overdose deaths that emerged in 2014 and, as of 2020, shows no signs of plateauing (6, 7). In recent years, fentanyl has overtaken prescription opioids and heroin as the most common substance contributing to fatal opioid overdoses, and overdoses involving multiple opioids and other drugs appear to be increasing (8).

Most research on opioid-related overdose relies on administrative data. Fatal overdoses are typically assessed using death certificate data and are categorized using International Classification of Diseases, 10^th revision (ICD-10) codes. Nonfatal overdoses in the United States are categorized through the use of codes with the clinical modification designation (ICD-9-CM/ICD-10-CM) (9). Using standardized, codebased definitions facilitates comparison of overdose rates across different states and counties but also imposes some limitations. For example, opioid-related overdoses are commonly categorized by the type of opioid involved. ICD-10 provides separate codes only for the following categories of opioids: opium (T40.0), heroin (T40.1), natural and semisynthetic opioids (T40.2), methadone (T40.3), synthetic opioids other than methadone (T40.4), and other or unspecified opioids (T40.6). Thus, analyses singling out other specific opioid compounds (e.g., hydrocodone, morphine, oxycodone) are not possible without additional data, such as toxicology reports. In recent years researchers and the Centers for Disease Control and Prevention (CDC) have used the ICD-10 code T40.4 as a proxy for overdoses involving illicit fentanyl (10). Overdoses are also frequently characterized by outcome (fatal or nonfatal), and manner of overdose (e.g., intentional, unintentional, or unknown). However, vital records data do not specify the source of opioids and often (with the exception of methadone and heroin) do not specify the precise chemical compound involved and so do not allow researchers to distinguish prescription opioids from opioids that are illicitly produced. Researchers can identify whether a deceased person was prescribed opioids prior to their death by linking vital records data with data from administrative claims or prescription drug monitoring programs, but these data sources would not capture opioids obtained from non-medical sources that may have contributed to that person’s overdose. Despite these limitations, use of single diagnosis codes can reliably identify opioid-related overdoses (11, 12).

Clinical and public health scholarship on opioid-related overdose continues to evolve and change rapidly. The objective of this article is to review studies that utilized patient-level data to construct prediction models for opioid-related overdose, and to assess the utility and limitations of these models. We also make recommendations for future studies in this area, including efforts to translate prediction models into clinically useful tools that can impact treatment decisions and, ultimately, reduce overdose rates and improve public health. This article should be useful for clinicians and health services researchers interested in understanding or advancing knowledge about overdose risk assessment.

Methods

To supplement our own clinical and research expertise and inform this narrative review, we conducted a literature search for studies of clinical prediction models for opioid-related overdose. Articles were identified by a PubMed search designed with input from an experienced research librarian. Details of the search terms utilized are available in Table S1. We included studies that were indexed as of October 20, 2020. We included studies published in English, conducted on populations in the United States, and that evaluated opioid-related overdose as an outcome. While both opioid prescribing and related harms have been associated with area-level socioeconomic factors (13, 14), our review focused on factors assessed at the individual (patient) level. Articles were chosen through sequential review. First, the primary author (IET) reviewed article titles and excluded those that clearly did not meet inclusion criteria. He then reviewed abstracts from the remaining articles and excluded those that clearly did not meet inclusion criteria. The remaining publications were fully reviewed to assess inclusion. We ultimately identified 42 articles that examined risk factors and 12 articles that either developed or validated clinical prediction models for opioid-related overdose. Table 1 summarizes the papers we identified reporting risk prediction models; articles examining specific risk factors associated with overdose risk are summarized in Table S2.

Table 1:

Characteristics of risk prediction models for opioid-related overdose in populations from the United States

Author, Year (Reference)	Study Type	Modelling Method	Data		Study Population	Type of Overdose	Predictors	Validation		Discrimination (c-statistic)	Calibration Assessed?
			Source	Years				Type	Method
Chang, 2019 (60)	Development + Validation	LR	Maryland PDMP data linked to hospital discharge and medical examiner data	2015-2016	Maryland residents, 18-80 years old, with at least one Rx for BUP for OUD	Nonfatal overdose identified by ICD9/ICD10 codes in hospital discharge data. Fatal overdose identified in medical examiner findings	Age, sex, non-BUP opioid characteristics (days supply, total prescribers, total dispensers, payment method), BUP characteristics (days supply, total prescribers, total dispensers, payment method), total BZD Rxs	Internal	Bootstrapping	0.69 (0.67, 0.70)	No
Chang, 2020 (59)	Development + Validation	LR	Maryland PDMP data linked to hospital discharge and medical examiner data	2015-2016	Maryland residents, 18-80 years old, with at least one opioid Rx	Nonfatal overdose identified by ICD9/ICD10 codes in hospital discharge data. Fatal overdose identified in medical examiner findings	Age, sex, opioid characteristics (days supply, total prescribers, total dispensers, payment method), total BZD Rxs	Internal	Bootstrapping	Nonfatal: 0.77 (0.77–0.78) Fatal: 0.83 (0.82–0.85)	No
Dong, 2019 (61)	Development + Validation	RF, RR, DT, DNN	Inpatient data from SPARCS and Cerner’s Health Facts modelled separately	SPARCS: 2005-2016 Cerner: 2000-2017	SPARCS: Patients with at least one historic encounter before first opioid poisoned related diagnosis. Randomly selected cohort of cases/controls with ratio of 1:10. Cerner: Like SPARCS data, included those w/at least one hospital visit with a clinical event record.	Opioid poisoning as identified by ICD-9 and ICD-10 codes.	Thousands of candidate predictors considered for inclusion. Different pools of candidates for each data source.	Internal	Random Split-Sample	SPARCS: RF: 0.95, RR: 0.81, DT: 0.85, DNN: 0.94 Cerner: RF: 0.95, RR: 0.74, DT: 0.93, DNN: 0.95	No
Ferris, 2019 (62)	Development + Validation	LR	Maryland PDMP data linked to hospital discharge and medical examiner data	2016	Maryland residents, 18-80 years old, with at least one opioid Rx	Any opioid-related fatal overdose. Subset for illicit fatal overdoses involving fentanyl or heroin with no Rx opioids present	Age, sex, total prescribers, total dispensers, total fills (methadone, other LA, SA Schedule 2, SA Schedule 3-4, MOUD, BZD, muscle relaxant, non-BZD sedative), >90 opioid days supply, concurrent BZD-opioid use, daily MME	Internal	Random Split-Sample	Any Fatal: 0.77 Illicit Fatal: 0.76	No
Geissert, 2018 (63)	Development + Validation	LR	Oregon PDMP data linked to vital records and hospital discharge data	Development: 2013 Validation: 2012	Development: Oregon residents, >12, who received an opioid Rx from an Oregon prescriber in 2013 Validation: Oregon residents, >12, who received an opioid Rx from an Oregon prescriber in 2012	Rx opioid-related fatal overdoses (identified by ICD-10 codes in vital health records) and nonfatal overdoses (identified by ICD-9 codes in discharge data)	Age, urban/rural, opioid characteristics (total Rxs, total prescribers, total pharmacies, any LA, Daily MME ≥90), non-opioid characteristics (Any BZD/sedative, any carisoprodol), any opioid overlap, any LA/SA opioid overlap, any opioid-BZD/sedative overlap, any opioid-BZD/sedativecarisoprodol overlap	External	Temporal	0.82	Not on validation cohort
Glanz, 2018 (64)	Development + Validation	CPH	Electronic health records data and pharmacy databases linked to state vital records	2006-2015	Development: Non-cancer patients, 18+, on long-term opioids enrolled in the Kaiser Permanente Colorado Health plan Validation: Non-cancer patients, 18+, receiving long-term opioid therapy at Denver Health and Hospital Authority	Any opioid-related fatal overdoses (identified by ICD-10 codes in vital health records) and nonfatal overdoses (identified by ICD-9 codes in hospital records)	Age, age-squared, mental health diagnosis, SUD diagnosis, tobacco-use or TUD diagnosis, any LA opioid	External	New Setting	0.75 (0.70–0.80)	Yes
Lo-Ciganic, 2019 (65)	Development + Validation	LR, LASSO, RF, GBM, DNN	Medicare database	2011-2015	A 5% random sample of Medicare beneficiaries who filled at least one opioid Rx	Any opioid-related fatal or nonfatal overdoses identified using ICD-10 codes	268 predictor candidates were assessed for inclusion. These included patient, prescriber, and regional factors. Opioid and non-opioid Rx factors were also included.	Internal	Random Split-Sample	DNN: 0.91, GBM: 0.90	No
Metcalfe, 2020 (66)	Validation	LR	Unidentified multi-state private insurer claims data	2015-2016	Patients with at least one opioid Rx. Controls randomly selected to achieve case-control ratio of 1:10	Any opioid-related overdose or serious opioid-induced respiratory depression (identified through ICD-9-CM/ICD-10-CM)	See Zedler, 2015 (61)	External	New Setting	0.88	Yes
Saloner, 2020 (67)	Development + Validation	LR	Maryland PDMP data linked to hospital discharge, medical examiner, and criminal justice data	2015-2016	Maryland residents, 18-80 years old, with at least one opioid Rx	Nonfatal overdose identified by ICD-9/ICD-10 codes in hospital discharge data. Fatal overdose identified in medical examiner findings	Age, sex, opioid characteristics (total SA, any LA, MOUD, any BZD, total prescribers, total MME), total ED visits, inpatient admission, visit w/diagnosis for OUD or other SUD, prior visit w/nonfatal OD, visit w/nonoverdose injury, mental health treatment, SUD treatment, OUD treatment, any arrest, arrest for misdemeanor drug charges, any parole/probation case, newly released from prison for drug or property crimes	Internal	Bootstrapping	Nonfatal: 0.85 (0.85–0.86) Fatal: 0.89 (0.88–0.90)	No
Sun, 2020 (68)	Development + Validation	LR, EN, RF	Optum Datamart	2011-2015	Patients, 18+, who filled at least one opioid Rx	Any opioid-related overdose identified by ICD-9-CM codes	78 candidate predictors considered. Demographics, medical diagnoses, Rx characteristics, and healthcare utilization.	Internal	Random Split-Sample	EN: 0.89 (0.87–0.90), LR: 0.88 (0.87–0.90), RF: 0.86 (0.85–0.88)	Yes
Zedler, 2015 (69)	Development	LR	Veterans’ Health Affairs database	2010-2012	Patients with at least one opioid Rx. Controls randomly selected to achieve case-control ratio of 1:10	Any opioid-related overdose or serious opioid-induced respiratory depression (identified through ICD-9-CM)	OUD, bipolar disorder or schizophrenia, chronic lung disease, chronic liver disease, chronic kidney disease, traumatic injury, sleep apnea, any Rxs (LA opioids, methadone, oxycodone, antidepressants, BZDs), maximum daily MME, one or more ED visits, hospitalized for one or more days			0.88	Yes
Zedler, 2018 (70)	Validation	LR	IMS Health claims data	2009-2013	Patients with at least one opioid Rx. Controls randomly selected to achieve case-control ratio of 1:4	Any opioid-related overdose or serious opioid-induced respiratory depression (identified through ICD-9-CM)	Old Risk Index: See Zedler, 2015 (65) New Risk Index based on data in this study: SUD, bipolar or schizophrenia, stroke, kidney disease, heart failure, pancreatic disease, chronic lung disease, migraines, any Rxs (fentanyl, morphine, methadone, hydromorphone, LA opioid, BZD, antidepressant), maximum daily MME	External	New Setting	Zedler, 2015 (65) Risk Index: 0.85 New Risk Index: 0.90	Yes

LR: Logistic Regression, RF: Random Forest, RR: Ridge Regression, DT: Decision Tree, DNN: Deep Neural Network, CPH: Cox Proportional Hazards Regression, LASSO: Least Absolute Shrinkage and Selection Operator, EN: Elastic Net Regularization, GBM: Gradient Boosting Machine, PDMP: Prescription Drug Monitoring Program, Rx: Prescription, BZD: Benzodiazepine, BUP: Buprenorphine, OUD: Opioid Use Disorder, SPARCS: New York State Statewide Planning and Research Cooperative System, LA: Long-Acting, SA: Short-Acting, MOUD: Medication for Opioid Use Disorder, MME: Milligram Morphine Equivalents, SUD: Substance Use Disorder, TUD: Tobacco Use Disorder, ED: Emergency Department

Results

Risk factors for opioid-related overdose

Multiple studies have examined individual factors associated with overdose risk. Studies have generally examined overdoses related to prescription opioids, illicit opioids (e.g., heroin), or overdoses involving any opioid. Studies have further examined overdoses by outcome (fatal or nonfatal) and/or intent. Researchers often examine all opioid-related overdoses in order to maximize their study’s power to detect significant associations with risk of overdose (a relatively rare event). Table S2 of the Supplement presents a detailed list of studies that have examined specific risk factors by overdose type organized by risk factor.

These studies have been conducted in a wide range of populations using a variety of data sources. Populations studied in the United States have focused on veterans, Medicaid beneficiaries, Medicare beneficiaries, commercially insured populations, emergency department patients, and, more generally, patients appearing in national databases or identified in state-specific prescription drug monitoring programs (PDMPs). PDMPs include databases that track all outpatient opioids dispensed in a given state and web interfaces clinicians can check in real time to review patients’ history of prescribed controlled substances. Studies using PDMP data to identify risk factors associated with overdose risk have linked PDMP data to overdose outcome data. Most risk factor studies have broad inclusion criteria and usually focus on adults without a cancer diagnosis and at least one opioid prescription during the study period. Some studies, however, have focused on patients in certain age groups (e.g., adolescents), with certain opioid receipt patterns (e.g., new versus long-term use), or with specific comorbidities (e.g., patients with cancer diagnoses).

Risk factors for opioid-related overdose have been studied extensively and fall into six general categories: patient demographics, mental health comorbidities, substance use disorders, physical health comorbidities, characteristics of opioids prescribed, and non-opioid medications prescribed. Age (15-24), sex (17-20, 22, 23, 25, 26), race (15,16, 18-21, 23, 24, 27-29), and socioeconomic status (15,18, 23, 25, 26, 28, 30) are the most commonly studied demographic risk factors. Positive associations between mental health diagnoses (16, 21, 22, 31-33), more specifically mood (17, 19, 24, 27-29, 34, 35) and thought disorders (25), and overdose risk have also been documented. Any substance use disorder (16-18, 22, 31-35), opioid use disorders (19, 21, 24, 28, 36, 37), non-opioid drug use disorders (19, 21, 24, 28), alcohol use disorder (18, 19, 25, 29, 32, 34, 36), and tobacco use disorder (17, 18, 28, 33, 36), have also been associated with increased overdose risk. Various physical health comorbidities, including chronic pain and lung disease, have also been identified as risk factors (24, 25, 27, 28, 33). Opioid prescription characteristics associated with greater overdose risk include higher prescribed dose (16, 17, 19, 20, 22, 24, 29, 32-34, 38-44), longer duration of treatment (20, 25, 31, 45, 46), receipt of long-acting/extended-release opioid formulations (20, 24, 31), no opioid tolerance (47), receipt of overlapping prescriptions (29, 45), receiving opioids from multiple prescribers (20, 41, 42, 45, 48, 49) and pharmacies (29, 32, 41, 42, 45, 49, 50), and living with people who are prescribed opioids (51, 52). Concurrent receipt of non-opioid controlled substances, particularly benzodiazepines, is a well-documented risk factor for opioid-related overdose risk (17-19, 23, 25, 31, 33, 39, 42, 43, 49, 53-55). Concurrent use of other controlled substances, such as muscle relaxants and other non-benzodiazepine sedatives (18-20, 44, 56), has also been associated with increased risk.

Prediction models developed for opioid-related overdose

Prediction models—models that assess an individuals’ risk of experiencing an opioid overdose—can be important resources for translating clinical and epidemiologic research into public health benefit. They are the precursors to prognostic tools that, when integrated into clinical workflow, can provide clinicians with critical information to help inform decisions about opioid prescribing and treatment in order to reduce opioid-related harms. Developing prediction models involves selecting individual predictors and then combining them into a multivariable model to estimate the probability of an outcome (overdose). To avoid overfitting the resultant model, model parameters should then be applied or validated on a previously held out sample; the results of this validation procedure are then used to describe model performance. External validation, validation using data from a different setting, is preferred over internal validation, validation using a subset of same data from which the model was derived, because external validation provides evidence for model generalizability (57, 58). Prediction models are typically assessed on two characteristics: discrimination and calibration. Discrimination refers to a model’s ability to differentiate between individuals who experience an overdose (cases) and those who do not (controls). Calibration describes the agreement of model predictions versus observed outcomes. Evaluation of discrimination and calibration are critical to proper characterization of how well a prediction model performs (57, 58). We identified twelve studies that reported prediction models for opioid-related overdose (59-70). The characteristics of these studies are described in Table 1.

Years of Data Analyzed

The years of data utilized to develop prediction models varied by study. Among the studies we identified, only two utilized more than 5 years of data to develop their prediction models (61, 64). Studies using PDMP data tended to use one year or less of data to generate predictors and then assess overdose outcomes over a subsequent follow-up period (59, 60, 62, 63, 67). All but one of the studies we identified were published in 2018 or later. Most studies used data from the early to mid-2010s (65, 68-70) with 2016 being the most recent year of data utilized in the literature (62, 66). Notably, none of these studies included data collected after 2016, the year during which the Centers for Disease Control and Prevention (CDC) published their influential opioid prescribing guidelines (71).

Populations/Data Sources

Models have been developed in different patient populations. Some studies utilized data sources that included information on both risk factors and outcomes and so, depending on the study question, may not require data linkage. These include data from specific populations such as the Veteran’s Health Administration (VHA) Corporate Data Warehouse (69), Medicare claims (65), data from commercial insurance plans (68), and data from an integrated regional health system (64). Other studies conducted person-level linkages of data from various sources and then used the linked dataset to develop their models. These studies used population-level data to develop models based on statewide populations, and linked PDMP, death certificate, and hospital discharge data (59, 60, 62, 63). One study also linked data from the criminal justice system (67). We did not identify any risk prediction models that were developed using Medicaid data.

Modelling Approaches

Most studies utilized traditional statistical approaches to create their multivariable model: standard logistic regression (59, 60, 62, 63, 66, 67, 69, 70) and Cox proportional hazards regression (64). Three studies examined alternative methods for building multivariable risk prediction models. Lo-Ciganic et al compared standard logistic regression, penalized logistic regression (specifically least absolute shrinkage and selection operator-type regression [LASSO]), random forest, gradient boosting machine, and deep neural network models. They found that the deep neural network model had the best performance (65). Sun et al compared elastic net regularization, a form of penalized logistic regression, and random forest models. They found comparable performance and selected the elastic net model as their final model (68). Dong et al compared random forest, penalized logistic regression (specifically a ridge regression), decision tree, and deep neural network models. They found that random forest and deep neural network models performed best when assessed by measures of recall and precision, terms commonly used in data science that are synonyms for sensitivity and positive predictive value, respectively (61).

Type of Overdose

Most models were developed to predict any opioid-related overdose without distinguishing the type of opioid involved or whether the overdose was fatal or intentional. Zedler et al (2015) and all associated external validation studies (66, 70) included any “serious opioid-induced respiratory depression” in their definition of opioid overdose when developing (69) and validating (66, 70) their model. Only one study we identified developed models specifically for overdoses involving illicit opioids. Ferris et al developed models to predict any heroin or fentanyl-related fatal overdose, without prescription opioid involvement, as well as fatal overdoses involving any opioid (62).

Predictors

Studies used a variety of different predictors to construct their models, though they all fell under the same broad categories mentioned above: patient demographics, mental health comorbidities, substance use disorders, physical health comorbidities, characteristics of opioids prescribed, and non-opioid medications prescribed. Studies that developed models using PDMP-based predictors did not include clinically derived predictors because PDMPs do not collect clinical data (59, 60, 62, 63). As is typical when constructing risk prediction models, studies first examined a large pool of candidate predictors and then created more parsimonious models by using different predictor selection approaches. One approach was to select predictors manually by first screening out predictors that did not meet a threshold value for significance in bivariate analysis (e.g., p-value < 0.25). Then, authors used a backwards selection approach by calculating pairwise comparisons of parameter estimates between the full model and the model with the candidate removed. If the candidate’s removal did not appreciably change parameter estimates (e.g., >20%) that predictor was excluded from the model (69, 70). A second approach was to use a similar procedure to screen candidate predictors through bivariate analysis, and then use a more automated backwards selection procedure. Pairwise comparisons of model fit statistics, such as the c-statistic or Akaike’s Information Criterion (AIC), when individual predictors were removed were used to determine predictor inclusion (60). A third approach, found in models that utilized complex machine learning techniques, was a completely automated procedure that involved compiling a large pool of candidate predictors informed by the literature (65) and/or bivariate associations (61) and then running algorithms to select the final model.

Validation

Although best practice guidelines recommend using external validation when developing risk prediction models (57), most studies we identified used internal validation methods. Four of these studies utilized a single random split of the data into development and validation cohorts (61, 62, 65, 68). Two others further split the data into another cohort to allow for testing of models in the development phase of the data to occur without utilizing validation data (65, 68). A series of studies from Maryland analyzing linked statewide PDMP data used bootstrapping validation with 300 iterations (59, 60, 67).

Two of the 12 risk prediction studies were externally validated (64, 69) and one, Geissert et al, was temporally validated (63). Temporal validation uses another year of data to evaluate model performance (57); it is considered superior to internal validation yet not as strong as external validation. Glanz et al employed data from a different health system in the same region to externally validate their model (64). The model developed by Zedler et al (2015), using VHA data (69), was externally validated in two later studies that independently applied and evaluated the model on commercially insured populations (66, 70).

Model Performance

All studies reported discrimination using the c-statistic; discrimination ranged from fair (c-statistic = 0.69) to excellent (c-statistic = 0.95). Models that included only PDMP-derived predictors tended to have lower discrimination (59, 60, 62, 63). The model with the lowest discrimination (c-statistic = 0.69) predicted overdose risk among patients prescribed buprenorphine for opioid use disorder (60). The model with the best discrimination was Geissert et al (c-statistic = 0.82) (63). Worse performance by PDMP-based models is likely due to their lack of clinical predictors, especially predictors related to substance use disorders.

Models that included clinical predictors typically displayed excellent discrimination. The model by Saloner et al, that utilized both clinical and criminal justice predictors, reported discrimination for nonfatal and fatal overdoses at 0.85 and 0.89, respectively (67). Zedler et al’s (2015) risk index was independently validated in two studies with c-statistics of 0.91 (70) and 0.88 (66). The model by Glanz et al, however, included clinical predictors but reported lower discrimination (c-statistic = 0.75) (64).

Models that utilized machine learning techniques also reported excellent discrimination. Lo-Ciganic et al utilized deep neural networks to develop a model with a c-statistic of 0.91. Sun et al developed models using standard logistic regression, penalized logistic regression, and random forest with high discrimination (c-statistics of 0.88, 0.89, and 0.86, respectively). Dong et al developed random forest and deep neural network models with very high discrimination (c-statistic = 0.95). These models were also constructed using clinical predictors.

Calibration was not consistently examined even though it is considered a necessary measure of prediction model performance (57). Of all the studies we reviewed, fewer than half assessed the degree to which model predictions were calibrated with observed outcomes (64, 66, 68-70). Sun et al examined mean predicted and observed probabilities in 29 strata: predicted probabilities were stratified into deciles with the tenth, and highest, decile further split into 20 additional percentile-based strata. Visual examination of the calibration plot showed that the model was well calibrated, though it slightly underestimated overdose risk in lower strata. Glanz et al categorized their cohort into tertiles of risk, and then plotted separate curves, one for each tertile, for mean cumulative observed and predicted risk against years of follow-up. When applied to the validation cohort, their model revealed miscalibration and substantially underestimated risk of overdose in the validation cohort for each tertile. Zedler et al (2015), and the two associated external validation studies (66, 70), assessed calibration by comparing mean predicted and observed risk for each stratum of risk (69). All three studies found good agreement between observed and predicted risk, indicating a well-calibrated risk index.

Clinical Utility

Studies commonly examined model diagnostics, specifically positive and negative predictive values, at predetermined thresholds. Almost across the board, models reported high negative predictive values and low positive predictive values (59, 60, 62-65, 67, 68). Unlike the c-statistic, positive and negative predictive values are influenced by the prevalence of the outcome. Overdoses are rare events, especially when the sample includes any person receiving a single opioid prescription. The prevalence of overdose outcomes in studies that included any person receiving an opioid prescription ranged from 0.05%-0.49% (59, 62, 63, 65, 67, 68) with one study of buprenorphine treated patients reporting an overdose prevalence of 3.2% (60). Dong et al developed models with very high positive predictive value in stark contrast to other studies. This is likely due in large part to Dong et al utilizing random selection to create a sample with a higher overdose prevalence (9.1%) (61).

Comparisons of model predictions to other risk prediction tools were minimal. Lo-Ciganic et al compared risk groups identified by their model, developed on a cohort of Medicare beneficiaries, to risk groups identified by the 2019 Center for Medicare and Medicaid Services (CMS) opioid safety measures. The high-risk group identified using the deep neural network model captured over 90% of all overdose events while the one from the CMS tool captured only 29% (65).

State of current research

Most of the published research on opioid overdose risk has involved the identification of individual risk factors. Only a handful of studies have developed multivariable prediction models assessing patients’ overall risk of opioid-related overdose. Models have been developed using a variety of data sources, though to our knowledge no models have been developed using Medicaid data. All the models we identified had good-to-excellent discrimination. However, many models did not assess calibration, and few have been externally validated. More studies are needed to advance, and improve upon, the models that have been already developed. External validation studies of published models would enhance their generalizability, particularly if performed in the Medicaid population which represents a large fraction of low-income Americans. Additionally, it is important to evaluate calibration when validating models. This enables researches to fully understand how a model performs in a certain population and, if necessary, how models can be adjusted to improve performance.

In recent years, rates of overdoses involving illicit opioids (particularly overdoses involving illicit fentanyl) have increased substantially, while rates of overdoses involving prescription opioids have stabilized and decreased in some areas (6, 7). These trends are likely to continue and may even accelerate due to economic and social disruptions related to the COVID-19 pandemic, which appears to have precipitated a spike in opioid-related overdoses (72, 73). The populations at risk for overdoses involving prescription opioids differ from those at risk for overdoses involving illicit opioids (23). Existing studies use data that predate these changes in overdose epidemiology and so future work will need to address the fact that most recent overdoses are due to illicit fentanyl and not prescription opioids. For example, future studies should explore the extent to which risk factors for overdoses involving prescription versus illicit opioids differ, and, if necessary, update existing prediction models to account for increases in overdoses involving illicit fentanyl. Reductions in overdoses involving prescription opioids likely reflect shifts in clinical practice away from using opioids to treat pain, greater awareness of opioid-related risks, and increased regulatory barriers to opioid prescribing implemented since the publication of the CDC’s opioid prescribing guidelines. The studies reviewed in this article utilized prescribing data from before the CDC guidelines were published. The influence of prescribing characteristics such as dose, duration, and overlapping prescription on overdose risk thus may be different when analyzing more recent years of data. Future research focused on updating existing models with newer years of data is needed to account for these changes and ensure that models continue to perform reliably.

Future studies that develop prediction models for overdose risk should ensure that they pay proper attention to model calibration. Calibration is an important property of any prediction model. It is possible for models to have high discrimination and yet still inaccurately predict outcomes due to poor calibration, and researchers are unable to identify miscalibration unless they specifically evaluate it. A poorly calibrated model can lead to misleading conclusions and inappropriate clinical decisions (74). For example, consider a model with good discrimination that is miscalibrated and so strongly overestimates the probability of an overdose. Such a model could so strongly overestimate a patient’s risk for overdose that clinicians using this model would falsely conclude that many patients are too high-risk to continue receiving opioids to treat pain, resulting in undertreatment of pain without any clinically significant reduction in overdose risk. In such a scenario, the patient may resort to non-medical use of opioids (e.g., diversion), self-medication with other substances, or even use of street drugs and therefore greatly increase their risk of an overdose. Thus, identifying and correcting model miscalibration is critical for avoiding unintended harms to patients.

External validation is an important aspect of clinical prediction modelling that increases confidence in the general applicability of a model. Models are strengthened when their generalizability extends beyond the populations in which they are developed, and their utility increases when they are applicable to multiple settings. For instance, a model developed using data from a single state’s PMDP may not translate well to another state. The populations of Kentucky and California, for example, are quite different from one another. Kentucky’s population has fewer racial and ethnic minorities, is older, and poorer than California’s population (75, 76). These two states are also different in terms of opioid prescribing and overdose; opioid prescribing rates in 2018 were twice as high (77) and opioid-related overdose deaths four times as high (78) in Kentucky than in California. Finally, policy differences between California and Kentucky may alter the effects of certain predictors. PDMP program features, such as clinical alerts and mandatory registration/use, may directly impact risk factors for opioid-related overdose. For example, laws requiring clinicians to register for their state’s PDMP are associated with reductions in certain overdose risk factors, such as prescribed daily dose (79), so risk prediction models that incorporate prescribed daily dose and were developed in states that require PDMP registration may not generalize to populations in states without this provision.

In the studies we identified, measures of clinical utility (e.g., sensitivity, specificity, positive predictive value) were reported at predetermined thresholds. Authors selected these thresholds based on sample characteristics (e.g., percentiles) or to maximize sensitivity and specificity. However, the preferred method for selecting thresholds is to use a decision-analytic approach that weighs the costs and benefits associated with rates of true/false positives and negatives (58). In practice it can be difficult to determine a clinical threshold. This is especially true in opioid prescribing as there is no clear consensus about the tradeoffs between preventing opioid-related harms and benefits related to pain management. Such considerations also are likely to vary across different populations of interest (e.g., previously opioid-naïve patients versus patients on buprenorphine for opioid use disorder). An alternative approach, decision curve analysis, can be used to assess clinical utility over a range of thresholds and therefore forego the need to decide an optimal threshold. Figure 1 shows an example of a decision curve analysis. Decision curve analysis involves determining the relative value of false-positive and negative results (i.e., “net benefit”) at specific thresholds. Net benefit is then plotted for the entire range of plausible thresholds with the added advantage of direct comparisons of competing prediction models (80). Decision curves provide a more informative assessment of a model’s clinical utility because they allow for comparisons across a range of thresholds.

Figure 1:

A hypothetical example of a decision curve analysis comparing various strategies for predicting overdose.

Net benefit is plotted for all thresholds, or cutoffs, for four different prediction strategies. The first two default strategies assume that either all (solid, red) or no (dash, red) patients will overdose. Model 1 (solid, black) and Model 2 (dash, black) are shown in comparison. In this scenario, Model 1 has the greatest net benefit, and therefore the greatest utility, up until a decision threshold of approximately 70%, and is preferred over Model 2 throughout.

Nearly all the models we identified reported low positive predictive values, which indicates that they have a high rate of false positives. In other words, the model will identify more patients predicted to experience an overdose than are actually observed in the data. This high rate of false positives limits model utility to clinical contexts or interventions where there would be negligent harm or cost to misclassifying low-risk patients as high risk. For example, a prediction model with a high rate of false positives would likely not be a problem if it was used for a program disseminating educational material, encouraging more follow-up visits, and prescribing naloxone to patients classified as having a high risk of overdose. On the other hand, high rates of false positives would be potentially harmful when identifying patients who should be tapered off opioids. The models we identified also reported high negative predictive values, which indicates that patients identified as low risk by a model are extremely unlikely to experience an opioid-related overdose. By extension, models with high negative predictive values can be useful for preventing use of resources on patients who would receive minimal benefit from them. Consider the previous example of a program identifying high-risk patients who would then receive educational material, close follow-up, and naloxone. If the model had high negative predictive values, then resources could safely be withheld from patients classified as low-risk to preserve program resources for high-risk patients that would actually benefit from them.

Implementation and translation to clinical and public health benefit

Findings from risk prediction models will not translate into public health benefits (e.g., reductions in overdose rates) unless models are used to develop risk prediction tools that can be implemented into clinical practice, made accessible to clinicians, and eventually evaluated to determine their effectiveness at reducing overdose outcomes. Among the studies reviewed for this article, only two presented a potentially usable clinical tool. Zedler et al (2015) used the results from the multivariable logistic regression model they developed to create an overdose risk score (69). They first identified predictors that met the following criteria: (a) showed a ‘statistical strength of association’ in the sample, (b) were supported in the literature, (c) were ‘likely’ to generalize to the greater population of patients receiving prescription opioids in the United States, and (d) were feasibly obtained from patients in a questionnaire. Predictors were then assigned a point-score by multiplying the regression coefficient by ten and rounding to the nearest integer. Each patient was assigned a score based on the point total for each predictor with a positive response. The validation studies that followed found that this VHA-developed risk tool was generalizable to commercially insured populations (66, 70). Lo-Ciganic et al stratified patients into low, medium, and high-risk categories based on their predicted probabilities (65). A cut-off for the categories was based on maximizing both sensitivity and specificity. Patients with predicted probabilities less than the cutoff were considered low risk. Those above the cut-off and below the 95^th percentile were considered medium risk, while those above the 95^th percentile were considered high risk.

These two studies utilized administrative health data sources containing all the information needed to identify outcomes and construct predictors for individual patients. Working with such robust datasets facilitates implementation because the features of the prediction model can be abstracted from a single database and implemented on the same platform. Indeed, the VHA, an integrated health system with a well-established integrated data warehouse (69) has already explored implementing a tool to identify patients at risk for serious adverse events, specifically drug overdose or suicide-related events, called the Stratification Tool for Opioid Risk Management (STORM). The STORM tool utilizes electronic health records extracted from national VHA data to estimate patient risk. The algorithm calculates a risk score that categorizes patient risk at low, medium, high, or very high levels. This data, which is updated nightly, is presented through a dashboard that informs clinicians of the score, risk level, relevant risk factors, and upcoming appointments for each patient (81).

For models developed on linked data, such as those based on PDMP data, the task of implementation can be more difficult. State-specific bureaucratic factors may make it difficult to both link data necessary to develop prediction models and to implement and update tools using these models into clinical practice. For example, our work developing opioid-related risk models in California has required coordination with and linking data from three different state agencies: California PDMP records are maintained by the California Department of Justice, death certificates documenting fatal overdoses are maintained by the Department of Public Health, and data on emergency department visits and hospital admissions involving nonfatal overdoses are maintained by the Office of Statewide Health Planning and Development. Each of these departments has separate procedures for obtaining data for research purposes and obtaining agreement to link data from these three departments required support from leadership in all 3 departments. Some states have responded to the opioid crisis by making organizational or statutory changes to facilitate creating linked datasets to assess overdose risk. For example, in 2015, Massachusetts passed legislation (“Chapter 55”) mandating that statewide databases related to opioid overdose be linked. The state-mandated this linked dataset specifically to overcome administrative barriers to developing evidence-based approaches for addressing the public health problem of opioid overdoses in Massachusetts (82).

Successful implementation of overdose prediction tools will require additional research to ensure that these tools do not have unintended and potentially harmful consequences for the patient population they were designed to help. Many of the published models developed to predict risk of overdose have high rates of false positives. A legitimate concern is that clinical prediction tools based on these models may, in practice, discourage clinicians from properly managing their patients’ pain by leading them to over-estimate patients’ overdose risk and thus, avoid prescribing opioids to patients at low risk of overdose who may derive clinical benefit from opioid pain medications. Additionally, clinicians may learn to simply ignore warnings if prediction tools identify too many patients as high risk (83). Follow-up studies, likely with input from patients and other stakeholders, will be needed to address these concerns and guide implementation to ensure these unintended consequences are minimized. The STORM tool implemented in the VHA system is currently the subject of an extensive evaluation aimed at analyzing the effectiveness of the tool itself (84) as well as identifying the most successful implementation strategies within the VHA system (85). The results of these studies may lay the groundwork for how models in other health systems can be implemented successfully.

Conclusion

In summary, models with good discrimination have been developed to predict opioid-related overdose in a variety of settings, though few studies assessed model calibration and even fewer were externally validated. Models that included clinical predictors (e.g., mental and physical health comorbidities) performed better than those that did not. Models that utilized machine learning techniques performed better than those that utilized standard logistic regression. An important gap in this field is the paucity of externally validated and calibrated risk prediction models. Most of extant models would benefit from external validation studies to improve their generalizability. Future studies that develop or validate prediction models for opioid-related overdose should ensure that model calibration is evaluated and reported. Another important gap is the lack of prediction models derived from, or validated using, Medicaid data. A novel model, or validation of an existing model, using Medicaid data is needed to identify models that have good discrimination and calibration for low income populations. Future studies should also consider using decision curve analysis to compare the clinical utility of competing models across the entire range of decision thresholds. Models reported high rates of false positives, and so careful attention to avoiding potential unintended consequences is needed when developing and implementing clinical prediction tools based on these models. Ongoing implementation of similar prediction tools, such as the VHA’s STORM, may provide a framework for similarly configured health systems to implement their own prediction tools. Finally, evaluation studies of implemented prediction tools will be needed to determine effectiveness in reducing overdose outcomes.

Abbreviations:

AIC

Akaike’s Information Criterion

CDC

Centers for Disease Control and Prevention

CMS

Center for Medicare and Medicaid Services

ICD-9

International Classification of Diseases, 9^th Edition

ICD-10

International Classification of Diseases, 10^th Edition

LASSO

Least Absolute Shrinkage and Selection Operator

PDMP

Prescription Drug Monitoring Program

STORM

Stratification Tool for Opioid Risk Management

VHA

Veterans’ Health Administration