Using contributing causes of death improves prediction of opioid involvement in unclassified drug overdoses in U.S. death records

Abstract

Background and Aims.

A substantial share of fatal drug overdoses is missing information on specific drug involvement, leading to underreporting of opioid related death rates and a misrepresentation of the extent of the opioid epidemic. We aimed to compare methodological approaches to predicting opioid involvement in unclassified drug overdoses in United States death records and to estimate the number of fatal opioid overdoses from 1999 to 2016 using the best performing method.

Design.

This was a secondary data analysis of the universe of drug overdoses in 1999–2016 obtained from the National Center for Health Statistics Detailed Multiple Cause of Death records.

Setting.

United States.

Cases.

A total of 632,331 drug overdose decedents. Drug overdoses with known drug classification comprised 78.2% of the cases (N=494,316) and unclassified drug overdoses (ICD-10 T50.9) comprised 21.8% (N=138,015).

Measurements.

Known opioid involvement was defined using ICD-10 codes T40.0–40.4 and T40.6, recorded in the set of contributing causes. Opioid involvement in unclassified drug overdoses was predicted using multiple methodological approaches: logistic regression and machine learning techniques, inclusion/exclusion of contributing causes of death, and inclusion/exclusion of county-level characteristics. Having selected the model with the highest predictive ability, we calculated corrected estimates of opioid related mortality.

Findings.

Logistic regression and random forest models perform similarly. Including contributing causes substantially improves predictive accuracy, while including county characteristics does not. Using superior prediction model, we found that 71.8% of unclassified drug overdoses in 1999–2016 involved opioids, translating into 99,160 additional opioid related deaths, or approximately 28% more than reported. Importantly, there is a striking geographic variation in undercounting of opioid overdoses.

Conclusions.

When aiming to correct opioid death counts in the United States, future reports and studies should include contributing causes of death as predictors; with respect to statistical modeling, logistic regression and random forests are equally effective in prediction.

Introduction

According to the Centers for Disease Control and Prevention (CDC), 47,600 people died from opioid overdoses in 2017 (1). Yet, fatal opioid overdoses are likely underreported. Undercounting occurs because the drug involved in an overdose is not always specified on death certificates. This is not a trivial issue; “other and unspecified drugs” were implicated in 21–25% of drug overdoses in 1999–2013 and 15–19% in 2014–2016 (2). The drivers of this phenomenon are not fully understood but are potentially related to inadequate training for coroners without medical experience and variation in substances included in toxicological testing across jurisdictions (3).

There are several studies that have attempted to predict opioid involvement in unclassified drug overdoses; however, questions remain. Buchanich and colleagues inferred opioid involvement by projecting proportions of specific drug involvement from cases where the drug was known into overdoses where it was not for 1999–2015 (4). This extrapolation assumes that the frequency of certain drug involvement is similar in classified versus unclassified drug overdoses. However, it is possible that different drugs are differentially reported on death certificates, perhaps due to difficulty of assignment, cost of test, or stigma associated with opioid use. In a more robust approach, Ruhm provided adjusted proportions of opioid involvement (opioid analgesics, heroin) in 1999 and 2012 (5) and in 2014 (6). In the most recent study providing corrected opioid overdose rates, Ruhm estimated year-specific logistic regression models of opioid involvement as functions of decedent characteristics and county socioeconomic characteristics and used these models to predict opioid involvement in unclassified drug overdoses for 1999–2015 (7).

This study aimed to 1) offer methodological comparisons and guidance for corrections used in empirical research, and 2) based on the superior methodology, provide more accurate estimates of fatal opioid overdoses in 1999–2016. With respect to the former, we compared predictive accuracy of various methodological approaches, including logistic regression and a machine learning algorithm, inclusion and exclusion of county-level socioeconomic characteristics, and inclusion and exclusion of contributing causes of drug poisoning deaths. Given the growing popularity and often superior performance of machine learning methods in prediction studies, we hypothesized that machine learning would outperform more traditional statistical methods, such as logistic regression. Based on previous research which showed little evidence of county-level socioeconomic characteristics playing a significant role in predicting unclassified drug overdoses (8), we hypothesized that incorporating county-level socioeconomic data would not offer a substantial improvement. With death records including multiple contributing causes of death, in addition to the underlying cause of death, we hypothesized that the inclusion of contributing causes would improve predictive accuracy.

Methods

Design and Study Population

This was a secondary data analysis of the universe of drug overdoses in the United States in 1999–2016. Our study population is all drug overdose decedents in this time period (N=632,331). Drug overdose data were obtained from the Multiple Cause of Death (MCOD) Research Files from the National Center for Health Statistics (9). We followed the CDC definition of drug overdoses, using ICD-10 underlying cause of death codes: X40-X44 (accidental drug poisoning), X60-X64 (intentional self-poisoning), X85 (assault by drugs), and Y10-Y14 (drug poisoning of undetermined intent).

Measures

Our main outcomes of interest are opioid involvement in drug overdoses as well as opioid-related and non-opioid related overdoses. Opioid involvement was identified using ICD-10 codes recorded in the set of contributing causes (record axis): opium (T40.0), heroin (T40.1), other opioids (T40.2), methadone (T40.3), other synthetic narcotics (T40.4), and other and unspecified opioids (T40.6). Non-opioid related overdoses are defined as overdoses involving other drugs (T36–39, T40.5, T40.7–9, T41-T50.8) but not opioids. While opioid related overdoses may also involve non-opioid drugs, the distinction between opioid related overdoses and non-opioid related overdoses is in opioid drug involvement.

We use three sets of predictors: decedent characteristics, contributing causes of death, and county-level characteristics. Decedent characteristics are available in the MCOD data and include age; race and Hispanic ethnicity; sex; education attainment (less than high school, high school degree, some college, and a bachelor’s degree or greater); marital status; day of the week and month of death; and place of death (inpatient, outpatient, dead on arrival at hospital, home, hospice facility, nursing home, other location, and unknown). The database also includes county of residence and county of death occurrence.

The MCOD data also contain up to twenty contributing causes of death. We identified causes that contributed to at least 500 drug overdoses in 1999–2016, for a total of 122 contributing causes (Table S1). We generated indicator variables equal to 1 if a contributing cause was reported in each death record.

County-level socioeconomic characteristics include circa-2010 poverty rates and median household income (Small Area Income and Poverty Estimates); shares of population 25 years and older in four education levels, including less than high school, high school, some college, and bachelor’s degree or greater (U.S. Census Bureau); shares of households headed by females (Area Health Resources File (AHRF)); population density and its square polynomial (Survey of Epidemiology and End Results); and the number of physicians per 1,000 people (AHRF).

Statistical Analysis

To achieve our first aim, we compared methodological approaches to predicting opioid involvement in classified drug overdoses. We tested our hypotheses by estimating total predictive accuracy of several models and making qualitative comparisons of temporal trends in predictive accuracy between these models. Our main reference model is a baseline logistic regression model, which only includes decedent information (Model 0). To test our first hypothesis (improvements from using machine learning), we estimated a random forest ensemble, using the same predictors, (Model 1) and compared its predictive accuracy to that of Model 0. To test our second hypothesis (improvements from using county-level characteristics), we estimated the same model as in Ruhm’s approach (7), i.e., logistic regression with decedent and county of residence socioeconomic characteristics (Model 2), and compared its predictive accuracy to that of Model 0. To test our third hypothesis (improvements from using contributing causes of death), we estimated the baseline logistic regression model adding contributing cause indicators (Model 3) and compared its predictive accuracy to Model 0. Additionally, we estimated a random forest ensemble which includes decedent information and contributing causes (Model 4). Compared to Model 3, Model 4 serves as another test of the first hypothesis. Compared to Model 1, Model 4 serves as another test of the third hypothesis. Figure S1 shows the models and hypotheses schematically. For an additional source of reference, we estimated a naïve accuracy rate representing what would be achieved if each overdose was classified as the majority classification (i.e., opioid).

The five models being compared are essentially combinations of two estimation methods and three sets of predictors. The two estimation methods compared are logistic regression and random forests. With each, we estimated separate models for each year of death, which allows the relationships between explanatory variables and opioid involvement to change over time. For a description of random forests, see Supporting Information in the Appendix. The three sets of predictors were decedent characteristics, county-level characteristics, and contributing causes, as described in Measures above. For a further examination of contributing causes that predict opioid involvement in classified drug overdoses, see Supporting Information in the Appendix.

To assess each model’s predictive accuracy, we separated each annual database into a training set (80%) and a test set (20%). The test set is not used for model fitting and thus acts as an independent source for assessing predictive accuracy. For both estimation approaches, drug overdoses with estimated probability of opioid involvement greater than or equal to 0.50 are considered an opioid overdose; those with estimated probability less than 0.50 are considered a non-opioid overdose (10). We estimated models on the training set and tested their predictive accuracy on the test set. Our main accuracy measure is total predictive accuracy, as follows:

$$\mathit{\text{Accuracy}}=\frac{\mathit{\text{True Positives }}\left(TP\right)+\mathit{\text{True Negatives }}\left(TN\right)}{\mathit{\text{Total Negatives}}+\mathit{\text{Total Positives}}+\mathit{\text{False Negatives }}\left(FN\right)+\mathit{\text{False Positives }}\left(FP\right)}$$

Total predictive accuracy is the rate at which the model yields a correct assessment of whether the drug overdose involved an opioid or not.

As an additional accuracy measure, we estimated receiver operating characteristic (ROC) curves. These show the model’s sensitivity or true positive rate (i.e., TP / (TP + FN)) versus the 1 – specificity or false positive rate (i.e., FP / (TN + FP)) across multiple classification thresholds, not just 0.50. Models with high sensitivity at given false positive rates have higher accuracy (10). Finally, we used F₁ score as another accuracy measure. This performance metric balances the model’s precision (i.e. TP / (TP + FP)) with its sensitivity (11).

In these analyses, we used data from drug overdoses with at least one specified drug, i.e., classified drug overdoses (N=494,316, 78.2% of total).

To achieve our second aim, we used the best performing model (from Models 0–4) in terms of total predictive accuracy to predict opioid involvement in unclassified drug overdoses. We calculated the number of total opioid-related drug overdoses in each year and in total. This was done by summing predicted probabilities of opioid involvement across all unclassified drug overdoses, following Ruhm (7). The number of predicted opioid overdoses was then divided by the total number of unclassified drug overdoses to obtain the proportion of estimated opioid involvement. Corrections are presented graphically over time and mapped to U.S. counties in 2016.

In these analyses, we used data from drug overdoses with no drug specified (ICD-10 T50.9), i.e., unclassified drug overdoses (N=138,015, 21.8% of total). All statistical analyses were conducted using R and Stata.

Results

Descriptive Statistics

Figure 1 shows trends in the number of drug overdoses by category of overdose: 1) opioid overdoses; 2) non-opioid overdoses; and 3) unclassified overdoses. The number of opioid overdoses increased dramatically from 8,050 to 42,249 between 1999 and 2016. Non-opioid overdoses also increased, though at a lower rate (5,233 to 12,117). In 1999–2016, 21.8% overdoses were unclassified, but this proportion declined from a high of 25.4% in 2008 to 14.6% in 2016 (Appendix Figure S2). Despite these improvements, the annual number of unclassified drug overdoses plateaued in 2008 at around 10,000. Table 1 reports summary statistics of decedent characteristics by category of overdose. Decedents in the three overdose groups are similar for most characteristics.

Figure 1:

Trends in drug overdoses from 1999 to 2016 by drug overdose category

Notes: This figure shows drug overdose counts from 1999 to 2016 based on the following category: 1) opioid-related overdoses; 2) non-opioid related overdoses; and 3) overdoses without a specific drug classification. Opioid overdoses may have also been partially caused by non-opioid drugs, though non-opioid caused overdoses are not caused by opioid drugs. These are estimates based on National Center for Health Statistics’ Multiple Cause of Death Data. The smoothed lines are estimated using local regression and are included to highlight trends in the data over time.

Table 1:

Decedent characteristics by drug overdose category

Variable	Non-Opioid	Opioid	Unclassified
	N = 142,686	N = 351,630	N = 138,015
Demographic Characteristics
Female, %	39.7	34.2	43.5
Age: 0–20, %	2.2	2.6	2.0
Age: 20–30, %	10.3	19.0	15.6
Age: 30–40, %	18.7	23.7	22.3
Age: 40–50, %	29.0	27.8	30.0
Age: 50–60, %	24.0	20.3	22.1
Age: 60–70, %	8.7	5.3	5.9
Age: 70–80, %	3.6	0.9	1.4
Age: 80+, %	3.4	0.4	0.7
Race: White, %	79.4	89.6	91.3
Race: Black, %	17.3	8.6	7.2
Race: American Indian, %	1.3	1.1	0.9
Race: Asian, %	2.0	0.6	0.6
Hispanic, %	8.5	7.9	5.6
Married, %	27.0	25.1	29.1
Education: Less Than High School, %	19.5	20.5	20.3
Education: High School, %	42.0	46.3	45.0
Education: Bachelor’s Degree or More, %	12.5	8.0	9.4
Education: Some College, %	20.5	21.1	21.4
Death Characteristics
Alcohol Involvement, %	11.6	13.3	4.2
Ethanol Involvement, %	7.9	8.9	1.6
Place of Death: Dead-on-Arrival at Hospital, %	2.4	3.0	3.4
Place of Death: Hospital Inpatient, %	19.1	7.3	10.6
Place of Death: Hospital Outpatient, %	17.1	15.0	12.6
Place of Death: Nursing Home, %	0.5	0.3	0.4
Place of Death: Residence, %	43.4	53.8	55.1
Occurred on Weekend, %	29.8	32.2	31.7

Notes: In this table, we display distribution (%) of decedent characteristics available from the death records and used as control variables in methodological comparisons.

Methodological Comparisons

Figure 2 displays predictive accuracy of alternative models used to predict opioid involvement in classified drug overdoses in each year, providing a test of our hypotheses. Our first hypothesis was that machine learning techniques would outperform logistic regression. However, logistic regression and random forests have very similar levels of predictive accuracy (Models 0 and 1; Models 3 and 4). This result is in agreement with the findings of a recent review which does not find evidence of superior performance of machine learning methods over logistic regression in clinical prediction models (12). Our second hypothesis stated that incorporating county-level characteristics would not offer a substantial improvement in predictive ability. Indeed, we found that the logistic model with decedent characteristics and county-level variables (Model 2, or Ruhm’s approach (7)) is similar to the logistic model with just decedent characteristics (Model 0). Finally, we hypothesized that incorporating contributing causes of death would improve predictive accuracy. In support of this hypothesis, models with contributing causes had a substantially higher predictive accuracy compared to those without, regardless of whether logistic regression or random forests were used (Models 0 and 3, Models 1 and 4). Of note, there is not a single year where the approach used in Ruhm (7) (Model 2) outperforms the predictive accuracy of either logistic regression or random forest model with contributing causes.

Figure 2:

Opioid involvement prediction accuracy rates from 1999 to 2016: methodological comparisons

Notes: This figure displays total predictive accuracy of year-level models of opioid involvement in drug overdoses with known drug classifications. In each year, we partitioned our data into training (80%) and test (20%) sets. We estimated the models on our training set and performed accuracy estimates using our test set. The “Naïve Estimate” corresponds to the predictive accuracy one would achieve if one predicted that every overdose was in the highest-frequency classification. In this case, this corresponds to predicting that every overdose was opioid-involved. Our reference model M0 is a logistic regression model which includes decedent information (Table 1) but does not include any contributing causes of death. The M1 model is a random forest ensemble with the same predictors as those in the M0 model. The M2 model is a logistic regression model including decedent information and county-level socieconomic characteristics from Ruhm (7). The M3 model is a logistic regression model with both decedent information and contributing cause indicators. Lastly, the M4 model is a random forest ensemble with both decedent information and contributing cause indicators. The smoothed lines are estimated using local regression and are included to highlight trends in the data over time.

These findings are confirmed when ROC curves and the F₁ score are used as alternative accuracy measures. Figures S4–21 show ROC curves for Models 0, 2, and 3 estimated using thresholds from prediction thresholds in 0.05 increments from 0.05 to 0.95 on an annual basis. Outside of 1999, all ROC curves show that Model 3 has higher true positive rates relative to false positive rates. Only in 1999 are the ROC curves for Model 3 and Model 2 similar. Figure S22 shows the F₁ score for Models 0, 2, and 3 from 1999 to 2016. Model 3 outperforms Model 2 and Model 0 for all years in our study period.

Estimates of Opioid Involvement in Unclassified Drug Overdoses

Appendix Figure S23 shows estimated percentages of opioid involvement in unclassified drug overdoses alongside the known percentage of opioid involvement in classified drug overdoses. Figure 4 displays estimated opioid-involved overdose counts calculated, as well as the reported opioid overdoses and total drug overdoses. The estimates of opioid overdoses are based on the logistic regression model using decedent characteristics and contributing causes (Model 3), the superior method identified in our study. Our logistic regression and random forest models have similar predictive accuracy in predicting opioid involvement across time, but logistic regression is simpler and more commonly used. We estimate that 99,160 unclassified drug overdoses in 1999–2016 involved an opioid (71.8% of total).

Figure 4:

Trends in opioid-involved drug overdoses from 1999 to 2016 with corrected estimates using the superior methodology

Notes: OD = overdose. This figure displays the number of total overdoses from 1999 to 2016 alongside estimates of opioid involvement projected using superior methodology identified in the study, i.e., a logistic regression with decedent characteristics and contributing causes of drug overdose deaths as predictors. The smoothed lines are estimated using local regression and are included to highlight trends in the data over time.

Figure 3 maps county-level differences between the uncorrected opioid overdose rate and the opioid overdose rate corrected using the superior method. We found that the absolute increase in opioid overdose rates with this correction is dramatic in certain U.S. states, including the Appalachian states of Kentucky, Pennsylvania, and Tennessee; and Florida and Louisiana in the Southeastern region, which is consistent with prior state-level research on corrected opioid mortality rates (13). Our findings further reveal that there is also considerable variation at the county level. Some counties have substantially higher corrected opioid mortality rates, compared to their neighbors, even in states that overall do not seem to have dramatically different corrected mortality. These county clusters correlate with the high rates of unclassified drug overdoses reported in prior research (8).

Figure 3:

Differences in opioid overdose rates between corrected rates using the superior methodology and uncorrected rates in 2016

Notes: This figure displays county-level increases in opioid overdose rates predicted using contributing causes of drug overdose death, relative to uncorrected estimates for 2016. We used the superior methodology identified in our study – a logistic regression model with decedent characteristics and contributing causes of death as predictors.

To directly compare our estimates with those from Ruhm (7), we calculated the number of predicted opioid overdoses from 1999–2015. We estimated that 71.5% (92,027) of unclassified overdoses involved opioids, which is slightly lower than Ruhm’s estimate over that same time period (73.2%, 94,256) (7).

Discussion

In this study, we aimed to 1) offer methodological comparisons and guidance for corrections of fatal opioid overdoses, and 2) based on the superior methodology, provide more accurate estimates of these overdoses in 1999–2016. Having compared several methodological approaches to predicting opioid involvement in unclassified drug overdoses, we found that 1) against our expectations, the random forest machine learning approach did not outperform logistic regression; 2) as expected, incorporating county-level characteristics did not improve predictive accuracy; and 3) as expected, including contributing causes of death substantially improved predictive accuracy. Using the best performing, yet simplest, prediction model – a logistic regression model with decedent characteristics and contributing causes of death, we found that 71.8% of unclassified drug overdoses in 1999–2016 involved opioids, translating into 99,160 additional opioid-related deaths, or approximately 28% more than reported. Our estimate is similar to that of Ruhm (7) and is lower than the estimate from a toxicological study done in Marion County, Indiana (14), which found that approximately 87% of unclassified drug overdoses involved an opioid drug.

We provide an important methodological contribution to the future study of the opioid epidemic. We recommend that researchers use contributing causes of death, in addition to decedent characteristics, to calculate corrected estimates of opioid overdoses in future research. Contributing causes are included in the Detailed MCOD Research Files and do not require additional data. With respect to the choice of a statistical method, logistic regression and random forest ensemble perform similarly and either one can be used, although logistic regression is simpler and familiar to most researchers. Lastly, the addition of county-level demographic and socioeconomic characteristics used does not offer an improvement in predictive accuracy. In fact, we caution that use of these characteristics in correcting opioid overdose counts can be problematic in some studies. If relationships between opioid overdoses and community characteristics are examined using opioid overdose counts corrected using county-level characteristics (highly correlated to community characteristics), then estimates of these relationships may be biased.

The identified superior approach to statistical modeling of opioid involvement in unclassified drug overdoses is an improvement on existing approaches (4–7). It relies neither on extrapolations of proportions of drug overdoses with known opioid involvement (4), nor on information from outside sources (7). Importantly, our findings indicate the predictive accuracy of each model, which is missing from previous studies. If methodological improvements to predicting opioid involvement are tested in future studies, they can be directly compared to our models.

Our corrected estimates are not trivial and show that the human toll of the opioid epidemic has been substantially higher than reported, by several thousand lives taken each year. Given the geographic variation in the prevalence of unclassified drug overdoses (3,8) and the county-level variation in the rate of additional opioid deaths that we report, several states and county clusters across the nation have particularly suffered from underreported opioid mortality and likely experienced a much higher human toll than currently understood. In 2018, the National Science and Technology Council’s Fast-Track Action Committee on Health Science and Technology Response to the Opioid Crisis (Opioid FTAC) made a recommendation to evaluate opioid mortality more accurately in order to improve responses to the epidemic (15). Corrected estimates at county and state levels can help public health officials, policy makers, and clinicians better understand the extent of local opioid epidemics and respond accordingly. Importantly, most states where unclassified drug overdoses are common and corrections are thus dramatic have decentralized coroner or hybrid (coroners and medical examiners) systems of death investigation. While future research is needed to further investigate this potential relationship, there may be important implications for the state and county medicolegal death investigation systems. Our findings also suggest that the economic burden of opioid overdoses is currently underestimated as well, which is in line with the conclusion of a report done by the Council of Economic Advisors (16).

Limitations

One limitation of our approach is that our empirical methodology implies a unidirectional relationship from contributing causes to opioid involvement in drug overdoses. Yet, the relationship between opioid involvement and other contributing causes of death may be more complicated. Another potential limitation is that we did not utilize highly complex machine learning techniques, but used a rather straightforward random forest approach. We made this choice because our goal in comparing different methods was to provide methodological guidance for future research and we thus wanted to avoid using highly complex models that would not be easily implemented by most researchers. We refrain from asserting that no machine learning methods perform better than logistic regression. Further, the set of county characteristics is far from being a comprehensive list of all county factors that could be at play. However, these are the county characteristics used in prior corrections of opioid overdose rates (7), and our analysis provides an assessment of improvements in predictive accuracy from using these characteristics. We also believe that use of county characteristics in correcting opioid overdose counts can be problematic, as described above.

Another limitation is that our estimates of opioid involvement in unclassified drug overdoses do not detect all variation in the local changes in the effectiveness of drug identification in overdoses over time. Such local changes may involve increasing expertise in opioid involvement detection among local medicolegal investigators, advancements and spread of additional toxicological tests, and changing political and societal pressures to either confirm or deny an opioid overdose, among others. However, this limitation stems from the lack of data on such changes and unavailability of more granular decedent location in the mortality data, and is not unique to our study. When our estimates are applied to describe the extent of local opioid epidemics, such extrapolations should be made with this understanding in mind. Yet another limitation is that our new estimates of the scale of the opioid epidemic may still be underestimates since we do not predict opioid involvement in deaths where no drug overdose was recorded. Lastly, our fundamental assumption is that statistical models of opioid involvement estimated in drug overdoses with classified drug involvement are relevant for drug overdoses without a drug classification. Yet, there may be unobserved factors that limit the comparability between drug overdoses with versus without a drug classification. In this case, relationships between opioid involvement and explanatory variables may not be consistent between the two groups. However, this seems to be the only viable approach to inferring opioid involvement using observational data. Other studies (4–7) also implicitly rely on the comparability between these two groups.

Conclusion

In modeling opioid involvement in unclassified drug overdoses, highest predictive accuracy was achieved using a statistical model – either logistic regression or a random forest ensemble – with decedent characteristics and contributing causes of death as predictors. This approach should be used in future empirical studies that involve opioid mortality data. Using logistic regression, we estimated that 71.8% of unclassified drug overdoses in 1999–2016 involved opioids, bringing the estimated human toll of the opioid epidemic to approximately 28% more lives than reported. The corrected estimates and the geographic variation in the additional opioid mortality rates should aid decision-makers in making more accurate assessments of the national and local opioid epidemics and improving responses to the crisis.