Customizing and fine-tuning opioid overdose machine learning models for community-specific dashboards

Abstract

Drug overdose is a leading cause of death in the United States. Targeted community interventions and allocation of resources for reducing overdose deaths require timely and actionable data to make a meaningful public health impact. We present our work on tuning geographic-specific machine learning forecast models to predict the number of opioid-involved overdoses involving Emergency Medical Services (EMS) in Kentucky, with a focus on optimizing covariates representing measurable opioid overdose risk and protective factors. We aggregated suspected opioid-involved overdoses in EMS data into monthly totals by county between January 2017 and September 2025. We further grouped these counties into larger regions, representing Area Development Districts (ADDs), service areas for Community Mental Health Centers (CMHCs), and local health departments (LHDs). We constructed long-term time-series forecasting models (NLinear LTSF) to predict opioid-involved overdoses three months into the future. We tested the impact of using eight covariate data sources: Kentucky State Police drug seizures, Medicaid claims, community naloxone distribution, Kentucky Department of Corrections intakes and releases, social determinants of health data, location of treatment venues, and weather. We evaluated our models using both an unseen 10% test set and an expanding window strategy, where we simulated predicting historical data and retrained the model using data available for every three-month window. We embedded predictions into community-specific dashboards where end-users can select individual counties, or larger regions covered by distinct ADDs, CMHCs, or LHDs. The dashboards are integrated in Kentucky’s Rapid Actionable Data for Opioid Response system designed to deliver ongoing near-real time analytics and forecasting at state and local levels.

Introduction

The opioid crisis is an ongoing national public health emergency (1). Most drug overdose deaths in the United States involve opioids (2,3). Kentucky ranked seventh in drug overdose deaths for 2023 with an age-adjusted rate of 50.1 deaths per 100,000 people (4). Overdose incidents vary across time and location; for example, sharp increases in opioid-involved overdose were observed during the COVID-19 pandemic, although these increases were not uniform across the United States (5,6). Differences are also observable within individual states; drug overdose mortality rates range from 0 to more than 100 per 100,000 people among counties in Kentucky (7). Substantial declines in overdose mortality were observable in Kentucky post-COVID, but these declines were not uniform across all populations (8). Similarly, estimates for prevalence of opioid use disorder also vary substantially by county, ranging from 1.3% to 17.7% (9). Accessibility to Food and Drug Administration (FDA)- approved medication treatment for opioid use disorder (MOUD) (i.e., methadone and buprenorphine), which reduce the risk of opioid overdose mortality and morbidity, also vary geographically with notable differences between rural and urban locations (10–14). In 2020, only 45% of counties in the United States had a facility offering MOUD (15). In response to these temporal and spatial dynamic shifts in public health challenges, Rapid Actionable Data for Opioid Response in Kentucky (RADOR-KY) was created to shorten the reporting time between data collection and the sharing of actionable insights (16). Our state and affiliated partners have historically led advances in opioid overdose surveillance (17–19); however, traditional surveillance focuses on reporting sudden spikes and dips in opioid overdose retrospectively and lacked comprehensive data sources and programming needed for predicting anticipated overdoses. RADOR-KY augments traditional surveillance efforts by adding a host of new data inputs and sophisticated modeling programs to forecast the severity of overdose in the upcoming months allowing for public health interventions aimed at preventing morbidity and mortality from these anticipated overdoses.

Actionable, data-driven insights are critical for effective and efficient community planning and resource allocation, including distribution of naloxone, the highly effective opioid overdose reversal medication, and assistance with linking to MOUD and harm reduction services (20). Dashboards play a critical role in delivering key information needed to understand and respond to the opioid overdose crisis (21). On our publicly available dashboards, RADOR-KY visualizes historical trends and uses machine learning to generate predictions of suspected opioid-related overdoses for key stakeholders such as policymakers, administrators of government programs, and leaders of public health departments. The predictions rely upon RADOR-KY’s timely, large-scale data sets, automated data processing, and machine learning models that are sensitive to both time and location. A wide variety of big data studies have explored substance use disorders (22–24); machine learning has been used to predict prolonged opioid exposure (25–27), opioid use disorder (28,29), and opioid overdose risk (30). We previously demonstrated the benefit of machine learning models over simpler regression models due to the complexity of overdose patterns (16). Machine learning models for other domains require customization and fine-tuning to support geographic variation (31,32).

We developed RADOR-KY to be sensitive to changes in frequency of suspected opioid-involved overdoses across time and location, where forecasting models can be customized and fine-tuned to work in areas with either frequent or infrequent overdose activity. In this manuscript, we present our methodology for customizing machine learning models for specific geographic regions by evaluating and selecting appropriate covariates for that type of geographic region. We also describe how we visualize these models as interactive dashboarding tools for our community stakeholders which are currently available on our website (33).

Methods

We developed and evaluated prediction models for the number of suspected opioid-involved overdose events treated by Emergency Medical Services (EMS) using data collected by the Kentucky Board of Emergency Medical Services (KBEMS) between January 2017 through September 2025. Our models are designed to predict the total number of events per month for the upcoming three months for our EMS dashboard. We developed custom models by measuring the influence of covariates representing known risk and protective factors for opioid overdose and evidence-based practices for opioid overdose prevention and control (20,34). The covariates were sourced from a wide range of data sources (Table 1) with the goal of selecting the best performing measures for each geographic region; implementation details for each covariate are described in Supplemental Table 1.

Table 1.

Data sources for predicting suspected opioid overdoses in EMS data

Variable	Data Source	Date Range	Measurement of variable
Overdoses (Base outcome variable)	Kentucky Board of Emergency Medical Services (KBEMS)	January 2017-September 2025	Monthly totals for weekly EMS run data of opioid-involved overdoses from the Kentucky State Ambulance Reporting System
Drug Seizures	Kentucky State Police	January 2021-August 2025	Monthly drug seizures across Kentucky, detailing the substances seized, timelines, and agencies involved
Medicaid Claims	Office of Data Analytics, Kentucky Cabinet for Health and Family Services	January 2017-August 2025	Monthly total of Medicaid beneficiaries with OUD with claims for medication for opioid use disorder (MOUD) or claims for behavioral health treatment (case management and peer support)
Naloxone Distribution	Kentucky Pharmacists Association	January 2020 - May 2024	Monthly count of two-dose naloxone units distributed to counties
Kentucky Department of Corrections (KDOC) Intakes and Releases	Kentucky Department of Corrections	June 2020 -May 2025	Monthly total of individuals newly under supervision by KDOC probation and parole and individuals released from custody by county, categorized by their substance use risk level over the past 24 months
Social Determinants of Health (SDOH)	Center for Disease Control/Agency for Toxic Substances and Disease Registry (CDC/ATSDR)	2020	Yearly CDC/ATSDR Social Vulnerability Index (SVI) identifies and measures at-risk communities
Treatment Venues	Find Help Now Kentucky	2024	Yearly locations of centers and resources related to treatment of opioid use disorder
Weather	Open-Meteo	1940 -Present	Monthly historical weather data

Our data sources are aligned with facets of information related to overdose exposure. We partnered with our state law enforcement administration to receive drug seizure data, which is associated with overdose mortality (35). We leveraged Medicaid claims for MOUD by repurposing prior specifications developed by our prior research project and expanding to all counties in Kentucky (36); higher rates of overdose mortality are linked to regions with lower capacity to connect patients to mortality-reducing MOUD (37). Areas successfully distributing naloxone to communities in need have lower rates of opioid overdose deaths (38). We received the number of intakes and releases from the Kentucky Department of Corrections; risk of opioid overdose is significantly higher after release from prison (39). Multiple social determinants of health (SDOH) indices are publicly available at county, state, and national levels; differences within these SDOH indices correlate to differences in drug overdose deaths. Weather data is publicly available at state, county, and local levels; colder weather is also linked to risk of opioid overdose (40).

We developed a common data model to streamline both input and output of our prediction process. For our source data, each row of our common data model represented a single observation (a specific numerator and denominator) for a geographic region (a specific county, ADD, CMHC, or LHD) for a point in time (a specific month, quarter, or year). We organized these rows with a unique identifier representing each type of measurement for covariates, such as a unique identifier for naloxone distribution or drug seizures. Our predictions used the same common data model, except the numerator is a prediction and the time points indicated what month in the future in being predicted. This study was approved by the University of Kentucky Institutional Review Board (IRB #81270).

Geographic Regions of Interest

Kentucky’s approximately 4.5 million residents live across 120 counties that exhibit dynamic overdose trends varying by rural and urban status (8). In designing RADOR-KY, we conducted stakeholder interviews to understand the analytical needs of the state agencies to effectively respond to the opioid overdose epidemic. The stakeholders and partners indicated a need for grouping counties together so overdoses could be monitored by specific geographic regions of interest. These geographic groupings are aligned with established jurisdictional resource allocation practices, such as naloxone distribution efforts. As illustrated in Figure 1, we created county groupings based on Area Development District (ADD), Community Mental Health Centers (CMHC), and local health department (LHD) districts and centers.

Figure 1:

Geographic regions of interest showing county groupings served by: (A) community mental health centers (CMHC), (B) area development districts (ADD), and (C) local health department (LHD) districts and independent centers, respectively. American Community Survey (ACS) 2022 5-year population totals are indicated by the semi-transparent circles (based on natural breaks). In A and B, each region is uniquely colored. In C, white counties indicate individual county-health department centers compared to colored multicounty health districts.

There are 15 ADDs, each of which contains several counties grouped by geographic positioning and other cultural and demographic similarities. There are fourteen CMHCs based on Kentucky’s regional Boards for Mental Health or Individuals with an Intellectual Disability (MHID). The counties served by each CMHC overlapped heavily with those served by each ADD; however, there are slight differences in county composition and the intended dashboard audience. Each ADD and CMHC is uniquely colored in Figure 1. There are 61 LHDs; larger counties often have their own LHD (white in Figure 1) and smaller counties that neighbor one another are served by a single LHD (uniquely colored in Figure 1).

Data Preprocessing

Counties in Kentucky have widely varying populations, from extremely rural to more dense, urban and suburban areas. As a result, some counties have few EMS suspected opioid-involved overdose incidents. In addition to aligning with stakeholder needs, the county groupings aided our prediction models as a preprocessing step by providing a way to aggregate counties with a potentially sparse number of overdose events into a meaningful less sparse group of events better suited for prediction tasks. We aggregated overdoses per month to support monthly predictions; yearly aggregations were too broad to be useful for our dashboard audience and weekly aggregates were too narrow with sparse occurrences of overdose events.

Each county grouping’s total overdose counts were individually normalized to a maximum of 1 and a minimum of 0 to account for differences in scale between different geographic regions. This technique mitigated the disproportionate influence of raw overdose count magnitudes in some counties over others driven by population differences and the influence of arbitrarily high rates when adjusted per population; normalization ensures more robust forecasting performance across diverse geographical scales by translating the signals to a range more easily understood by the model. Testing confirmed predictions were improved after normalization which validates the need for normalization which may not be universal for all prediction tasks (16,41). Covariate data using the data sources listed in Table 1 were incorporated for each region and similarly normalized to convert raw counts from source systems (drug seizures, Medicaid claims, naloxone distribution) into scaled results. In some cases, additional pre-processing was required to fully utilize the covariate. For example, the KDOC intake/release raw data consisted of counts of individuals, each of whom was assigned a score from 1 to 5 indicating their substance use risk level. To aggregate this data, weighted averages of risk scores were calculated for each timestep to produce a single overall intake score and a single release score per geographic region; the intake risk score was subtracted from the release score to yield a single scaled value for that region and timestep. When any covariate data source did not span the same period as the EMS data, missing covariate data were imputed using a simple exponential smoothing model to forecast the incomplete parts of the data, ensuring that the filled data would match the overall trend and seasonality of the covariate data(42). Additionally, a flag variable was created to indicate if the data was originally missing. Additional covariate implementation details regarding aggregation strategies are available in Supplemental Table 1.

Model Choice

For our prediction task, we constructed a variation of long-term time-series forecasting models (NLinear LTSF), which is a simple one-layer neural network specifically created for time series forecasting with normalization (43). This model used a normalization and de-normalization process to address potential distribution shift in the time series data by automatically removing non-stationary information, such as a changing mean or variance over time, and then re-applying it to the forecasts to produce more accurate predictions. Unlike complex models like transformers, which rely on self-attention mechanisms to find complex relationships, the N-Linear model leveraged a straightforward linear architecture to capture temporal trends in the long-term, year to year, season to season, and in the short near-term. We chose this model over more complex architectures, such as the Temporal Fusion Transformer (44), after testing demonstrated that the simpler architecture allowed the model to adapt better to the scale and requirements of our data (16).

More specifically, we chose to create NLinear LTSF models because they support past, future, and static covariates. Past covariates are additional data that is known in the past but not necessarily at prediction time. This can be due to the type of data represented or differing lags in administrative data collection. Most covariates used in our model are past covariates as their values are not known into the future. Some data, such as weather, are future covariates, because we can use future values of the covariate at prediction time to improve the quality of the forecasts. Some data, such as social determinants of health (SDOH), are relatively static covariates that do not change much over time but can still help the model to distinguish behavior among different regions of interest. N-Linear models also support multivariate forecasting which allows the model to be trained on data from all geographical regions at once and produce individual predictions for each. This allowed our model to learn the overall trends that all series have in common, while adjusting for individual differences between regions at prediction time.

Training and Evaluation

Two different methods of training and evaluation were compared to fully capture the performance of the models. In the first method, the dataset was simply split into a training and testing set, with the last 10% of timestamps withheld for testing. We used this split for evaluating accuracy of forecasting models to determine the most optimal set up of model parameters and covariate choices. When the model was used to generate real predictions for each area as part of the dashboard visualization, it was trained on the entire time series and produced predictions for the three months in the future following the end of the series.

The second method used an expanding window technique where the model is initially trained on the first two years of data available and produces predictions for the next three months; subsequently, those three months are incorporated into the training set and the model is refit. It predicts the next three months, and this cycle of prediction and training continues until the entire time series has been used. This yielded a full evaluation of the model’s performance using simulated time periods and demonstrated how well the model adapts to new data. The results of the models were evaluated by averaging two primary error metrics across all regions. First, we used the Root Mean Squared Error (RMSE), a measure of the average difference between the predicted and actual values (45). A value closer to 0 indicates a better-performing model with predictions very close to the observed values, while an RMSE closer to 1 suggests poor performance. Generally, an RMSE below 0.2 indicates strong performance, and below 0.1 indicates an accurate model. To complement RMSE, the Mean Absolute Scaled Error (MASE) was also calculated (46). If a region has mainly zero values in the absence of events, the model will safely predict no incidents, producing a model that has a good RMSE but is not useful in practice. We calculated MASE as the ratio of Mean Absolute Error (MAE) of the model’s predictions to the MAE of a baseline prediction that uses the average of the training set to predict on the test set. These metrics measure whether the model’s predictions are improved from a baseline naive prediction. A very sparse county would have a MASE close to 1 because the model’s predictions aren’t significantly different from a baseline prediction. Thus, this metric accounts for differences in sparsity between regions better than the RMSE.

We deliberately chose not to perform hyperparameter tuning in this evaluation. Given our limited sample size, extensive tuning would risk overfitting to this specific dataset, potentially compromising the generalizability of our findings. Additionally, the N-Linear model’s simple architecture provided inherent robustness, unlike more complex models that are highly sensitive to hyperparameter choices (43). Finally, our comprehensive evaluation strategy, which employed multiple independent assessment methods, provided sufficient evidence of model performance without requiring a separate validation set for parameter optimization. This approach prioritized model generalizability and reproducibility over dataset-specific optimization because data sources, predictions, and dashboards are constantly updated.

Customizing Covariates per Geographic Region

To maximize forecasting accuracy, we developed more than 10 data use agreements with state agencies for timely (weekly or monthly) flow of data for RADOR-KY to capture relevant covariates from a wide variety of data sources (Table 1) and to customize our models per region (Figure 1). During preliminary testing, our forecasting models performed best when using no more than a couple of relevant covariates. We observed that it was impractical to include all covariates in the same model, as this provided the model with too much information and noise for it to be able to use effectively, thereby lowering the forecast accuracy.

Our forecasting system was constructed so that each covariate could be tested independently to determine which set is the most effective for predictions for each region. Each covariate may perform differently for each geographical region; for example, some covariates are sparser than others and contribute more information to the model at larger geographical groupings. Once separate models were trained on data using different covariates, we compared the forecasting error of each to determine which covariates had the greatest impact. From there, combinations of the best performing covariates were tested to determine both which covariates helped the most and how many covariates were adequate to include. Because a single model is trained across all regions, each individual geographical grouping used the same covariates.

Interactive Dashboards

We developed interactive, dynamic dashboards to show our predictions to community stakeholders and partners; our dashboard was implemented using D3.js (Data-Driven Documents), an open-source JavaScript library for creating interactive visualizations. Our dashboards visualize both the original data and the predictions; the graph displayed is based on which region was selected by the user. Users can adjust the displayed interval and review previous predictions on historical data. The dashboards used the same common data model used by our prediction process.

Results

We extracted suspected opioid-involved overdoses from our Kentucky EMS data for events occurring from January 2017 to September 2025. We aggregated these overdoses for each county in Kentucky and calculated totals over time for ADDs, CMHCs, and LHDs. These overdose totals represent the basis of our prediction task and evaluation processes.

Covariates were defined based on relevant variables provided by each of our data sources (Table 1). Each individual covariate was tested and evaluated to determine its practical impact on our prediction task. Additional notes on implementation details are available as Supplemental Table 1. The top performing covariates from each data source were selected for further evaluation. For example, several Medicaid measures related to OUD and treatment were evaluated across all regions; ultimately, two provided acceptable performance: (1) number of Medicaid recipients with OUD receiving MOUD and (2) number of Medicaid recipients receiving behavioral health treatment including case management and peer support. Social Vulnerability Index (SVI) data contains indices representing four thematic components: socioeconomic status, household characteristics, racial and ethnic minority status, housing and transport, and overall vulnerability. Preliminary evaluation determined these three variables as the best performing covariates: percentage in multi-unit housing, percentage of individuals with access to a vehicle, and percentage of cost-burdened housing.

Customizing Covariates per Geographic Region

Table 2 shows a comparison of average RMSE scores within different geographic regions for each covariate when evaluating the model on the last 10% of the time series data; the best performing covariate geographic region is bolded (Table 2). Each covariate was tested independently of each other, which allowed us to rank the impact on base model performance.

Table 2.

Average RMSE of individual covariates across geographic region

	Geographic Region
Covariate	County	ADD	CMHC	LHD
Overdose (Base)	0.1956	0.1452	0.1382	0.1701
Drug Seizures	0.1836	0.1391	0.1376	0.1713
Medicaid Claims	0.1819	0.1266	0.1243	0.1639
Naloxone Distribution	0.2009	0.1458	0.1421	0.1879
KDOC Intakes and Releases	0.1811	0.1326	0.1289	0.1643
SDOH	0.1865	0.1396	0.1555	0.1763
Treatment Venues	0.1852	0.1335	0.1332	0.1704
Weather	0.1864	0.1335	0.1357	0.1794

Table 3 shows the best performing covariate or covariate combinations for each geographic region; covariate combinations not involving the single best covariates did not perform well and are ignored. The best performing covariates were determined by comparing average RMSE within a geographic region; MASE was used as a secondary check for the rankings and agreed with RMSE findings. For the best models, the county-level models achieved an average MASE of 0.8298, ADD averaged 0.6461, CMHC averaged 0.6468, and LHD averaged 0.7705. In all cases, using the single best-performing covariate was slightly more effective than using combinations.

Table 3.

Average RMSE for best-performing covariate combinations for each geographic group

	Geographic Region
Covariates	County	ADD	CMHC	LHD
Best Single Covariate	0.1811 (Intakes/Releases)	0.1266 (Medicaid Claims)	0.1243 (Medicaid Claims)	0.1639 (Medicaid Claims)
Medicaid, Intake/Release	0.1863	0.1392	0.1412	0.1746
Medicaid, Drug Seizures	0.1999	0.1352	0.1377	0.1883
Drug Seizures, Intake/Release	0.1825	0.1372	0.1396	0.1835
Medicaid, Drug Seizures, Intake/Release	0.1962	0.1641	0.1407	0.1705

Impact of Sparseness

The sparseness of overdose data impacted RMSE and MASE differently. Supplemental Figure 16 shows two plots to compare RMSE (A) and MASE (B) scores for counties according to sparsity level (percent of zeros in the data); these zeros represent places where no overdoses occurred for that time period. These plots show that while low-sparsity counties perform similarly well between the two metrics, high-sparsity counties do not. The MASE plot shows a stronger correlation, where higher sparsity counties were associated with higher MASEs, because the predictions do not improve much upon a baseline forecast in those cases.

Because some counties (the smallest geographic regions of interest) do not have many events, the larger geographic regions containing them may have better performing models. When comparing the RMSE of each county with the RMSE of their parent region, 85.0% of the counties had better performance when rolled into ADDs and 85.83% had better performance when rolled into CMHCs. Only 52.5% of counties had an improvement in RMSE when rolled into LHDs, which was likely due to the numerous counties serving as their own LHDs, hence an equal RMSE. Similar trends were observed when measuring the MASE improvements observed when rolling counties into their parent region (82.5% for ADD, 78.33% for CMHC, 60.0% for LHD). Using the Kentucky River ADD as an example, the ADD model achieved an RMSE of 0.1339 and all eight counties within this ADD have an improved RMSE at this level compared to the individual county level; several of these counties had individual RMSEs higher than 0.2. We chose Kentucky River as an example because it had the lowest variation in county population across all counties in the ADD, which ensured that the ADD’s observation totals were not dominated by one or two of the biggest counties. For these cases with a sparse number of events, aggregating into their parent region provided a better model for forecasting incidents in the area for visualization in dashboards.

Expanding Window Evaluation

Each geographic region was tested with the expanding window method of training and evaluation. For each geographic region, the best performing covariates summarized in Table 3 were used, where each training run was evaluated with both RMSE and MASE. Two variations of each metric are given in Table 4: the error across all windows spanning the entire period and the error in the last year (four testing windows of three months). We calculated this using the mean for RMSE and the median for MASE; this was because for some counties with very consistently low overdose counts, the baseline error was low or zero. Because this error was used as the denominator in the calculation of MASE, this yielded hugely inflated error numbers regardless of the actual model’s performance; the median was used instead of the mean to address this. The error typically began high and decreased as the model is trained on more data; hence, performance in the last year showed an improvement.

Table 4.

Average RMSE and median MASE per geographic region when expanding the evaluation window across time

Metric	Evaluation Window	County	ADD	CMHC	LHD
RMSE	All Windows	0.1763	0.1349	0.1343	0.1717
	Last Year	0.1440	0.1046	0.1088	0.1352
MASE	All Windows	0.9657	0.8107	0.8207	0.9516
	Last Year	0.8294	0.5067	0.7013	0.8594

As the evaluation window expands as data increases and contributes to training the model, RMSE changed for all four geographic regions (Figure 2, A). In every case, the trend showed an improvement in RMSE as more data was used for training. Similar trends are also observed for MASE (Figure 2, B).

Figure 2:

Comparison of (A) RMSE and (B) MASE scores throughout expanding window evaluation

An example is shown in Figure 3 of the Kentucky statewide predictions over the course of the expanding window evaluation. This shows the predictions begin to match the real trends more closely as more data is used to train the model. Individual plots for each ADD showing the overdose counts compared to the predictions are provided in Supplemental Figures 1–15.

Figure 3:

Kentucky statewide EMS data, with expanding window predictions over time

Interactive Dashboards

Figure 4 shows screen captures of two example dashboards visualizing predictions three months into the future. Users of our dashboards may filter by time and geographic region; we continue our example of Kentucky River ADD (Figure 4A) and show two example counties (Figure 4B) where the orange line represents a regular county, and the blue line represents a sparse county where there are not many overdose events. The gray area contains predictions for three months into the future, where each monthly prediction is presented as a diamond. Note that the sparse county predictions were mostly zeros.

Figure 4:

Visualizing the predictions for suspected opioid-involved overdoses in EMS data for (A) Kentucky River ADD and (B) a sparse county (blue) and a regular county (orange) within Kentucky River ADD.

Discussion

The dashboards presented are part of a larger suite of dashboards and tools designed for our team’s public health partners and stakeholders who oversee and monitor the opioid overdose crisis for specific geographic region (a county, a grouping of counties, or the whole state). Our results demonstrate that machine learning models can be customized to these specific geographic regions by using custom covariates and combinations of covariates demonstrated to have improved baseline model performance. Covariate performance varied between different geographic regions; measures derived from drug seizures, intake/release data, and Medicaid claims data were the most informative covariates. Synthetic opioid seizures provided insights into the changes on drug markets as increases of synthetic opioids in the drug supply are associated with overdose deaths (35). In Ohio, a state neighboring Kentucky, increases in seizures of fentanyl and carfentanil have been associated with increases with opioid-involved overdose deaths (47). Not surprisingly, the Medicaid population is highly impacted by social drivers of health and is at increased risk for fatal opioid-involved overdose (48). In most cases during our model development, we observed that using the single best-performing covariate was slightly more effective than using combinations of covariates.

We also tested naloxone distribution due to naloxone being a highly effective tool against opioid overdose (34); surprisingly, monthly naloxone distribution was not a strong covariate for predicting future increases or decreases in overdoses. In a prior study, volume of naloxone distribution was positively and significantly associated with fatal overdose rates, while changes in distribution rates had no relationship with overdose mortality (49). Kentucky Medicaid has covered naloxone since 2021 and Kentucky’s Cabinet for Health and Family Services has partnered with the Kentucky Pharmacists Association to support community opioid education and naloxone distribution programs (50). Despite naloxone’s effectiveness, stigma associated with naloxone is problematic in Kentucky (51) and efforts to scale-up distribution of naloxone are needed to mitigate opioid-involved overdose mortality (52).

Our work demonstrates that dashboards could be updated rapidly with actionable data in a real-world setting with real-world stakeholders, which is consistent with other studies indicating dashboards are an important element to responding to the opioid overdose crisis (21,53). Our work also highlights the challenges and opportunities of using data from counties that have a wide range of populations and number of events. Covariate performance varied across different geographic regions due to sparseness and realistically prevents a ‘one size fits all’ solution for forecasting opioid-involved overdoses; this is consistent with findings suggesting that overdose trends for our state vary by rural and urban areas which supports our decision to customize models per geographic region (8). A single model is unlikely to fit to the diversity represented by these geographic regions equally well, so customizing and fine-tuning is required to more accurately adapt models to these regions. Additionally, regular re-testing is recommended to ensure that we can adapt to changes in the data over time. When this work was first performed a year ago, the best performing covariates were slightly different than they are now, due to the inclusion of an additional year of both EMS data and covariate data for data sources with data availability for 2025.

Aggregation of counties into larger counties groupings associated with particular ADDs, CMHCs, and LHDs combats sparsity issues by merging small counts together; however, the appropriate choice of geographical aggregation for a dashboard needs to consider both modeling performance and the needs of the intended audience. County membership for the ADDs and CMHCs was very similar with only slight variations in certain regions; given this, we did not observe significant differences in model performance between the ADD and CMHC geographic regions. LHD performance was generally better when the LHD covered several counties instead of a single county; as expected, the models performed better for larger geographical areas (e.g. large metro counties, or groupings of counties).

The expanding window evaluation demonstrated the difficulty of accurately forecasting events during COVID-19. In Figure 3, a large spike in RMSE can be observed at the beginning of 2020, coinciding with the start of the COVID-19 pandemic. In Kentucky, there was an unprecedented spike in drug overdoses during the first months after the COVID-19 public health emergency declaration, with more than 50% increase in the fatal opioid-involved overdoses during the first years of the COVID-19 pandemic, as well as an increase in patients treated by EMS responders for suspected opioid overdose who refused transportation (5,54). Although the model was unable to predict this drastic change in the quantity of opioid-involved overdose events, the error steadily declined after this point and eventually stabilized.

In addition to considering performance, we chose geographic regions already used by various government and community organizations for decision-making; our stakeholders and partners requested integration of ADDs, CMHCs, and LHDs. In addition to this, users may need to view dashboards for a single county to support an initiative or to monitor overdose activity; this use-case prevents us from automatically adjusting sparse counties into larger regions in cases where data from the larger region yields a better model; user goals and practical needs outweigh the importance of performance. These dashboards are a straight-forward method to enable transparency and data sharing across different health jurisdictions; this may enable easier collaborations between jurisdictions which was shown to benefit public health initiatives (55). Community engagement and collaboration between public health stakeholders are both necessary elements to successfully implement evidence-based practices known to reduce opioid-involved overdose mortality (20). One of the biggest pain points in community engagement is access to meaningful data (53); our goal was to minimize the time between data availability and having actionable information and predictions available in our dashboards.

Limitations

Our data are limited to a single state and may not be representative of other jurisdictions. Stakeholders may ultimately receive a different version of the data and models discussed in this paper due to RADOR-KY being in active development; data sources may change, and new data sources may be added which will influence predictions on the dashboard. Missing data were imputed using constant fill values based on an adjacent time period when data sources for covariates did not cover the same period as our EMS overdose data. Our training process creates a single model for a geographic region, where the same covariates are used for each county contained within it. The customized models are part of the RADOR-KY opioid overdose forecasting system and required intensive automation efforts to receive data into our secure environment, ingest data into a database, analysis, and immediate dissemination via dashboards to stakeholders; for this reason, further customization of the models per each individual county may not be a sustainable process and is unnecessary given that the current regional models perform well for our purposes.

Conclusion

We demonstrated that models for predicting the number of opioid-involved overdoses benefit from customization through selection of covariates and through aggregation of geographic regions. These models will better serve our community partners and stakeholders, giving them access to rapid and actionable data. These dashboards assist stakeholders in the allocation of finite resources, such as naloxone, designed to reduce overdose deaths and ensure help is given where it is needed the most. As future work, we hope to monitor our predictions into the future, and evaluate their accuracy as well the effectiveness of any interventions launched based on these predictions to better understand the public health utility of RADOR-KY on the ongoing opioid overdose crisis.

List of Abbreviations

ADD

area development districts

CMHC

community mental health centers

EMS

emergency medical services

LHD

local health department

KBEMS

Kentucky Board of Emergency Medical Services

KDOC

Kentucky Department of Corrections

MASE

Mean Absolute Scaled Error

OUD

Opioid use disorder

RADOR-KY

Rapid Actionable Data for Opioid Response in Kentucky

RMSE

Root Mean Squared Error

Availability of data and materials

The machine learning models discussed in this manuscript are implemented for the RADOR-KY website in publicly available dashboards: https://www.rador-ky.uky.edu. Access to customized data and dashboards is available for those with RADOR-KY accounts.