Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Feb 1.
Published in final edited form as: J Hosp Med. 2022 Dec 16;18(2):154–162. doi: 10.1002/jhm.13023

External Validation of a Model to Predict Future Chronic Opioid Use among Hospitalized Patients

Susan L Calcaterra 1,2, Eric Grimm 2, Angela Keniston 2
PMCID: PMC9899308  NIHMSID: NIHMS1856285  PMID: 36524583

Abstract

Background:

Previous research demonstrates an association between opioid prescribing at hospital discharge and future chronic opioid use. Various opioid guidelines and policies contributed to changes in opioid prescribing practices. How this affected hospitalized patients remains unknown.

Objective:

Externally validate a prediction model to identify hospitalized patients at highest risk for future chronic opioid therapy (COT).

Design:

Retrospective analysis of health record data from 2011 to 2022 using logistic regression.

Patients:

Hospitalized adults with limited to no opioid use one-year prior to hospitalization in a statewide healthcare system.

Main Measurements:

Used variables associated with progression to COT in a derivation cohort from a different healthcare system to predict expected outcomes in the validation cohort.

Key Results:

The validation cohort included 17,060 patients, of whom 9,653 (56.6%) progressed to COT one-year after discharge. Compared to the derivation cohort, in the validation cohort, patients who received indigent care (Odds Ratio [OR] = 0.40, 95% confidence interval [CI] 0.27 – 0.59, p < 0.001) were least likely to progress to COT. Among variables assessed, opioid receipt at discharge was most strongly associated with progression to COT (OR = 3.74, 95% CI 3.06 – 4.61, p < 0.001). Receiver operating characteristic curve for the validation set using coefficients from the derivation cohort performed slightly better than chance (AUC = 0.55).

Conclusions:

Our results highlight the importance of externally validating a prediction model prior to use outside of the derivation population. Periodic updates to models are necessary as policy changes and clinical practice recommendations may affect model performance.

Keywords: electronic health record, chronic opioid therapy (COT), prediction model, chronic non-cancer pain (CNCP)

Background

In 2016, approximately twenty percent of U.S. adults reported having chronic noncancer pain (CNCP). CNCP is associated with depression,1, 2 anxiety,3 loss of mobility,4 and a reduced quality of life.1, 5 Guidelines for management of CNCP recommend non-opioid analgesics as first line therapy with opioid analgesics as second line therapy.6 The decision to use opioid therapy for CNCP should involve consideration of the benefits versus harms of prescription opioids for patients who have not benefited from nonopioid analgesics for pain relief. Reported benefits of chronic opioid therapy (COT) for CNCP are a reduction in perceived pain7 and functional improvements.8 While COT may be beneficial for some, in others, COT may lead to complex persistent opioid dependence,9 opioid use disorder,10 or overdose.11 Identifying factors that predict unhealthy outcomes related to COT progression could inform decisions regarding opioid prescribing by clinicians and patients.

Previous research demonstrates an association between opioid prescribing at hospital discharge and progression to COT.12 Identifying patient characteristics associated with progression to COT could inform opioid prescribing to reduce opioid-related risks. One such method is the use of prediction models which involves the use of multiple predictors commonly identified by clinical reasoning and existing literature, sequentially limited based on stepwise variable selection algorithms to estimate the probability that an outcome will occur in a specific patient population.13 Prediction model development includes development and internal validation, model testing in an internal and/or external population, and an assessment of the model’s impact on clinical practice and patient outcomes. A valuable prediction model allows for reasonable classification of patients into risk groups with different prognoses in the development data (derivation dataset) and in data from other healthcare systems (validation dataset).14

We externally validated our previously developed prognostic statistical model to predict future COT among hospitalized patients15 using data from a statewide healthcare system to understand how variations in study populations, electronic health record (EHR) systems, policy, and guideline recommendations may affect the ongoing utility of a prediction model in clinical practice.

Methods

Study Design and Setting

This was a retrospective cohort study of hospital discharges from August 15, 2011 to April 14, 2022 at any of 13 UCHealth-affiliated hospitals located in urban, suburban, and rural Colorado. The largest of these hospitals is a 650-bed university-affiliated Level 1 trauma center and is the primary referral center for the region. This study received Colorado Multiple Institutional Review Board approval.

Data Source and Participants

We obtained data from a multi-institutional data warehouse16 that contains patient data for diagnoses, procedures, medications, laboratory results from two EHRs and links it to state-level Colorado All-Payers Claims Data (CO-APCD).17

Inclusion and Exclusion Criteria for the Validation and Derivation Cohorts

We intended to externally validate our previously developed model to predict progression to COT among hospitalized adults with limited to no opioid use at an initial hospitalization.15 We developed our study cohort applying the same inclusion and exclusion criteria used in the derivation cohort unless noted below (Figure 1, Table 1).15 Patients in the validation cohort received care at a UCHealth-affiliated site and patients in the derivation cohort received care at a Denver Health-affiliated site. The two healthcare systems are distinct. The validation cohort included hospitalized patients ≥18 years and the derivation cohort included hospitalized patients 15 to 85 years.15 The patient’s first hospitalization following ≥2 face-to-face encounters within the healthcare system was defined as the “index hospitalization”. In both cohorts, we excluded patients with <2 face-to-face encounters because they may be less likely to follow-up within the healthcare system following discharge. In both cohorts, we excluded patients on COT at the index hospitalization because this was our primary outcome.15 In the validation cohort, we identified COT by capturing opioid prescriptions ordered through the EHR or dispensed from a UCHealth-affiliated pharmacy one year prior to the index hospitalization. We also linked patient identifiers to claims data for dispensed opioid prescriptions in the CO-APCD to capture opioid prescriptions filled outside of a UCHealth-affiliated pharmacy. In the derivation cohort, we identified COT using prescription data available from the Denver Health data warehouse. At that time, we did not have access to CO-APCD, however, approximately 80% of patients filled their prescriptions at a Denver Health-affiliated pharmacy because they received discount pharmacy pricing.18 In both cohorts, we excluded patients on buprenorphine/naloxone at the time of index hospitalization.15 In both cohorts, we excluded obstetrics encounters, however we used the next non-obstetric-related encounter during the study period to qualify as the index hospitalization. In both cohorts, we excluded patients who died within one-year following index hospitalization because they would not be able to meet criteria for the primary outcome, progression to COT one year after index hospitalization.15 In both cohorts, we excluded subsequent hospitalizations following the index hospitalization to ensure our dataset only included each patient’s first “index” hospitalization within the study period.15 In the validation cohort we did not exclude patients under police custody nor did we exclude persons who were undocumented and received emergent hemodialysis because we did not have a way to identify them in the dataset.

Figure 1.

Figure 1.

Open in a new tab

Inclusion Criteria for Study Cohort

Table 1.

Comparison of the Exclusion Criteria between the Derivation and Validation Cohorts

Derivation Cohort Validation Cohort
Excluded patients aged <15 or >85 Excluded patients <18 years old
Excluded patients with <2 healthcare encounters to the healthcare system where the deviation cohort was drawn from in the 3 years preceding the index encounter Excluded patients with <2 healthcare encounters to the healthcare system where the deviation cohort was drawn from in the 3 years preceding the index encounter
Excluded patients on COT defined as “≥10 opioid prescriptions ordered or dispensed within any 365-day period of time” or “receipt of a ≥90-day opioid supply without a 30-day gap within any 180-day period of time” *, ± Excluded patients on COT defined as “≥10 opioid prescriptions ordered or dispensed within any 365-day period of time” or “receipt of a ≥90-day opioid supply without a 30-day gap within any 180-day period of time” *, ±
Past year opioid use was identified using pharmacy prescription fill data available from the healthcare system EHR (Denver Health) Past year opioid use was identified by:
1) Querying all opioid prescriptions ordered through the healthcare system’s ERH one year prior to the index hospitalization (UCHealth)
2) Linking patient identifiers to pharmacy claims data for dispensed opioid prescriptions available in the Colorado All Payer Claims Database (CO-APCD)
Excluded patients on buprenorphine/naloxone in the one year preceding their index admission Excluded patients on buprenorphine/naloxone at the time of index hospitalization
If the index hospitalization was an obstetric-related encounter, we used the next non-obstetric related encounter to qualify as the index hospitalization If the index hospitalization was an obstetric-related encounter, we used the next non-obstetric related encounter to qualify as the index hospitalization
Excluded patients who died one year following index discharge Excluded patients who died one year following index discharge
Excluded subsequent hospitalizations following the index hospitalization to ensure our dataset only included each patient’s first “index” hospitalization within the study period Excluded subsequent hospitalizations following the index hospitalization to ensure our dataset only included each patient’s first “index” hospitalization within the study period
Excluded patients in prison, jail, or police custody because ongoing medication refills would occur with corrections and not tracked in this study. The derivation cohort was derived from a hospital with a locked unit under police supervision. Did not exclude this patient population because the hospitals assessed in the validation cohort do not have a locked unit for patients in custody (as compared to the derivation cohort). If a patient was brought into the hospital in police custody, there was no way to identify this in the dataset.
Excluded patient who lacked documentation of US citizenship (at the time of this study, patients requiring hemodialysis for ESRD were admitted every week for emergent HD) Not applicable. In 2019, the Colorado Department of Health Care Policy and Finance opted to include end stage renal disease in Colorado’s definition of “emergency medical conditions thereby expanding access to scheduled thrice-weekly hemodialysis to undocumented immigrants€
Open in a new tab
*

COT definition from Vanderlip et al. and Von Korf et al.19,20

±

In the derivation, we reviewed all opioid prescriptions filled at a Denver Health affiliated pharmacy. In the validation cohort, we obtained APCD to identify patients with COT.

€

Cervantes L, Johnson T, Hill A, Earnest M. Offering Better Standards of Dialysis Care for Immigrants: The Colorado Example. Clin J Am Soc Nephrol. 2020 Oct 7;15(10):1516–1518. doi: 10.2215/CJN.01190120. Epub 2020 May 22. PMID: 32444395; PMCID: PMC7536747.

Primary Outcome

The primary outcome, progression to COT, intended to capture variations in opioid prescribing which could contribute to COT. COT was defined as “≥10 opioid prescriptions ordered or dispensed within any 365-day period of time” or “receipt of a ≥90 day opioid supply without a 30-day gap within any 180 day period of time”.19, 20 To identify COT in the validation cohort, we used UCHealth pharmacy data, opioid orders from the EHR, and CO-APCD pharmacy claims data. In the derivation cohort, we identified opioid prescriptions filled 365 days after the index discharge using Denver Health pharmacy data.

Predictor Selection

Predictor selection matched those reported in the original study.15 We extracted demographic data from the index hospitalization. We determined past medical history by querying patient encounters over the three years preceding the index hospitalization using International Classification of Diseases codes (ICD-9-CM/ICD-10-CM) (Appendix, Table 2). From these diagnoses, we calculated a Charlson Comorbidity Index.21 We grouped diagnoses into larger descriptive categories including cardiovascular, respiratory, or renal diseases, neoplasms, acute pain, CNCP. We defined CNCP by grouping ICD codes for musculoskeletal, back and neck, neuropathic, psychogenic, and headache pain. We derived discharge diagnoses from encounter diagnoses, billing data, and hospital problem lists from the index hospitalization. We used ICD codes to identify surgical procedures performed during hospitalization. We identified opioids ordered, administered, or dispensed during hospitalization including route, e.g., intravenous, oral, epidural, etc., and morphine milligram equivalent (MME) per hospital day using National Drug Codes and pharmacy data. We identified opioid prescriptions within 72 hours of discharge using EHR and CO-APCD claims data. We used EHR data to identify frequency of non-opioid analgesics (NSAIDS, capsaicin, lidocaine, acetaminophen), neuropathic medications (gabapentin, pregabalin, amitriptyline, nortriptyline), and benzodiazepines (alprazolam, clonazepam, diazepam, lorazepam) ordered one-year preceding the index hospitalization. We identified outpatient and inpatient encounters over 12-months preceding and proceeding the index hospitalization.

Statistical Analysis

We compared characteristics of patients using chi-square or Fisher exact tests for categorical variables and t-tests for continuous variables. We applied a logistic regression model to the validation cohort based on variables retained using stepwise selection in the derivation cohort.15 We calculated odds ratios with confidence intervals and prediction metrics such as AUC after fitting the model to the validation cohort. We used model coefficients from the derivation cohort in a logistic regression model to predict expected outcomes in the validation cohort, and prediction metrics were subsequently evaluated after applying coefficients from the previously established model to the new patient population. All analyses were performed in R version 3.6.3.

Results

The validation cohort included 17,060 patients compared to 27,705 patients in the derivation cohort (Table 2).15 The majority of index hospitalizations were at the University Hospital (n=8,755, 51.3%) (Appendix, Table 1). Of the 17,060 patients, 9,653 (56.6%) progressed to COT one-year following discharge. The derivation cohort included 27,705 patients, of whom 1,457 (5%) progressed to COT.15 In the validation cohort, compared to patients who did not progress to COT, patients who progressed to COT were slightly younger (mean age 59.11 vs. 60.7 years old, p < 0,001), had more past year healthcare encounters (mean 11.6 vs. 9.3, p < 0.001), more subsequent one-year hospitalizations (mean 0.3 vs. 0.1, p < 0.001), and underwent fewer surgical procedures during hospitalization (58.7% vs. 67.8%, p < 0.001). In the validation cohort, compared to patients who did not progress to COT, patients who progressed to COT were more likely to receive opioids at discharge (695 vs. 117, p < 0.001), have past year receipt of benzodiazepines (2,830 vs. 1,922, p < 0.001), and have CNCP (5,270 vs. 3,619, p < 0.001). Similar findings were noted in the derivation cohort.15 In the validation cohort, patients who progressed to COT were more likely to have a substance use disorder compared to patients who did not progress to COT (1,464 vs. 1,014, p < 0.001). This was not noted in the derivation cohort.15 Generally, compared to patients who did not progress to COT, patient who progressed to COT had greater past year opioid prescriptions, received higher daily average MMEs during hospitalization, and were less likely to receive indigent healthcare (Table 1). Similar findings were noted in the derivation cohort.15

Table 2.

Patient Characteristics and Distribution of Potential Predictors of Chronic Opioid Use

Variable Chronic Opioid Use (N=17,060)
Yes N = 9,653 (56.6%) No N = 7,407 (43.4%) P-value
Age, Mean (SD) 59.1 (16.0) 60.7 (16.7) < 0.001
Age, Median (25th, 75th) 61 (49, 71) 64 (51, 73)
CCI*, Mean (SD) 3.0 (2.4) 3.1 (2.4) 0.011
CCI, Median (25th, 75th) 3 (1, 4) 3 (1, 4)
Encounters in Prior 12 Months†, Mean (SD) 11.6 (12.9) 9.3 (10.0) < 0.001
Encounters in Prior 12 Months, Median (25th, 75th) 8 (4, 15) 7 (3, 12)
Post Index Hospitalizations, Mean (SD) 0.3 (0.8) 0.1 (0.4) < 0.001
Post Index Hospitalizations, Median (25th, 75th) 0 (0, 0) 0 (0, 0)
Surgical Procedure at Index Hospitalization, n (%) 5,670 (58.7%) 5,020 (67.8%) < 0.001
Opioid Receipt at Discharge n (%) 695 (7.2%) 117 (1.6%) < 0.001
Past Year Benzodiazepine Receipt, n (%) 2,830 (29.3%) 1,922 (25.9%) < 0.001
Past Year Receipt of Non-Opioid Analgesics n (%) 4,001 (41.4%) 3,247 (43.8%) 0.002
Three-year History of a CNCP Diagnosis‡, n (%) 5,270 (54.6%) 3,619 (48.9%) < 0.001
Three-year History of Any Substance Use Disorder, n (%) 1,464 (15.2%) 1,014 (13.7%) 0.007
Past Year Number of Opioid Prescriptions Filled, n (%)
 0 7,901 (81.9%) 6,314 (85.2%) < 0.001
 1 966 (10.0%) 642 (8.7%)
 2 407 (4.2%) 266 (3.6%)
 3 136 (1.4%) 83 (1.1%)
 4+ 243 (2.5%) 102 (1.4%)
Index Hospitalization MME§ per day, n (%)
 0 3,823 (39.6%) 2,884 (38.9%) < 0.001
 0.01 < 10 3,057 (31.7%) 2,499 (33.7%)
 10 < 51 2,214 (22.9%) 1,739 (23.5%)
 51 < 100 267 (2.8%) 116 (1.6%)
 100+ 292 (3.0%) 169 (2.3%)
Financial Classification, n (%)
 Discount Payment Plan (Indigent) 49 (0.5%) 80 (1.1%) < 0.001
 Medicaid 1,220 (12.6%) 825 (11.1%)
 Medicare 5,295 (54.9%) 3,949 (53.3%)
 Commercial 2,472 (25.6%) 2,005 (27.1%)
 Other/Unknown/Self-Pay 617 (6.4%) 548 (7.4%)
Open in a new tab
*

Charlson Comorbidity Index

†

Outpatient and inpatient encounters

‡

Includes arthritis/joint pain/bony pain/muscular pain/back pain/neck pain/nerve-related pain/psychogenic pain/headache syndromes

§

Milligram of morphine equivalents

Similar to the derivation cohort, in the validation cohort, older age (odds ratio [OR] = 1.04, 95% confidence interval [CI] 1.03 – 1.06, p < 0.001), receipt of higher daily average MMEs during hospitalization (i.e., 51 – 100 MME, OR = 2.18, 95% CI 1.72 – 2.77, p < 0.001), and greater number of past year (OR = 1.02, 9% CI 1.01 – 1.02, p < 0.001) and subsequent one-year hospitalizations (OR = 2.58, 95% CI 2.36 – 2.82, p < 0.001) were associated with progression to COT. Opioid receipt at discharge (OR = 3.74, 95% CI 3.06 – 4.61, p < 0.001), receipt of ≥ 4 past year opioid prescriptions (OR = 1.64, 95% CI 1.29 – 2.11, p < 0.001), and three-year history of CNCP (OR = 1.23, 95% CI 1.15 – 1.31, p < 0.001) were associated with progression to COT in both cohorts. Similar to the derivation cohort, in the validation cohort, surgery at hospitalization was inversely associated with progression to COT (OR = 0.68, 95% CI 0.63 – 0.74, p < 0.001). In contrast to the deviation cohort, in the validation cohort, patients who received indigent care (OR = 0.40, 95% CI 0.27 – 0.59, p < 0.001), had past year non-opioid analgesic receipt (OR = 0.89, 95% CI 0.83 – 0.95, p < 0.001), and had a higher Charlson Comorbidity Index (CCI) (OR = 0.96, 95% CI 0.94 – 0.98, p < 0.001) were less likely to progress to COT, while patients with Medicare were more likely to progress to COT (OR = 1.42, 95% CI 1.25 – 1.61, p < 0.001) (Table 2). The area under the curve for the model fit to the validation cohort data was (AUC: 0.673). The area under the curve for the model fit to the validation cohort data was (AUC: 0.673). The receiver operating characteristic (ROC) curve for the validation set using the coefficients from the derivation cohort data performed only slightly better than chance (AUC = 0.55).

We conducted secondary analyses to further investigate our results. We recreated our validation cohort excluding CO-APCD and only included UCHealth pharmacy data. The resulting cohort included 362 patients, of whom 60% (n=225), progressed to COT. We limited our dataset to include data from 2011 to 2014, the earliest data available, to approximate the derivation cohort study period of 2008 to 2014. This reduced the validation cohort size to 4,358 patients, of whom 67% (n=2,939), progressed to COT. The area under the curve for both the model fit to the validation cohort data (AUC: 0.676) and the derivation cohort coefficients applied to the validation cohort data (AUC: 0.562) were similar to the area under the curve when using the entire validation cohort time frame from 2011 to 2022.

Discussion

We externally validated a prediction model to identify hospitalized patients at highest risk for progression to COT following hospital discharge. The prediction model, originally developed in a closed, safety-net healthcare system, failed to predict progression to COT when applied to a cohort of patients in a quaternary referral, statewide healthcare system. Many variables from the prediction model developed in the derivation cohort remained independently associated with progression to COT in the validation cohort. The association between opioid receipt at discharge and COT,12, 22 higher dose of initial opioid exposure and COT,23 CNCP and COT,24 and COT and greater healthcare utilization25 are documented in the literature. While many patients in both cohorts received care in metropolitan Denver, and both healthcare organizations used the same EHR, many inter-cohort differences likely limited our ability to predict progression to COT in the validation cohort.

The prediction model developed in the derivation cohort failed to predict progression to COT in the validation cohort for multiple reasons. Despite using similar inclusion criteria for the two cohorts, the total number of people meeting inclusion criteria, and the percentage of people who progressed to COT within each cohort, were significantly different. First, more patients overall progressed to COT in the validation cohort, suggesting they represented a more at-risk population to develop COT than the derivation cohort. Patients in the validation cohort were more medically complicated and older than patients in the derivation cohort (validation cohort: mean CCI ≈ 3; mean age ≈ 60 years; mean past year healthcare encounters ≈ 11; derivation cohort: mean CCI ≈ 2; mean age ≈ 48 years; mean past year healthcare encounters = 0.2).15

Next, we did not have access to CO-APCD when identifying patients who progressed to COT in the derivation cohort, leading to possible underascertainment bias.

Patients in the validation cohort who received indigent healthcare were less likely to progress to COT. This may be because these patients are less likely to present for healthcare due to an inability to pay resulting in a lack of opioid exposure. Poor access to healthcare, stigma with opioid prescribing, and undertreatment of pain among Black and Hispanic/Latino people is a phenomenon reported in previous research studies.26 In our validation cohort, patients with Medicare were more likely to progress to COT than patients on Medicaid. Previous research demonstrates that new opioid use after hospitalization is common among Medicare beneficiaries27 and that COT is common among Medicare beneficiaries.28 This association observed in our validation cohort is unsurprising since Medicare beneficiaries are more likely to have CNCP,29 have a greater burden of chronic medical conditions,30 and have polypharmacy,30, 31 all of which are associated with COT32 compared to younger people who do not qualify for Medicare.

Despite limiting our validation dataset to the years 2011 to 2014 to better match the derivation cohort period of 2008 to 2014, temporal confounders related to opioid prescribing likely contributed to our model’s failure to predict progression to COT in the validation cohort. Opioids filled nationwide at retail pharmacies peaked in 2012 representing a shift in opioid prescribing throughout the U.S.33 Patients in the validation cohort were less likely to receive a discharge opioid prescription (4.8%) compared to patients in the derivation cohort (29%) representing a shift in opioid prescribing within our metropolitan area over time.15 In 2018, Colorado law limited an initial opioid prescription to a 7-day supply; a second prescription required the prescriber to check the Prescription Drug Monitoring Program (PDMP) database with a recommended 7-day limit.34 This intended to curb the opioid epidemic by reducing prescription opioid exposure and reflected a national trend in state-based restrictions on opioid prescribing.35 Temporal changes in opioid prescribing influenced by state legislation limiting opioid prescribing,36 greater access PDMP data,37 and various public health campaigns warning of risk associated with opioid use38 likely reduced initial opioid prescribing in the validation cohort. Future work should examine if laws and policies encouraging a shorter duration of opioid prescribing was associated with reduced prescription opioid availability, chronic opioid use, the development of opioid use disorder, or overdose.

Variations in the underlying data model across the two EHRs likely contributed to the failure of our model to predict COT in our validation cohort. Common data models are developed by adhering to non-proprietary, pre-defined standardized models to EHRs. The safety-net healthcare system used to develop the derivation cohort and the statewide healthcare system used to develop the validation cohort both used the same commercial EHR. Differences in the design of the two EHRs likely affected data categorization, collection, and reporting. For example, EHRs are commonly customized to meet the needs of each healthcare system, e.g., incorporation of decision making tools,39 inclusion of alerts or nudges to avoid medical errors or to encourage delivery of medical interventions,40 and restrictions on sensitive patient data.41 The structure of data elements, e.g., codes, classifications, and nomenclature varies across EHRs42 and impacts how data are categorized, queried, and reported for clinical care and research purposes. Ideally, when comparing EHR data across healthcare systems, data must be complete and uniformly coded, preferably at the time of documentation during provision of patient care.43 The requirements of coding uniformity and data completeness are especially relevant and particularly challenging when using data from multiple EHRs.

Limitations

We did not include CO-APCD in the derivation cohort and may have underestimated progression to COT. The use of ICD codes to classify disease is limited to the accuracy of the code itself.

Conclusion

The use of prediction models to complement clinical reasoning and decision-making is valuable and pertinent in clinical practice. Previous research has demonstrated both benefit and harm related to COT. Understanding patient-level factors associated with progression to COT can guide decisions about ongoing opioid prescribing. Our results highlight the importance of externally validating a prediction model prior to clinical use outside the derivation population. Periodic updates to predictive models should be performed as policy changes and clinical practice recommendations may affect model performance. Many predictive models are developed using data from administrative databases or registries, which incorporate biological and clinical data. Recognizing that variations in data collection and reporting is essential to understand how a prediction model will perform across systems and population.44

Supplementary Material

Supplemental Appendix

Table 3.

Logistic Regression Parameter Estimates from the Development Dataset and the External Dataset

Derivation Cohort15 Validation Cohort
Variable Odds Ratio (95% Confidence Interval) P-value Odds Ratio (95% Confidence Interval) P-value
Intercept 0 (0 – 0.01) < 0.001 0.38 (0.28 – 0.52) < 0.001
Age 1.19 (1.114 – 1.25) < 0.001 1.04 (1.03 – 1.06) < 0.001
Age Squared 1.0 (1.0 – 1.0) < 0.001 1.00 (1.00 – 1.00) < 0.001
Financial Classification (Ref = Medicaid)
 Indigent 1.04 (0.79 – 1.36) 0.78 0.40 (0.27 – 0.59) < 0.001
 Medicare 0.73 (0.52 – 1.03) 0.07 1.42 (1.25 – 1.61) < 0.001
 Commercial 0.43 (0.25 – 0.73) < 0.001 0.96 (0.85 – 1.08) 0.493
 Other/Unknown/Self-Pay 0.54 (0.31 – 0.92) 0.02 0.94 (0.80 – 1.10) 0.417
MME* per day (Ref = 0 mg)
 0.01 < 10 mg 1.65 (1.09 – 2.52) 0.02 1.17 (1.07 – 1.27) < 0.001
 10 – 50 mg 2.08 (1.47 – 2.93) < 0.001 1.29 (1.17 – 1.42) < 0.001
 51 – 100 mg 2.23 (1.49 – 3.35) < 0.001 2.18 (1.72 – 2.77) < 0.001
 101 + mg 3.37 (2.1 – 5.41) < 0.001 1.69 (1.38 – 2.09) < 0.001
Number of Healthcare Encounters in the Prior 12 Months 0.63 (0.47 – 0.84) < 0.001 1.02 (1.01 – 1.02) < 0.001
Opioid Receipt Within 72 Hours of Discharge (Ref = No) 2.33 (1.78 – 3.04) < 0.001 3.74 (3.06 – 4.61) < 0.001
Past Year Number of Opioid Prescriptions Filled (Ref = 0)
 1 1.99 (1.46 – 2.71) < 0.001 1.21 (1.08 – 1.35) 0.001
 2 3.31 (2.26 – 4.83) < 0.001 1.20 (1.02 – 1.42) 0.028
 3 4.19 (2.47 – 7.12) < 0.001 1.19 (0.90 – 1.59) 0.228
 4+ 9.87 (6.33 – 15.37) < 0.001 1.64 (1.29 – 2.11) < 0.001
Past Year Receipt of Non-Opioid Analgesics (Ref = No) 1.92 (1.49 – 2.48) < 0.001 0.89 (0.83 – 0.95) 0.001
Past Year Benzodiazepine Receipt (Ref = No) 1.89 (1.26 – 2.82) < 0.001 1.08 (1.00 – 1.17) 0.048
Three-year History of Substance Use Disorder (Ref = No) 1.24 (0.98 – 1.56) 0.07 0.97 (0.88 – 1.06) 0.461
Three-year History of a CNCP Diagnosis (Ref = No) 1.79 (1.41 – 2.26) < 0.001 1.23 (1.15 – 1.31) < 0.001
Surgical Procedure at Index Hospitalization (Ref = No) 0.57 (0.44 – 0.74) < 0.001 0.68 (0.63 – 0.74) < 0.001
Charlson Comorbidity Index 1.11 (1.05 – 1.17) < 0.001 0.96 (0.94 – 0.98) < 0.001
Number of Subsequent Hospitalizations within 12 Months after Discharge 1.51 (1.39 – 1.64) < 0.001 2.58 (2.36 – 2.82) < 0.001
Open in a new tab
*

Milligram of morphine equivalents

Funding

Dr. Calcaterra is supported by the National Institute on Drug Abuse (NIDA), National Institutions of Health, grant award number K08DA049905. Health Data Compass is supported by the Health Data Compass Data Warehouse project (healthdatacompass.org).

Footnotes

Declaration of Interest

All authors listed on this manuscript have no conflicts of interest to declare.

References

  • 1.Outcalt SD, Kroenke K, Krebs EE, et al. Chronic pain and comorbid mental health conditions: independent associations of posttraumatic stress disorder and depression with pain, disability, and quality of life. Journal of behavioral medicine. 2015;38(3):535–543. [DOI] [PubMed] [Google Scholar]
  • 2.Miller LR, Cano A. Comorbid chronic pain and depression: who is at risk? The journal of pain. 2009;10(6):619–627. [DOI] [PubMed] [Google Scholar]
  • 3.McCracken LM, Spertus IL, Janeck AS, Sinclair D, Wetzel FT. Behavioral dimensions of adjustment in persons with chronic pain: pain-related anxiety and acceptance. Pain. 1999/03/01/ 1999;80(1):283–289. doi: 10.1016/S0304-3959(98)00219-X [DOI] [PubMed] [Google Scholar]
  • 4.Baird CL, Sands L. A pilot study of the effectiveness of guided imagery with progressive muscle relaxation to reduce chronic pain and mobility difficulties of osteoarthritis. Pain Management Nursing. 2004;5(3):97–104. [DOI] [PubMed] [Google Scholar]
  • 5.Ataoğlu E, Tiftik T, Kara M, Tunc H, Ersöz M, Akkuş S. Effects of chronic pain on quality of life and depression in patients with spinal cord injury. Spinal cord. 2013;51(1):23–26. [DOI] [PubMed] [Google Scholar]
  • 6.Dowell D, Haegerich TM, Chou R. CDC guideline for prescribing opioids for chronic pain—United States, 2016. Jama. 2016;315(15):1624–1645. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Noble M, Treadwell JR, Tregear SJ, et al. Long-term opioid management for chronic noncancer pain. Cochrane Database Syst Rev. Jan 20 2010;2010(1):Cd006605. doi: 10.1002/14651858.CD006605.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Furlan AD, Sandoval JA, Mailis-Gagnon A, Tunks E. Opioids for chronic noncancer pain: a meta-analysis of effectiveness and side effects. Cmaj. 2006;174(11):1589–1594. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Manhapra A, Sullivan MD, Ballantyne JC, MacLean RR, Becker WC. Complex Persistent Opioid Dependence with Long-term Opioids: a Gray Area That Needs Definition, Better Understanding, Treatment Guidance, and Policy Changes. J Gen Intern Med. Dec 2020;35(Suppl 3):964–971. doi: 10.1007/s11606-020-06251-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Boscarino JA, Rukstalis M, Hoffman SN, et al. Risk factors for drug dependence among out-patients on opioid therapy in a large US health-care system. Addiction. Oct 2010;105(10):1776–82. doi: 10.1111/j.1360-0443.2010.03052.x [DOI] [PubMed] [Google Scholar]
  • 11.Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study. Ann Intern Med. Jan 19 2010;152(2):85–92. doi: 10.7326/0003-4819-152-2-201001190-00006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Calcaterra SL, Yamashita TE, Min S-J, Keniston A, Frank JW, Binswanger IA. Opioid prescribing at hospital discharge contributes to chronic opioid use. J Gen Intern Med. 2016;31(5):478–485. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Brookhart MA, Schneeweiss S, Rothman KJ, Glynn RJ, Avorn J, Stürmer T. Variable selection for propensity score models. American journal of epidemiology. 2006;163(12):1149–1156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. doi: 10.1136/bmj.b605 [DOI] [PubMed] [Google Scholar]
  • 15.Calcaterra SL, Scarbro S, Hull ML, Forber AD, Binswanger IA, Colborn KL. Prediction of Future Chronic Opioid Use Among Hospitalized Patients. J Gen Intern Med. 2018;33(6):898–905. doi: 10.1007/s11606-018-4335-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Compass HD. Referencing Compass in a Grant. February 3, 2021. Accessed September 16, 2021. https://pub.healthdatacompass.org/data-delivery-services/health-data-compass-resources
  • 17.Healthcare CfIVi. CO APCD Overview. Accessed September 17, 2021. https://www.civhc.org/get-data/co-apcd-info/
  • 18.Plan DHM. Your Pharmacy Options. Denver Health Medical Plan. Accessed October 18, 2022. https://www.denverhealthmedicalplan.org/pharmacy
  • 19.Vanderlip ER, Sullivan MD, Edlund MJ, et al. National study of discontinuation of long-term opioid therapy among veterans. Pain®. 2014;155(12):2673–2679. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Von Korff M, Saunders K, Ray GT, et al. Defacto long-term opioid therapy for non-cancer pain. The Clinical journal of pain. 2008;24(6):521. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of chronic diseases. 1987;40(5):373–383. [DOI] [PubMed] [Google Scholar]
  • 22.Mosher HJ, Hofmeyer BA, Hadlandsmyth K, Richardson KK, Lund BC. Predictors of long-term opioid use after opioid initiation at discharge from medical and surgical hospitalizations. Journal of hospital medicine. 2018;13(4):243–248. [DOI] [PubMed] [Google Scholar]
  • 23.Shah A, Hayes CJ, Martin BC. Characteristics of Initial Prescription Episodes and Likelihood of Long-Term Opioid Use - United States, 2006–2015. MMWR Morb Mortal Wkly Rep. Mar 17 2017;66(10):265–269. doi: 10.15585/mmwr.mm6610a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Nury E, Schmucker C, Nagavci B, et al. Efficacy and safety of strong opioids for chronic non-cancer pain and chronic low back pain: a systematic review and meta-analyses. Pain. 2021; [DOI] [PubMed] [Google Scholar]
  • 25.Kern DM, Zhou S, Chavoshi S, et al. Treatment patterns, healthcare utilization, and costs of chronic opioid treatment for non-cancer pain in the United States. Am J Manag Care. 2015;21(3):e222–e234. [PubMed] [Google Scholar]
  • 26.Meghani SH, Byun E, Gallagher RM. Time to take stock: a meta-analysis and systematic review of analgesic treatment disparities for pain in the United States. Pain Med. Feb 2012;13(2):150–74. doi: 10.1111/j.1526-4637.2011.01310.x [DOI] [PubMed] [Google Scholar]
  • 27.Jena AB, Goldman D, Karaca-Mandic P. Hospital Prescribing of Opioids to Medicare Beneficiaries. JAMA Internal Medicine. 2016;176(7):990–997. doi: 10.1001/jamainternmed.2016.2737 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Morden NE, Munson JC, Colla CH, et al. Prescription opioid use among disabled Medicare beneficiaries: intensity, trends, and regional variation. Med Care. Sep 2014;52(9):852–9. doi: 10.1097/mlr.0000000000000183 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Bialas P, Maier C, Klose P, Häuser W. Efficacy and harms of long-term opioid therapy in chronic non-cancer pain: Systematic review and meta-analysis of open-label extension trials with a study duration≥ 26 weeks. European Journal of Pain. 2020;24(2):265–278. [DOI] [PubMed] [Google Scholar]
  • 30.Almodóvar AS, Nahata MC. Associations between chronic disease, Polypharmacy, and medication-related problems among Medicare beneficiaries. Journal of Managed Care & Specialty Pharmacy. 2019;25(5):573–577. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Maust DT, Strominger J, Kim HM, et al. Prevalence of central nervous system–active polypharmacy among older adults with dementia in the US. Jama. 2021;325(10):952–961. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Silva Almodovar A, Nahata MC. Potentially unsafe chronic medication use among older adult chronic opioid users. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. 2019;39(2):140–149. [DOI] [PubMed] [Google Scholar]
  • 33.Schieber LZ, Guy GP, Seth P, et al. Trends and patterns of geographic variation in opioid prescribing practices by state, United States, 2006–2017. JAMA network open. 2019;2(3):e190665–e190665. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Society CM. SB18–022 Summary for Physicians: What you need to know about new opioid bill:. Colorado Medical Society. Accessed October 29, 2021. http://files.constantcontact.com/42cad56c001/ab8ab6d1-181f-4336-b89d-0bb07136b945.pdf
  • 35.Davis CS, Lieberman AJ, Hernandez-Delgado H, Suba C. Laws limiting the prescribing or dispensing of opioids for acute pain in the United States: A national systematic legal review. Drug Alcohol Depend. Jan 1 2019;194:166–172. doi: 10.1016/j.drugalcdep.2018.09.022 [DOI] [PubMed] [Google Scholar]
  • 36.Rutkow L, Vernick JS, Alexander GC. More states should regulate pain management clinics to promote public health. American Public Health Association; 2017. p. 240–243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Haffajee RL, Jena AB, Weiner SG. Mandatory use of prescription drug monitoring programs. Jama. 2015;313(9):891–892. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Blendon RJ, Benson JM. The Public and the Opioid-Abuse Epidemic. New England Journal of Medicine. 2018;378(5):407–411. doi: 10.1056/NEJMp1714529 [DOI] [PubMed] [Google Scholar]
  • 39.Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. Jul 3 2012;157(1):29–43. doi: 10.7326/0003-4819-157-1-201207030-00450 [DOI] [PubMed] [Google Scholar]
  • 40.Hsiang EY, Mehta SJ, Small DS, et al. Association of an Active Choice Intervention in the Electronic Health Record Directed to Medical Assistants With Clinician Ordering and Patient Completion of Breast and Colorectal Cancer Screening Tests. JAMA Network Open. 2019;2(11):e1915619–e1915619. doi: 10.1001/jamanetworkopen.2019.15619 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.McCarty D, Rieckmann T, Baker RL, McConnell KJ. The Perceived Impact of 42 CFR Part 2 on Coordination and Integration of Care: A Qualitative Analysis. Psychiatr Serv. Mar 1 2017;68(3):245–249. doi: 10.1176/appi.ps.201600138 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Häyrinen K, Saranto K, Nykänen P. Definition, structure, content, use and impacts of electronic health records: A review of the research literature. International Journal of Medical Informatics. 2008/05/01/ 2008;77(5):291–304. doi: 10.1016/j.ijmedinf.2007.09.001 [DOI] [PubMed] [Google Scholar]
  • 43.Vuokko R, Mäkelä-Bengs P, Hyppönen H, Lindqvist M, Doupi P. Impacts of structuring the electronic health record: Results of a systematic literature review from the perspective of secondary use of patient data. International journal of medical informatics. 2017;97:293–303. [DOI] [PubMed] [Google Scholar]
  • 44.Ristevski B, Chen M. Big data analytics in medicine and healthcare. Journal of integrative bioinformatics. 2018;15(3) [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Appendix
NIHMS1856285-supplement-Supplemental_Appendix.docx (37KB, docx)

RESOURCES