Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network

Abstract

Background

Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability.

Methods

Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts.

Results

Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P < .05).

Conclusions

Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.

Introduction

Model generalizability in healthcare is a challenging task due to various factors, such as the heterogeneity of patient populations, the complexity of medical data and its storage, privacy and security concerns, and the need for regulatory compliance. As a result of such factors, models’ deployment is limited.^1,2 To assess the generalizability of ML models, real-world validation, including rigorous testing and evaluation in several clinical settings, is required. Machine learning (ML) models that perform well in research settings often struggle to adapt to diverse patient populations and varying healthcare systems, hindering their effective deployment in clinical practice.^3,4 The lack of model generalizability can be indicative of hidden biases in the data or ML algorithms, leading to potential harm for specific patient groups.^5,6 To ensure successful and ethical model deployment in healthcare, it is crucial to focus on improving model generalizability, which involves externally validating models on diverse datasets and accounting for potential single-site biases.^7,8

A globally accepted Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), paired with Observational Health Data Sciences and Informatics (OHDSI) tools, facilitated the development and validation of healthcare ML models by standardizing and harmonizing electronic health records (EHRs) into a unified format. While patient-level prediction models have utilized the OMOP CDM, most focus on external model performance.^9–18 Some studies propose small, clinician-customized models for improved generalizability,^14,19,20 but these small models have scalability challenges and potential biases.

A notable gap in these studies pertains to the selection of features that are robust across all sites, crucial for developing universally effective ML models.²¹ Healthcare data often exhibits significant variability due to variations in patient populations, clinical practices, and data collection methods across different sites and regions.²² If models rely on features that are site-specific or not universally applicable, their performance may suffer when applied to new or diverse healthcare settings.²³ Robust features that hold their predictive power consistently across various sites are essential for creating ML models that can provide valuable insights and predictions in a broader range of clinical contexts, ultimately enhancing their practical utility and impact in healthcare decision-making.

To address this gap, we leverage the OMOP CDM and OHDSI tools to advance external validation of ML models in healthcare. Our novel approach involves independent evaluation in both development and validation databases, safeguarding against data leakage. This approach enables the incorporation of various evaluation metrics into model development, capturing potential significant features from various statistical perspectives across all databases.²⁴ Furthermore, we examined the generalizability of the models, considering both site-specific and cross-site features and their impact on different learning algorithms. As a proof of concept, we developed a predictive model using a cross-site feature selection approach for prolonged opioid use (POU) and conducted validations across five distinct healthcare settings using the OMOP/OHDSI infrastructure. To ensure generalizability and the development of equitable models, we additionally assessed ML models across vulnerable populations, such as those with diabetes, depression, or obesity.^25–27 This innovative approach contributes to the broader understanding of universal model development, further emphasizing the impact of site-specific and cross-site features on model generalizability and applicability across diverse healthcare landscapes.

Methods

This study is a retrospective analysis of observational health data that has received approval from the Institutional Review Board (IRB) at Stanford University and all institutions involved in the external validation. We developed 15 models using 5 ML algorithms with 3 sets of covariates from a development database and 2 combinations of development and validation databases. The target cohort was divided into train (80%) and test (20%) sets, and then the feature selection process was applied to the train set. All training models were validated on 20% of the target cohort internally and on 100% of the target and risk group cohorts in the external databases. Additionally, the models trained using a feature superset were only validated on 20% of the target cohorts in the external validation databases, leaving out data for patients used in the feature selection step. The model development and validation are summarized in Figures S1 and S2.

Source of data

This study relies on multisite observational data from EHRs mapped to the OMOP CDM. The CDM was proposed by the OMOP community and aims to standardize the structure and content of observational data while also enabling efficient analyses that produce reliable evidence (https://www.ohdsi.org/data-standardization). The data were analyzed in a federated manner, where the patients’ data remained with the data owners and only grouped cohort demographic characteristics, model parameters, performance metric values, calibration summary statistics, and selected features were shared. The study utilized 5 CDM databases, namely STARR, IMRD-UK (THIN, A Cegedim Database), AUSOM, FinnGen, and CUIMC from the United States, United Kingdom, Korea, Finland, and the United States, respectively (Table S1).

Participants

The study included adult patients who underwent surgery between 2008 and 2019 and were prescribed at least one opioid 30 days before or after the surgery. Patients were included if they had at least 2 visits between 2 years and 30 days before the surgery and 2 visits within 30 days to 2 years after the surgery. Patients were excluded if they had other surgery between 2 and 7 months after the index surgery, as well as those who died within 1 year of surgery. To quantify the biases introduced by the follow-up inclusion criteria,²⁸ we excluded the last 3 rules shown in Figure S3 to validate the performance of the models in the presence and absence of the follow-up inclusion criteria. In the target cohort, 9 groups of RxNorm opioid drug ingredients were used to identify opioid prescriptions (Table S2), and 19 groups of procedures were considered to identify surgeries (Table S3). Diabetes was defined using type 1 or 2 diagnosis, as well as blood glucose-lowering drugs and hemoglobin A1C measurement. For the depression subgroup, the depressive disorder diagnosis was used, and for the obesity subgroup, the obesity diagnosis and body mass index were utilized (Method S1).

Outcome

The primary outcome was POU, which is defined as at least one opioid drug exposure between 90 days and 180 days after surgery.^25,29,30

Feature construction

Five groups of prediction features were defined based on CDM concepts (Table S4). We considered demographic features in combination with clinical features over 4 CDM domains, including condition, procedure, measurement, and drug, in 2 observation periods: 180 days (medium term) and 30 days (short term) prior to index surgery. Missing values were not present in the data representation because all the features were binary or count, and their occurrences were considered during the patients’ observation period.

Model development and construction

The target cohort was divided into train and test sets (80:20) at each site using stratified random sampling based on the study outcome. The chi-square metric was calculated for all features in the training set based on their distribution between POU and non-POU. Features significant at P < .01 were chosen for each site. Following that, 2 supersets of features were generated by cross-evaluating local features from 4 databases: STARR, IMRD-UK, FinnGen, and AUSOM.

The first superset, denoted as “superset-union”, was derived from a thorough analysis of features from multiple sites using chi-square methods, and the union of features was selected. The term “superset” was chosen to signify its origin from an exhaustive analysis spanning multiple sites. The second superset was created by excluding features unique to a single database, termed “site-specific”. This exclusion aimed to evaluate the impact of site-specific features on the generalizability of ML models. The exclusion of site-specific features dramatically reduced the number of features. We employed an additional metric, the positive-negative frequency (PNF),³¹ to evaluate features only in the second superset. Features with PNF > 1.5 in each database were retained, while site-specific features were excluded. As a result, the intersections of all site-specific features (called cross-site features) were selected by chi-square and PNF. Finally, these 2 intersections were unioned to create the second superset, named “superset-intersect” (ie, superset-intersect-chi-square + superset-intersect-PNF). This name was chosen to reflect its utilization of the intersection of features from different datasets while excluding site-specific features. The PNF metric proposes a distinct evaluation approach that can effectively capture different features by taking into account the disproportionate distribution of features in the development database (STARR) with a 14% incidence rate and can augment the number of cross-site features. By defining the 2 supersets, we aimed to comprehensively assess the influence of site-specific features versus cross-site features derived from multiple databases on model generalizability.

Statistical analysis methods

Five ML algorithms were applied using the Patient-Level Prediction (PLP) R package,³² version 6.0.4: Lasso logistic regression (LR), random forest (RF), AdaBoost (AB), eXtreme gradient boosting (XG), and Naive Bayes (NB). The best hyperparameters were chosen using a 5-fold cross-validation with a grid search strategy and then used to train the models. To evaluate the model discrimination, we reported the performance of models based on the area under the receiver operating characteristic curve (AUROC) with a 95% confidence interval (CI) and the area under precision recall (AUPRC). The 95% CI was calculated using 2000 stratified bootstrap replicates. We applied a LR estimator to the logit of the model outputs to update the calibration intercept and slope through external validation.³³ The calibration plots with Brier scores were used to show the model calibration. The paired t-test was utilized to determine the significance of performance differences between ML models, feature sets, and risk subgroups. All study source codes are available in the OHDSI’s network studies GitHub repository (https://github.com/ohdsi-studies/PORPOISE).

In regard to the OHDSI PLP package, we have improved it by providing sub-cohort validation in the external validation module with independent validation settings. This enhancement enables researchers to efficiently validate ML models across multiple sub-cohorts within a single external database, contributing to the release of PLP package version 6.0.4. Additionally, we introduced supplementary feature selection metrics such as PNF, expanding the PLP package’s versatility in developing ML models with diverse learning capabilities. Moreover, our implementation of subgroup evaluation on training, validation, and test sets complements the PLP package, providing a comprehensive approach to model assessment.

Results

Participants

The target cohort in STARR consisted of 41 929 patients, of whom 55.6% were female, 64.9% were white, and 84.9% were non-Hispanic. Figure S3 displays patient counts based on inclusion/exclusion criteria in the target cohort and subgroups.

Table 1 characterizes the target cohort in the STARR and 4 validation databases by subgroups, age groups, race, and ethnicity. Of the 41 929 patients who underwent surgery, 6121 (14.6%) had opioid drug exposure from 90 days to 180 days following surgery, of which 58.2% were female. While the mean age of prolonged opioid users was 56.6 (SD 15.64) years at the time of surgery, it was 57.4 (SD 16.53) years for non-prolonged opioid users. Among all patients in the target cohort, 17.7% were diabetic, with 18.6% being prolonged opioid users. Patients with any depressive disorder prior to surgery made up 17.3% of the target cohort, with 20.7% having POU. The target cohort also included 18% of obese patients, of whom 16.6% were prolonged opioid users (Table S5). The characteristics of the validation databases are also reported in Table 1. The IMRD-UK database had the highest rate of prolonged opioid users (44.2%), while STARR had the lowest rate (14.6%) over the target cohort.

Table 1.

Characterization of the target cohort on the STARR database used in the model development and external validation databases.

Characteristic	STARR	CUIMC	IMRD-UK	AUSOM	FinnGen
	% (n = 41 929)	% (n = 23 100)	% (n = 31 932)	% (n = 7295)	% (n = 3934)
Country	United States	United States	United Kingdom	Korea	Finland
Prolonged opioid use	14.6	22.0	44.2	15.8	21.8
Subgroups
Diabetes	17.7	26.5	10.7	20.6	23.6
Depression	17.3	8.3	7.2	1.6	5.8
Obesity	18.0	7.5	0.9	0.4	3.2
Age group
15-19	1.3	1.5	0.5	0.2	0.4
20-24	2.8	2.9	2.6	1.1	1.9
25-29	3	3.2	4.3	1.7	3
30-34	4.1	3.9	5.5	2.9	4.1
35-39	5.1	4.6	7.3	4.7	4.8
40-44	5.9	6	9.9	7.1	6.4
45-49	7.5	7.3	10.6	9.6	8.3
50-54	9.6	9.1	10	10.6	9.8
55-59	11	10	8.8	10.5	10.9
60-64	11.5	11.9	9.8	11.4	13.3
65-69	12.8	12.8	10.3	12.1	13.2
70-74	10.9	10.7	8.8	12.4	12.9
75-79	7.8	8.8	6.6	9.4	7.4
80-84	4.6	4.9	3.5	4.8	2.9
85-89	2.2	2.5	1.2	1.5	0.7
Gender: female	55.6	59.5	72.6	49	54.6
Race
Asian	13.3	2.4	0.1	98.5	NA
Black or African American	3.6	12.1	0.4	NA	NA
White	64.9	46.9	31.2	NA	100
Native Hawaiian	0.9	0.5	NA	NA	NA
American Indian or Alaska Native	0.4	0.1	NA	NA	NA
Ethnicity
Hispanic or Latino	13.5	26.5	NA	NA	NA
Not Hispanic or Latino	84.9	47.7	NA	NA	NA

Feature selection

Figure 1 indicates the number of local features selected by the chi-square metric: 6482, 1845, 548, and 1869 features from the STARR-US, IMRD-UK, FinnGen-FI, and AUSOM-KR databases, respectively. All these features were considered our local feature sets for developing ML models on each given database. The numbers of features selected by the PNF metric whose values are greater than 1.5 were 7747, 4365, 1086, and 2248 for the STARR-US, IMRD-UK, FinnGen-FI, and AUSOM-KR databases, respectively. We obtained 107 chi-square features (Table S12), which were shared by all databases, compared to 5 for PNF-selected features (Table S13).

Figure 1.

Cross-site feature selection results achieved from four CDM databases: (A) site-specific and cross-site features selected by the chi-square metric; (B) site-specific and cross-site features selected by the PNF metric; (C) cross-site features selected by the combination of chi-square and PNF metrics by excluding site-specific features.

The superset-union features included 8795 features from 4 CDM databases. By excluding site-specific ones, 1163 cross-site features were achieved (superset-intersect-chi-square). We generated the same one using the PNF metric, achieving 767 features (superset-intersect-PNF), and considered the union of both sets (1771) as superset-interest features. Figure 1C depicts the details of the combination of both sets generated from superset-intersect-chi-square and superset-intersect-PNF. As a result of generating superset-interest features, we obtained 159 features (Table S14), which were common between the chi-square and PNF features. We considered this set to be common risk factors shared between databases.

Internal and external validation: model discrimination

Table 2 shows the AUROC and AUPRC values obtained from internal validation (STARR) through the target cohort and 3 subgroups, as well as the average of the external validation over the 3 external databases, AUSOM, IMRD-UK, and FinnGen, for the 15 models developed. The AUROC values with 95% CIs have been presented for each of the databases in the Supplementary Section through Tables S6-S10, as well as the AUPRCs through Figures S4-S8. The internal validation results with 95% CIs showed that there is no significant difference between the models developed by local chi-square and superset features, whereas the performances of most models were improved through the external validation using superset-union features.

Table 2.

Internal (IN) and external (EX) validation results of the developed models using local (chi-square) and cross-site (superset-union, superset-intersect) feature selection across the target cohort and 3 risk subgroups.

Model	Feature set	Target cohort		Diabetes		Depression		Obesity
		AUROC (SD)	AUPRC (SD)	AUROC (SD)	AUPRC (SD)	AUROC (SD)	AUPRC (SD)	AUROC (SD)	AUPRC (SD)
Logistic Regression	IN (chi-square)	0.744 (··)	0.370 (··)	0.747 (··)	0.450 (··)	0.754 (··)	0.496 (··)	0.718 (··)	0.345 (··)
	IN (superset-union)	0.746 (··)	0.371 (··)	0.748 (··)	0.451 (··)	0.752 (··)	0.493 (··)	0.722 (··)	0.347 (··)
	IN (superset-intersect)	0.728 (··)	0.351 (··)	0.736 (··)	0.433 (··)	0.752 (··)	0.470 (··)	0.704 (··)	0.337 (··)
	EX (chi-square)	0.686 (0.02)	0.448 (0.16)	0.675 (0.03)	0.490 (0.21)	0.698 (0.03)	0.634 (0.08)	0.699 (0.06)	0.491 (0.26)
	EX (superset-union)	0.693 (0.02)	0.447 (0.16)	0.669 (0.02)	0.488 (0.21)	0.695 (0.01)	0.597 (0.10)	0.724 (0.07)	0.508 (0.28)
	EX (superset-intersect)	0.640 (0.05)	0.417 (0.14)	0.615 (0.06)	0.465 (0.20)	0.674 (0.03)	0.583 (0.06)	0.580 (0.07)	0.531 (0.13)
Random Forest	IN (chi-square)	0.729 (··)	0.358 (··)	0.739 (··)	0.441 (··)	0.746 (··)	0.468 (··)	0.708 (··)	0.332 (··)
	IN (superset-union)	0.730 (··)	0.358 (··)	0.738 (··)	0.448 (··)	0.749 (··)	0.467 (··)	0.711 (··)	0.336 (··)
	IN (superset-intersect)	0.717 (··)	0.349 (··)	0.721 (··)	0.440 (··)	0.748 (··)	0.470 (··)	0.707 (··)	0.336 (··)
	EX (chi-square)	0.644 (0.03)	0.405 (0.18)	0.622 (0.05)	0.444 (0.22)	0.654 (0.03)	0.558 (0.14)	0.631 (0.10)	0.452 (0.30)
	EX (superset-union)	0.650 (0.02)	0.409 (0.18)	0.615 (0.02)	0.448 (0.22)	0.678 (0.05)	0.529 (0.15)	0.709 (0.09)	0.559 (0.22)
	EX (superset-intersect)	0.648 (0.02)	0.401 (0.18)	0.604 (0.03)	0.428 (0.23)	0.677 (0.02)	0.537 (0.13)	0.705 (0.10)	0.550 (0.24)
XGBoost	IN (chi-square)	0.744 (··)	0.382 (··)	0.757 (··)	0.459 (··)	0.749 (··)	0.484 (··)	0.720 (··)	0.362 (··)
	IN (superset-union)	0.747 (··)	0.382 (··)	0.754 (··)	0.459 (··)	0.758 (··)	0.499 (··)	0.716 (··)	0.353 (··)
	IN (superset-intersect)	0.725 (··)	0.346 (··)	0.730 (··)	0.411 (··)	0.757 (—)	0.486 (··)	0.709 (··)	0.347 (··)
	EX (chi-square)	0.673 (0.03)	0.427 (0.15)	0.667 (0.03)	0.475 (0.19)	0.675 (0.06)	0.582 (0.14)	0.610 (0.10)	0.462 (0.27)
	EX (superset-union)	0.672 (0.05)	0.426 (0.13)	0.651 (0.05)	0.471 (0.17)	0.701 (0.05)	0.564 (0.09)	0.647 (0.13)	0.436 (0.31)
	EX (superset-intersect)	0.660 (0.03)	0.410 (0.13)	0.656 (0.03)	0.470 (0.17)	0.677 (0.05)	0.547 (0.07)	0.693 (0.11)	0.471 (0.31)
AdaBoost	IN (chi-square)	0.730 (··)	0.355 (··)	0.731 (··)	0.439 (··)	0.727 (··)	0.445 (··)	0.692 (··)	0.329 (··)
	IN (superset-union)	0.730 (··)	0.356 (··)	0.727 (··)	0.436 (··)	0.726 (··)	0.451 (··)	0.697 (··)	0.310 (··)
	IN (superset-intersect)	0.718 (··)	0.345 (··)	0.715 (··)	0.431 (··)	0.738 (··)	0.465 (··)	0.674 (··)	0.316 (··)
	EX (chi-square)	0.624 (0.03)	0.398 (0.13)	0.626 (0.04)	0.450 (0.19)	0.671 (0.06)	0.592 (0.07)	0.651 (0.04)	0.541 (0.12)
	EX (superset-union)	0.685 (0.04)	0.438 (0.13)	0.692 (0.03)	0.495 (0.18)	0.689 (0.06)	0.579 (0.10)	0.664 (0.07)	0.536 (0.15)
	EX (superset-intersect)	0.633 (0.05)	0.394 (0.14)	0.627 (0.06)	0.457 (0.18)	0.639 (0.03)	0.552 (0.06)	0.576 (0.07)	0.397 (0.24)
Naïve Bayes	IN (chi-square)	0.596 (··)	0.225 (··)	0.625 (··)	0.289 (··)	0.615 (··)	0.307 (··)	0.586 (··)	0.232 (··)
	IN (superset-union)	0.597 (··)	0.223 (··)	0.623 (··)	0.282 (··)	0.612 (··)	0.299 (··)	0.584 (··)	0.224 (··)
	IN (superset-intersect)	0.645 (··)	0.240 (··)	0.656 (··)	0.290 (··)	0.655 (··)	0.323 (··)	0.638 (··)	0.244 (··)
	EX (chi-square)	0.506 (0.00)	0.288 (0.16)	0.518 (0.01)	0.359 (0.20)	0.513 (0.01)	0.434 (0.10)	0.545 (0.05)	0.436 (0.13)
	EX (superset-union)	0.549 (0.06)	0.322 (0.13)	0.562 (0.06)	0.421 (0.15)	0.544 (0.05)	0.410 (0.09)	0.540 (0.03)	0.366 (0.21)
	EX (superset-intersect)	0.511 (0.03)	0.296 (0.15)	0.525 (0.08)	0.357 (0.22)	0.514 (0.03)	0.386 (0.13)	0.527 (0.08)	0.362 (0.24)

For external validation, the mean values achieved from 3 databases, AUSOM, IMRD-UK, and FinnGen (standard deviation shown in parentheses).

Abbreviations: AUPRC, area under precision recall; AUROC, area under the receiver operating characteristic curve.

The LR model achieved the highest mean AUROC of 0.69 and AUPRC of 0.45 through external validation with superset-union features over the target cohort. Compared to its internal validation, the mean AUPRC increased by 22%, and the AUROC was reduced by 7%, showing acceptable generalizability. The mean AUROCs over the 4 evaluation cohorts and 3 databases were 0.70, 0.68, 0.67, 0.66, and 0.55 for LR, AB, XG, RF, and NB models trained with superset-union features, respectively. Performance difference between NB and other models, as well as LR and RF, were significant (paired t-test, P < .05). The mean AUROCs of the LR, when considering only cross-site features in the superset-interest, had a significant reduction to 0.63 (P = .0015) compared to the superset-union (0.70).

There was no significant difference between the target cohort and each of the risk subgroups through the external validation (P > .05). The best mean AUROC was achieved by LR for the target cohort (0.69) and obesity (0.72), AB for diabetes (0.69), and XG for depression (0.70).

In our target cohort, we selected patients based on additional post-index criteria that may introduce bias when evaluating the model in a real-life scenario. To evaluate the impact of this bias, we validated the performance of each model on 2 groups of patients: one selected with post-index criteria and one without, in both internal and external validation (Table 3). The lack of significant differences (P > .05 in most cases) in the external performance of the models in both groups indicates that the developed models can perform well on 2 different groups of patients. It suggests that the post-index criteria do not negatively affect model performance in external validation and thus do not introduce any significant validation bias.

Table 3.

Internal and external validation AUROC with 95% CI for the developed models using superset-union features on the target cohorts with and without post-index criteria

Model	Target cohort	Internal validation (STARR)	External validation (AUSOM)	External validation (IMRD-UK)	External validation (FinnGen)
Logistic Regression	With post-criteria	0.746 (0.731-0.761)	0.714 (0.698-0.730)	0.676 (0.663-0.689)	0.689 (0.644-0.734)
	Without post-criteria	0.717 (0.706-0.727)	0.729 (0.715-0.744)	0.676 (0.664-0.688)	0.698 (0.657-0.738)
Random Forest	With post-criteria	0.730 (0.714-0.746)	0.627 (0.609-0.645)	0.658 (0.644-0.671)	0.664 (0.617-0.711)
	Without post-criteria	0.706 (0.696-0.717)	0.647 (0.631-0.664)	0.670 (0.658-0.682)	0.683 (0.642-0.724)
XGBoost	With post-criteria	0.747 (0.731-0.762)	0.724 (0.708-0.740)	0.640 (0.626-0.654)	0.652 (0.604-0.700)
	Without post-criteria	0.721 (0.711-0.731)	0.743 (0.728-0.758)	0.646 (0.634-0.658)	0.659 (0.616-0.702)
AdaBoost	With post-criteria	0.730 (0.715-0.746)	0.718 (0.701-0.735)	0.644 (0.631-0.658)	0.692 (0.647-0.737)
	Without post-criteria	0.706 (0.696-0.717)	0.728 (0.712-0.743)	0.641 (0.629-0.653)	0.694 (0.653-0.735)
Naïve Bayes	With post-criteria	0.597 (0.584-0.611)	0.612 (0.597-0.627)	0.512 (0.505-0.518)	0.522 (0.502-0.542)
	Without post-criteria	0.587 (0.579-0.596)	0.628 (0.614-0.642)	0.515 (0.510-0.521)	0.522 (0.504-0.540)

We also validated our models on the CUIMC database, which was not included in the cross-site feature selection due to joining the project after the selection was performed. The best AUROC with 95% CI, 0.650 (0.641-0.658), was achieved by the XG model with superset-union features over the target cohort. Nevertheless, there was no significant difference between the local and supersets (Table S10).

Following feature selection, the most relevant features for predicting POU were identified from the top-performing model, which was the LR trained with superset-union. Figure 2 depicts the top 20 features associated with higher POU risk. These features include condition and procedure count 6 months prior to surgery, breast procedures, enterostomy, laryngoscopy, bypass ileum, black race, prior conditions of breast diseases, chronic pain, gastrointestinal complications, muscle disorders, and prior long-term use of codeine, which were the most predictive of POU.

Figure 2.

Top 20 features associated with higher POU risk achieved from Lasso logistic regression trained with superset-union features.

Internal and external validation: Model calibration

Figure 3 illustrates the calibration plots and Brier scores of the 3 developed models through internal and external validation over the target cohort and 3 risk subgroups after recalibration on the validation databases. The recalibration of the model outputs moderated the intercept and gradients of prediction scores compared to those without recalibration (Figure S9) and improved the model risk estimation by reducing the Brier score. All 3 models achieved more calibrated predictions on the target cohort than the other risk subgroups. In addition, the Lasso LR produced more calibrated results on the external validation databases and achieved an average Brier score of 0.168 (SD = 0.04) compared to its internal Brier of 0.110.

Figure 3.

Calibration plots and Brier scores of the 3 models trained with superset-union features on the 4 external validation databases over the target cohort and 3 risk subgroups.

Model generalizability

The AUROC and AUPRC values of the LR and XG models developed using different databases are compared in Table S11 over the target cohort. The IMRD-UK, AUSOM, and FinnGen models outperformed the STARR models in internal validation. However, they did not preserve this superiority in external validation when transferring to another site, and the STARR-developed models outperformed them. STARR models had the lowest reduction rate when transferring to the IMRD-UK, AUSOM, and FinnGen databases. While the AUROC of the STARR LR model was reduced by 9% through external validation, this reduction was 26% for the IMRD-UK model. Table S11 also shows the performance between internal and external validation on the same database. The LR internal performances on the IMRD, AUSOM, and FinnGen databases were 0.82, 0.77, and 0.81, respectively, which were greater than their external performances of 0.68, 0.71, and 0.69 achieved by the STARR LR model. This may reveal the effect of site-specific biases on ML models’ performance as well as data shifts in the training sets.

Discussion

In this global study of POU across 4 countries, we developed a generalizable prediction model for POU using the OMOP CDM and community derived OHDSI tools. By introducing a cross-site feature selection approach and conducting extensive cross-site feature evaluations, we have illuminated the intricate interplay between feature selection, learning algorithms, and model generalizability. Of the 15 ML models developed, the one including both site-specific and cross-site features demonstrated the best generalizability and had minimal bias when applied to vulnerable populations. These results suggest that using cross-site feature selection from standardized CDM systems enhances model generalizability. The findings underscore the potential of standardized data frameworks for reliable prediction in different healthcare systems on a global scale. In addition, this work serves as compelling proof of concept for the OHDSI community. Through the utilization of these open-source tools, we not only advanced the understanding of POU globally but also contributed to the OHDSI toolkit regarding validation across sub-populations, ensuring the broader applicability of OHDSI-derived ML models.

This study found that including universally shared features improved model generalizability. This was evident in the performance of the models on the AUSOM database, which shared the most features with the development database. Generalizability arises from models’ transportability to clinical settings beyond development, which cannot be evaluated through internal validation due to its inherent optimism-corrected estimate of performance for the originating setting.⁷ Several factors affect generalizability, including oversampling, feature selection, and data augmentation, which can inflate internal performance but hinder new dataset generalization.^34–36 Our findings contribute to the existing body of literature, demonstrating that a combination of site-specific features alongside cross-site features can improve model generalizability, aligning with previous suggestions.^36,37 However, our results also underscore the critical influence of learning algorithms on the impact of site-specific features on generalizability. Overall, the cross-site feature evaluation approach highlights the importance of incorporating both types of features selected across diverse datasets to enhance the model’s ability to generalize on unseen data, which has important implications for the reliable and robust application of ML models in healthcare.

Furthermore, this study demonstrated the potential of CDM-based ML models to predict POU in vulnerable populations globally. Patient characteristics can introduce biases that affect model generalizability, such as demographic or site-specific biases.^35,38 On the other hand, populations subjected to human and structural biases are vulnerable to harm from incorrect predictions. Another type of bias that may cause performance disparities when deploying models in real clinical settings is the use of post-index criteria in patient selection.²⁸ We carried out a sensitivity analysis to verify that the models trained on surgical patients with post-index criteria are generalizable across all postoperative patients. Together, this evidence underscores the importance of leveraging standardized data models like CDM to enhance model generalizability, here in the context of opioid use prediction on a global scale.

While our feature selection and ML approaches successfully identified relevant features associated with the risk of POU, it is important to acknowledge the inherent challenges in comprehensively assessing the clinical relevance of these features. Large ML models that use EHR data often operate as black boxes, revealing unexpected features that may influence the outcome of interest. Given this, we found that a significant portion of the common features shared across all databases stemmed from recent medication use, particularly analgesics, cardiovascular, immunomodulators, and antimicrobials. Condition-wise, musculoskeletal, pain-related, mental health, neurological, and oncological disorders were strongly linked to higher risk. Conditions like rheumatoid arthritis, hip osteoarthritis, carpal tunnel syndrome, chronic pain, and generalized anxiety disorder were associated with higher POU risk, suggesting a need for tailored pain management strategies. Notably, breast pathology and its treatments were commonly linked to increased risk, highlighting the need for enhanced pain management in breast disease care. Overall, the findings underscore the importance of individualized treatment plans for effective pain management, weighing potential risks, and exploring alternative therapies whenever suitable.

Our findings indicate that boosting (specifically AdaBoost and XGBoost) outperforms RF when using superset features. Random forest’s reliance on combining decision trees can introduce biases and reduce generalizability.³⁹ In contrast, AB’s sequential training with weighted samples and trees reduces biases, while XGBoost’s gradient boosting improves calibration and flexibility. Lasso and XGBoost models demonstrated superior generalizability and calibration in our global multisite study compared to RF and other boosting methods. This knowledge assists healthcare practitioners in selecting effective ML approaches for predicting extended opioid use post-surgery and enhancing pain management outcomes.

Global cross-site feature evaluation emerged as a potential method for improving generalizability; however, more accurate models are needed before deployment at the point of care. Inconsistencies in feature distributions across EHRs can impede the development of high-performance models. Solutions like transfer/federated learning or adjusting decision thresholds can mitigate this challenge.^35,36,38 Our approach to cross-site feature selection across OMOP CDM databases, along with the use of transfer or federated learning, may lead to high-performing models with enhanced generalizability. This is crucial for most clinical applications, such as POU. Accurate prediction of POU at the time of surgery initiation may lead to better pain control and reduced opioid consumption, minimizing the risk of opioid-related complications and addiction. It can also help healthcare providers implement safer opioid prescribing practices or consider alternative pain management methods.

Limitation

This study had several limitations. First, it defined features based on a fixed observation period prior to surgery, which made it unable to capture temporal and modifiable risk factors. Most features were based on individual CDM concepts and used standard drug-group features, which may be less generalizable. Despite this, the model achieved acceptable generalizability with a comparable AUROC through external validation. Future work could include temporal features and more relevant drug-group features based on opioid use for improvement. It is important to note that while we employed established ML algorithms through the PLP package, which follows standardized and transparent implementation practices, the field of predictive modeling continually evolves. While our study focuses on the generalizability of models using a common data model (OMOP CDM), researchers are encouraged to explore and contribute to the development of cutting-edge algorithms within this standardized framework to enhance the field’s collective understanding and application of predictive analytics in healthcare. Additionally, EHRs lack functional measures like pain scores, but we compensate by utilizing a combination of prescriptions, encounters, and diagnosis codes to capture the patient care episode. However, the diagnosis codes related to POU might be under-reported and lack differentiation between prescribed opioids and illicit opioids. This hinders our ability to examine issues specific to prescribed opioids versus illicit ones. Reducing subjectivity in the outcomes could potentially improve the performance of our models. In acknowledging the limitations of our study, it is important to note that while our feature selection approach demonstrated effectiveness in conjunction with the utilized ML techniques, its performance when applied to more advanced ML methods remains uncertain. Future investigations employing sophisticated algorithms could provide insights into the adaptability and generalizability of the proposed feature selection approach. Additionally, it is crucial to recognize that our study serves as a foundational exploration, and further refinements to the feature selection methodology may be warranted for optimal performance in diverse clinical scenarios.

Conclusions

This study highlighted the pivotal role of the cross-site feature evaluation approach across international cohorts in the generalizability of ML models. This approach incorporated site-specific and cross-site features into model development, leading to enhanced model generalizability. Our work builds on community tools and highlights the importance of global feature selection and recalibration in enhancing model generalizability, ensuring robust performance across different healthcare systems. However, the variation in feature distribution across external EHRs poses challenges for developing high-performing models. Yet, this work serves as an important proof-of-concept for developing generalizable models using cross-site feature evaluation. Cross-site analysis allows researchers and clinicians to leverage a broader and more representative dataset, which may enhance the generalizability and applicability of the developed models.

Author contributions

T.H.-B. takes responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: T.H.-B., C.M.C. Methodology: B.N., C.Y., and T.E.-H. Statistical analysis: B.N. Collection of data: B.N., C.Y., T.E.-H., B.C., R.W.P., J.G.T., and T.F. Interpretation of data: All authors. Drafting of the manuscript: B.N. Critical revision of the manuscript: All authors. Administrative, technical, or material support: T.H.-B. Study supervision: T.H.-B.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

Research reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Number R01LM013362. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflicts of interest

The authors have no competing interests to declare.

Data availability

Data cannot be shared for ethical/privacy reasons. However, all source codes are available in the OHDSI’s network studies GitHub repository called PORPOISE (PostOpeRative Prolonged OpioId uSE) (https://github.com/ohdsi-studies/PORPOISE).