Machine learning models for predicting new-onset diabetes following acute pancreatitis using real-world data

Abstract

Introduction

About one-quarter of patients with acute pancreatitis (AP) will develop diabetes mellitus (DM) within 3 years, but risk factors remain unclear. This study aims to determine whether machine learning models (ML) can be trained to accurately predict new-onset DM following AP and identify key clinical features using real-world data.

Methods

This retrospective cohort study used de-identified data from the TriNetX federated electronic health records (EHR) network from 1 January 2017 to 11 March 2024. A total of 58 746 patients with AP (International Classification of Diseases-10 code K85) and no prior diagnosis of DM were included. New-onset DM following AP was the main outcome of interest. Five ML models were trained across four prediction windows, including logistic regression (LR), eXtreme Gradient Boosting, Adaptive Boosting, Random Forest and support vector machine. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC).

Results

Among the 58 746 patients with AP (mean (SD) age, 50.2 (16.6) years), the LR model demonstrated the highest overall performance, with a mean accuracy of 0.72 (SD, 0.009) and an AUROC of 0.79 (SD, 0.009). Key clinical features across models included age, pancreatic necrosis, body weight, body mass index, systolic blood pressure, number of medical visits and prior AP laboratory values such as glucose, anion gap, blood urea nitrogen (BUN) and total protein.

Conclusions

In this first real-world evidence study using EHR data, we have developed and demonstrated the feasibility of using ML to predict the new onset of DM after AP. Clinical features such as age, pancreatic necrosis, prior glucose, BUN and anion gap had the highest overall importance scores and may inform tailored prevention strategies.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Acute pancreatitis (AP) is a common gastrointestinal disorder, and approximately one-quarter of patients develop diabetes mellitus within a few years after an AP episode. Although previous studies have identified several potential risk factors for post-AP diabetes, most have been limited by relatively small sample sizes or reliance on conventional statistical methods that may not adequately capture complex clinical risk patterns.

WHAT THIS STUDY ADDS

• Using a large real-world electronic health record cohort of 58 746 patients with AP, we developed and evaluated machine learning models to predict new-onset diabetes following AP. Logistic regression demonstrated the best overall performance, and key features included age, pancreatic necrosis, body weight, body mass index, healthcare utilisation and laboratory markers such as glucose, blood urea nitrogen, anion gap and total protein.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• These findings demonstrate the feasibility of using routinely collected electronic health record data to identify patients at elevated risk of diabetes following AP. Early risk stratification may support targeted surveillance and preventive interventions, while the identified features provide a foundation for future studies aimed at improving prediction models and understanding mechanisms underlying post-pancreatitis diabetes.

Introduction

Acute pancreatitis (AP) is an inflammatory disease of the exocrine pancreas associated with tissue injury and sometimes necrosis.¹ AP is the third most common gastrointestinal diagnosis requiring admission in the USA, accounting for more than 300 000 hospitalisations per year.^2–4 Pancreatogenic diabetes (or type 3c diabetes) is an underdiagnosed form of secondary diabetes that can occur in patients with any disease of the exocrine pancreas.^5,6 AP is likely to be the most common cause of pancreatogenic diabetes, but the mechanisms of hyperglycaemia are poorly understood.⁷ It is likely the most common metabolic complication following AP and may develop more often than previously known, with a cumulative incidence up to 40%.^8–10 Diabetes mellitus (DM) is a metabolic disorder characterised by dysregulation of blood glucose levels.¹¹ Hyperglycaemia is a common early feature in patients with AP, but the likelihood of persistence of hyperglycaemia or progression to diabetes is not well-delineated. Hyperglycaemia in AP was historically considered a transient phenomenon.¹² However, recent meta-analyses revealed that one-quarter of patients with AP develop DM within 3 years following hospital discharge.^8,9 While many risk factors for AP and DM have been described extensively using traditional epidemiological studies, such as gallstones, advancing age, alcohol intake and cigarette smoking, among others,^13–17 both DM and AP risk are also independently affected by other clinical risk factors, including comorbid conditions, metabolic markers of inflammation and medications.^18–22 A recent meta-analysis encompassing 50 studies evaluated 79 potential risk factors for diabetes following AP. The analysis identified severe and moderately severe AP, alcoholic and hypertriglyceridaemic aetiologies, pancreatic necrosis, organ failure, recurrent AP (RAP) and comorbid conditions including obesity, chronic kidney disease, liver disease and dyslipidaemia as significant features of post-pancreatitis DM.²³

The identification of patients who are at a higher risk of developing new-onset DM following AP diagnosis can help clinicians prioritise and adopt prevention plans for those at greater risk.

Two fairly recent studies have used prospective data to define the incidence and risk factors for DM following AP.^24,25 Nevertheless, both studies had sample sizes of fewer than 200 participants. Given the complexity of post-AP metabolic outcomes, these limited cohorts may not fully capture the features of new-onset DM. In contrast, a large multivariable analysis using the Nationwide Readmission Database identified several independent risk factors for AP-related DM, including age, sex, an Elixhauser Comorbidity Index ≥3, presence of metabolic syndrome components, as well as severe and RAP.²⁶ However, there are limitations to prior traditional predictive models due to the limited scope of variables that could be included without over-adjustment.^26–29 This limitation can be overcome by using machine learning (ML) for predictive modelling of DM in AP, which permits inclusion of a wide range of variables that can be used without risk of overfitting or compromising the performance of the models. Furthermore, ML can identify complex, high-dimensional and non-linear risk among clinical features. To date, no studies have applied ML approaches using foundation models (FMs) to predict new-onset diabetes DM following AP, including the identification of key features. This is a substantial gap in using advanced predictive modelling to risk-stratify high risk patients and understand the complicated clinical and demographic aspects related to DM following AP.

Thus, the objective of this study was to build effective ML-based predictive models with high sensitivity and specificity to identify patients with AP at risk of developing new-onset DM based on patient clinical features using large real-world data.

Methods

Study design

This study emulates a prospective cohort design using large real-world data to build innovative, ML predictive models with high sensitivity and specificity by identifying risk factors (ie, features) to predict DM following AP. All data were collected retrospectively, but individuals were followed forward in time to determine new-onset DM following AP. This study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis with Artificial Intelligence (TRIPOD+AI) guideline.³⁰ The Penn State University Institutional Review Board considered this study not to be research involving human participants; thus, informed consent was not required.

Data source description: TriNetX research network

This study was based on the extracted data from the TriNetX research network, including patients from the four regions of the USA (Northeast, Midwest, South, West), non-USA (ex-USA) and unknown, a web-enabled technology multi-institutional network of electronic health records (EHR) and clinical-based data repositories to identify patients with AP diagnosis. The ex-USA data originate from participating healthcare organisations outside the USA. To preserve patient and site confidentiality, the exact countries and site-level locations are not disclosed. These data include multiple international healthcare settings. The database is managed at Penn State College of Medicine through its Clinical and Translational Science Institute Award, funded by the National Center for Advancing Translational Sciences. It is a federated, multi-institutional health research network that compiles de-identified data from EHR across a broad spectrum of healthcare organisations. The database provides researchers access to de-identified patient data from a network of healthcare organisations (HCOs). Database includes clinical patient data such as demographics (age, race, sex, geographical location), diagnoses, procedures, laboratory values and medications—commonly referred to as real-world data. The data are primarily from HCOs collected for the primary purpose of providing care to patients. It is a global network that has supported over 19 000 randomised trials and many peer-reviewed publications.³¹ TriNetX research datasets represent more than 220 HCOs and over 250 million patients.³¹

Patient and public involvement

Patients and the public were not involved in the design or planning of this secondary data analysis.

Study population

The study population included adults aged 18–90 years (TriNetX caps age at 90 years to protect patient privacy) with AP (diagnosed using International Classification of Diseases, 10th revision, Clinical Modification, ICD-10 codes: K85), with no prior history of DM diagnosis at the study baseline. We used the TriNetX research network database (1 January 2017 to 11 March 2024) to identify the index AP. The database’s earliest date of AP diagnosis was defined as the index date for study participants. For this study, preexisting DM at baseline was defined when any of the following criteria were present: ICD diagnostic codes E08-E14, random glucose ≥200 mg/dL, fasting plasma glucose level of ≥126 mg/dL, glycated haemoglobin (HbA1c) level of ≥6.5% or being prescribed diabetes-related medication before or on the index date of AP diagnosis. Patients with unreliable glucose values reported in mmol/L due to coding errors in the EHR database were excluded from the analysis. Further exclusions included patients with pancreatic cancer (ICD-10: C25).³² The flowchart for selection of the final study population is depicted in online supplemental figure 1.

Endpoint assessment

The outcome of interest was new-onset DM following AP. Incident diabetes was ascertained using ICD-10 diagnosis codes (E08–E14) or laboratory-based glycaemic measures. To improve robustness against short-term variability and potential transient hyperglycaemia, repeated laboratory values were summarised at the patient level using mean values across all available measurements during the observation period. Individuals were classified as having diabetes based on either ICD-10 diagnosis codes or mean laboratory values meeting the following thresholds: mean random glucose ≥200 mg/dL, mean fasting plasma glucose ≥126 mg/dL or mean HbA1c ≥6.5%.^33–37 As in previous studies, E08, E09 and E14 are included to avoid missing incident cases of diabetes due to errors in coding.^33–35,37 We identified the incidence of DM across four prediction windows: at any time point during the 7.2-year follow-up period (total diabetes) and specifically at 24, 36 and 48 months following the index date of AP.

Feature extraction/assessment

Data on age, sex, geographical region (Northeast, Midwest, South, West, ex-USA and unknown), race/ethnicity and marital status on the index date of AP, and length of stay (LOS) were collected directly from the TriNetX database. Emergency department (ED) visits in the year prior to AP were extracted using the current procedure terminology (CPT-4) codes. Based on a comprehensive literature review,^23,26,29 the following features, representing risk factors for AP and DM, were also captured using their corresponding ICD-10, CPT-4 Logical Observation Identifiers Names and Codes (LOINC) or RxNorm codes (online supplemental notes 1–3). The baseline features assessment window for the comorbidities and medications was defined as 1 year prior to the index date, during which we extracted the following: (1) comorbidities features: over 40 conditions were identified using ICD-10-CM codes and history of AP necrosis was assessed using the entire medical records for each patient with AP during the study period (complete list in the online supplemental note 1); (2) medications features: use of selected medications was identified using the RxNorm codes (online supplemental note 2).³⁸

Blood glucose and HbA1c values were identified during the follow-up period using the LOINC codes. Key laboratory features, including biochemical and nutritional-related biomarkers and vital signs, were determined using LOINC within 13 months before the index date to avoid missing the annual physical examination. We identified common vital signs features, including body height, weight, body mass index (BMI), systolic blood pressure (SBP) and diastolic blood pressure (DBP). While BMI is a standardised measure of adiposity, body weight may additionally capture aspects of overall body mass not fully reflected by BMI alone. Therefore, we evaluated the inclusion of both features in the ML models. The complete list of laboratory test features and vital signs using the LOINC codes and corresponding codes used in this study is presented in online supplemental note 3.

Data pre-processing

Initial data pre-processing involved excluding patients with LOS values exceeding 100 days to remove potential outliers. Sex as a biological feature, ethnicity and marital status were treated as label encoding variables and were directly extracted from TriNetX.

Imputation for missing data

Missing data imputation employed multiple ML algorithms tailored to the nature of the features.³⁹ For continuous features, eXtreme Gradient Boosting (XGBoost) and Ridge regression were implemented, while categorical features were imputed using Random Forest (RF) and XGBoost classifiers. The selection of specific imputation models was based on a comprehensive performance evaluation across various algorithms, including Linear Regression, Ridge, Lasso, ElasticNet, RF and XGBoost. In this analysis, we evaluated the quality of imputation using Cohen’s d to measure mean shifts and the Kolmogorov-Smirnov (KS) D statistic to evaluate overall distributional overlap. By convention, we consider |Cohen’s d|<0.20 to be very small/negligible, 0.20≤|Cohen’s d|<0.50 to be small, 0.50≤|Cohen’s d|<0.80 to be medium and |Cohen’s d|≥0.80 to be large. For the KS D metric, values below 0.10 indicate very small/negligible differences, 0.10≤D<0.20 indicate small differences, 0.20≤D<0.30 indicate moderate differences and D≥0.30 indicate large differences. Variables with more than 80% missingness were excluded before imputation. Variable-level missingness percentages and the corresponding imputation quality metrics are reported in online supplemental table 3A and table 3B.

Training-testing set

The dataset was split using fivefold cross-validation with an 80:20 ratio for training and testing sets using scikit-learn (V.1.4.2) in Python (V.3.11.8). The cross-validation process was implemented using SciPy (V.1.13.1) for numerical computations. This process was repeated 50 times to ensure robust model evaluation and account for potential variability in performance metrics. The repeated cross-validation approach helped minimise bias from random sampling and provided robust estimates of model performance across different data partitions. All partitions were created at the patient level, so all records from a given patient remained within a single fold and no patient contributed data to both training and test sets.

To address class imbalance, we applied random oversampling of the minority class within each training split only (no synthetic feature generation; duplication with replacement). Held-out test folds were not resampled, and performance was evaluated on the unaltered class distribution and averaged over cross-validation.

Model development and testing

The study evaluated five distinct ML algorithms in a consistent order across the manuscript, figures and supplementary tables. The models were trained across four prediction windows, including logistic regression (LR), XGBoost, Adaptive Boosting (AdaBoost), RF and support vector machine (SVM). Model assessment incorporated multiple performance metrics, including area under the receiver operating characteristic curve (AUROC), precision, sensitivity, specificity, accuracy and F1 score. Additionally, model performance was compared using age-based stratification that was dichotomised at the mean age of the study population (ie, <51 vs ≥51 years), as well as by sex and race/ethnicity. All resampling for class imbalance was performed within the training folds only; held-out test folds were not resampled.

Statistical analysis

Demographic characteristics, laboratory test results, vital signs, comorbidities and medication features were summarised by new-onset diabetes status (yes vs no) at four different prediction windows (online supplemental table 1). Subgroup (age) and time-window comparisons were descriptive only; no formal hypothesis tests were conducted across groups, so no multiplicity correction was applied. Continuous features were presented as mean (SD) values, and categorical features were summarised as numbers and percentages. Performance uncertainty was summarised from out-of-fold estimates over fivefold cross-validation (CV) repeated 50 times, reporting 2.5th–97.5th percentile intervals across the 250 fits. All the data manipulation and extraction were conducted using SAS software V.9.4 (SAS Institute, Cary, North Carolina, USA), and ML prediction models were performed using Python software.

We performed two additional sensitivity analyses for the incident diabetes outcome. First, we repeated model evaluation in a complete-case cohort, excluding participants with missing values in any retained analytic feature or the outcome. Second, we repeated the analysis using the restricted laboratory dataset, linked by patient ID and used laboratory-defined diabetes outcome among participants with glucose or HbA1c data. The same fivefold cross-validation repeated 50 times, training-fold-only balancing and performance metrics were used for these analyses.

Feature analysis

We began by converting the analytical baseline table of 84 features into compact numeric representations with the Tabular Prior-data Fitted Network (TabPFN), a foundation model pretrained on synthetic tabular tasks and able to generate informative embeddings without large in-domain training sets.⁴⁰ In our workflow, TabPFN/FM was applied after construction of the analytical feature matrix, including preprocessing and imputation for the primary analysis. Therefore, the foundation-model step did not replace the missing-data imputation procedure. Each patient was represented as one row from the 1-year look-back window, and the resulting patient-level vectors were passed to five classifiers.

To understand why the foundation-model pipeline flags certain patients as high risk, we set two linked objectives: first, to pinpoint the hidden (‘latent’) directions within the TabPFN embedding that carry the strongest signal for incident diabetes, and second, to trace those directions back to the clinical features that clinicians recognise for improved model interpretation. We trained a random-forest classifier on those vectors to predict new-onset diabetes after AP. The model’s importance scores revealed which latent directions within the embedding space carried the strongest predictive signal; we retained the top 5% of those dimensions for further analysis. To translate this information back into familiar clinical terms, each high-impact latent dimension was modelled in turn using the original 84 baseline features as features in separate random-forest regressions. Averaging the resulting importance scores produced a single, consolidated ranking for every baseline feature. This two-stage approach connects the abstract embedding space to concrete EHR features, giving a global ranking of features. However, these importances are relative and non-causal, and the embedding step limits the model’s interpretability; clinically familiar variables may rank lower due to collinearity, measurement frequency/missingness or scaling rather than absence of relevance. In simple terms, each latent dimension represents a learnt combination of patient characteristics rather than one directly observed clinical variable. Mapping the high-signal dimensions back to the original EHR variables identifies the patient characteristics most strongly associated with the model-derived diabetes risk patterns, without implying causality (online supplemental figure 2).

Results

Baseline characteristics of the study population

We included a total of 58 746 patients with a diagnosis of AP, age (mean±SD): 50.2±16.6 years; 49.5% male. During 7.2 years of maximum follow-up, 4084 patients (mean age 53.7 ± 14.8 years; 56.2% male) developed incident diabetes. Examining the four different follow-up windows, we found that 2777 (54.1 ± 14.9 years; 55.2% male) developed diabetes within 24 months, 3324 (53.9 ± 14.8 years; 55.9% male) within 36 months, and 3710 (53.7 ± 14.7 years; 56.0% male) within 48 months. Patients with incident diabetes were more likely to be older, obese, have higher proportions of prior alcohol abuse (8.7% vs 6.6%), metabolic dysfunction-associated steatotic liver disease (4.4% vs 2.7%), hypertriglyceridaemia (0.9% vs 0.4%) in the year prior to AP and a history of pancreatic necrosis during the study (13.7% vs 5.3%). Furthermore, patients with incident diabetes had higher proportions for most of the comorbidity conditions, frequent medical visits and ED visits and higher number of medications used in the year prior to AP, as compared with those without diabetes (online supplemental table 1).

Missing data distribution

Prior mean glucose exhibited a Cohen’s d of 0.14 (negligible) and a KS D of 0.09 (negligible), suggesting that the mean glucose after imputation (113.7 mg/dL) remains essentially unchanged from the original mean (110.5 mg/dL) and that the empirical cumulative distributions nearly overlap. BMI shifted from a mean of 28.96 kg/m² to 29.24 kg/m², corresponding to a Cohen’s d of 0.05 (negligible) and a KS D of 0.12 (small). These statistics show that imputation has not significantly altered patients’ body mass profiles. Although weight, SBP and DBP also demonstrated Cohen’s d values between 0.11 and 0.17 (all <0.20) and KS D values ≤0.13 (either negligible or small), our primary focus on glucose and BMI confirms that the imputed values faithfully reproduce the critical distributional characteristics of these diabetes-predictive features (more details in online supplemental table 3).

Model performance

Among the tested models, LR demonstrated the highest performance; however, its discriminative ability was moderate with an AUROC of 0.79±0.009, precision of 0.72±0.007, accuracy of 0.72±0.009, sensitivity of 0.72±0.020 and specificity of 0.72±0.008 (figure 1). Figure 2 illustrates the comparative performance of all models through AUROC curves, with LR consistently outperforming other algorithms across different performance metrics. To benchmark practicality, we trained a minimal-feature model (age, BMI, sex, ethnicity, history of pancreatic necrosis) using the identical training process. The results (median AUROC 0.56–0.63 across algorithms; online supplemental figure 3) showed the accuracy–simplicity trade-off. Furthermore, a reliability (calibration) curve for the best model (LR on FM embeddings) shows close agreement between predicted and observed risk across bins (online supplemental figure 4).

Figure 1. AUROC variation using different machine learning models. AdaBoost, Adaptive Boosting; AUROC, area under the receiver operating characteristic curve; XGBoost, eXtreme Gradient Boosting.

Figure 2. Radar plots of the comparative performance of all models. AdaBoost, Adaptive Boosting; AUROC, area under the receiver operating characteristic curve; LR, logistic regression; RF, Random Forest; SVM, support vector machine; XGBoost, eXtreme Gradient Boosting.

Sensitivity analyses for the incident diabetes outcome are summarised in online supplemental table 4. The primary imputed analysis included 58 746 participants and 4084 diabetes events. The complete-case analysis included 2201 participants and 194 diabetes events, reflecting a large reduction in sample size after excluding participants with any missing retained feature. The laboratory-data-restricted analysis included 33 304 participants and 3162 diabetes events. In these analyses, the highest AUROC was 0.654 (95% empirical interval 0.636 to 0.671) for LR in the primary imputed analysis, 0.646 (0.567 to 0.726) for random forest in the complete-case analysis and 0.653 (0.632 to 0.672) for random forest in the laboratory-data-restricted analysis. The laboratory-data-restricted results remained similar to the primary imputed analysis. The complete-case results were broadly consistent but less precise because of the substantial reduction in analytical sample size. Results were compared with the primary imputed analysis and are reported in online supplemental table 4 and figure 5.

Important features

Our analysis revealed that patient age and history of pancreatic necrosis during the study period exhibited the highest mean importance across all 11 embedding models across the 4 prediction windows, followed by mean glucose prior to the AP index event and anion gap in the year prior to the index date of AP. The full set of top features (including total protein, blood‐urea nitrogen, weight, visit count, race and SBP) accounted for most of the aggregated importance mass. Figure 3 presents a notched boxplot of the importance distributions for the top K real‐space features, demonstrating both their high median values across embedding‐space targets, underscoring the robustness of these clinical features in driving the learnt representations.

Figure 3. Feature importance using Random Forest. AP, acute pancreatitis; BMI, body mass index.

Stratified analysis

Age-stratified analysis revealed differential performance patterns across age groups, as detailed in online supplemental table 2. Models consistently showed higher AUROC values for the younger age groups compared with the older age groups, though performance remained robust across both groups. LR maintained the best performance metrics in both age stratifications, as visualised in figure 4, which presents the stratified AUROC curves for different age groups (all incident diabetes: younger age group: mean±SD, 0.80±0.011, older age group: 0.74±0.011; incident diabetes in 24 months: younger age group: 0.79±0.014, older age group: 0.74±0.013; incident diabetes in 36 months: younger age group: 0.80±0.012, older age group: 0.74±0.013; incident diabetes in 48 months: younger age group: 0.80±0.012, older age group: 0.74±0.013) (online supplemental table 2). Moreover, analysing model performance across sex and racial/ethnic groups, we observed slightly higher AUROC values for females, as well as for individuals identifying as Hispanic and black or African American, which may suggest potential disparities in model performance across demographic subgroups (figure 5).

Figure 4. AUROC variation considering stratifications on age (defined as <51 years and ≥51 years). AdaBoost, Adaptive Boosting; AUROC, area under the receiver operating characteristic curve; XGBoost, eXtreme Gradient Boosting.

Figure 5. Model performance across sex and racial/ethnic groups. Subgroups represent sex and race/ethnicity categories. AUROC is shown for model performance across these subgroups. AUROC, area under the receiver operating characteristic curve.

For figure 4, younger and older age groups are defined as <51 years and ≥51 years, respectively, based on the mean age of the study population.

Discussion

This is the first study to use large real-world data and novel ML models to predict new-onset DM risk following AP. We leveraged a large-scale EHR database (TriNetX) to emulate a prospective cohort study to perform robust feature identification to generate five ML models (LR, XGBoost, AdaBoost, RF and SVM) to predict diabetes following AP. We were able to examine the ML algorithms at different prediction windows using real-world data. A total of up to 84 features were used as the full set, and the interpretability analysis identified the top 10 clinical features shown in Figure 3. We employed advanced ML techniques by leveraging FMs using TabPFN to project data into a learnt vector space, followed by training an RF on these embeddings to predict the new onset of DM. Eventually, we found that the performance ranking of the five ML models was AdaBoost<XGBoost<RF<SVM<LR. The LR model consistently had the highest performance among the five ML models with an AUROC of 0.79. LR’s advantage over other models likely stems from training on foundation-model embeddings that already encode interactions; in this setting, a regularised linear decision boundary can generalise better than higher-capacity learners. Prior attempts using ‘regular’ approaches on similar problems showed lower AUROC, consistent with this explanation. These models consistently revealed that patient age and history of pancreatic necrosis exhibited the highest mean importance across all 11 embedding models.

While models have been used to predict DM, no studies have focused mainly on the high-risk population of AP and the future risk of DM using ML techniques. Using real-world data, ML offers a potential pathway to more accurate and precise clinical predictions. This stratification of patient risk is meaningful to provide opportunities for early intervention and ultimately improved patient outcomes, including reducing the healthcare burden of DM. In this study, we use AUROC, precision, sensitivity, specificity, accuracy and F1 score to evaluate different performance metrics for our ML algorithms. Analysing the feature importance revealed that, except for age and history of pancreatic necrosis, in general, laboratory and vital signs values in the year prior to AP were highly influential in the prediction models. The pattern of feature importance remained similar when considering different prediction windows, with many comorbidities and medications showing minimal relative impact. This could be because some comorbidity features may be under control or well managed and therefore may have low importance. The results may be more representative of the patient’s overall present health status and risk.

Although previous studies have used regular ML to predict DM in the general population, none of those studies have used the most recent techniques using FMs such as TabPFN, which frequently outperform traditional models.⁴⁰ Given the high rate and current burden of DM, there is an urgent need for better ML models of long-term prediction of incident DM following AP. Our ML models demonstrated consistent performance in this specific cohort of patients with AP using EHR data, comparable to other ML models developed for predicting DM in the general population, achieving high AUROC values ranging from 0.71 to 0.84.^41–46 A previous study conducted using a private dataset of women in Bangladesh and different ML models, including decision tree, SVM, RF, LR, K‑Nearest Neighbor (KNN) and XGBoost found that using the ADASYN approach, the XGBoost classifier achieved the highest performance.⁴³ Another ML study using the National Health and Nutrition Examination Survey database, a nationally representative sample of the US population, found that Categorical Boosting (CATboost) achieved the best performance with an accuracy of 82.1% and an AUROC of 0.83, with age, total protein, BMI and SBP being among the most important features in predicting DM.⁴² A study conducted by Joshi and Dhakal used a combined approach of LR and ML to predict the risk factors of type 2 diabetes mellitus, found that glucose, BMI, pregnancy and age were among the most important features with an accuracy of 78.26% and a cross-validation error rate of 21.74%.⁴¹ Furthermore, we observed slightly higher AUROC values for females, Hispanic and black or African American individuals, suggesting potential disparities in model performance across demographic groups.

Study strengths, limitations and future directions

Our study has several strengths, including the use of large real-world data and ML models using different time windows to predict new-onset DM risk following AP with high AUROC. We also employed advanced ML techniques by leveraging FMs using TabPFN to predict our targeted outcome.⁴⁰ In addition to its strong predictive performance, TabPFN demonstrates core capabilities of foundation models, including data generation, density estimation, learning reusable embeddings and fine-tuning.⁴⁰ We tried using regular approaches, which did not work as well as in a similar study (AUC score of 71%)⁴⁴; however, using FMs significantly improves both ML performance and reproducibility. While LR performed best among evaluated models, its discriminative ability remains moderate and performance declines in older adults. Consistent with this, we acknowledge that LR is often viewed as a conventional statistical approach in public health and epidemiological research, although it is also widely considered a supervised ML method in predictive modelling literature. In this study, LR was included as one of the ML predictive models evaluated.^47–49 Specifically, the model leverages routinely collected clinical and demographic features including age, pancreatic necrosis, blood urea nitrogen (BUN), total protein (as a proxy for nutritional status), weight, healthcare utilisation, race, SBP and BMI according to their post-AP diabetes risk. This is especially important in resource-constrained settings, where long-term follow-up care after AP diagnosis may be challenging due to limited access to specialised care in AP and DM. Thus, the selected ML model in this study is best suited for risk stratification and clinical decision support tools, especially in resource-limited settings and to inform follow-up monitoring and screening. There are some limitations for consideration. While we emulated a prospective cohort design by identifying patients with AP and no prior diabetes at baseline, our study employs ML-based predictive models to identify individuals who developed new-onset DM following the AP index date. Because our outcome is a binary classification and not based on time-to-event or survival analysis, censoring and competing risks such as death do not apply to our ML modelling framework. EHR databases, including TriNetX, may include selection bias and missing data,⁵⁰ such as information about AP severity using the Revised Atlanta Classification. There are also inherent biases where an EHR-based database may include misclassifications due to inaccurate coding, under-reporting or incomplete documentation of diagnoses and medication use. In our study, the history of pancreatic necrosis was identified using ICD-10 diagnosis codes, which may be incomplete and fail to capture cases of extrapancreatic necrosis, potentially introducing misclassification bias. The use of repeated laboratory measurements to define DM may reduce sensitivity for detecting cases identified on the basis of isolated abnormal laboratory values; however, this approach provides a more stable estimate than reliance on a single measurement, which may be influenced by transient physiological or clinical factors. We did not have imaging-based biomarkers, such as an MRI or ultrasound, to directly assess pancreatic tissue damage, fat infiltration, fibrosis and necrosis, as well as a suitable dataset to perform external validation. Future studies should consider including imaging-based biomarkers and genetic features, and an external dataset in their ML prediction models. There is at least one ongoing prospective cohort study, the Diabetes RElated to Acute Pancreatitis and its Mechanisms (DREAM) study (ClinicalTrials.gov identifier: NCT05197920), which aims to collect MRI biomarkers and integrate data from these different domains.^51,52 Our next steps will involve training and validating our predictive ML models using suitable external datasets different from the TriNetX databases. Moreover, in the absence of diagnostic criteria and due to the current study design, we could not distinguish between type 3c DM and other types of DM or perform a model against insulin versus non-insulin-treated type 3c, respectively. We used both USA and non-USA (ex-USA) TriNetX research data, to enhance geographical and healthcare system diversity, helping to address concerns related to implementation equity and improve generalisability. After data preprocessing, cohort selection and imputation of missing values, the final analytical cohort may not fully represent the original TriNetX population. In addition, the imputation process may introduce uncertainty or bias if the missing data were not completely random.

Our 80% feature-level missingness filter was a pragmatic choice. EHR studies commonly exclude variables once missingness approaches 80%, but there is no universal standard.^53–56 We therefore report this threshold transparently and added complete-case and laboratory-data-restricted sensitivity analyses to evaluate whether performance estimates were driven by imputed features or by availability of laboratory data.

Interpretability is another limitation. Our embedding-based pipeline with two-stage feature mapping provides population-level, relative importance and does not imply causality or patient-level explanation. Consequently, some clinically plausible features may show low global importance because of confounding, collinearity or data sparsity. In addition, we did not conduct economic evaluations or decision-curve analysis. Moreover, the cost data were not available during the data request. These require site-specific costs and will be considered in future work related to clinical implementation of a final, refined model. Lastly, most DM cases in our cohort were identified using validated indicators like ICD-10 codes or HbA1c levels. To further reduce the risk of misclassification from transient stress hyperglycaemia, we used the average values of metabolic markers, including glucose and HbA1c, instead of relying on single elevated measures.

Conclusions

Leveraging comprehensive longitudinal data from large EHRs and advanced FMs across various prediction windows, we have demonstrated the potential of ML algorithms using LR for predicting DM following AP. The key features for predicting the new onset of DM following AP were age, pancreatic necrosis, BUN, total protein (nutritional biomarker), weight, frequency of medical visits, race, SBP and BMI. Based on our findings, we recommend that future studies aimed at using ML to predict new onset of DM should incorporate innovative FMs to improve performance and reproducibility due to optimal performance compared with traditional approaches. Our findings could be used to develop and test personalised intervention plans to identify high-risk patients with AP. Potential interventions could include more intensive follow-up with monitoring and/or prevention strategies (eg, lifestyle modifications). The model may ultimately inform a research-prototype decision-support tool. However, deployment is premature pending external validation, calibration and prospective impact and economic evaluation.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.