The Impact of Multi-Institution Datasets on the Generalizability of Machine Learning Prediction Models in the ICU

Abstract

OBJECTIVES:

To evaluate the transferability of deep learning (DL) models for the early detection of adverse events to previously unseen hospitals.

DESIGN:

Retrospective observational cohort study utilizing harmonized intensive care data from four public datasets.

SETTING:

ICUs across Europe and the United States.

PATIENTS:

Adult patients admitted to the ICU for at least 6 hours who had good data quality.

INTERVENTIONS:

None.

MEASUREMENTS AND MAIN RESULTS:

Using carefully harmonized data from a total of 334,812 ICU stays, we systematically assessed the transferability of DL models for three common adverse events: death, acute kidney injury (AKI), and sepsis. We tested whether using more than one data source and/or algorithmically optimizing for generalizability during training improves model performance at new hospitals. We found that models achieved high area under the receiver operating characteristic (AUROC) for mortality (0.838–0.869), AKI (0.823–0.866), and sepsis (0.749–0.824) at the training hospital. As expected, AUROC dropped when models were applied at other hospitals, sometimes by as much as –0.200. Using more than one dataset for training mitigated the performance drop, with multicenter models performing roughly on par with the best single-center model. Dedicated methods promoting generalizability did not noticeably improve performance in our experiments.

CONCLUSIONS:

Our results emphasize the importance of diverse training data for DL-based risk prediction. They suggest that as data from more hospitals become available for training, models may become increasingly generalizable. Even so, good performance at a new hospital still depended on the inclusion of compatible hospitals during training.

KEY POINTS.

Question: Do deep learning (DL) models trained to detect acute deterioration of ICU patients generalize to new hospitals?

Findings: This retrospective cohort study showed that the performance of DL models may drop significantly when applied at new hospitals, although multicenter training resulted in considerably more robust models. Importantly, sophisticated computational approaches meant to improve generalizability did not outperform simple multicenter training.

Meaning: Purely computational approaches are unlikely to result in reliable risk prediction for the ICU and interoperable training data from a large set of diverse hospitals is needed instead to develop DL-based decision support that ultimately enhances patient care.

Deep learning (DL) is increasingly considered for clinical decision-support (1). Applications in ICUs are especially prevalent, owing to the abundant availability of high-density data like those generated by devices for life support (2). DL prediction models have been proposed for a range of ICU events including ICU mortality (3, 4), acute kidney injury (AKI) (5, 6), and sepsis (7, 8).

Despite their promise, many current DL models do not perform reliably at new hospitals. Models are still overwhelmingly developed on data from a single hospital (9). Exclusive training and evaluation at a single location may have overestimated the performance of models in real-world practice. Hospitals differ markedly in terms of the patients that they treat or the organization of their clinical pathways (10). A DL model may exploit such local patterns of care (11). For example, a local sepsis treatment pathway may prescribe a set of investigations when there is suspicion of infection. Models that merely learn to look for these investigations—rather than the underlying biology of sepsis—perform well at the training hospital but do much worse at new hospitals with different pathways (7, 9). As a result, much of the clinical decision-support software that is currently investigated for use in intensive care may require costly and often infeasible retraining for every new hospital.

Training on data from several hospitals may alleviate this issue (12). Associations that are present at multiple hospitals are more likely to apply to new hospitals too (13). Yet, multicenter studies remain rare due to difficulties of accessing and harmonizing data (1). What is more, little is known on how many hospitals may actually be needed for robust training (12). Multicenter data alone may not even suffice to train robust DL models, with need for additional techniques that explicitly optimize for generalizability (14–18). A better understanding of how we can improve the applicability of DL models for the ICU is crucial to developing trustworthy applications that have an honest chance of being implemented into clinical practice at scale.

In this work, we used carefully harmonized data from four ICU data sources across the United States and Europe to investigate how reliably DL models can be transferred from one hospital to another. We provide results for the three most common ICU prediction tasks: ICU mortality, AKI, and sepsis (9). Using these tasks, we investigate whether training on data from multiple centers improves model performance at new hospitals, and quantify any benefits derived from explicitly optimizing for generalizability. Our results guide the development of ICU risk prediction models that can adapt effectively to diverse clinical settings and ultimately enhance patient care.

MATERIALS AND METHODS

Data Sources

We used retrospective ICU data from four public databases: the Amsterdam University Medical Centers database (AUMCdb), Version 1.0.2 (Amsterdam, The Netherlands) (19), the High Time Resolution ICU Dataset (HiRID), Version 1.1.1 (Bern, Switzerland) (20), the eICU Collaborative Research Database (eICU), Version 2.0 (Boston, MA) (21), and the Medical Information Mart for Intensive Care IV (MIMIC), Version 2.0 (Boston, MA) (22). All datasets represent a single center except for eICU, which combines data from 208 hospitals using the same software system. However, small sample sizes per hospital preclude the study of between-hospital differences in eICU (23) and eICU was considered a single—if diverse—center for the purposes of this study.

Since the datasets were created independently of each other, they did not share the same data structure or vocabularies. To make them usable within a common prediction model, we used data harmonization utilities provided by the ricu R package, Version 0.5.3 (24, 25).

Ethics Statement

Datasets were observational and had preexisting ethical approvals or exemptions that allowed for retrospective analysis (19-22). Access was obtained following the respective requirements for each dataset.

Study Population

We included all patients 18 years old or older that were admitted to the ICU. Each ICU stay was discretized into hourly bins. To guarantee adequate data quality, we excluded stays with: 1) invalid admission and discharge times, 2) less than 6 hours in the ICU, 3) measurements in less than 4 hourly bins, and 4) more than 12 consecutive hours in the ICU without a single measurement (Figs. S1 and S2, http://links.lww.com/CCM/H561). The unit of observation was the ICU stay. As some datasets did not allow to identify which ICU stays belonged to the same patient, all available ICU stays were included in the analysis. Patients could therefore contribute more than one ICU stay. For example, a patient that was discharged and subsequently readmitted to the ICU contributed two ICU stays.

Features and Preprocessing

We extracted 52 clinical features (four static, 48 time-varying) recorded during or before admission to the ICU (Fig. S3, http://links.lww.com/CCM/H561). Static information included age at admission, sex, height, and weight. Time-varying features included vital signs and laboratory measurements (Table S1, http://links.lww.com/CCM/H561). No information on comorbidities, medication, or procedures was used in the models. Features were chosen based on prior literature (5, 8) and availability across datasets.

Biologically implausible feature values were automatically set to missing by the ricu package (24). The raw feature values were screened for systematic differences between datasets. Values were centered and scaled to unit variance. Measurements could be missing, for example, due to the infrequent measurement of expensive laboratory tests. Emulating existing modeling practices in the literature (9), missing values were imputed using a last-observation-carried-forward scheme with missing indicators.

Outcomes

We investigated model generalizability for three prediction tasks (9): all-cause ICU mortality, onset of AKI, and onset of sepsis.

ICU mortality was defined as death while admitted to the ICU. We predicted a patient’s future risk of dying using information from the first 24 hours of the stay, including only patients who were alive and in the ICU after 24 hours. To exclude patients that were about to be discharged or die around the time of prediction at 24 hours—and for which a clinician may be unlikely to use the model in practice—we further excluded patients who died or were discharge between 24 and 30 hours after ICU admission.

Onset of AKI was defined as Kidney Disease Improving Global Outcomes stage greater than or equal to 1, either due to increased serum creatinine or low urine output (Table S2, http://links.lww.com/CCM/H561) (26). At every hour, we predicted a patient’s risk of AKI within the next 6 hours. We excluded stays in which AKI occurred less than 6 hours after ICU admission. We additionally excluded patients with end-stage renal disease defined as baseline creatinine greater than 4 mg/dL (27).

Onset of sepsis was defined according to the Sepsis-3 criteria as organ dysfunction due to infection (28). Organ dysfunction was identified through an increase in Sequential Organ Failure Assessment score of greater than or equal to 2 points. Clinical suspicion of infection was defined as co-occurrence of antibiotic administration and microbiological culture. Due to limited microbiology data in HiRID and eICU, infection in those datasets was defined using only antibiotic administration (8). Again, the patient’s risk of sepsis was predicted hourly for the next 6 hours, excluding sepsis onset before the 6th hour in the ICU.

Further details on outcome definitions can be found in Supplement C (http://links.lww.com/CCM/H561). For AKI and sepsis, all hours until discharge, death, or 7 days were included. If disease onset was determined, the previous 6 hours were labeled positive. The following 6 hours were also labeled positive, penalizing the model for late false negatives, after which follow-up was stopped. Finally, because data quality among hospitals in eICU can be variable (8), we excluded hospitals with no AKI or sepsis cases (two [1.0%] hospitals for AKI and 80 [38.5%] for sepsis).

Statistical Analysis

Prediction Models

We used a neural network consisting of gated recurrent units (29) followed by a simple fully connected network for prediction. We trained all models in PyTorch, Version 1.12.1 (The Linux Foundation, San Francisco, CA), using the implementations by Yeche et al (30). As a sensitivity analysis, we reran the experiments using temporal convolutional networks (31) and transformers (32), again followed by a fully connected network. Further details on the models can be found in Figure S4 (http://links.lww.com/CCM/H561).

Model Training and Hyperparameters

We used five-times repeated random subsampling to train and evaluate our models. At each repetition, we randomly held out 20% of each ICU dataset as test set and split remaining data into training (64%) and validation (16%) sets. Ten random draws of hyperparameters were run for each model type and each of the five resamples (Tables S3 and S4, http://links.lww.com/CCM/H561). Models were trained for a maximum of 1000 epochs using binary cross-entropy and an Adam optimizer. We employed early stopping with a patience of ten epochs. The hyperparameters with the lowest average validation loss across the five resamples were chosen. This model was then evaluated on the held-out test set using the area under the receiver operating characteristic (AUROC), the area under the precision-recall curve (AUPRC), sensitivity, and positive predictive value (PPV). Model calibration was assessed using calibration plots. Calibration was assessed after recalibration using isotonic regression on the validation set.

Generalization Across Data Sources

We assessed each model’s ability to perform well on a dataset it has not previously encountered. This is referred to as “domain generalization” (13). We compared the following training and evaluation strategies (Fig. 1):

Figure 1.

Schematic overview of the experimental setup. CORAL = correlation alignment.

Internal validation

Often, models are merely evaluated on a randomly chosen set of held-out patients from the training hospital (9). This does not assess the model’s likely performance at a future hospital, as the evaluation hospital is already known to the model. We considered two common variants of this practice: 1) training and evaluating on the same hospital (single-center training with internal validation) and 2) training a model on all datasets and evaluating on each hospital separately (multicenter training with internal validation).

Single-center training with external validation

Increasingly, models trained on data from one hospital are evaluated on an external hospital (9). We emulated this practice by evaluating each single-center model on all other hospitals.

Multicenter training with external validation

Multicenter training allows the model to see more data and a more varied pattern of care, which may improve generalizability compared with single-center training. We investigated this by training on data from three hospitals and evaluating on the fourth, left-out hospital (pooled [n–1]).

Multicenter training with generalizability criterion

Several advanced training criteria have been proposed in the literature that promise improved model generalizability through the use of additional loss terms during training (13). Similar in principle to regularization, these additional loss terms penalize models for working very differently at different hospitals. Here, we assessed the ability of five such state-of-the-art criteria to further improve model performance in external data: correlation alignment (CORAL) (14), variance risk extrapolation (VREx) (15), Fishr (16), meta-learning domain generalization (MLDG) (17), and group distributionally robust optimization (GroupDRO) (18), adapting them for the first time to ragged time series typical of ICU data. An in-depth technical description of each criterion and why it was chosen can be found in Supplement D (http://links.lww.com/CCM/H561).

We formally tested for differences in performance between conventional multicenter training and multicenter training with generalizability criterion using paired Wilcoxon rank-sum tests across resampling folds.

RESULTS

We combined and harmonized data for 334,812 ICU stays that occurred between 2003 and 2019. After removing stays from children or from patients with insufficient data (Fig. S1, http://links.lww.com/CCM/H561), we included 311,400 stays in our analysis (AUMCdb: 20,636; HiRID: 32,338; eICU: 182,774; and MIMIC: 75,652).

The median age at admission was 65 years and was the same across datasets (Table 1). There were fewer women than men, with a more pronounced imbalance at European hospitals (~35% women) compared with the United States (~45% women). Not every dataset provided the same information. Patients’ race was only available in U.S. datasets, where three out of four stays were from White patients. ICU stays in European datasets lasted for a median of 24 hours, whereas U.S. stays were roughly twice as long at 48 and 42 hours (MIMIC and eICU respectively).

TABLE 1.

Patient Characteristics

	Amsterdam University Medical Centers Database	High Time Resolution ICU Dataset	eICU Collaborative Research Database	Medical Information Mart for Intensive Care IV
Number of patients	19,790	—a	160,816	53,090
Number of ICU stays	20,636	32,338	182,774	75,652
Age at admission (yr)	65 (55–75)b	65 (55–75)	65 (53–76)	65 (53–76)
Female	7,699 (35)	11,542 (36)	83,940 (46)	33,499 (44)
Race
Asian	—c	—c	3,008 (3)	2,225 (3)
Black	—c	—c	19,867 (11)	8,223 (12)
White	—c	—c	140,938 (78)	51,575 (76)
Other	—c	—c	16,978 (9)	5,514 (8)
Unknown	—c	—c	1,983	8,115
ICU length of stay (hr)	24 (19–77)	24 (19–50)	42 (23–76)	48 (26–89)
Hospital length of stay (d)	—d	—d	6 (3–10)	7 (4–13)
ICU mortality
Number of included stays	10,535	12,859	113,382	52,045
Died	1,660 (15.8)	1,097 (8.2)	6,253 (5.5)	3,779 (7.3)
Onset of acute kidney injury
Number of included stays	20,290	31,772	164,791	66,032
Kidney Disease Improving Global Outcomes ≥ 1	3,776 (18.6)	7,383 (23.2)	62,535 (37.9)	27,509 (41.7)
Onset of sepsis
Number of included stays	18,184	29,894	123,864	67,056
Sepsis-3 criteria	764 (4.2)	1,986 (6.6)	5,835 (4.7)	3,730 (5.6)

^aHigh Time Resolution ICU Dataset (HiRID) only provides stay-level identifiers.

^bSince Amsterdam University Medical Centers Database (AUMCdb) only includes age groups, we calculated the median of the group midpoints.

^cAUMCdb and HiRID did not include information on patients’ race.

^dAUMCdb and HiRID did not include information on hospital length of stay. Numeric variables are summarized by median (interquartile range), and categorical variables are summarized by n (%).

Datasets also differed in terms of outcomes. ICU mortality was lowest in eICU at 5.5% and highest in AUMCdb at 15.8% (Table 1; and Fig. S2, http://links.lww.com/CCM/H561). AKI rates ranged from 18.6% in AUMCdb to 41.7% in MIMIC and were generally lower in European datasets. Sepsis rates ranged from 4.2% in AUMCdb to 6.6% in HiRID.

Internal Validation

Single-center models were able to predict ICU mortality well, with AUROCs ranging from 0.842 ± 0.009 in HiRID to 0.861 ± 0.002 in MIMIC (mean ± se across resampled runs; diagonal in Fig. 2A). Discriminative performance for AKI onset was comparable, ranging from 0.826 ± 0.003 in HiRID to 0.885 ± 0.004 in AUMCdb (Fig. 3A). Sepsis was the most difficult task, with lower performance ranging from 0.741 ± 0.002 in eICU to 0.816 ± 0.004 in MIMIC (Fig. 4A). Due to differences in outcome rates, AUPRC, sensitivity, and PPV varied notably (Figs. 2–4). There was little overfitting in the source data (Table S5, http://links.lww.com/CCM/H561) and no difference between neural network types (Tables S6–S8, http://links.lww.com/CCM/H561). After recalibrating models on the validation set, calibration in the training dataset was reasonable (Fig. S5, http://links.lww.com/CCM/H561), although calibrated probabilities were almost exclusively less than 25% for sepsis. Multicenter training on all datasets performed similar to single-center training, with notable improvements only for sepsis (final row of Figs. 2–4).

Figure 2.

Performance of the ICU mortality prediction model when trained on one dataset (rows) and evaluated on others (columns). A, Area under the receiver operating characteristic (AUROC). B, Area under the precision-recall curve (AUPRC). C, Sensitivity. D, Positive predictive value (PPV). The diagonal represents internal validation, that is, training and test samples were taken from the same dataset. In pooled (n–1), the model was trained on combined data from all except the test dataset. In all, training data from all datasets (including the test dataset) was used during model development. Sensitivity and PPV were evaluated on the stay level and thresholds were chosen based on the validation portion of the training dataset. Performance was averaged across five random resamples. All models used a gated recurrent unit featurizer. AUMCdb = Amsterdam University Medical Centers database, eICU = eICU Collaborative Research Database, HiRID = High Time Resolution ICU Dataset, MIMIC = Medical Information Mart for Intensive Care IV.

Figure 3.

Performance of the acute kidney injury prediction model. A, Area under the receiver operating characteristic (AUROC). B, Area under the precision-recall curve (AUPRC). C, Sensitivity. D, Positive predictive value (PPV). All models used a gated recurrent unit featurizer. AUMCdb = Amsterdam University Medical Centers database, eICU = eICU Collaborative Research Database, HiRID = High Time Resolution ICU Dataset, MIMIC = Medical Information Mart for Intensive Care IV.

Figure 4.

Performance of the sepsis prediction model. A, Area under the receiver operating characteristic (AUROC). B, Area under the precision-recall curve (AUPRC). C, Sensitivity. D, Positive predictive value (PPV). All models used a gated recurrent unit featurizer. AUMCdb = Amsterdam University Medical Centers database, eICU = eICU Collaborative Research Database, HiRID = High Time Resolution ICU Dataset, MIMIC = Medical Information Mart for Intensive Care IV.

External Validation

As expected, discriminative performance decreased when models were applied to datasets that they had not seen before (off-diagonal in Figs. 2A–4A and Tables S6–S8, http://links.lww.com/CCM/H561)—sometimes by as much as Δ_AUROC = 0.210 (AKI: AUMCdb to eICU). Models trained using MIMIC or eICU tended to be more generalizable, particularly among each other. Multicenter training on three datasets (pooled [n–1]) mitigated the performance drop on the fourth, left-out dataset. Pooled models roughly performed on par with the best single-center model. Pooled models frequently overestimated risks, although, leading to worse calibration (Fig. S5, http://links.lww.com/CCM/H561). Changes in other performance metrics were dominated by the differences in outcome rates and more difficult to interpret (Figs. 2–4).

In our analysis, additional generalizability criteria only occasionally performed better in external data than conventional multicenter training. Importantly, no algorithm consistently outperformed multicenter training across tasks and datasets (Table 2). GroupDRO was the only algorithm that never performed worse than conventional multicenter training, achieving statistically significantly higher performance in two of 12 experiments (p < 0.1). The largest improvement from AUC 0.683 ± 0.008 to AUC 0.754 ± 0.002 was seen for VREx when predicting AKI on HiRID data. Notably, the same algorithm led to a considerable drop in performance when applied to other datasets.

TABLE 2.

Performance of Generalization Algorithms Compared With Standard Multicenter Training (Pooled [n–1])

Algorithm	Test Domain
	Amsterdam University Medical Centers Database	High Time Resolution ICU Dataset	eICU Collaborative Research Database	Medical Information Mart for Intensive Care IV
ICU mortality
Pooled (n–1)	0.819 ± 0.005	0.791 ± 0.006	0.827 ± 0.004	0.833 ± 0.002
CORAL	0.816 ± 0.006	0.785 ± 0.011	0.828 ± 0.002	0.835 ± 0.004
VREx	0.822 ± 0.005	0.791 ± 0.007	0.818 ± 0.002	0.824 ± 0.003
Fishr	0.803 ± 0.009	0.679 ± 0.006	0.803 ± 0.003	0.690 ± 0.029
MLDG	0.825 ± 0.003	0.801 ± 0.003	0.820 ± 0.003	0.837 ± 0.001
GroupDRO	0.823 ± 0.007	0.801 ± 0.008	0.826 ± 0.002	0.848 ± 0.002
Acute kidney injury
Pooled (n–1)	0.856 ± 0.004	0.683 ± 0.008	0.804 ± 0.002	0.835 ± 0.001
CORAL	0.851 ± 0.003	0.713 ± 0.003	0.723 ± 0.018	0.822 ± 0.003
VREx	0.802 ± 0.003	0.754 ± 0.002	0.656 ± 0.001	0.746 ± 0.003
Fishr	0.699 ± 0.016	0.730 ± 0.003	0.636 ± 0.011	0.696 ± 0.016
MLDG	0.840 ± 0.006	0.682 ± 0.011	0.743 ± 0.017	0.825 ± 0.003
GroupDRO	0.856 ± 0.004	0.686 ± 0.002	0.815 ± 0.001	0.833 ± 0.001
Sepsis
Pooled (n–1)	0.742 ± 0.009	0.746 ± 0.003	0.711 ± 0.003	0.787 ± 0.004
CORAL	0.738 ± 0.004	0.744 ± 0.003	0.712 ± 0.004	0.783 ± 0.003
VREx	0.674 ± 0.008	0.688 ± 0.006	0.672 ± 0.004	0.728 ± 0.004
Fishr	0.562 ± 0.026	0.616 ± 0.018	0.596 ± 0.013	0.584 ± 0.033
MLDG	0.731 ± 0.008	0.726 ± 0.005	0.711 ± 0.004	0.778 ± 0.004
GroupDRO	0.728 ± 0.009	0.734 ± 0.003	0.714 ± 0.002	0.785 ± 0.004

CORAL = correlation alignment, GroupDRO = group distributionally robust optimization, MLDG = meta-learning domain generalization, VREx = variance risk extrapolation.

Algorithms are trained using data from all datasets except the test dataset. All models shown in this table used a gated recurrent unit featurizer. Performances were compared with that of standard multicenter training using a paired Wilcoxon rank-sum test across folds. Boldface values indicate statistically significantly better, whereas italics indicate statistically worse, performance at p < 0.1.

DISCUSSION

In this large retrospective study, we systematically evaluated how reliably DL models can be transferred from one hospital to another, using four publicly available ICU datasets. Compared with estimates from the hospitals at which the models were trained, we observed a drop in model performance when models were applied at new hospitals. Using data from multiple hospitals during training attenuated the performance gap, providing performance roughly on par with the best single-center model. Dedicated algorithms that optimize for generalizability unfortunately did not further improve performance compared with simply training on data from multiple hospitals.

It is well established that predictive models trained on data from a single-center tend to perform worse at others. We showed that this is also the case for state-of-the-art DL models across the three most common ICU risk prediction tasks. Yet, this lack of model generalizability often goes unnoticed since most models are never externally validated (2, 9, 33). Where external validation is performed, it is usually limited to a single dataset chosen for convenience rather than applicability. As a result, validation studies overwhelmingly rely on the same two U.S. data sources: MIMIC and eICU (9).

While some validation is certainly better than none, we show that good performance in one dataset does not imply similarly good performance in others. Fundamental differences between hospitals or healthcare systems may affect the models’ ability to generalize to a given dataset. For example, it has been previously suggested that European datasets cover sicker patient populations (10). The current practice of validating on arbitrary datasets may thus be of limited practical utility. Instead, targeted validation in datasets representative of the intended hospitals and patient populations will be required to obtain the meaningful estimates of model performance required for trustworthy model development (34).

It is further accepted that models developed on data from many centers likely perform better at new centers compared with those trained only at a single center (12). How many centers are required for robust training of DL-based ICU risk prediction is not immediately clear, though. We showed that including as few as two additional centers during training can already benefit model generalizability. This agrees with previous results that ensembled several single-center models for sepsis prediction (8). Importantly, multicenter models in our study matched the performance of the most generalizable single center, alleviating the need to preselect a single compatible center. If this relationship persists, these results promise increasingly generalizable models as more and more centers are added to the training data.

The use of sophisticated training strategies intended to further improve generalizability—while enticing and relatively easy to apply in practice—did not improve performance over simply training on multiple hospitals. This is in keeping with a recent comprehensive benchmarking study on image classification (13). Early attempts exploring generalizable training for ICU data did not show improvements either, although these early results remained inconclusive due to a lack of sufficiently heterogeneous data (23, 35). Using diverse datasets representative of real-world applications, our results confirm that there is currently no “free lunch” for ICU model generalizability. Existing training schemes are no substitute for data that adequately represents the range of clinical contexts encountered at future hospitals. Unfortunately, this means that if representative training data cannot be obtained, few alternatives remain. Generalizability may still be achieved through causal approaches (36) but the complexity of intensive care medicine often prevents their successful application in practice.

We performed an in-depth evaluation of the generalizability of several state-of-the-art DL architectures across the three most common ICU risk prediction tasks (9). To ensure a diverse and representative sample, we included harmonized data from the most used ICU datasets (10). All models were trained and evaluated via a peer-reviewed benchmarking framework to ensure reproducible results (13). Although the availability of rich, open-source ICU data has improved lately, there were only four such datasets available at the time of analysis (10). It is therefore likely that even the most generalizable model developed in this study will inevitably show poor performance in some future ICU dataset. All data used in this study were from tertiary hospitals in developed Western countries, and generalizability to other settings is likely limited. The selection of inputs was driven by their availability across the included datasets. Inputs were restricted to demographics, vital signs, and laboratory tests, with no information on comorbidities, interventions, or medications. It is worth noting that access to further information could have improved performance at (25) or accounted for differences between individual centers. Conversely, omitting variables that exhibited the largest variations between centers—for example, because they are linked to sepsis pathways—could have improved generalizability at the expense of model performance. The use of more complex imputation techniques such as multiple imputation may have similarly benefited generalizability. Simpler methods were chosen instead to mirror current modeling practices in the literature (9). Owing to its dominance in the domain generalization literature, our experiments were limited to DL. Generalizability of boosting trees—another popular algorithm—was not evaluated. Boosting trees were previously found to have similar same-hospital performance for our tasks (25), and there is little reason to assume that generalizability would differ systematically.

CONCLUSIONS

Our study underscores the critical need for diverse training data when aiming for reliable risk prediction in the ICU. Even with just a few additional centers, we found that DL models became notably more robust compared with single-center training. These results emphasize the importance of ongoing initiatives, such as those by the European Union, to create large multinational datasets. Nevertheless, the models in our study were only as good as the most compatible center included during training. Care must still be taken to curate training sets that reflect the hospitals at which a model is deployed, including smaller nonacademic hospitals and hospitals in rural or underdeveloped areas.

ACKNOWLEDGMENTS

We thank the Scientific Computing of the Information Technology Division at the Berlin Institute of Health and Charité—Universitätsmedizin Berlin for providing computational resources that have contributed to the research results reported in this article.