External Validation, Recalibration, and Extension of a Prediction Model of Early Acute Kidney Injury in Critically Ill Children Using Multicenter Data

Abstract

BACKGROUND:

Acute kidney injury (AKI) is common in critically ill children and is associated with high morbidity and mortality. Risk prediction models designed for clinical decision support implementation can facilitate early identification and proactive mitigation of AKI risk. Existing models have primarily been validated using single-center data, partly because of the lack of appropriately detailed multicenter datasets.

OBJECTIVE:

To determine the performance of a single-center model to predict new AKI at 72 hours of ICU admission in children across two multicenter datasets and refine this model to improve prediction performance while maintaining acceptable alert burden.

DERIVATION AND VALIDATION COHORTS:

We analyzed two datasets: the Pediatric Learning Health System Network–Virtual Pediatric Systems (PEDSNET-VPS) dataset, created through the linkage of PEDSnet electronic health record (EHR) extraction with VPS (LLC, http://www.myvps.org), and the PICU Data Collaborative dataset, created through EHR extraction and harmonization from eight participating institutions. We divided each dataset into a derivation and test split.

PREDICTION MODEL:

We first recalibrated an existing single-center model and measured discrimination (area under the receiver operating characteristic curve [AUROC] and area under the precision-recall curve [AUPRC]) and performance at multiple cutpoints. We next added features available at 12 hours of ICU admission, optimizing by precision and recall. We measured discrimination and performance at multiple cutpoints and identified the features contributing most to the risk score.

RESULTS:

In total we analyzed 186,540 ICU admissions. We found early AKI by serum creatinine criteria within 72 hours of admission in 2.2–2.7%. Initial recalibration of an existing single-center model demonstrated poor discrimination (AUROC 0.65–0.78; AUPRC 0.10–0.12). Following the addition of new features, the model had higher AUROC (0.80–0.88) and AUPRC (0.13–0.22).

CONCLUSIONS:

In this first use of two new multicenter datasets, we found improved performance in a model designed using features available at 12 hours of ICU admission, balancing sensitivity and precision to predict patients at risk for AKI development.

KEY POINTS.

Question: This study aims to leverage two new granular, multi-institution datasets to validate and update an existing single-center model to predict early acute kidney injury (AKI), with a focus on prospective implementation, to improve prediction performance while maintaining acceptable alert burden.

Findings: We validated an existing single-center model in two multi-institution datasets with 55,890 and 130,650 ICU admissions, respectively. Applying existing model coefficients resulted in moderate-to-poor discrimination in both datasets, which improved slightly (area under the receiver operating characteristic curve [AUROC]: 0.65–0.78) by recalibrating using the same feature set. We then updated the model through iterative feature selection and retraining, and improved discrimination in both datasets (AUROC: 0.80–0.88, area under the precision-recall curve: 0.13–0.22). Sensitivity and precision across multiple cutpoints in the test data split were similar or improved compared with existing single-center models.

Meaning: Machine learning models updated in granular, multi-institution datasets may help predict new early AKI when implemented prospectively. Models may be more transportable to sites with new patients and decreased informatics expertise. Implemented models may help clinicians reduce the burden of AKI through intervention and prevention strategies.

Acute kidney injury (AKI) occurs in up to 25% of critically ill children and is associated with high morbidity and mortality (1–3). Importantly, AKI is also more frequent among racial minorities and uninsured children (4). New-onset AKI within the first 72 hours of PICU admission, occurring in only 4% of PICU admissions in one single-center study, may be an important target for intervention (5). Despite calls for more automated and data-driven approaches to preventing AKI (6), its identification largely remains reactive, based on serum creatinine and urine output, which reflect established injury (7). As a result, early opportunities for mitigation strategies, including limiting nephrotoxic medications, fluid balance optimization, and early renal replacement therapies, are frequently missed (6, 8).

Risk prediction models leveraging machine learning techniques coupled with user-centered clinical decision support (CDS) offer an opportunity to identify at-risk patients and intervene earlier (9–11). Existing AKI risk prediction models for critically ill children have been developed and validated primarily on single-center data (5, 12–14). The Renal Angina Index (RAI) is an AKI risk scoring tool that has been evaluated in a multicenter trial (15), but it was developed using expert opinion instead of a data-driven approach, relies on intake and output data which may be inaccurate in the electronic health record (EHR), may represent already established injury in many cases, and missed 18.9% of severe AKI cases during validation (16). Similarly, a quality improvement program to screen nephrotoxic medication exposure demonstrated success when translated across institutions, but is limited only to nephrotoxin-induced AKI and specifically excluded critically ill patients (17, 18).

A key barrier to developing and validating sophisticated AKI prediction models for critically ill children is the lack of sufficiently granular multicenter datasets. Quality improvement and administrative datasets, such as Virtual Pediatric Systems (VPS) and Pediatric Health Information System, while voluminous and standardized, are not sufficiently granular for model training for complex conditions like AKI (19). In addition, these datasets are completed upon patient discharge, but when designing prediction models for implementation, one must rely only on data available at the time the prediction will be made (20). Large adult critical care datasets such as the Medical Information Mart for Intensive Care III (21) and the electronic ICU Collaborative Research Database (22) have enabled development and validation of machine learning (ML) algorithms and have been the basis for a multitude of AKI prediction models in adults (23), but no equivalent dataset exists yet in pediatric critical care.

In this study, we leveraged two new granular, multi-institution datasets comprised of harmonized EHR data to validate and update an existing model to predict early AKI (5). This initial model demonstrated adequate performance (7% precision at the 50th percent cutpoint, 16% precision at the 90th percent cutpoint) but was trained and evaluated only on single-center data. We updated and validated this existing model with a focus on feasibility of implementation in live EHRs. We sought to improve prediction performance as measured by precision and recall. Developing and externally validating this updated model is an important first step toward its prospective real-world implementation to help clinicians reduce the burden of AKI in this population through proven intervention and prevention strategies, such as optimizing fluid balance, renal perfusion, and other modifiable AKI risk factors.

METHODS

Study Population

We included two multi-institution datasets in this study to demonstrate the generalizability and cross-validation of our results. Although the two datasets included different date ranges, available features, and rates of missingness, there were three sites that overlapped within the two datasets. Both datasets are convenience samples of all children admitted to a PICU during the data collection period. In both datasets, we excluded patients younger than 30 days and older than 18 years. The unit of analysis in each dataset is an ICU admission, and to maximize generalizability, we included all ICU admissions for individual patients (i.e., we did not exclude repeat admissions).

The first (Pediatric Learning Health System Network–Virtual Pediatric Systems [PEDSNET-VPS]) was created through the probabilistic linkage of the VPS (LLC, http://www.myvps.org) and the PEDSnet pediatric learning health system (PEDSNET) (24). VPS is a quality improvement dataset comprised of all admissions into the PICU at participating sites. Data are abstracted manually by trained abstractors. Limited laboratory and vital sign data are recorded; however, diagnoses and treatments are recorded reliably through manual chart review. PEDSnet is an EHR extracted and harmonized dataset following a standard common data model (25). We have previously described this probabilistic linkage validation using single-center deterministic linkage as the gold standard (26). The same linkage algorithm was applied across data from six sites (Supplementary eTable 1, https://links.lww.com/CCX/B632). Briefly, site matches were identified by pairwise matching patients from every site in VPS with patients from every site in PEDSNET and identifying the site pair with the lowest aggregate likelihood ratio (suggesting the most accurate matches). Using these site-wise pairs, we then matched patients between VPS and PEDSnet, applying the same likelihood ratio cutoff calculated in our single-center linkage to link patients. Individual sites chose their date range for submission, with most spanning greater than 6 years and the most recent data being from 2017. More contemporary data were unavailable due to funding limitations. One site was excluded due to missing laboratory result timestamps, which were necessary to calculate hours from ICU admission.

The second dataset is comprised of an eight-site dataset from the PICU Data Collaborative (PDC) (27). Briefly, this collaborative is an initiative among academic PICUs from across the United States to aggregate and harmonize granular EHR data for data science use cases. Each site contributes its own extracted and anonymized clinical data for patients admitted to their PICU using an agreed-upon data model. All date elements are converted into time difference measurements (e.g., seconds from hospital admission), and all local identifiers are removed. Data elements are harmonized centrally on an Azure cloud compute instance (Microsoft Azure, Redmond, WA) by content experts from each site. Research computing occurs in Microsoft ML Studio. Individual sites chose the date range for their data submission spanning 2–11 years, with most sites submitting at least 8 years’ worth of data. The dataset presented in this study includes only the first eight PDC sites (the Collaborative has since grown to 19 sites), with admissions up to and including 2022 (Supplementary eTable 1, https://links.lww.com/CCX/B632). Although three sites overlap datasets, due to differences in date ranges, only a small proportion of total ICU admissions will exist in both.

The institutional review board for the University of Rochester determined use of the first dataset to be not human subjects research, and the second dataset to be an exempt study (Study Number 00007184). All procedures associated with this research were conducted in accordance with the ethical standards delineated in the Helsinki Declaration of 1975. We followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD + AI) reporting guidelines for predictive models (Supplemental eTable 2, https://links.lww.com/CCX/B632) (28) as well as specialty-specific reporting guidelines (29).

Study Variables

We analyzed both datasets using the same approach. All features were available in both datasets, unless specifically noted. We collected demographic data, including age at PICU admission and sex, for all admissions. We identified all laboratory measures that resulted during the PICU admission and selected the 44 most common measures across both datasets. We selected the most commonly recorded features, including Fio₂, blood pressure (systolic blood pressure [SBP], diastolic blood pressure, and mean arterial pressure), oxyhemoglobin saturation, heart rate, respiratory rate, temperature, weight, and height. We identified diagnoses for immunocompromised conditions based on the International Classification of Diseases (ICD), Ninth Revision and ICD-10 codes (30, 31). Several features were only available in the PEDSNET-VPS dataset, including preadmission cardiac arrest and admission for postoperative recovery, both of which were coded in the VPS dataset. In addition, from the PEDSNET-VPS dataset we were able to reliably abstract medications and obtain counts of nephrotoxic medications by matching each medication name to its Anatomical Therapeutic Chemical classification (32). Medication administration data were not available in the PDC dataset at the time of analysis. Neither dataset includes measures of geocoded socioeconomic status (e.g., Child Opportunity Index). We a priori chose not to include race or ethnicity as abstracted variables in our predictive model given the lack of biological plausibility related to race and ethnicity being associated with AKI, as opposed to social constructs potentially reflecting issues such as poor access to care and systemic racism, and to avoid including social determinants of health in a clinical decision-making use case that could exacerbate systemic bias (33).

Data Split

Each dataset was divided into three splits, termed training, validation, and test, using temporal and site-specific sampling. In the PEDSNET-VPS dataset, the training (60%) and validation (20%) splits included all ICU admissions before June 1, 2016. The remaining 20% of ICU admissions after June 1, 2016 were included in the test split. In the PDC datasets, the training split included all ICU admissions before 2019, the validation split included ICU admissions in 2019–2020, and the test split included ICU admissions in 2021 and beyond, as well as admissions from all dates at one site (Supplemental eFig. 1, https://links.lww.com/CCX/B632). Although datasets represent a convenience sample, we compared the sample sizes within each split to previous reports of early AKI incidence to ensure adequate cases for prediction.

AKI Definition and Outcomes

We followed the Early AKI definition used in the original model by Sanchez-Pinto and Khemani (3, 5), based on the Kidney Disease Improving Global Outcomes (KDIGO) staging criteria using serum creatinine (SCr). Briefly, we calculated baseline SCr using a staged approach. We first attempted to identify the most recent SCr value within the prior 90 days. If that measure was unavailable, we calculated the height-dependent SCr; if no height measure was available, we calculated the height-independent SCr (34). Outliers were removed based on exceeding two SDs from median (35). From these baseline SCr measures, we calculated the AKI stage at each time point a new SCr was obtained during the first 3 days of the ICU stay. Patients were excluded from subsequent analysis if they met criteria for AKI on ICU admission or within 12 hours after admission (based on SCr at 12 hr), or if their ICU length of stay was less than 12 hours. Each remaining ICU admission was assigned the maximum stage AKI achieved between hours 12 and 72 after admission, from stage 0 (or no AKI) through stage 3. For modeling purposes, we defined AKI as stage 1, 2, or 3 and did not separate out stage 2 of 3, given the very low incidence. Admissions with no measured creatinine values during this period were classified as stage 0 (no AKI). Neither dataset includes reliable intake and output measures, we did not include measures of fluid balance or urine output-based AKI.

Single-Center Model Recalibration

We first measured the performance of the Early AKI prediction model in our two datasets (5). We used the initial coefficients in the original publication, and then recalibrated the same model (5). Importantly, two of the seven variables (postoperative care and preadmission cardiac arrest) were not available in the PDC dataset. We did not impute or rescale any data for this validation; similar to the initial single-center model, missing variables were simply not added to the score. Only features available up to 12 hours after ICU admission were included in the model, choosing the value that resulted in the worse score if multiple values were available. We then recalibrated and updated the coefficients by training a multivariable logistic regression model on our training split in each dataset, and reporting results of the model on the test split in each dataset. For this and subsequent models, we computed the area under the receiver operating characteristic curve (AUROC) to assess discrimination and the area under the precision-recall curve (AUPRC) to assess precision, or positive predictive value (PPV), which may be more informative for modeling problems with class imbalance (36). We report the sensitivity, specificity, and PPV at the 50th and 90th percentiles of the risk score in each dataset, and compare these to the validation cohort reported in the original study (5).

Additional Feature Selection and Model Training

We next identified additional features, retrained and re-evaluated a set of models to improve performance. We first a priori defined additional physiologically plausible features for inclusion. Additional features include laboratory values, vital sign measurements, and the presence of an immunocompromised diagnosis as described above. Again, we only included results and measurements available up to 12 hours after ICU admission. We aggregated repeated measures such as vital signs in several ways (e.g., maximum, minimum, mean, median), resulting in multiple candidate feature values for each measure. We measured pairwise collinearity between all feature pairs using the maximum of Pearson, Kendall, and Spearman correlation coefficients, and removed one feature from any pair where this coefficient was above 0.8. We encoded categorical features, such as the presence of an immunocompromised diagnosis, using a binary encoding approach (i.e., yes vs. no). All continuous features were scaled to minimize magnitude bias.

We performed feature selection alongside imputation and model selection within the development split of each dataset. We imputed using either median imputation or median + last recorded value imputation, whereby the last recorded laboratory value was used instead of the median value from the test split. We compared model performance when targeting optimal AUROC vs. AUPRC, and we compared four different model types: decision tree, random forest, penalized logistic regression, and support vector machines. We began the feature selection process with an initial feature set comprised of the seven features with the best univariate performance as independent predictors of AKI. We then added one feature at a time to identify those that increased the target measure performance. Then, using backward elimination, we removed features if their absence did not decrease model performance. Finally, we trained each of the different model types on each feature set. Through this search space, we were able to identify the optimal feature set, imputation method, and model type. Once we identified these optimal parameters, we performed final model hyperparameter tuning in the validation split of the datasets.

Statistical Analysis

We performed initial dataset linkage (PEDSNET-VPS) and descriptive analysis in R, Version 4.2.1 (R Foundation for Statistical Computing, Vienna, Austria), including packages “dplyr” and “ggplot” (37, 38). Subsequent feature engineering, selection, model creation, and analysis were completed in Python version 3.8 using packages “NumPy,” “pandas,” and “SciPi” (39–41).

We assessed model performance in the test split of each dataset. We measured model discrimination as AUROC and AUPRC and calculated 95% CIs using bootstrapping. We tested differences between curves on the same dataset using DeLong’s test; however, we could not test significance between datasets as they are maintained on separate, secure resource compute environments (42). We calculated plot metrics including sensitivity, PPV, and number needed to alert (NNA, the inverse of PPV) across all potential cutpoints. We also assessed these metrics in the validation and test splits at each of the 50th and 90th percentile cutpoints, defined in the development split within each dataset. We cross-validated the final PDC model in the PEDSNET-VPS dataset and report the same metrics at the 50^th and 90^th percentile cutpoints. We report feature significance as a contribution to the model based on Shapley Additive exPlanation (SHAP) scores. All source code for this project is available under the MIT license on GitHub (Microsoft Corporation, Redmond, WA): https://github.com/Lab-for-Integrated-Decision-Support/vps-pdc-aki-public.

RESULTS

Epidemiology and Demographics

We identified a total of 67,167 ICU admissions across five sites in the PEDSNET-VPS dataset. Similarly, we identified a total of 154,174 ICU admissions across eight sites in the PDC dataset. After excluding admissions based on patient age, length of ICU stay less than 12 hours, and AKI at admission or within the first 12 hours, the datasets had 55,890 (83.2%) and 130,650 (84.7%) eligible ICU admissions, respectively. The incidence of new AKI by 72 hours was 2.2% (1,240 admissions) and 2.7% (3,535 admissions) in the datasets, with greater than 60% being stage 1 AKI in both datasets. The PDC dataset contained patients who were generally younger at admission, had longer median ICU length of stay, and higher mortality (Table 1).

TABLE 1.

Demographic Features of the Two Cohorts of ICU Admissions

Variable	Pediatric Learning Health System Network–Virtual Pediatric Systems	PICU Data Collaborative
All ICU encounters	82,457	154,174
Excluded Encounters
Incomplete laboratory result timesa	15,290 (18.5%)	—
ICU length of stay < 12 hra	7,837 (11.7%)	15,454 (10.0%)
AKI < 12 hr after ICU admissiona	3,440 (5.7%)	8,070 (5.8%)
ICU encounters meeting inclusion (% of all ICU encounters)	55,890 (67.8%)	130,650 (84.7%)
Unique patients	34,686	86,202
Early AKI by 72 hr (%)	1,240 (2.2%)	3,535 (2.7%)
KDIGO AKI stage 1 (%)	762 (1.4%)	2,927 (2.2%)
KDIGO AKI stage 2 or 3 (%)	478 (0.8%)	630 (0.5%)
Age, median years (interquartile range)	4.2 (1.2–10.9)	3.2 (0.7–10.2)
Gender, female (%)	24,679 (44%)	58,247 (45%)
PICU length of stay, median days (IQR)	1.6 (0.8–3.5)	2.0 (1.1–4.7)
Hospital mortality (% of hospital encounters)	708 (1.3%)	2,002 (2.1%)
Cohort split
Development	33,300 (60%)	76,762 (59%)
Validation	8,326 (15%)	27,132 (21%)
Test	14,264 (20%)	26,756 (20%)

AKI = acute kidney injury, KDIGO = Kidney Disease Improving Global Outcomes.

^aPercentage is calculated as a percent of the remaining encounters after subtracting the prior exclusions. For example, the percentage of Pediatric Learning Health System Network–Virtual Pediatric Systems encounters with “ICU length of stay < 12 hours” is as a percentage of those eligible after excluding those encounters with “incomplete laboratory result times.”

Dash indicate that there were no incomplete laboratory result times in the PICU Data Collaborative dataset.

Single-Center Model Performance

We applied existing coefficients from the original early AKI prediction model in both datasets. The AUROC was 0.74 in the PEDSNET-VPS dataset and 0.39 in the PDC dataset, which is missing two of the seven features as previously described. After recalibration in the training split, the AUROC in the PEDSNET-VPS dataset was 0.78 [95% CI, 0.78–0.78] and in PDC was 0.65 [95% CI, 0.64–0.65]. Similarly, at both the 50th and 90th percentile cutpoints, sensitivity and PPV were lower than the original single-center model’s validation dataset (Table 2).

TABLE 2.

Discrimination and Accuracy of the Single-Center Model Recalibrated on the Development Split of Each Dataset

Variable	Validation #2 Dataset, Sanchez-Pinto and Khemani (5)a		Pediatric Learning Health System Network–Virtual Pediatric Systems		PICU Data Collaborativeb
Total, n	2,832		14,264		26,756
Early acute kidney injury (%)	114 (4)		303 (2)		574 (2)
Area under the receiver operating characteristic curve (95% CI)	0.86 (0.85–0.88)		0.78 (0.78–0.78)		0.65 (0.64–0.65)
Area under the precision-recall curve (95% CI)	Not reported		0.12 (0.12–0.12)		0.10 (0.10–0.11)
Cutpoint percentile	50th	90th	50th	90th	50th	90th
Test positive at cutpoint (%)	1,578 (56)	493 (17)	7134 (50)	1427 (10)	13,277 (50)	2,723 (10)
Sensitivity % (95% CI)	96 (90–98)	71 (62–79)	85 (81–89)	37 (32–42)	66 (63–70)	29 (25–32)
Specificity % (95% CI)	50 (50–50)	86 (86–86)	51 (50–52)	91 (90–91)	50 (50–51)	90 (90–91)
Positive predictive value % (95% CI)	7 (7–7)	16 (14–18)	4 (3–4)	8 (6–9)	3 (3–3)	6 (5–7)

^aFor comparison we include results from the Validation Number 2 dataset from the original publication, Sanchez-Pinto and Khemani (5)

^bTwo of the seven features from the single-center model (preoperative status and cardiac arrest before admission) were unavailable in this dataset.

Results are reported from the test split in each dataset.

Expanded Model Derivation and Performance

We identified 45 total laboratory tests for possible inclusion, along with 10 vital signs (Supplemental eTable 2, https://links.lww.com/CCX/B632). Imputation and target metric search identified last value + median imputation targeting AUPRC, resulting in the best performance measures (Supplementary eTable 3, https://links.lww.com/CCX/B632). We chose logistic regression for the final model as it provided an optimal combination of metric performance and explainability. In the test splits, AUROC was 0.80 (95% CI, 0.80–0.80) in PEDSNET-VPS and 0.88 (95% CI, 0.88–0.88) in PDC. However, the AUPRC was higher in the PDC dataset (0.22, 95% CI, 0.22–0.22) compared with PEDSNET-VPS (0.13, 95% CI, 0.13–0.13) (Fig. 1 and Table 3). Accordingly, at the 90th percentile cutpoint, PPV was higher in the PDC dataset (12%) compared with the PEDSNET-VPS dataset (10%) (Fig. 2). Sensitivity and PPV were lower at both cutpoints in the cross-validated PEDSNET-VPS dataset when applying the PDC model (Table 3). Features most contributing to the prediction score included calcium, lactate, creatinine, and mean SBP, all measured at 12 hours of admission (Supplementary eTable 4 and eFig. 2, https://links.lww.com/CCX/B632).

Figure 1.

Area under the precision-recall curve (AUPRC) and area under the receiver operating characteristic curve (AUROC) of the best-performing logistic regression model following the addition of new features. Columns represent the receiver operating characteristic (ROC) curves (A, C) and precision-recall curves (B, D) for each dataset. Rows represent the Pediatric Learning Health System Network–Virtual Pediatric Systems (PEDSNET-VPS) dataset (A, B) and the PICU Data Collaborative (PDC) dataset (C, D). A, PEDSNET-VPS Test split AUROC: 0.80. B, PEDSNET-VPS Test split AUPRC: 0.13. C, PDC Test split AUROC: 0.88. D, PDC Test split AUPRC: 0.22. All curves report data from the validation (solid line) and test (dashed line) splits. FPR = false positive rate, TPR = true positive rate.

TABLE 3.

Final Model Diagnostic Performance for the Optimal (“Best”) Model With the Optimal Feature Set, Including Only Those Features Able to be Extracted From the Electronic Health Record Using Fast Healthcare Interoperability Resource-Based Calls Within the First 12 Hours of ICU Admission

Variable	Pediatric Learning Health System Network–Virtual Pediatric Systems				PICU Data Collaborative
	Validation		Test		Validation		Test		Test 2 (Pediatric Learning Health System Network–Virtual Pediatric Systems)
Total, n	8,326		14,264		27,132		26,756		14,264
Area under the receiver operating characteristic curve (95% CI)b	0.80 (0.80–0.80)		0.80 (0.80–0.80)		0.89 (0.89–0.89)		0.88 (0.88–0.88)		0.62 (0.62–0.62)
Area under the precision-recall curve (95% CI)b	0.13 (0.13–0.14)		0.13 (0.13–0.13)		0.24 (0.23–0.24)		0.22 (0.22–0.22)		0.03 (0.03–0.03)
Cutpoint percentileb	50th	90th	50th	90th	50th	90th	50th	90th	50th	90th
Risk test positive (%)	4,135 (51)	854 (10)	6,924 (49)	1,418 (10)	13,007 (48)	3,734 (14)	14,617 (55)	3,365 (13)	7,756 (54)	873 (6)
Sensitivity	87	48	83	47	94	75	95	72	82	39
Specificity	51	91	52	91	53	88	46	89	46	95
Precision	4	10	4	10	5	14	4	12	3	13
Number needed to alert or 1/Precision	26	11	26	11	20	7	27	9	32	8

^bCIs were calculated by ten-fold sampling (with replacement) of 3000 samples, followed by sorting results and calculating the 0.025 and 0.975 percentiles.

^bFor each model (Pediatric Learning Health System Network–Virtual Pediatric Systems [PEDSNET-VPS] and PICU Data Collaborative [PDC]), cutpoint percentiles were established on development split of each dataset and applied to additional datasets. For PDC model, cutpoints from development split were applied to PDC Test split as well as PEDSNET-VPS Test split.

Figure 2.

Sensitivity, specificity, and precision across the range of cutpoints for the best-performing logistic regression model in each dataset, reported from the test split. Datasets are indicated for each column (PEDSNET-VPS: A, C, E; PDC: B, D, F). In each dataset, the 50th and 90th percentile cutpoints are indicated by vertical lines. All curves report data from the validation (solid line) and test (dashed line) splits. PDC = PICU Data Collaborative, PEDSNET-VPS = Pediatric Learning Health System Network–Virtual Pediatric Systems.

DISCUSSION

We present the first use of two new multicenter pediatric critical care datasets to evaluate, recalibrate, extend, and validate a machine learning-based prediction algorithm for new early AKI. We first applied an existing single-center model of early AKI prediction across the datasets and found poor discrimination and performance compared with the initial single-center validation. After the addition of new features that would be available at the time of the prediction, we found improved discrimination and performance. The development of the new model focused on implementability, selecting features commonly available at 12 hours after admission and targeting metrics balancing both sensitivity and PPV.

Several other risk prediction algorithms have been developed to identify AKI, but each has limitations (11). Sanchez-Pinto and Khemani (5) created a parsimonious score to predict early AKI from single-center data; when we applied this score across multicenter data, the performance decreased. In addition, this algorithm relies on data that may not be stored in discrete variables within the EHR (e.g., preadmission cardiac arrest), limiting its ability to be deployed in real-time. Similarly, Wang et al (12) developed a single-center derived model to predict AKI risk throughout the ICU stay. They report a much higher proportion of AKI (470/ 866, or 54%) in their validation cohort, as they included cases beyond the first 72 hours of the ICU admission and reported a lower AUROC of 0.74 (95% CI 0.71–0.77) in the ICU cohort. Dong et al (14) created a model based on three independent centers’ EHR data using engineered features to predict AKI development within 30 hours. Although they report excellent discrimination (AUROC 0.89) and an impressive PPV (0.47), these results may not translate to real-time implementation. They focused on encounter-level analysis of patients who did and did not develop AKI, such that control encounters (e.g., those ICU admissions where no AKI was observed) were only counted once. To provide an accurate measure of implemented PPV, and thus alert burden, authors must account for this model running in real-time across every analysis window for every admitted patient, which would substantially increase the number of possible true and false negatives.

In addition to the above machine learning-based prediction algorithms, risk scores have been developed and validated in general PICU patients as well as in specific subpopulations. The RAI combines patient risk factors and potential renal injury early in the PICU admission to identify patients at risk of developing or having persistent AKI three days after PICU admission (e.g., 72–96 hr). The score incorporates SCr measurement and percent fluid overload as well as features of critical illness (e.g., mechanical ventilation and inotropy use). Initial validation studies included patients with AKI at admission as well as patients with sepsis and thus higher rates of early AKI (10.2–19.4%) (15). A follow-up validation study from prospectively obtained multicenter data also included sicker patients with higher mortality (5.1%) and higher rates of severe AKI by ICU day 3 (23.1%) (16). In this population, less reflective of the general ICU admissions, whereas RAI demonstrated a reasonable PPV (121/286, or 42%), sensitivity was poor (121/368, or 32%), and 18.9% of patients with severe AKI were missed. Incorporating biomarker data and focusing on specific subpopulations (e.g., sepsis) improved performance but limits use in broader ICU admissions (43, 44).

Multicenter validation of single-center predictive models is critical for generalization outside of the original development site. Because AKI occurs across the sociodemographic spectrum, prediction solutions should be generalizable and shareable to cross the “digital divide” and improve care equity (45). As we show here, and others have shown, single-center models often do not show adequate performance in external validation, even with recalibration (46). Rockenschaub et al (46) compared single-center derived models across two adult ICU EHR-derived datasets, and even the addition of newer generalizability criteria did not improve performance substantially. Unfortunately, we are limited by the availability of sufficiently complete, harmonized datasets in pediatric critical care to perform this kind of external validation (19). We anticipate that efforts like the PDC may serve an important role in the development, recalibration, and validation of generalizable models (27). Transportability, or adequate performance across multiple sites, is critical for trial design and testing as well as implementation in new patient populations, specifically at sites without informatics expertise, such as potentially less-well-resourced institutions (47).

We intentionally focused model development and feature selection to design for implementation. Only a very small percentage of published models are translated into the clinical environment through real-time CDS (29). As highlighted in the recent Phoenix Criteria for Sepsis and Septic Shock in Children study, the AUPRC is more informative than the AUROC for imbalanced problems (i.e., with rare events of interest) and in acute care use cases such as ours, where sensitivity and PPV provide more actionable insights (e.g., NNA, false alert burden, missed cases, etc) (36). We report PPV and NNA in addition to sensitivity, as these measures are important to clinicians who may be receiving these alerts. Specifically, at our highest sensitivity cutoff, with a PPV of 13% and NNA of 8%, clinicians would be exposed to 7% false positives for every 1% true positive result. Future work includes user-focused decision support design based on model results, guided by the need to minimize alert burden given our PPV and NNA results, to improve the actionability and usability of this predictive model (48). The addition of urine biomarkers in a tiered approach, similar to the TAKING FOCUS 2 trial, may allow for acceptance of higher rates of false positives to enrich the population for subsequent biomarker stratification and intervention (e.g., furosemide stress test) assessments (49). We could not include biomarkers (e.g., Neutrophil gelatinase-associated lipocalin, NGAL) as features or endpoints in our model, given their sparse representation in both datasets. We have implemented our current model prospectively using a Fast Healthcare Interoperability Resource-based interface and are currently silently evaluating model performance prospectively. Although CDS-based prediction models may improve screening rates (50, 51), not all early identification improves clinical outcomes (52). Future studies should incorporate pragmatic clinical trial design across multiple institutions, using generalized, shareable, user-designed design support, possibly incorporating biomarkers and protocolized care bundles.

This study had several limitations. First, one of our datasets (PEDSNET-VPS) included a large proportion of excluded subjects due to missing data and did not include data more recent than 2017. Neither dataset currently includes intake or output (IO) fluid measurements, and therefore, we could not compute AKI based on all KDIGO criteria, nor could we compare the model to the RAI score. Both datasets currently lack harmonized medication data to accurately report continuous infusions of vasoactive medications as well as harmonized mechanical ventilation data. There was overlap among three of the sites in these multi-institution datasets, with the reduced time window (e.g., up to 2017 data) in the PEDSNET-VPS dataset; we did not exclude these overlapped sites. We anticipate that with the continued addition of new data into the PDC dataset, including IO fluid measurements, harmonized medications, and harmonized mechanical ventilation, we will be able to update, recalibrate, and revalidate our model.

CONCLUSIONS

In the first use of two new multicenter datasets specifically designed for machine learning development and validation, we demonstrate improved discrimination and performance over recalibration of a single-center model. We designed our model and selected features for implementation, balancing sensitivity and PPV to both capture patients at risk for new early AKI while balancing alert burden to clinicians. Ongoing work includes prospective silent validation in our intensive care environment as well as continuous model improvement through the addition of new temporal and site-specific data.

ACKNOWLEDGMENTS

The authors would thank all those who supported data collection, aggregation, and harmonization at each site. We thank the Virtual PICU participating centers and research coordinators, as well as the Pediatric Learning Health System Network (PEDSnet) data coordinating center analysts, programmers, and administrative support personnel. In addition, we thank the PICU Data Collaborative project manager, Jeffrey Terry, and data team members, including Alysia Flynn, Ruiqi Huang, Eugene Laksana, Ishmael Obeso, Michael Reilly, Alexsandra Kretsu, and Long Ho.

Virtual Pediatric Systems (VPS) data was provided by VPS, LLC (http://www.myvps.org). No endorsement or editorial restriction of the interpretation of these data or opinions of the authors has been implied or stated. This article has been reviewed by the VPS Research Committee.

The research reported in this publication was conducted using PEDSnet, A Pediatric Clinical Research Network. This publication includes data from the PEDSnet institutions listed in Supplementary eTable 1 (https://links.lww.com/CCX/B632).

Supplementary Material

cc9-8-e1425-s001.pdf ^{(2MB, pdf)}