Significance Statement
Because AKI in hospitalized children is associated with poor outcomes, a tool allowing early identification of children at risk of developing AKI may facilitate timely interventions. The authors describe various machine learning techniques used to build a parsimonious model predictive of pediatric AKI. From an initial pool of 720 potential variables, they evaluated multiple feature selection techniques to create a ten-feature logistic regression model that could predict, in time-updated fashion, the risk of AKI in the next 48 hours. A machine learning-based genetic algorithm (reflecting the process of natural selection) was the best variable selection method, using ten factors extracted from electronic health records to use for AKI prediction. Risk-stratifying hospitalized children might allow clinicians to implement targeted and timely interventions prior to AKI development.
Keywords: acute kidney injury, pediatrics, electronic health records, risk, feature selection
Visual Abstract
Abstract
Background
Timely prediction of AKI in children can allow for targeted interventions, but the wealth of data in the electronic health record poses unique modeling challenges.
Methods
We retrospectively reviewed the electronic medical records of all children younger than 18 years old who had at least two creatinine values measured during a hospital admission from January 2014 through January 2018. We divided the study population into derivation, and internal and external validation cohorts, and used five feature selection techniques to select 10 of 720 potentially predictive variables from the electronic health records. Model performance was assessed by the area under the receiver operating characteristic curve in the validation cohorts. The primary outcome was development of AKI (per the Kidney Disease Improving Global Outcomes creatinine definition) within a moving 48-hour window. Secondary outcomes included severe AKI (stage 2 or 3), inpatient mortality, and length of stay.
Results
Among 8473 encounters studied, AKI occurred in 516 (10.2%), 207 (9%), and 27 (2.5%) encounters in the derivation, and internal and external validation cohorts, respectively. The highest-performing model used a machine learning-based genetic algorithm, with an overall receiver operating characteristic curve in the internal validation cohort of 0.76 [95% confidence interval (CI), 0.72 to 0.79] for AKI, 0.79 (95% CI, 0.74 to 0.83) for severe AKI, and 0.81 (95% CI, 0.77 to 0.86) for neonatal AKI. To translate this prediction model into a clinical risk-stratification tool, we identified high- and low-risk threshold points.
Conclusions
Using various machine learning algorithms, we identified and validated a time-updated prediction model of ten readily available electronic health record variables to accurately predict imminent AKI in hospitalized children.
AKI develops in approximately 30% of hospitalized children in intensive care units (ICUs) and 5% in non-ICU settings.1–5 Hospital-acquired AKI is associated with a longer hospital stay, higher mortality, and an increased economic burden.1–4,6 AKI may also increase the risk of long-term complications such as CKD, proteinuria, and hypertension.3 AKI management strategies are largely supportive and include avoiding nephrotoxin exposure, optimizing volume status, and addressing electrolyte imbalances.4 Prior studies have suggested that the early identification of patients at risk of developing AKI and subsequent nephrotoxin avoidance can decrease the AKI rate by 64%.7
The AKI prediction workgroup of the Acute Dialysis Quality Initiative recommended creating electronic health record (EHR)-integrated, real-time AKI prediction models that combine risk factors from prototype prediction models with novel risk factors using machine learning methods.8 Current AKI prediction models have helped build our understanding of the field of AKI prediction in children, but they have limitations as they mainly rely on baseline admission data, were developed with a limited set of AKI predictors, and do not include neonates.9,10 Although neonates have a lower GFR as compared to older children, they share many similar AKI risk factors.4,11 Additionally, none of the current prediction models were created in an unbiased clinically agnostic approach leveraging the diversity of variables available in the EHR.9,10,12
The EHR contains a wealth of variables that may predict AKI, but brute-force methods to evaluate parsimonious models are computationally infeasible. For example, there are 9.69*1021 models containing 10 variables that could be created from a set of 720 variables (as we had in this study). Even assuming one could evaluate 1,000,000 such models per second using logistic regression (infeasible with even modern supercomputers), it would take more than 300 million years to find the best possible combination of 10 variables in a space with 700+ potential variables. Therefore, researchers are often forced to create parsimonious models based on prior research or clinical intuition, which is biased against novel predictors. Several algorithms, including forward selection, Least Absolute Shrinkage and Selection Operator (LASSO) regression, and others, have been proposed to more efficiently find parsimonious models that may allow for novel features to be included.13 This study presents several such methods of narrowing the difficulty of the problem. We aimed to create a novel AKI prediction model that is time-updated, is parsimonious, and could be used in all hospitalized neonates and children to predict AKI within the next 48 hours.
METHODS
Study Design and Population
We retrospectively reviewed the medical records of all children younger than 18 years old, who had at least two creatinine values measured at any time during a hospital admission from January 2014 to January 2018 to two hospitals in the Yale-New Haven Health System, a large, academic, tertiary care center. The two hospitals are Yale-New Haven Children’s Hospital and Bridgeport Hospital, the latter being a hospital affiliated with the same Health System, but staffed by other physicians and caring for a significantly less acutely ill pediatric population. Patients with an initial serum creatinine level <4 mg/dl or an eGFR of >15 ml/min per 1.73 m2 were excluded. We calculated eGFR using the modified Schwartz equation, or if height was not available, using the Full Age Spectrum equation.14 The Yale Human Investigation Committee reviewed and approved the study protocol under a waiver of informed consent.
Dependent (Outcome) Variables
The primary outcome was the development of AKI within the next 48 hours, updated throughout the hospital stay. We used the Kidney Disease Improving Global Outcome creatinine definition using a rolling window that compared the current creatinine to the lowest in the past 48 hours or 7 days. Stage 1 AKI was defined as a 0.3 mg/dl increase in creatinine within 48 hours or an increase of 1.5 times the lowest measured creatinine within the prior 7 days.4 Secondary outcomes included severe AKI (stage 2 or 3) on the same time scale, inpatient mortality, and length of stay. Stage 2 and 3 AKI are defined as an increase in creatinine level by 2–2.9 times and an increase ≥3 times baseline, respectively.4 Urine output was not used to define AKI due to missingness and expected difficulty in application in real time. To qualify for our AKI definition, an absolute value of serum creatinine >0.5 mg/dl was also required.11,15 This avoids clinically dubious changes in creatinine (such as from 0.2 mg/dl to 0.3 mg/dl) being considered diagnostic of AKI. To limit the number of imputed values and to make predictions similar to what would be done in prospective implementation, we excluded timepoints prior to the first creatinine measurement.
Independent (Predictor) Variables Selection
We included a total of 720 candidate variables extracted from the EHR including patient demographics, vital signs, patient locations, diagnoses, laboratory results, and medication exposures (Figure 1, Supplemental Table 1). We excluded from analysis highly collinear variables with a correlation coefficient r of 0.95 or greater. For continuous variables with data missing for <25% of patient encounters, measured values were carried forward until remeasurement occurred. For timepoints with no previous measurement, we imputed derivation-set medians. We transformed continuous variables with >25% missing data into binary markers indicating if the variable had ever been measured up to the current timestamp. For example, chloride level, which was missing in 1.6% of encounters, was modeled as the measured value of the derivation cohort median (102 mEq/L), while lactate (missing in 66% of encounters) was modeled simply as having been measured at the current point in the hospitalization or not yet having been measured (regardless of the lactate value). This simple imputation strategy was chosen to facilitate adoption of the model into the EHR, as multiple imputation techniques are not applicable in real-time, prospective analyses. We transformed medication exposure (and procedure) variables into binary markers to indicate exposure to the medication at the current timestamp and once exposed, this status was continued until the end of encounter.
Figure 1.
Candidate variable groups. A total of 720 candidate variables available in the EHR were included to create a parsimonious predictive model.
We used multiple feature selection strategies to select a parsimonious set of ten features from the entire feature set. We selected ten features as each additional feature introduces greater complexity to EHR-integration (as over time, data points may be changed or require different cleaning strategies). We also performed logistic regression analyses of incremental numbers of variables and observed an increasing model performance as additional variables were added, but there were diminishing returns after the first ten variables (Supplemental Figure 1). Selection strategies were as follows:
Univariable Area under the Receiver Operating Characteristic Curve Ranking
Each of the 720 features were ranked according to the Univariable Area under the Receiver Operating Characteristic Curve (AUC) after a univariable logistic regression. The top ten features were then chosen.
Forward Selection
We built a logistic regression model that sequentially added the most statistically significant variables from the remaining pool, one at a time, until a model of ten variables was obtained.16
LASSO Logistic Regression
This is a form of penalized logistic regression that promotes sparsity over the model coefficients.13,17 In LASSO regression, the process of feature selection occurs when a penalty (λ) level is chosen and applied on regression coefficients. The least significant variables will be forced to have coefficients of zero and therefore will be eliminated. We chose a penalty level that allowed for selection of only ten variables to create this model.
Clinical Model
We manually selected ten variables highly associated with the development of AKI in children as determined by the research team’s clinical expertise and prior research.2,4 Previous research has identified that mechanical ventilation, acidosis, elevated BUN, and vasoactive support are associated with AKI in children.1,2,18 Studies have also shown that children admitted to an ICU are at an increased risk of AKI as compared with those not admitted to an ICU.1,4,11,12,19 Hyper- and/or hypocalcemia were previously reported to be associated with AKI development.20 Lastly, recent EHR-based AKI prediction models and AKI severity scores have included acidosis, BUN, changes in weight, fluid overload, chemotherapy medications, ICU admission, and ventilatory and inotropic support in their prediction models.9,10,12,13
Genetic Algorithm
Genetic algorithms are a machine learning technique in the domain of evolutionary optimization; to prevent confusion, we stress that this study’s usage of genetic algorithms does not use any biologic genetic data. Our genetic algorithm models parsimonious groups of ten features as “individuals” or genotypes in a population. Each genotype corresponds to 10 features chosen, initially at random, from the 720-feature pool. Each genotype is assigned a fitness score; in this case, this corresponds to the AUC with logistic regression using only the feature set given by the genotype. The fittest individuals are preferentially allowed to exchange genetic information (mate) to create offspring, creating a next generation with the same population size. Algorithmically, this translates to each child having ten features randomly selected from the combined features present in the parents (genetic recombination). Akin to genetic mutation, single features are also substituted at random with low probability from the 720-feature pool. This process terminates when the median fitness score in the population has not improved for 20 generations. Finally, we estimate a mean fitness score using fivefold crossvalidation for each genotype in the final population, and return the feature set that produces the highest score (Supplemental Figures 2 and 3 and Supplemental Methods).21 Genetic algorithm code can be found at https://github.com/Yale-PATR/PopFS.
Statistical Analysis
We divided the study population at the Yale-New Haven Children’s Hospital into derivation (70%) and internal validation (30%) cohorts, randomized at the patient level to ensure the same patient would never appear in both data sets. We used the cohort of patients admitted to Bridgeport Hospital as an external validation cohort. We summarized the data as median [interquartile range (IQR)] for continuous variables and count (percentage) for categorical variables. Univariable comparisons of categorical data were analyzed with the Pearson’s chi-squared or Fisher’s exact tests. Continuous data were analyzed using Wilcoxon rank-sum tests. We set the threshold of statistical significance at P<0.05.
We performed logistic regression analyses on the five 10-feature models. All models were developed entirely in the derivation set, and evaluated in the internal and external validation sets. Model AUC was calculated by applying the c-statistic transformation to the rank-based Somers D statistic, accounting for within-patient clustering as multiple predictions are made over time in each patient.22 We compared the AUC across models by computing the linear combination of the difference in AUC from the above process. We also evaluated the area under the precision-recall curve and calibration of the prediction models. We performed subgroup analyses of age groups, patient locations, medical diagnoses, and stages of AKI.
A sensitivity analysis was performed on high- and low-risk cutoff threshold levels of the genetic algorithm model predictive probability of 0.24 and 0.08, respectively. These analyses included sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, odds ratios, and median time from crossing cutoff to AKI development with IQRs. Additionally, we performed a sensitivity analysis on different pediatric age groups using the high- and low-risk cutoff threshold levels.
Statistical analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC), Stata version 15 (StatCorp, College Station, TX), R version 3.3.3 (R Foundation for Statistical Computing, Vienna, Austria), and the scikit-learn in Python version 3.6.
RESULTS
From January 1, 2014 to January 30, 2018, a total of 6328 children (8608 encounters) were hospitalized and had at least two serum creatinine values measured (Figure 2). One hundred and thirty-five encounters were excluded due to an initial serum creatinine level of >4 mg/dl and/or an eGFR of <15 ml/min per 1.73 m2. Median age was 4.4 years (IQR, 0.04–13.1), 4614 (54.4%) were male, and 2608 (30.8%) encounters had at least some time in an ICU [13% in the pediatric ICU (PICU) and 13.5% in the neonatal ICU] (Table 1). The baseline median serum creatinine was 0.51 mg/dl (IQR, 0.32–0.77) and baseline eGFR was 96.7 ml/min per 1.73 m2 (IQR, 44–129). AKI occurred in 516 (10.2%), 207 (9%), and 27 (2.5%) encounters in derivation, and internal and external validation cohorts, respectively. AKI was associated with higher inpatient mortality (11% compared with 0.9% without AKI, P<0.001) and increased length of stay (average of 17.1 days compared with 4.2 days without AKI, P<0.001) (Supplemental Table 2).
Figure 2.
Patient population and cohort allocation. To build the prediction model, study population was divided into derivation, and internal and external validation cohorts.
Table 1.
Baseline characteristics of patient groups
| Variables | Derivation (N=5072) | Internal Validation (N=2299) | External Validation (N=1102) |
|---|---|---|---|
| Demographic | |||
| Age, yr (range) | 5 (0.2–13.6) | 5.2 (0.2–12.8) | 0.2 (0–10.1) |
| Ethnicity (Hispanic), n (%) | 1347 (26.6) | 577 (25.1) | 404 (36.7) |
| Race (black), n (%) | 1048 (20.7) | 466 (20.3) | 296 (26.9) |
| Sex (male), n (%) | 2756 (54.3) | 1252 (54.5) | 606 (55) |
| Medical history, n (%) | |||
| Congenital heart disease | 674 (13.3) | 248 (10.8) | 89 (8.1) |
| CKD | 77 (1.5) | 23 (1) | 2 (0.2) |
| Malignancy | 841 (16.6) | 405 (17.6) | 24 (2.2) |
| Inpatient location, n (%) | |||
| Pediatric ICU | 758 (14.9) | 362 (15.7) | 0 (0) |
| Neonatal ICU | 818 (16.1) | 348 (15.1) | 292 (26.5) |
| General pediatrics | 1288 (25.4) | 602 (26.2) | 546 (49.5) |
| Pediatric surgery | 518 (10.2) | 252 (11) | 0 (0) |
| Hematology/Oncology | 693 (13.7) | 315 (13.7) | 0 (0) |
| Nursery | 249 (4.9) | 106 (4.6) | 218 (19.8) |
| Laboratory (first values) | |||
| Serum creatinine, mg/dl | 0.5 (0.3–0.8) | 0.5 (0.3–0.7) | 0.7 (0.5–0.8) |
| eGFR, ml/min per 1.73 m2 | 99.4 (55.3–132.1) | 101.9 (58.7–135.6) | 61.2 (25.6–105.3) |
| BUN, mg/dl | 11 (8–16) | 11 (8–16) | 11 (8–15) |
| Glucose, mg/dl | 100 (83–128) | 101 (83–126) | 82 (53–102) |
| Bicarbonate, mmol/L | 21 (18.7–23.5) | 21 (18.5–23.4) | 23 (20–25) |
| Calcium, mg/dl | 9.3 (8.7–9.8) | 9.3 (8.7–9.8) | 9.3 (8.4–9.8) |
| Platelet count, ×1000/µl | 253 (183–341) | 259 (185–351) | 251 (206–314) |
| Lymphocyte percent (%) | 21.6 (9–37) | 24 (10–42) | 4 (2.2–7.1) |
| INR measured (%) | 1656 (32.6) | 772 (33.6) | 103 (9.3) |
| Fibrinogen measured (%) | 497 (9.8) | 219 (9.5) | 8 (0.7) |
| Lactate measured (%) | 1947 (38.4) | 883 (38.4) | 89 (8.1) |
| Medication,a n (%) | |||
| Loop diuretic | 922 (18.2) | 348 (15.1) | 117 (10.6) |
| Vasopressor | 529 (10.4) | 218 (9.5) | 23 (2.1) |
| Chemotherapy | 527 (10.4) | 253 (11) | 0 (0) |
| Sodium bicarbonate | 353 (7) | 133 (5.8) | 4 (0.4) |
| Calcium gluconate | 216 (4.3) | 86 (3.7) | 3 (0.3) |
| Alprostadil | 43 (0.8) | 13 (0.6) | 0 (0) |
| Carboplatin | 13 (0.3) | 13 (0.6) | 0 (0) |
| Foscarnet | 5 (0.1) | 2 (0.1) | 0 (0) |
| Paclitaxel | 2 (0) | 0 (0) | 0 (0) |
| Nimodipine | 1 (0) | 0 (0) | 0 (0) |
| Procedure, n (%) | |||
| Mechanical ventilation | 475 (9.4) | 193 (8.4) | 48 (4.4) |
| RBC transfusion | 1150 (22.7) | 457 (19.9) | 136 (12.3) |
Data are presented as median (IQR) or proportion. All comparisons across the groups have P<0.01. INR, international normalized ratio; RBC, red blood cell.
If a medication was ever given during hospitalization for non-AKI encounters or prior to AKI for AKI encounters.
Table 2 lists the results of the feature selection strategies we employed. Notably, several variables were selected by multiple feature selection methods: BUN, creatinine, glucose, sodium bicarbonate use, and the time since admission (Table 3). Supplemental Table 3 shows the prediction model coefficients and odds ratios. The genetic algorithm model outperformed other logistic regression-based models with an AUC of 0.76 [95% confidence interval (95% CI), 0.72 to 0.79]. This was significantly better performance than the other feature selection algorithms (P<0.05 for comparing the genetic algorithm model with clinical, LASSO, and forward models in the internal validation cohort). The genetic algorithm model was well calibrated in both the derivation and internal validation cohorts (Supplemental Figure 4). The area under the precision-recall curve of the genetic algorithm model was 0.11 in the internal validation cohort (Supplemental Figure 5).
Table 2.
Feature selection methods with comparison based on AUCs
| Model Name | Feature Selection Method | Model AUC (95% CI) | ||
|---|---|---|---|---|
| Derivation | Internal Validation | External Validation | ||
| Clinical | Clinical experience and literature review | 0.73 (0.71 to 0.76) | 0.72 (0.69 to 0.76) | 0.76 (0.66 to 0.87) |
| Lasso | LASSO | 0.77 (0.75 to 0.79) | 0.73 (0.69 to 0.77) | 0.79 (0.70 to 0.88) |
| Top AUC | Highest AUCs in univariable analysis | 0.76 (0.73 to 0.78) | 0.74 (0.70 to 0.77) | 0.83 (0.76 to 0.91) |
| Forward | Forward selection | 0.78 (0.75 to 0.80) | 0.73 (0.69 to 0.77) | 0.84 (0.78 to 0.91) |
| Genetic | Genetic algorithm | 0.79 (0.77 to 0.81) | 0.76 (0.72 to 0.79) | 0.87 (0.81 to 0.92) |
The genetic algorithm significantly improves prediction of AKI as compared with other feature selection methods (P<0.05 for comparing the genetic algorithm model with clinical, LASSO, and forward models in internal validation cohort).
Table 3.
Variables selected by different feature selection models
| Variable | Clinical | Lasso | Top AUC | Forward | Genetic |
|---|---|---|---|---|---|
| BUN | + | + | + | + | + |
| Time since admission | — | — | — | — | |
| Sodium bicarbonate use | + | + | + | ||
| Glucose | + | + | + | ||
| Creatinine | + | + | + | + | |
| Calcium | — | — | — | ||
| RBC transfusion | + | + | |||
| Loop diuretic use | + | + | |||
| Alprostadil use | + | + | |||
| Δ creatinine 48 h | + | + | |||
| Chemotherapy use | + | + | |||
| General pediatrics admission | — | ||||
| Carboplatin use | + | ||||
| Foscarnet use | + | ||||
| Nimodipine use | + | ||||
| Paclitaxel use | + | ||||
| Fibrinogena | + | ||||
| Oxygen saturation | — | ||||
| Lymphocyte percent | — | ||||
| Platelet count | — | ||||
| Respiratory rate | — | ||||
| INRa | + | ||||
| Calcium gluconate use | + | ||||
| Bicarbonate | — | ||||
| Lactatea | + | ||||
| ICU admission | + | ||||
| Mechanical ventilation | + | ||||
| Pressor use | + | ||||
| Weight | + |
Variables at the top were selected by several feature selection methods. RBC, red blood cell; Δ creatinine 48 h, change in serum creatinine within the last 48 h; INR, international normalized ratio.
Transformed into categorical variables (measured versus not measured) due to missingness >25%; +, positive correlation with AKI; –, negative correlation with AKI.
Figure 3 demonstrates an example of the predictive probability of the genetic algorithm model along with the rise in serum creatinine and development of AKI in an individual patient.
Figure 3.
An example in one patient of the predictive probability of the genetic algorithm model along with the rise in serum creatinine until the development of AKI. In this example, our prediction model identified the patient to be at risk of AKI 10 hours before creatinine criteria were met.
In evaluating secondary outcomes, the genetic algorithm model AUC increased to 0.79 (95% CI, 0.74 to 0.83) in predicting severe AKI (stage 2 or 3) and to 0.80 (95% CI, 0.72 to 0.88) for only stage 3 AKI (Supplemental Figure 6). In a subgroup analysis, the genetic algorithm model performed the best among newborns (AUC 0.81; 95% CI, 0.77 to 0.86) and worse in the adolescent age group (AUC 0.67; 95% CI, 0.60 to 0.73), better in children with CKD (AUC 0.91; 95% CI, 0.78 to 0.99) than children with malignancy (AUC 0.61; 95% CI, 0.52 to 0.69), and better in ICU (AUC 0.77; 95% CI, 0.75 to 0.82) than non-ICU populations (AUC 0.68; 95% CI, 0.62 to 0.74).
To apply the predictive model in clinical practice, we took into consideration the burden of alerting physicians and the development of alert fatigue versus the risk of missing too many high-risk patients; therefore, we chose low- and high-risk cutoff threshold points using the highest-performing model: the genetic algorithm model (Supplemental Tables 4 and 5). The low-risk cutoff threshold point of 0.08 predictive probability would identify the vast majority of patients at risk of AKI with a negative predictive value of 97.9% in the internal validation cohort, but would include many patients who do not develop AKI and is thus well suited for low-level interventions (such as further monitoring of serum creatinine). The high-risk cutoff threshold point of 0.24 predictive probability would identify a smaller percentage of patients who are at a substantially higher risk of developing AKI with a 99.6% specificity, 12-fold increase in baseline positive predictive value to 26.6% (positive likelihood ratio of 12.5), would provide a median time of 23.7 hours (IQR 5.3–343.7) prior to AKI development, and would be well suited toward more intensive or costly interventions (such as empirical volume restoration or pharmacist consultation) (Figure 4). The suggested low-risk threshold point of 0.08 showed nearly similar performance in the external validation cohort as compared with the internal validation cohort, with a specificity of 97.3% and negative predictive value of 99.5%. On the other hand, the suggested high-risk cutoff threshold point of 0.24 did not identify any patients at risk of AKI in the external validation cohort (no true positives) (Supplemental Table 5). Supplemental Tables 6 and 7 show the sensitivity analysis of these cutoff threshold points at different age groups. Our web-based AKI risk prediction calculator uses the features from the genetic algorithm model and can be accessed at https://yalepatr.shinyapps.io/TRACK.
Figure 4.
Diagnostic test characteristics of the genetic model at two cutoff threshold points. A low-risk cutoff threshold point can be used to identify patients suitable for low-risk interventions (such as monitoring serum creatinine). A high-risk cutoff threshold point can be used to apply costly, risky, or intensive interventions (such as medication and fluid changes). LR+, positive likelihood ratio; LR-, negative likelihood ratio; NPV, negative predictive value; OR, odds ratio; PPV, positive predictive value; TP:FP, true positive to false positive ratio.
DISCUSSION
The utility of the EHR has expanded from its inception as primarily a data storage medium to its potential today to guide clinical decision making and augment prediction of important outcomes. Leveraging the EHR to predict AKI in children may be a low-cost approach to prevent incident AKI or limit the severity of AKI. We used multiple feature selection techniques to develop and validate a parsimonious risk prediction model for AKI in hospitalized neonates and children. We determined that a genetic feature selection algorithm was the highest-performing model. We also identified potential AKI predictors such as sodium bicarbonate use and glucose. All of the information required by this genetic model is reported in the EHR and could be used in real time to identify children at the highest risk of impending AKI, allowing clinicians to intervene prior to AKI development.
A few studies have attempted to develop EHR-based AKI prediction models. Sanchez-Pinto et al.9 created a pediatric EHR-based AKI risk prediction model, which had good discrimination for AKI, but was limited to children admitted to the PICU and did not include neonates. Additionally, their prediction model only uses data available in the first 12 hours of a PICU admission to predict AKI within the first 72 hours of a PICU stay. Another EHR-based prediction model was created by Wang et al.,10 which had modest discrimination for AKI with AUCs of 0.74 in the ICU and 0.69 for non-ICU patients. However, this study only included predictor variables 3 days before AKI and patients younger than 28 days old were excluded.
In contrast to the previously published AKI prediction models, our prediction approach uses time-varying data, allowing the model to incorporate clinical events that occur during hospitalization and making predictions that are continuously updated in real time. We also used a clinically agnostic approach in selecting predictor variables, including a wide array of variables available in the EHR. This approach allows for the identification of novel predictors of AKI and, insofar as some of those predictors are strong, may improve overall predictive ability. To build a model that can be widely used in EHR systems, we decided to include all children younger than 18 years and admitted in all units of the hospital.
Neonates represent a unique pediatric population in which their glomerular and tubular function is still maturing, and their creatinine levels initially reflect maternal creatinine. As a result, neonates are often excluded from pediatric AKI research studies. However, neonates are subjected to many of the same risk factors as other children such as kidney ischemia, surgery, sepsis, and nephrotoxin exposure.4,11 Furthermore, AKI in neonates occurs at similar rates as PICU patients, and is strongly associated with length of stay and mortality.
Whereas including more variables in the model can improve performance and increase the AUC, we aimed to create a prediction model that can be practically applied in the EHR. This is an effort to recognize that for prediction methods to be useful, they must be prospectively implemented into the EHR and every variable is a potential break point. New or modified laboratory tests, medication names, and hospital locations may change over time and result in inaccurate predictions.
In this study, we noted that prediction modeling using machine learning feature selection methods had higher predictive ability, for example an AUC of 0.76 (95% CI, 0.72 to 0.79) in the genetic algorithm model, compared with a clinical experience-based model with an AUC of 0.72 (95% CI, 0.69 to 0.76), emphasizing the increased efficacy of clinically agnostic machine learning-based prediction models over that of current models based on clinical intuition and experience.
Despite the generally good discriminatory ability of the models, we observed a less robust precision recall AUC. This suggests that a significant proportion of AKI in our population occurred without warning, at least insofar as warning signals can be derived from the EHR. This has important implications for targeting interventions to at-risk children, as the models may identify those with an AKI prodrome and miss those with more sudden AKI.
Using a clinically agnostic approach of feature selection by including 720 predictor variables from the EHR allowed us to identify clinical parameters not typically thought to be associated with or causative of AKI. Sodium bicarbonate use and red blood cell transfusions are two examples of variables selected by machine learning algorithms that are not necessarily causative, but can reflect the severity of a patient’s illness and predict the development of AKI. A clinically agnostic feature selection approach may thus generate hypotheses that would not have been investigated under a purely hypothesis-driven paradigm. This paradigm has no ability to assess causality. The model may be predictive of the outcome, but there’s no current indication to change clinical management to affect the features that appear in the model. This model is to be used purely for prognostic purposes and not for treatment decisions at the variable level.
Our study has several strengths. We created a continuously updating AKI prediction model using EHR data. This model requires only ten variables that are commonly available in the EHRs. We used a clinically agnostic approach in selecting predictor variables from a large number of candidate variables utilizing several machine learning algorithms, which allowed us to identify several novel predictors of AKI.
We also acknowledge several limitations of our study. First, including all variables in the EHR forced us to impute a large number of laboratory variables due to missingness of categorical variables. However, predictive potential would only be reduced by imputation, implying that more robust measurements of risk factors would further improve predictive performance. Additionally, medication variables were treated as exposures regardless of their doses. We chose to not include medication doses to avoid increasing the complexity of the model in prospective implementation, because medication doses vary based on age, weight, treatment indication, and kidney function in children. We acknowledge the limited calibration performance in the external validation cohort (Bridgeport Hospital), however the model maintained the discrimination measures. As noted, patients in this cohort are of lower acuity and there were only 27 AKI events, which highlights the need to recalibrate such prediction models when used at different institutions. Furthermore, most patients in this cohort did not cross the high-risk threshold point of 0.24, and therefore using a high threshold point can have limitations in a low-acuity patient population. We recommend examining different threshold points on the population of interest before implementing the model. Lastly, the nature of retrospective studies limits our ability to make conclusions on the causal relationship between predictor variables and outcome.
In conclusion, we present different feature selection paradigms to create a time-updated, parsimonious, clinically agnostic prediction modeling approach for pediatric AKI that can be applied in real-time to all pediatric patients. After prospectively assessing the prognostic performance of the model, the best test of clinical utility is to determine if interventions, targeted to those crossing a given risk threshold, can meaningfully change clinical outcomes. Prediction models of AKI could also be further studied to identify children at high risk of AKI for enrollment in clinical trials. These high-risk individuals would be the most likely to develop AKI and may benefit from real-time interventions, such as alerts for nephrotoxin avoidance and best-practice advisories.
Disclosures
Dr. Testani reports grants and personal fees from Sequana Medical, BMS, 3ive labs, Boehringer Ingelheim, Sanofi, and FIRE1; personal fees from AstraZeneca, Novartis, Cardionomic, Bayer, MagentaMed, Renalguard, and W.L. Gore; and grants from Otsuka and Abbott outside the submitted work. Dr. Mansour reports grants and other from the American Heart Association and the Patterson Trust Fund during the conduct of the study. Dr. Moledina reports grants from the National Institutes of Health/National Institute of Diabetes and Digestive Kidney Diseases during the conduct of the study. In addition, Dr. Moledina has a patent system and methods for diagnosing acute interstitial nephritis pending. Dr. Wilson reports grants from the National Institute of Diabetes and Digestive Kidney Diseases during the conduct of the study. Dr. Wilson is the founder of Efference, LLC, a medical communications company.
Funding
Dr. Wilson received support from National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) grants R01 DK113191 and P30 DK079310. Dr. Greenberg is funded by career development grant K08DK110536. This research is funded by a Charles H. Hood Foundation, Inc. grant (to Dr. Greenberg). Dr. Moledina is funded by NIDDK grant K23DK117065.
Supplementary Material
Acknowledgments
The authors acknowledge Boian Etropolski for his assistance in developing the online risk prediction calculator.
I. Sandokji, J.H. Greenberg, and F.P. Wilson designed the study; I. Sandokji, Y. Yamamoto, A. Biswas, and I. Saran reviewed medical charts; I. Sandokji, Y. Yamamoto, A. Biswas, and F.P. Wilson analyzed the data; I. Sandokji, Y. Yamamoto, and M. Simonov made the figures; I. Sandokji, Y. Yamamoto, A. Biswas, T. Arora, U. Ugwuowo, I. Saran, M. Martin, J.M. Testani, S. Mansour, D.G. Moledina, J.H. Greenberg, and F.P. Wilson drafted and revised the paper; and all authors approved the final version of the manuscript.
Footnotes
Published online ahead of print. Publication date available at www.jasn.org.
Supplemental Material
This article contains the following supplemental material online at http://jasn.asnjournals.org/lookup/suppl/doi:10.1681/ASN.2019070745/-/DCSupplemental.
Supplemental Table 1. Predictor variables used to create parsimonious prediction models.
Supplemental Table 2. Comparing demographic and clinical characteristics according to development of AKI in the derivation cohort.
Supplemental Table 3. Regression-based prediction models coefficients and odds ratios.
Supplemental Table 4. Genetic algorithm model operating points applied to the internal validation cohort.
Supplemental Table 5. Genetic algorithm model operating points applied to the external validation cohort.
Supplemental Table 6. Low-risk cut-off threshold (0.08) analysis using the genetic algorithm model in age subgroups (on the internal validation cohort).
Supplemental Table 7. High-risk cut-off threshold (0.24) analysis using the genetic algorithm model in age subgroups (on the internal validation cohort).
Supplemental Figure 1. Logistic regression analyses of incremental numbers of variables of the stepwise forward selection model.
Supplemental Figure 2. Genetic algorithm feature selection process.
Supplemental Figure 3. Convergence in evaluation set (of derivation cohort) area under the receiver operating characteristic curve (AUC) of the genetic algorithm’s population.
Supplemental Figure 4. Genetic algorithm model calibration curves.
Supplemental Figure 5. Genetic algorithm model performance illustrated by a precision-recall curve.
Supplemental Figure 6. Genetic model for the prediction of severe AKI and subgroup analyses in the internal validation cohort.
Supplemental Methods. Genetic Algorithm Methodology.
References
- 1.Kaddourah A, Basu RK, Bagshaw SM, Goldstein SL; AWARE Investigators : Epidemiology of acute kidney injury in critically ill children and young adults. N Engl J Med 376: 11–20, 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Sutherland SM, Ji J, Sheikhi FH, Widen E, Tian L, Alexander SR, et al. : AKI in hospitalized children: Epidemiology and clinical associations in a national cohort. Clin J Am Soc Nephrol 8: 1661–1669, 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Greenberg JH, Coca S, Parikh CR: Long-term risk of chronic kidney disease and mortality in children after acute kidney injury: A systematic review. BMC Nephrol 15: 184, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kwiatkowski DM, Sutherland SM: Acute kidney injury in pediatric patients. Best Pract Res Clin Anaesthesiol 31: 427–439, 2017. [DOI] [PubMed] [Google Scholar]
- 5.Jetton JG, Boohaker LJ, Sethi SK, Wazir S, Rohatgi S, Soranno DE, et al. ; Neonatal Kidney Collaborative (NKC) : Incidence and outcomes of neonatal acute kidney injury (AWAKEN): A multicentre, multinational, observational cohort study. Lancet Child Adolesc Health 1: 184–194, 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Silver SA, Chertow GM: The economic consequences of acute kidney injury. Nephron 137: 297–301, 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Goldstein SL, Mottes T, Simpson K, Barclay C, Muething S, Haslam DB, et al. : A sustained quality improvement program reduces nephrotoxic medication-associated acute kidney injury. Kidney Int 90: 212–221, 2016. [DOI] [PubMed] [Google Scholar]
- 8.Sutherland SM, Chawla LS, Kane-Gill SL, Hsu RK, Kramer AA, Goldstein SL, et al. ; 15 ADQI Consensus Group : Utilizing electronic health records to predict acute kidney injury risk and outcomes: Workgroup statements from the 15(th) ADQI Consensus Conference. Can J Kidney Health Dis 3: 11, 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Sanchez-Pinto LN, Khemani RG: Development of a prediction model of early acute kidney injury in critically ill children using electronic health record data. Pediatr Crit Care Med 17: 508–515, 2016. [DOI] [PubMed] [Google Scholar]
- 10.Wang L, McGregor TL, Jones DP, Bridges BC, Fleming GM, Shirey-Rice J, et al. : Electronic health record-based predictive models for acute kidney injury screening in pediatric inpatients. Pediatr Res 82: 465–473, 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Selewski DT, Charlton JR, Jetton JG, Guillet R, Mhanna MJ, Askenazi DJ, et al. : Neonatal acute kidney injury. Pediatrics 136: e463–e473, 2015. [DOI] [PubMed] [Google Scholar]
- 12.Basu RK, Zappitelli M, Brunner L, Wang Y, Wong HR, Chawla LS, et al. : Derivation and validation of the renal angina index to improve the prediction of acute kidney injury in critically ill children. Kidney Int 85: 659–667, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Sanchez-Pinto LN, Venable LR, Fahrenbach J, Churpek MM: Comparison of variable selection methods for clinical predictive modeling. Int J Med Inform 116: 10–17, 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Pottel H, Dubourg L, Goffin K, Delanaye P: Alternatives for the bedside Schwartz equation to estimate glomerular filtration rate in children. Adv Chronic Kidney Dis 25: 57–66, 2018. [DOI] [PubMed] [Google Scholar]
- 15.Selewski DT, Cornell TT, Heung M, Troost JP, Ehrmann BJ, Lombel RM, et al. : Validation of the KDIGO acute kidney injury criteria in a pediatric critical care population. Intensive Care Med 40: 1481–1488, 2014. [DOI] [PubMed] [Google Scholar]
- 16.Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating, Berlin, Germany, Springer Science & Business Media, 2008 [Google Scholar]
- 17.Tibshirani R: Regression shrinkage and selection via the Lasso. J R Stat Soc B 58: 267–288, 1996 [Google Scholar]
- 18.Choi SJ, Ha EJ, Jhang WK, Park SJ: Factors associated with mortality in continuous renal replacement therapy for pediatric patients with acute kidney injury. Pediatr Crit Care Med 18: e56–e61, 2017. [DOI] [PubMed] [Google Scholar]
- 19.Selewski DT, Symons JM: Acute kidney injury. Pediatr Rev 35: 30–41, 2014. [DOI] [PubMed] [Google Scholar]
- 20.Thongprayoon C, Cheungpasitporn W, Mao MA, Sakhuja A, Erickson SB: Admission calcium levels and risk of acute kidney injury in hospitalised patients. Int J Clin Pract 72: e13057, 2018. [DOI] [PubMed] [Google Scholar]
- 21. Vafaie H, Jong KD: Genetic algorithms as a tool for feature selection in machine learning. Presented at the Fourth International Conference on Tools with Artificial Intelligence 1992, Arlington, VA, November 10–13, 1992. [Google Scholar]
- 22.Newson R: Parameters behind “nonparametric” statistics: Kendall’s tau, Somers’ D and median differences. Stata J 2: 45–64, 2002 [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.