Skip to main content
NCBI home page
JAMA Network logoLink to JAMA Network
. 2025 Feb 13;8(2):e2459684. doi: 10.1001/jamanetworkopen.2024.59684

Validation of 30-Day Pediatric Hospital Readmission Risk Prediction Models

Alison R Carroll 1,2,✉, Matthew Hall 3, Mitch Harris 3, Michael S Carroll 4, Katherine A Auger 5, Matthew M Davis 6, Denise M Goodman 7, Derek J Williams 1
PMCID: PMC11826366  PMID: 39946127

This prognostic study assesses the multihospital-level accuracy and variability of readmission risk prediction models.

Key Points

Question

How do single center–derived models for predicting all-cause 30-day readmission risk perform within a large sample of US children’s hospitals?

Findings

In this prognostic study involving 851 499 children discharged from 48 children’s hospitals, the models’ predictive performance declined in both temporal validation at the derivation hospital and external validation across 47 children’s hospitals. The models retained discriminatory power, but there was variability in hospital-level performance.

Meaning

Single center–derived readmission risk models showed reduced predictive accuracy across time and variability in hospital-level performance, suggesting the need for more accurate and generalizable models and rigorously designed impact analyses to ascertain clinical utility in reducing preventable use and improving outcomes.

Abstract

Importance

Accurate identification of hospital readmission risk during a current hospitalization may enhance decision-making, facilitate targeted systems-level interventions, and avoid preventable readmissions.

Objective

To temporally and externally validate a suite of readmission risk prediction models across 48 children’s hospitals to assess their generalizability and feasibility for future clinical implementation.

Design, Setting, and Participants

This prognostic study analyzed data from the Pediatric Health Information System (PHIS) database, which contains billing and resource use data from 48 US children’s hospitals, including the derivation hospital (DH) and 47 hospitals participating in the PHIS database (hereafter other PHIS hospitals). Children aged 18 years or younger discharged from these hospitals between January 1, 2016, to December 31, 2019, were included. This cohort was divided as specified into the 3 prediction models at the DH: 6 months or older with no recent hospitalizations (new admission model [NAM]), 6 months or older with 1 or more prior hospitalizations within the last 6 months (recent admission model [RAM]), and 6 months or younger (young infant model [YIM]). Data were analyzed from August 9 to December 1, 2023.

Main Outcomes and Measures

The primary validation outcome was hospital-level discrimination measured with area under the receiver operating characteristic curve (AUROC). Predictors included demographic, clinical, and utilization variables. All-cause 30-day readmission was modeled for each hospital using logistic regression and parameter estimates from the DH. Calibration plots examined observed vs predicted outcome frequencies for each hospital.

Results

In external validation, a total of 851 499 children were discharged from 48 hospitals (16 330 DH discharges and 835 169 other PHIS hospital discharges). The largest group of children was aged 5 to 14 years (281 193 [33.0%]). In temporal validation, the DH PHIS 2016-2018 cohort included 45 682 discharges. All-cause 30-day readmission rates were 7.2% for NAM, 35.5% for RAM, and 11.7% for YIM. The 2019 DH PHIS cohort included 16 330 discharges. All cause 30-day readmision rates were 7.2% for NAM, 35.1% for RAM, and 11.1% for YIM. Temporal validation demonstrated reduced discrimination across all 3 models (median AUROC, 0.65 [95% CI 0.62-0.67] for the NAM; 0.73 [95% CI 0.72-0.75) for RAM; 0.67 [95% CI 0.63-0.70) for the YIM compared with the original estimates (median AUROC 0.76 [95% CI 0.85-0.78] for the NAM; 0.84 [95% CI 0.83-0.84] for the RAM; 0.79 [95% CI 0.77-0.80] for the YIM). Overall readmission rates were 5.9% for NAM, 30.1% for RAM, and 7.6% for YIM. External validation yielded similiar findings as the temporal validation, although with demonstrable variation in performance across hospitals (median [range] AUROC, 0.64 [0.60-0.68] for the NAM; 0.73 [0.64-0.80] for the RAM; 0.65 [0.53-0.74] for the YIM). Most hospitals were poorly calibrated, with both significant overestimation and underestimation of observed risk. Of 47 other PHIS hospitals, only 3 for the RAM (6.4%) and 9 for both the NAM and YIM (19.1%) were adequately calibrated.

Conclusions and Relevance

This prognostic study found that the readmission risk prediction models had reduced predictive accuracy across time and variability in hospital-level performance. These findings stress the importance of local validation prior to clinical implementation and suggest opportunities to improve generalizability, including multicenter derivation and expansion of candidate predictors.

Introduction

If readmission were a diagnosis, it would outrank all other hospital diagnoses for children outside of the newborn period in the US.1 As of 2020, pediatric 30-day readmission rates were 11.4%, with costs nearly 1.4 times the cost of an index admission, a ratio higher than for all adult age groups.2 As such, much attention is paid to readmission rates by health systems and payers.3,4,5,6,7,8,9 Readmissions are also burdensome and disruptive for patients and families. Considerable efforts have focused on reducing readmissions, including a variety of hospital-to-home transition interventions.10 While 1 large randomized clinical trial found that nurse home visits did not prevent readmissions, other studies targeting higher-risk populations reported significantly reduced emergency department (ED) revisits and readmissions.10,11,12,13,14,15

Solutions that harness real-time health system data (ie, clinical, demographic, and socioeconomic data in the electronic health record [EHR]) to objectively define readmission risk may be a factor in improved effectiveness of hospital-to-home transitions by identifying high-risk patients early in a hospitalization. Challenges with existing readmission prediction models include being limited in scope (ie, focus on a single diagnosis),16 being proprietary,17,18 having poor discriminative ability,19 or lacking validation.18,20 Multicenter-derived readmission models have also included administrative data not routinely available at the point of care,8,21,22,23 limiting their utility for informing clinician decision-making during a current admission.

Goodman et al20 used EHR data from 48 019 hospitalizations at a single, freestanding children’s hospital (ie, derivation hospital [DH]) to predict all-cause 30-day readmission across 3 cohorts: (1) children age 6 months or older with no prior hospitalizations in the last 6 months (new admission model [NAM]), (2) children age 6 months or older with 1 or more hospitalizations within 6 months (recent admission model [RAM]), and (3) children age younger than 6 months (young infant model [YIM]). The resulting readmission risk prediction models included 4 to 7 predictors, and each demonstrated good to excellent discrimination as measured by area under the receiver operating characteristic curve (AUROC) ranging from 0.76 to 0.83 (eTable 1 in Supplement 1). These models are novel because they were designed to be integrated into the EHR and used at the point of care to inform disposition and hospital-to-home transition decisions. The generalizability of these models to other settings is unknown. Rigorous model validation requires evaluating the model’s performance using the originally specified parameters in a new population or setting.24 This often overlooked step is critical to avoid promulgation of badly performing models that do not help, or worse, are harmful.25 However, translating these off-the-shelf models, when validated, to other hospitals could help overcome common time, expertise, and resource barriers to facilitate more widespread adoption. Therefore, we sought to temporally and externally validate this suite of readmission risk prediction models across 48 US children’s hospitals to assess their generalizability and feasibility for future clinical implementation.24

Methods

Design, Data Source, and Population

We sourced data from the Children’s Hospital Association Pediatric Health Information System (PHIS) administrative database, which contains billing and resource use data from 48 US children’s hospitals, including the DH, across 27 states and 17 of the 20 largest metropolitan areas. To distinguish the DH from the remaining 47 hospitals in the PHIS database, we refer to them as other PHIS hospitals. All children aged 18 years or younger hospitalized (inpatient or observation status) at the DH or other PHIS hospitals between January 1, 2016, and December 31, 2019, were considered for inclusion. We excluded encounters resulting in death or transfer to another facility. The Vanderbilt University Medical Center Institutional Review Board deemed this prognostic study exempt from ethics review and informed consent requirement because it was not human participant research. We adhered to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.26

Predictor Variables

Candidate predictors for the original models20 were chosen according to literature review18,27,28,29,30 and clinical judgment. Final predictors were selected using bivariate significance testing for parsimony (Goodman et al20 provides a full methodological explanation). The final predictor set included the following EHR-derived demographic and clinical characteristics: age at admission, race and ethnicity (Hispanic, non-Hispanic Black, non-Hispanic White, and non-Hispanic other [including American Indian or Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, or multiracial] or unknown/unspecified), and insurance type (government, commercial, and other [including uninsured, self-insured, and multiple insurances] or unspecified); as well as medical history, including prior acute care use (hospital and ED discharges) and select prior procedures (invasive or noninvasive ventilation, central venous catheter placement, or transfusion) within 6 months of index admission; and measures of acute care use for the index admission, such as principal diagnosis (10 common diagnosis groups and 1 other), length of stay (LOS), admission location (ED vs other), service line (medical vs surgical), and use of intensive care. Race and ethnicity data were collected because they were included as predictor variables in the original models.

Modification for the service line variable, which was unavailable in the PHIS database, was required. Instead, surgical encounters were identified as those with operating room charges or with any International Statistical Classification of Diseases and Related Health Problems, Tenth Revision operating room procedure code.31 All other encounters were considered medical encounters. eTable 1 in Supplement 1 provides the model cohort definition, predictors, and predictive performance. Approximately 6% of the data had missing race and ethnicity information, and this missing variable was categorized as other. No hospitals were excluded due to data integrity issues.

Readmission Outcome

The models were designed to predict all-cause 30-day readmission, defined as any hospitalization (observation or inpatient) occurring within 30 calendar days of the index discharge to the same hospital, including both planned (ie, elective) and unplanned readmissions. Each hospitalization was treated as an index encounter. Thus, it was possible for readmission encounters to also serve as a new index encounter.

Validation Steps

As the original models20 were developed using EHR data at a single, tertiary care children’s hospital, we first validated the model construct (hereafter DH PHIS) by fitting the DH models as originally specified to the 2016 to 2018 PHIS data (the derivation study period) for the DH. Index admission dates between January 1, 2016, and December 31, 2018, were included, and 30-day follow-up required following some cases into 2019 to measure 30-day readmission. We anticipated that discrimination (ie, AUROC) should be essentially identical (Figure 1, EHR to PHIS validation).

Figure 1. Steps for Statistical Analysis .

Figure 1.

Open in a new tab

AUROCs for electronic health record (EHR) to Pediatric Health Information System (PHIS) validation were 0.76 (95% CI, 0.75-0.77) for new admission model, 0.83 (95% CI, 0.82-0.84) for recent admission model, and 0.80 (95% CI, 0.79-0.82) for young infant model. DH indicates derivation hospital.

aAUROCs from Goodman et al.20

bTemporal validation AUROCs are provided in eTable 2 in Supplement 1.

cExternal validation AUROCs are provided in Table 2.

We then proceeded with temporal validation, fitting the DH PHIS models estimated using the 2016 to 2018 data to the 2019 DH PHIS data (Figure 1, temporal validation). Index admission dates were January 1 to December 31, 2019, and 30-day follow-up required following some cases into 2020 to measure 30-day readmission. Given the substantial changes in use during the COVID-19 pandemic, we did not include index encounters after 2019.32

For external validation, we fit the DH PHIS models to the 2019 data for other PHIS hospitals (Figure 1, external validation). Characteristics of each population were summarized as counts and percentages for categorical variables and as medians and IQRs for continuous variables.

Statistical Analysis

For each analysis, logistic regression models were fit using the same modeling procedures as described for the DH cohorts, substituting the parameter estimates specified by the DH PHIS models. Patient was used as a random effect, as was done in the original models. Predictive performance for each model was evaluated using discrimination and calibration. Discrimination quantifies each model’s ability to distinguish between encounters with and encounters without all-cause 30-day readmission, and it was measured in this study using the AUROC, wherein a value of 1 indicates perfect discrimination and 0.5 is equivalent to chance (ie, coin flip). In general, an AUROC of 0.7 to 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent, and more than 0.9 is considered outstanding.33,34 Median AUROCs with 95% CIs were estimated across the other PHIS hospitals to summarize the aggregate performance of the DH PHIS models.

Calibration measures the degree of agreement between predicted and observed outcome frequencies across the full range of predicted risk.35 We created calibration plots of predicted vs observed risk for all-cause 30-day readmission for each PHIS hospital, evaluating for significant differences in calibration slope and intercept (null hypotheses of intercept = 0; slope = 1).

All analyses were performed from August 9 to December 1, 2023, using SAS, version 9.4 (SAS Institute Inc). Two-sided P < .05 was considered statistically significant.

Results

In external validation, a total of 851 499 children were discharged from 48 children’s hospitals. In 2019, there were 16 330 discharges from the DH and 835 169 discharges from the other PHIS hospitals (Table 1). The largest group of children was aged 5 to 14 years (281 193 [33.0%]). There were 165 020 (19.3%) Hispanic children, 409 419 (48.0%) non-Hispanic Black children, and 165 364 (19.4%) non-Hispanic White children.

Table 1. Patient Characteristics in the DH vs Other PHIS Hospitals, 2019.

Characteristic Patients, No. (%) Other PHIS hospitals, median (IQR), %a
DH Other PHIS hospitals
No. of discharges 16 330 835 169 16 111 (13 056-21 449)
Age at admission
<2 mo 1430 (8.8) 145 075 (17.4) 13.7 (8.8-21.3)
2-5 mo 1049 (6.4) 49 967 (6.0) 5.9 (5.2-6.8)
6-11 mo 1186 (7.3) 49 790 (6.0) 6.0 (5.4-6.7)
1-4 y 4831 (29.6) 200 740 (24.0) 25.1 (21.7-26.7)
5-14 y 5525 (33.8) 275 668 (33.0) 33.2 (29.5-37.1)
≥15 y 2309 (14.1) 113 929 (13.6) 13.7 (11.6-15.1)
ED admittance 9009 (55.2) 451 312 (54.0) 55.5 (45.8-62.3)
Insurance type
Government 8820 (54.0) 449 385 (53.8) 54.0 (48.7-61.0)
Commercial 7312 (44.8) 335 211 (40.1) 38.3 (32.9-45.7)
Other or unspecifiedb 198 (1.2) 50 573 (6.1) 4.9 (2.7-7.3)
LOS, d
<3 10 487 (64.2) 600 612 (71.9) 72.3 (68.2-74.6)
≥3 5843 (35.8) 234 557 (28.1) 27.7 (25.4-31.8)
PICU admission 3495 (21.4) 112 161 (13.4) 11.7 (10.0-16.4)
Primary diagnosis category, age <6 mo
Otherc 1174 (47.4) 130 547 (66.9) 53.5 (48.6-71.4)
ALTE or BRUE 60 (2.4) 2085 (1.1) 1.0 (0.6-1.7)
Cardiac 171 (6.9) 6503 (3.3) 3.7 (2.1-5.1)
Congenital anomalies 199 (8.0) 7610 (3.9) 4.2 (2.1-7.9)
Esophageal reflux 37 (1.5) 2650 (1.4) 1.5 (0.9-2.1)
Fever 85 (3.4) 3285 (1.7) 1.9 (1.4-2.5)
Respiratory, lower (bronchiolitis and pneumonia) 507 (20.5) 27 637 (14.2) 17.8 (10.7-21.6)
Respiratory, upper 52 (2.1) 2555 (1.3) 1.6 (0.9-2.1)
Neonatal jaundice 84 (3.4) 7080 (3.6) 3.6 (2.4-5.0)
Pyloric stenosis 42 (1.7) 2393 (1.2) 1.5 (0.8-1.8)
UTI and pyelonephritis 68 (2.7) 2697 (1.4) 1.3 (0.8-2.4)
Primary diagnosis category, age ≥6 mo
Otherc 8931 (64.5) 444 112 (69.4) 69.0 (64.7-72.5)
Appendicitis 69 (0.5) 12 919 (2.0) 1.8 (1.2-2.6)
Asthma 938 (6.8) 29 814 (4.7) 4.8 (3.1-6.3)
CNS shunt 84 (0.6) 2650 (0.4) 0.4 (0.3-0.5)
Dehydration or GI infection 569 (4.1) 19 753 (3.1) 2.8 (2.3-3.7)
Fever 128 (0.9) 3289 (0.5) 0.5 (0.3-0.7)
Respiratory, lower (bronchiolitis) 489 (3.5) 24 694 (3.9) 3.8 (2.4-5.1)
Respiratory, lower (pneumonia) 1483 (10.7) 45 352 (7.1) 6.7 (5.5-8.4)
Respiratory, upper 304 (2.2) 10 220 (1.6) 1.5 (1.2-2.0)
Seizure 663 (4.8) 38 736 (6.1) 5.7 (4.5-7.2)
Sickle cell anemia 193 (1.4) 8588 (1.3) 1.0 (0.6-1.8)
Any prior select procedures within 6 mo before admission 5387 (33.0) 196 507 (23.5) 23.4 (18.7-27.3)
Prior use within 6 mo before admission 6448 (39.5) 289 950 (34.7) 35.2 (31.7-38.9)
Race and ethnicityd
Hispanic, any race 5518 (33.8) 159 502 (19.1) 13.2 (3.8-31.6)
Non-Hispanic Black 6027 (36.9) 403 392 (48.3) 48.4 (33.6-64.1)
Non-Hispanic White 2849 (17.4) 162 515 (19.5) 17.5 (10.5-23.8)
Non-Hispanic other or unknown/unspecifiede 1936 (11.9) 109 760 (13.1) 10.0 (7.2-16.9)
Service line
Medical 13 804 (84.5) 683 821 (81.9) 85.5 (77.5-88.3)
Surgical 2526 (15.5) 151 348 (18.1) 14.5 (11.7-22.5)
Open in a new tab

Abbreviations: ALTE, apparent life-threatening event; BRUE, brief, resolved, unexplained event; CNS, central nervous system; DH, derivation hospital; ED, emergency department; GI, gastrointestinal; LOS, length of stay; PHIS, Pediatric Health Information System; PICU, pediatric intensive care unit; UTI, urinary tract infection.

a

Median percentage for each characteristic at the other PHIS hospitals excluding DH.

b

Other insurance type included uninsured, self-insured, and multiple insurances.

c

Any other diagnosis category not otherwise listed.

d

Race and ethnicity were derived from the PHIS database.

e

Other race and ethnicity included American Indian or Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, and multiracial.

EHR to PHIS Validation

The 2016 to 2018 DH cohort included 45 682 discharges. All-cause 30-day readmission rates for the model subpopulations were 7.2% (1855 of 25 750 discharges) for the NAM, 35.5% (4539 of 12 794 discharges) for the RAM, and 11.7% (831 of 7128 discharges) for the YIM (eTable 2 in Supplement 1). As expected, the predictive performance of the DH PHIS models was essentially identical to the previously published values for the DH cohort (Figure 1). Specifically, the AUROC for the NAM was 0.76 (95% CI, 0.75-0.78) in the DH PHIS models compared with 0.76 (95% CI, 0.75-0.77) in the original DH cohort, AUROC for the RAM was 0.84 (95% CI, 0.83-0.84) compared with 0.83 (95% CI, 0.82-0.84), and AUROC for the YIM was 0.79 (95% CI, 0.77-0.80) compared with 0.80 (95% CI, 0.79-0.82).20

Temporal Validation

The 2019 DH cohort included 16 330 discharges. All-cause 30-day readmission rates were 7.2% (676 of 9363 discharges) for the NAM, 35.1% (1574 of 4488 discharges) for the RAM, and 11.1% (275 of 2479 discharges) for the YIM. The AUROC was 0.65 (95% CI, 0.62-0.67) for the NAM, 0.73 (95% CI, 0.72-0.75) for the RAM, and 0.67 (95% CI, 0.63-0.70) for the YIM (Figure 1, temporal validation; eTable 2 in Supplement 1). These AUROCs appeared lower than the 2016 to 2018 estimates.

External Validation

The 2019 other PHIS hospitals cohort included 835 169 discharges, with median (range) hospital-level discharges of 16 111 (13 056-21 449). The readmission rates were 5.9% (27 798 of 470 202 discharges) for the NAM, 30.1% (51 177 of 169 925 discharges) for the RAM, and 7.6% (14 842 of 195 042 discharges) for the YIM (Table 2). Overall, 2.7% of patients had more than 1 readmission. Table 1 compares the characteristics of 2019 DH vs other PHIS hospital encounters and demonstrates differences in age at admission, race and ethnicity, LOS, and frequency of pediatric intensive care unit admission, prior select procedures, and prior use.

Table 2. External Validation: Primary Outcome and Discrimination for the DH PHIS and Other PHIS Hospitals, 2019.

DH PHIS (n = 16 330) Other PHIS hospitals (n = 835 169)
NAM RAM YIM NAM RAM YIM
No. of discharges 9363 4488 2479 470 202 169 925 195 042
30-d Readmission, No. (%) 676 (7.2) 1574 (35.1) 275 (11.1) 27 798 (5.9) 51 177 (30.1) 14 842 (7.6)
Other PHIS hospital–level median readmission (IQR), % NA NA NA 5.9 (5.4-6.3) 29.7 (27.4-33.0) 8.4 (6.5-9.9)
DH-level AUROC (95% CI) 0.65 (0.62-0.67) 0.73 (0.72-0.75) 0.67 (0.63-0.70) NA NA NA
Other PHIS hospital–level median AUROC (range) NA NA NA 0.64 (0.60-0.68) 0.73 (0.64-0.80) 0.65 (0.53-0.74)
Open in a new tab

Abbreviations: AUROC, area under the receiver operating characteristic curve; DH, derivation hospital; NA, not applicable; NAM, new admission model; PHIS, Pediatric Health Information System; RAM, recent admission model; YIM, young infant model.

Median AUROC values for the 2019 other PHIS hospitals were similar to those estimated for the 2019 DH cohort (Table 2 and Figure 1, external validation). However, variation in median (range) AUROCs was observed across individual hospitals (Figure 2), with 0.64 (0.60-0.68) for the NAM, 0.73 (0.64-0.80) for the RAM, and 0.65 (0.53-0.74) for the YIM. To explore the possible reasons for hospital-level variability in model performance, we used the RAM to compare hospital-level characteristics across tertiles of AUROC values for the other PHIS hospitals (eTable 3 in Supplement 1). We noted differences in median (IQR) readmission rate between the lowest and highest tertiles (28.3% [26.4%-29.3%] vs 32.9% [28.4%-35.4%]). There was a higher percentage of patients with 4 or more prior uses in the highest vs lowest tertile (33.8% vs 28.4%) and patients with 1 or more prior select procedures (49.9% vs 45.9%).

Figure 2. Violin Plot of Hospital-Level Median Area Under the Receiver Operating Characteristic Curves (AUROCs) of the External Validation of the 2019 Other Pediatric Health Information System (PHIS) Hospitals Cohort.

Figure 2.

Open in a new tab

The horizontal line in the middle of each plot represents the hospital-level median AUROC for each derivation hospital (DH) model. The shaded area above and below the median represents the 25th and 75th IQR, respectively. The width of each plot represents the density of observations across all other PHIS hospitals. NAM indicates new admission model; RAM, recent admission model; and YIM, young infant model.

Calibration

Calibration plots were generated for each of the PHIS hospitals to compare agreement between observed and predicted risk probabilities. Of the 47 other PHIS hospitals, only 3 for the RAM (6.4%) and 9 for both the NAM and the YIM (19.1%) were adequately calibrated. Figure 3 shows calibration plots for the RAM cohort at 4 hospitals to demonstrate variation in calibration performance, including risk underestimation as observed risk increased (Figure 3A), underestimation across the full range (Figure 3B), well-calibrated estimates (Figure 3C), and risk overestimation as observed risk increased (Figure 3D).

Figure 3. Four Calibration Plots for the Recent Admission Model (RAM) of the 2019 Other Pediatric Health Information System Hospitals.

Figure 3.

Open in a new tab

Ideal calibration is represented by the 45-degree dotted line through the origin. The circles represent observed risk, with whiskers representing 95% CIs at each decile of predicted risk.

Discussion

Using a multicenter cohort of children discharged from 48 US children’s hospitals, we evaluated the performance of a suite of single center–derived models for predicting all-cause 30-day readmissions. Temporal validation of the original readmission risk prediction models20 within the DH yielded reduced discriminatory performance (Figure 1, temporal validation). When the models were applied externally across 47 other children’s hospitals in the PHIS database, model discrimination varied at individual hospitals, although aggregate performance approximated the results from the temporal validation (Figure 1, external validation). The RAM demonstrated the best discrimination (median [range] AUROC, 0.73 [0.64-0.80]), which is on par or better than other published pediatric16,18,19,21,36,37 and adult27,28,38,39 readmission models.

Degradation in predictive performance is expected when models are applied in new settings,40 and several possibilities may explain the noted declines in this study. In the temporal DH validation (presumably stable population-level risk), seasonal differences in circulating respiratory viruses and other common infectious pathogens, including variation in risk for severe disease outcomes, may be factors in the risk for reuse that were not well captured in the models. Other population-level differences may further play a role in drift in discriminatory power. The DH models also included just a few predictors and imposed categorical variable transformations for parsimony and ease of interpretation, decisions that may be associated with information loss, especially if predictors are omitted or oversimplified.

These same factors may be associated with reduced discrimination in external settings. For example, the prevalence of all-cause 30-day readmission varied between the DH and each of the other PHIS hospitals. Median predictive performance across the other PHIS hospitals was overall consistent with performance under previously published pediatric risk prediction models23,41 but was suboptimal, with AUROCs ranging from 0.65 to 0.73.33 Performance was best for the RAM (AUROC, 0.73) likely because prior use, a requirement for RAM cohort inclusion, was one of the strongest predictors of subsequent use. Supporting this assertion was that the RAM cohort had the highest prevalence of children with medical complexity, a population at high risk for recurrent use.42 The YIM had the widest variability in model performance across hospitals and low overall risk for readmission. In addition, only 4 predictors were included in the YIM, suggesting that unmeasured heterogeneity in young infant care was associated with the observed variation. Considering new predictors to better capture important variation within this population would likely play a role in improved performance. Regardless of the reasons, the observed temporal and hospital-level variability suggests each model could potentially be improved and highlights the importance of local validation prior to clinical implementation.

Two studies have applied the LACE Index (LOS, acuity of admission, comorbidities, and number of ED visits in the 6 months before admission), an adult-derived 30-day readmission model, to pediatric data.29,43 In both studies, discrimination (AUROCs of 0.68 [single center] and 0.70 [multicenter]) was similar to discrimination in the DH PHIS models in the current study.18,44 However, there may be limited clinical utility of the LACE Index in pediatric cohorts given its reliance on an adult-specific comorbidity measure (ie, Charlson Comorbidity Index) and that it is calibrated to use variables collected at discharge. Additionally, efforts to adapt the LACE Index by incorporating an expanded set of demographic variables have yielded suboptimal results in children.45 The DH PHIS models are specific to pediatric cohorts, suggesting they may be a preferable starting point over adult-derived models.

While the DH models overcame several limitations noted in prior studies, important opportunities to improve performance remain. Multicenter derivation of new models could increase generalizability and discriminatory power. Advanced modeling approaches, such as maintaining continuous predictors vs dichotomizing variables, leveraging data transformation (eg, spline functions) to relax linearity assumptions, and modeling higher-order interactions, are techniques that help to avoid information loss. The inclusion of additional predictors, such as the presence and degree of medical complexity, illness severity measures, and health-related social needs, may play a role in enhanced performance. For example, medical complexity, and comorbidities more generally,36,37,46 are key risk factors for readmission, but these constructs were not well captured in the original models. While these factors can be difficult to define using EHR data from an index encounter, querying prior encounters (ie, 1-year lookback) could be conducted to classify comorbidities using validated diagnostic classification systems, such as the pediatric Complex Chronic Conditions classification system47 or the Pediatric Medical Complexity Algorithm.48 Similarly, reporting area- and patient-level measures of social needs and assets is increasingly common within EHRs, and these data may help predict reuse unexplained by baseline health status or illness acuity. Large language models that mine notes and other unstructured EHR fields could also be factors in enhanced performance.49 The predictors in the DH models may serve as a useful starting point for hospitals wanting to use risk models to guide resource allocation and target hospital-to-home transition interventions where they are most needed. However, differences in model performance highlight the need for local validation, recalibration, and potentially model updating prior to clinical adoption.

Limitations

This study has several limitations. First, we were unable to ascertain readmissions to hospitals not participating in the PHIS database, resulting in potential misclassification and underreporting of readmissions. However, prior studies have demonstrated that different hospital readmissions are less common within children’s hospitals than general hospitals.50 We did not include general hospitals that provide care to children, and the models may perform differently in those settings. Second, patterns of hospital use changed after the onset of the COVID-19 pandemic. Future studies should include contemporary data when designing, validating, or updating new models. Third, the original models20 did not differentiate between planned and unplanned readmissions. No differentiation may affect generalizability for hospitals that have substantially more or fewer planned readmissions.

Conclusions

In this prognostic study, temporal validation of a suite of readmission risk prediction models at the same DH and external validation at 47 other children’s hospitals demonstrated reduced discriminatory performance compared with performance in the original derivation study. Hospital-level variability in predictive performance suggests the need to create more accurate and generalizable models as well as the need for rigorously designed impact analyses to ascertain clinical utility in reducing preventable use and improving outcomes.

Supplement 1.

eTable 1. Model Cohort Definition, Predictors and Predictive Performance of Original Goodman et al. Study

eTable 2. Temporal Validation: Readmission Prevalence and Performance of Prognostic Models Comparing Derivation Hospital (DH) PHIS Models, 2016-2018 to DH PHIS, 2019

eTable 3. Patient-Level Encounter Characteristics for RAM Cohort by AUROC Tertile

jamanetwopen-e2459684-s001.pdf (102.9KB, pdf)
Supplement 2.

Data Sharing Statement

jamanetwopen-e2459684-s002.pdf (13.3KB, pdf)

References

  • 1.Kaiser SV, Rodean J, Coon ER, Mahant S, Gill PJ, Leyenaar JK. Common diagnoses and costs in pediatric hospitalization in the US. JAMA Pediatr. 2022;176(3):316-318. doi: 10.1001/jamapediatrics.2021.5171 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Agency for Healthcare Research and Quality. HCUP, Healthcare Cost and Utilization Project. Accessed March 4, 2024. https://datatools.ahrq.gov/hcupnet/
  • 3.Oklahoma Health Care Authority . Hospital Potentially Preventable Readmissions (PPR) program. April 30, 2019. Accessed May 7, 2024. https://oklahoma.gov/content/dam/ok/en/okhca/documents/a0302/23350.pdf
  • 4.New York State Health Foundation . Reducing hospital readmissions in New York State: a simulation analysis of alternative payment incentives. September 2011. Accessed May 7, 2024. https://nyhealthfoundation.org/wp-content/uploads/2017/12/reducing-hospital-readmissions-payment-incentives-september-2011.pdf
  • 5.Texas Medicaid and Healthcare Partnership. Potentially preventable events. August 18, 2023. Accessed May 7, 2024. https://www.hhs.texas.gov/about/process-improvement/improving-services-texans/medicaid-chip-quality-efficiency-improvement/potentially-preventable-events
  • 6.Illinois Department of Healthcare and Family Services . Potentially preventable readmissions policy. 2016. Accessed May 7, 2024. https://hfs.illinois.gov/medicalproviders/hospitals/pprreports.html
  • 7.National Quality Forum . All-cause admissions and readmissions measures. Accessed February 1, 2023. https://www.qualityforum.org/Publications/2015/04/All-Cause_Admissions_and_Readmissions_Measures_-_Final_Report.aspx
  • 8.Berry JG, Toomey SL, Zaslavsky AM, et al. Pediatric readmission prevalence and variability across hospitals. JAMA. 2013;309(4):372-380. doi: 10.1001/jama.2012.188351 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Toomey SL, Peltz A, Loren S, et al. Potentially preventable 30-day hospital readmissions at a children’s hospital. Pediatrics. 2016;138(2):e20154182. doi: 10.1542/peds.2015-4182 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Hamline MY, Speier RL, Vu PD, et al. Hospital-to-home interventions, use, and satisfaction: a meta-analysis. Pediatrics. 2018;142(5):e20180442. doi: 10.1542/peds.2018-0442 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Auger KA, Kenyon CC, Feudtner C, Davis MM. Pediatric hospital discharge interventions to reduce subsequent utilization: a systematic review. J Hosp Med. 2014;9(4):251-260. doi: 10.1002/jhm.2134 [DOI] [PubMed] [Google Scholar]
  • 12.Coller RJ, Klitzner TS, Lerner CF, et al. Complex care hospital use and postdischarge coaching: a randomized controlled trial. Pediatrics. 2018;142(2):e20174278. doi: 10.1542/peds.2017-4278 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.deJong NA, Kimple KS, Morreale MC, Hang S, Davis D, Steiner MJ. A quality improvement intervention bundle to reduce 30-day pediatric readmissions. Pediatr Qual Saf. 2020;5(2):e264. doi: 10.1097/pq9.0000000000000264 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Auger KA, Shah SS, Tubbs-Cooley HL, et al. ; Hospital-to-Home Outcomes Trial Study Group . Effects of a 1-time nurse-led telephone call after pediatric discharge: the H2O II randomized clinical trial. JAMA Pediatr. 2018;172(9):e181482. doi: 10.1001/jamapediatrics.2018.1482 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Auger KA, Simmons JM, Tubbs-Cooley HL, et al. ; H2O Trial Study Group . Postdischarge nurse home visits and reuse: the Hospital to Home Outcomes (H2O) trial. Pediatrics. 2018;142(1):e20173919. doi: 10.1542/peds.2017-3919 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Leary JC, Price LL, Scott CER, Kent D, Wong JB, Freund KM. Developing prediction models for 30-day unplanned readmission among children with medical complexity. Hosp Pediatr. 2019;9(3):201-208. doi: 10.1542/hpeds.2018-0174 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Pugh K, Granger D, Lusk J, et al. Targeted clinical interventions for reducing pediatric readmissions. Hosp Pediatr. 2021;11(10):1151-1163. doi: 10.1542/hpeds.2020-005786 [DOI] [PubMed] [Google Scholar]
  • 18.Ehwerhemuepha L, Finn S, Rothman M, Rakovski C, Feaster W. A novel model for enhanced prediction and understanding of unplanned 30-day pediatric readmission. Hosp Pediatr. 2018;8(9):578-587. doi: 10.1542/hpeds.2017-0220 [DOI] [PubMed] [Google Scholar]
  • 19.Brittan MS, Martin S, Anderson L, Moss A, Torok MR. An electronic health record tool designed to improve pediatric hospital discharge has low predictive utility for readmissions. J Hosp Med. 2018;13(11):779-782. doi: 10.12788/jhm.3043 [DOI] [PubMed] [Google Scholar]
  • 20.Goodman DM, Casale MT, Rychlik K, et al. Development and validation of an integrated suite of prediction models for all-cause 30-day readmissions of children and adolescents aged 0 to 18 years. JAMA Netw Open. 2022;5(11):e2241513. doi: 10.1001/jamanetworkopen.2022.41513 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Taylor T, Altares Sarik D, Salyakina D. Development and validation of a web-based pediatric readmission risk assessment tool. Hosp Pediatr. 2020;10(3):246-256. doi: 10.1542/hpeds.2019-0241 [DOI] [PubMed] [Google Scholar]
  • 22.Jovanovic M, Radovanovic S, Vukicevic M, Van Poucke S, Delibasic B. Building interpretable predictive models for pediatric hospital readmission using Tree-Lasso logistic regression. Artif Intell Med. 2016;72:12-21. doi: 10.1016/j.artmed.2016.07.003 [DOI] [PubMed] [Google Scholar]
  • 23.Nakamura MM, Toomey SL, Zaslavsky AM, et al. Potential impact of initial clinical data on adjustment of pediatric readmission rates. Acad Pediatr. 2019;19(5):589-598. doi: 10.1016/j.acap.2018.09.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Riley RD, Archer L, Snell KIE, et al. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ. 2024;384:e074820. doi: 10.1136/bmj-2023-074820 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Collins GS, Dhiman P, Ma J, et al. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ. 2024;384:e074819. doi: 10.1136/bmj-2023-074819 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-73. doi: 10.7326/M14-0698 [DOI] [PubMed] [Google Scholar]
  • 27.Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. 2013;173(8):632-638. doi: 10.1001/jamainternmed.2013.3023 [DOI] [PubMed] [Google Scholar]
  • 28.Donzé JD, Williams MV, Robinson EJ, et al. International validity of the hospital score to predict 30-day potentially avoidable hospital readmissions. JAMA Intern Med. 2016;176(4):496-502. doi: 10.1001/jamainternmed.2015.8462 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182(6):551-557. doi: 10.1503/cmaj.091117 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Feudtner C, Levin JE, Srivastava R, et al. How well can hospital readmission be predicted in a cohort of hospitalized children? a retrospective, multicenter study. Pediatrics. 2009;123(1):286-293. doi: 10.1542/peds.2007-3395 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.World Health Organization . ICD-10: International Statistical Classification of Diseases and Related Health Problems, Tenth Revision. 2nd ed. World Health Organization; 2004. Accessed January 1, 2024. https://iris.who.int/handle/10665/42980
  • 32.Antoon JW, Williams DJ, Thurm C, et al. The COVID-19 pandemic and changes in healthcare utilization for pediatric respiratory and nonrespiratory illnesses in the United States. J Hosp Med. 2021;16(5):294-297. doi: 10.12788/jhm.3608 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9):1315-1316. doi: 10.1097/JTO.0b013e3181ec173d [DOI] [PubMed] [Google Scholar]
  • 34.Hosmer D, Lemeshow S. Applied Logistic Regression. 2nd ed. John Wiley and Sons; 2000. doi: 10.1002/0471722146 [DOI] [Google Scholar]
  • 35.Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer; 2009. doi: 10.1007/978-0-387-77244-8 [DOI] [Google Scholar]
  • 36.Zhou H, Della PR, Porter P, Roberts PA. Risk factors associated with 30-day all-cause unplanned hospital readmissions at a tertiary children’s hospital in Western Australia. J Paediatr Child Health. 2020;56(1):68-75. doi: 10.1111/jpc.14492 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Niehaus IM, Kansy N, Stock S, Dötsch J, Müller D. Applicability of predictive models for 30-day unplanned hospital readmission risk in paediatrics: a systematic review. BMJ Open. 2022;12(3):e055956. doi: 10.1136/bmjopen-2021-055956 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: model development and comparison. J Hosp Med. 2016;11(7):473-480. doi: 10.1002/jhm.2568 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Shaffer BK, Cui Y, Wanderer JP. Validation of the LACE readmission and mortality prediction model in a large surgical cohort: comparison of performance at preoperative assessment and discharge time points. J Clin Anesth. 2019;58:22-26. doi: 10.1016/j.jclinane.2019.04.039 [DOI] [PubMed] [Google Scholar]
  • 40.Harrell FE Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Springer; 2015. doi: 10.1007/978-3-319-19425-7 [DOI] [Google Scholar]
  • 41.Zhou H, Roberts PA, Dhaliwal SS, Della PR. Risk factors associated with paediatric unplanned hospital readmissions: a systematic review. BMJ Open. 2019;9(1):e020554. doi: 10.1136/bmjopen-2017-020554 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Berry JG, Hall M, Hall DE, et al. Inpatient growth and resource use in 28 children’s hospitals: a longitudinal, multi-institutional study. JAMA Pediatr. 2013;167(2):170-177. doi: 10.1001/jamapediatrics.2013.432 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Hakim MA, Garden FL, Jennings MD, Dobler CC. Performance of the LACE index to predict 30-day hospital readmissions in patients with chronic obstructive pulmonary disease. Clin Epidemiol. 2017;10:51-59. doi: 10.2147/CLEP.S149574 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Ehwerhemuepha L, Gasperino G, Bischoff N, Taraman S, Chang A, Feaster W. HealtheDataLab - a cloud computing solution for data science and advanced analytics in healthcare with application to predicting multi-center pediatric readmissions. BMC Med Inform Decis Mak. 2020;20(1):115. doi: 10.1186/s12911-020-01153-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Douillard J, Lentz S, Ganjian S, et al. Predictive value of LACE scores for pediatric readmissions. Perm J. 2024;28(2):9-15. doi: 10.7812/TPP/23.114 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Berry JG, Hall DE, Kuo DZ, et al. Hospital utilization and characteristics of patients experiencing recurrent readmissions within children’s hospitals. JAMA. 2011;305(7):682-690. doi: 10.1001/jama.2011.122 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Feudtner C, Feinstein JA, Zhong W, Hall M, Dai D. Pediatric Complex Chronic Conditions Classification System version 2: updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatr. 2014;14(1):199. doi: 10.1186/1471-2431-14-199 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Simon TD, Haaland W, Hawley K, Lambka K, Mangione-Smith R. Development and validation of the Pediatric Medical Complexity Algorithm (PMCA) version 3.0. Acad Pediatr. 2018;18(5):577-580. doi: 10.1016/j.acap.2018.02.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Jiang LY, Liu XC, Nejatian NP, et al. Health system-scale language models are all-purpose prediction engines. Nature. 2023;619(7969):357-362. doi: 10.1038/s41586-023-06160-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Khan A, Nakamura MM, Zaslavsky AM, et al. Same-hospital readmission rates as a measure of pediatric quality of care. JAMA Pediatr. 2015;169(10):905-912. doi: 10.1001/jamapediatrics.2015.1129 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement 1.

eTable 1. Model Cohort Definition, Predictors and Predictive Performance of Original Goodman et al. Study

eTable 2. Temporal Validation: Readmission Prevalence and Performance of Prognostic Models Comparing Derivation Hospital (DH) PHIS Models, 2016-2018 to DH PHIS, 2019

eTable 3. Patient-Level Encounter Characteristics for RAM Cohort by AUROC Tertile

jamanetwopen-e2459684-s001.pdf (102.9KB, pdf)
Supplement 2.

Data Sharing Statement

jamanetwopen-e2459684-s002.pdf (13.3KB, pdf)

Articles from JAMA Network Open are provided here courtesy of American Medical Association

RESOURCES