Prediction of 1 and 2 week nonelective hospitalization and sepsis hospitalization risk in adults

Abstract

We developed and validated models to predict 1- and 2-week risk of non-elective hospitalization (NEH) and sepsis hospitalization following outpatient clinic, emergency department treat and release (EDTR), or hospitalization encounters. We employed data from 4,488,579 adults with 1,481,430 hospital, 6,035,296 EDTR, and 86,013,893 clinic encounters. Predictors included administrative, clinical (laboratory tests, vital signs), utilization, and prescription pattern data. We employed 2012–2018 data for development and 2019 data for validation. In validation datasets, discrimination (area under the receiver operator characteristic curve) ranged from 0.687 for NEH within 1 week of hospital discharge to 0.904 for sepsis hospitalization within 2 weeks of clinic visits. At a sensitivity of 40%, numbers needed to evaluate (NNE) ranged from 4.3 for NEH within 2 weeks of hospitalization to 45 for sepsis hospitalization within 1 week of a clinic visit. Our models have potentially clinically actionable NNEs and could support clinical programs for the prevention of short-term hospitalizations and sepsis.

Introduction

Widespread adoption of comprehensive electronic health records (EHRs) has resulted in increased availability of detailed clinical data. These data provide new opportunities for the development of predictive models that can be used for multiple purposes. One important area needing research attention is that of predicting non-elective hospitalization (NEH). While considerable efforts have focused on the prediction of short-term (within 30 days) rehospitalization^1–4 or medium-term (6–12 months) hospitalization^5–7 risk, it stands to reason that prevention strategies must also focus on preventing hospitalization in the first place. Prediction of hospitalization risk could facilitate efforts to mitigate adverse consequences of hospitalization (including rehospitalization). Further, given highly granular clinical data, prediction of hospitalization within short time frames from some T₀ may be feasible.

In a recent study, our team examined the incidence and timing of infection and sepsis hospitalization within 7 days of living hospital discharge, discharge from an emergency department treat and release (EDTR) encounter, and ambulatory visit settings⁸. Our results suggested that prediction of future sepsis hospitalizations within 7 days of some T₀ was feasible, in that theoretical numbers needed to evaluate (NNEs) in these 3 settings could be as low as 15–25, a potentially clinically actionable level. Sepsis is a major contributor to hospital morbidity and mortality, and delays in its care are associated with worsened outcomes. Thus, opportunities to identify patients earlier in their trajectory of infection and organ dysfunction could potentially facilitate prevention efforts and impact sepsis mortality and utilization.

In this report, we describe the continuation and expansion of this work. Our goal was to examine the feasibility of predicting not just sepsis hospitalizations but also non-elective hospitalizations (NEHs) in general. The predictive models we developed could be used to identify patients at high risk of being hospitalized within 1 or 2 weeks of a specific clinical encounter. Given that KPNC already has programs to identify patients at risk for hospitalization within 30 to 365 days, short-term models could be used by case managers for prioritization, thus enhancing the efficiency of such programs. A challenge identified by our end-users with prediction models designed to risk-stratify longer-term outcomes is that the correlation between interventions and risk reduction sometimes does not seem as actionable. Thus, predictions with shorter latency might have more acceptance by case managers and clinicians. Further, such models also have potential use for risk adjustment and for understanding the clinical epidemiology of multiple conditions, not just sepsis.

In our previous work on prediction of rehospitalization (i.e., prediction based on living hospital discharge), we went beyond just using hospital length of stay and diagnoses, and our predictive model, still in successful operational use, also includes laboratory test results and vital signs^9,10. In this report, we also describe how we evaluated other predictors now available in comprehensive EHRs (e.g., outpatient medication orders).

Results

Data description

During the study period, a total of 4,488,579 adults had an eligible encounter at a KPNC facility, which included 1,481,430 hospital, 6,035,296 EDTR, and 86,013,893 clinic encounters. Figure 1 displays the distribution of encounters among these adults and the apportionment of these encounters in the derivation and temporal validation cohorts (additional data are provided in the Supplementary Information). Rates of the outcome of interest ranged from 0.13% (for a sepsis hospitalization within 1 week of an outpatient encounter) to 9.2% (for a non-elective hospitalization within 2 weeks of discharge from the hospital).

Fig. 1. Flow diagram delineating study cohort assembly.

The unit of analysis was an individual patient encounter, which could be a hospital discharge, emergency department treat and release (EDTR) visit, or a scheduled clinic visit. Outcome rates (incidence rates for non-elective [NEH] or sepsis hospitalizations within 1 and 2 weeks from the T₀) are those for the entire cohort.

Table 1 characterizes the three encounter cohorts. Nonmembers do not come to KPNC clinics but are treated in KPNC EDs (where 79.0% of the encounters involved KFHP members) and hospitals (where 92.5% of the encounters involved KFHP members). In all cohorts, males were slightly underrepresented (48.6, 43.2, and 38.1% of the patients of hospital, EDTR, and clinic encounters, respectively). With respect to automated indices (COPS2, abLAPS, LAPS2) that capture comorbidity burden and acute physiology (where higher numbers indicate higher short- and medium-term mortality risk), hospital encounters involved sicker patients than EDTR encounters, and these, in turn, involved sicker patients than those with clinic encounters. Similar progression was seen across all variables we measured: hospital encounters had patients with higher utilization, frequency of laboratory tests, medications, and disease conditions compared to EDTR encounters. These encounters, in turn, had patients with higher utilization, laboratory tests, medications, and disease conditions compared to those with clinic encounters.

Table 1.

Characteristics of study cohort^a

Variable	Hospital	EDTRb	Clinic
Number of encounters	1,481,430	6,035,296	86,013,893
Unique patients	690,996	2,161,082	4,085,884
KFHPc member at the time of the encounter	92.5%	79%	100%
Age (mean, SDd)	66.4 (16.8)	52.5 (19.2)	55.6 (17.7)
Sex (% male)	719,500 (48.6)	2,606,769 (43.2)	32,812,055 (38.1)
COPS2e	44.4 (38.5)	19.1 (22.4)	16.9 (18.4)
abLAPSf	25.6 (20.1)	11.7 (11.9)	7.1 (11.3)
LAPS2g	44.1 (27.8)	30.2 (20.9)	NAg
Utilization in preceding 12 months [mean (SD)]
Number of encounters	15.4 (14.6)	10.1 (14.1)	11.7 (14.5)
Clinic visits	13.3 (12.9)	8.1 (11.4)	11.1 (14.0)
EDTR visits	1.2 (2.7)	1.8 (5.8)	0.4 (1.2)
Non-elective hospitalizations	0.7 (1.6)	0.2 (0.9)	0.1 (0.5)
All hospitalizations	0.9 (1.8)	0.3 (1.0)	0.2 (0.6)
Non-elective hospitalization LOSh	3.1 (8.6)	0.9 (4.3)	0.4 (2.4)
All hospitalization LOS	4 (10.5)	1.1 (5.1)	0.6 (3.4)
Test ordered in preceding 30 days (N, %)
Potassium	541,647 (36.6)	556,106 (9.2)	9,077,396 (10.6)
Aspartate transferase	767,520 (51.8)	1,418,185 (23.5)	5,056,102 (5.9)
Alkaline phosphatase	714,818 (48.3)	1,340,658 (22.2)	3,669,607 (4.3)
Ferritin, iron, and TIBCi	150,576 (10.2)	137,945 (2.3)	2,073,138 (2.4)
Medication order in the preceding yearj (N, %)
Antibiotic	1,042,543 (70.4)	3,433,401 (56.9)	42,226,316 (49.1)
Gastrointestinal	1,132,392 (76.4)	2,908,171 (48.2)	32,144,110 (37.4)
Analgesic	1,064,343 (71.8)	3,026,843 (50.1)	30,731,677 (35.7)
Psychotherapeutic drugs	592,733 (40.0)	1,847,870 (30.6)	23,301,072 (27.1)
Antiasthmatic	441,538 (29.8)	1,356,305 (22.5)	15,883,054 (18.5)
Diuretic	474,977 (32.1)	797,934 (13.2)	11,078,226 (12.9)
Antihyperglycemic	370,065 (25.0)	740,771 (12.3)	10,470,248 (12.2)
Diagnosis assigned in the previous yeark (N, %)
Vascular disease	333,644 (22.5)	248,455 (4.1)	2,110,243 (2.5)
Specified arrhythmias	300,660 (20.3)	227,663 (3.8)	2,045,870 (2.4)
Diabetes without complication	243,785 (16.5)	207,513 (3.4)	1,801,351 (2.1)
Septicemia/sepsis	293,103 (19.8)	179,072 (3.0)	1,310,177 (1.5)
COPDk	181,337 (12.2)	148,327 (2.5)	1,154,971 (1.3)
Major depressive, bipolar, and paranoid disorders	123,612 (8.3)	140,071 (2.3)	1,039,603 (1.2)
Acute renal failure	216,964 (14.6)	131,748 (2.2)	987,919 (1.1)
Breast, prostate, and other cancers and tumors	50,614 (3.4)	41,046 (0.7)	840,591 (1.0)

^aTable shows characteristics of the entire study cohort prior to its division for modeling purposes (2012–2018 cohorts were randomly split into training, validation and test datasets; 2019 cohort was used for the final temporal validation dataset).

^bEDTR emergency department treat and release visit.

^cKFHP Kaiser Foundation Health Plan, Inc. Note that non-KFHP members are not seen in Kaiser Permanente Northern California outpatient clinics but can be cared for in the organization’s emergency departments or hospitals.

^dSD standard deviation.

^eCOPS2 the COmorbidity Point Score, version 2 [Escobar, 2013] is assigned every month to all adults with a Kaiser Permanente Northern California medical record number. The range is from 0 to 1010; higher scores indicate worse mortality risk. The univariate relationship between the COPS2 and 1-year mortality is as follows: 0 to 39, 0.3%; 40 to 64, 5.3%; ≥65, 17.2%.

^fabLAPS the abbreviated laboratory-based acute physiology score, is a monthly score using 14 laboratory tests that is based on the LAPS score [Escobar, 2020]. The range is from 0 to 256; higher scores indicate increasing physiologic abnormalities in the preceding month. In recent internal analyses, the univariate relationship between the abLAPS and 30-day mortality is as follows: 0 to 4, 0.06%; 4 to 9, 0.18%; ≥10, 1.32%.

^gLAPS2 the admission laboratory-based acute physiology score, version 2 [Escobar, 2013] is assigned on the basis of a patient’s worst vital signs, pulse oximetry, neurologic status, and 16 laboratory test results in the 72 h preceding hospital entry. The univariate relationship of an admission LAPS2 with 30-day mortality is as follows: 0 to 59, 1.0%; 60 to 109, 5.0%; 110, 13.7%. After age, sex, diagnosis, and comorbid conditions are controlled for, the adjusted odds ratio for inpatient mortality for an increase in LAPS2 of five points is 1.13. Note that this hospital score is not assigned to outpatient visits.

^hLOS total hospital length of stay in days for non-elective hospitalizations and all hospitalizations.

ⁱTIBC total iron binding capacity.

^jRefers to prescription orders in the preceding year. See text for details.

^kRefers to coded diagnoses in any encounter in the preceding year. See text for details. COPD chronic obstructive pulmonary disease.

Model predictors and their contribution

Table 2 summarizes the broad categories (dimensions) of the 354 variables initially considered, as well as the relative contribution to the final models’ overall explanatory power. Automated indices contributed most to the predictive ability of the final NEH 2-week models (31 to 42%). Prior utilization contributed most to the hospital cohort (25%) compared to clinic and EDTR cohorts (20 and 16%, respectively), whereas vital signs contributed more to the EDTR cohort (21%) relative to the hospital and clinic cohorts (17 and 12%, respectively). Laboratory tests contributed little to the clinic cohort (1%), slightly more for EDTR and hospital cohorts (6.5 and 10.5%), and the contribution of medications to the three cohorts was somewhat similar (11.5, 14, and 10.5% for hospital, EDTR and clinic cohorts). Our final models did not include Hierarchical Condition Categories, as incorporating these variables did not result in substantive performance improvement. However, the addition of laboratory tests, medications, and vital signs did result in improvements in all models, as evidenced by increased AUROCs and AUPRCs and decreased NNEs. For example, among hospital patients, the base model NNE at a sensitivity of 40% was 7.8 for NEH within 1 week and 4.6 within 2 weeks. In contrast, the corresponding NNEs for our final models were 7.3 for NEH within 1 week (a 6.4% decrease) and 4.3 (a 6.5% decrease) within 2 weeks.

Table 2.

Variables evaluated and retained In NEH 2-week models^a

Dimension	Variables	Evaluated	Retained	Hospital Contributionb	EDTR Contribution	Clinic Contribution
Demographics	Age, gender	2	2	4.0	10.3	13.5
Membershipc	Membership status	1	1	0.04	0.7	NA
Utilizationd	Counts of outpatient visits, EDTRs, and hospitalizations and length of stay of hospitalizations prior to T0;	40	40	25.0	15.9	20.3
Laboratory testse	Laboratory test orders in the 30 days prior to T0	21	21	10.5	6.5	1.2
Medications	Outpatient prescriptions in 2 weeks and year before T0; new orders month before T0	147	145	11.5	14.2	10.5
Diagnosesf	Hierarchical Condition Categories	70	0	NA	NA	NA
Vital signsg	BP, HR, temperature, RR, FiO2, saturation, weight, height, BMI, neurologic score	70	70	17.3	21.4	12.3
Automated indicesh–j	LAPS2, abLAPS, COPS2	3	3	31.8	31.0	42.2

^aSee text and Supplementary Information for additional details on the methodology and variables evaluated. NEH non-elective hospitalization. Results for sepsis models were similar.

^bThis refers to the relative contribution (expressed as a percent) of the retained variable(s) in a dimension to the overall predictive ability of the model. See the text for details on how this was calculated.

^cRefers to whether the patient was a member of Kaiser Foundation Health Plan, Inc., or not at the time of the encounter.

^dSee text and Supplementary Information for details on how utilization data were structured for analysis.

^eThis dimension refers to whether or not a test was ordered (independent of the test result). Test results are considered in two of the automated indices described in footnote h, below.

^fDiagnoses incurred by the patient in the 12 months preceding the T₀ were grouped into 70 Hierarchical Condition Categories (HCCs) defined by the Centers for Medicare and Medicaid Services (CMS-HCC version 22 ICD10, available at www.CMS.gov).

^gSee Supplementary Information for details of how these variables were structured for analysis.

^hLAPS2 the admission laboratory-based acute physiology score, version 2 [Escobar, 2013] is assigned on the basis of a patient’s worst vital signs, pulse oximetry, neurologic status, and 16 laboratory test results in the 72 h preceding hospital entry. The univariate relationship of an admission LAPS2 with 30-day mortality is as follows: 0 to 59, 1.0%; 60 to 109, 5.0%; 110, 13.7%. After age, sex, diagnosis, and comorbid conditions are controlled for, the adjusted odds ratio for inpatient mortality for an increase in LAPS2 of 5 points is 1.13.

ⁱCOPS2 the COmorbidity Point Score, version 2 [Escobar, 2013] is assigned every month to all adults with a Kaiser Permanente Northern California medical record number. Range is from 0 to 1010; higher scores indicate worse mortality risk. The univariate relationship between the COPS2 and 1-year mortality is as follows: 0 to 39, 0.3%; 40 to 64, 5.3%; ≥65, 17.2%.

^jabLAPS the abbreviated laboratory-based acute physiology score, is a monthly score using 14 laboratory tests that is based on the LAPS score[Escobar, 2020]. The range is from 0 to 256; higher scores indicate increasing physiologic abnormalities in the preceding month. In recent internal analyses, the univariate relationship between the abLAPS and 30-day mortality is as follows: 0 to 4, 0.06%; 4 to 9, 0.18%; ≥10, 1.32%.

Different variable dimensions affected AUPRCs to different degrees depending on the outcome and prediction locus. Thus, for patients being discharged from the hospital or ED (where laboratory tests are routine), little improvement was seen from the addition of laboratory tests and medications (~2% increase in AUPRC) for NEH models, whereas the AUPRC increased by ~7 to 9% after the addition of laboratory tests and medications for sepsis models. Among clinic patients, the AUPRC increased by ~9% after the addition of laboratory tests and medications to the NEH 1-week model but by only ~3% for the NEH 2-week model. All models were well calibrated at a predicted risk of <30% (see Supplementary Information).

Model performance

Table 3 and Fig. 2 summarize key findings when the model trained with data from 2012–2018 was applied to the temporal validation cohort (data from 2019). Table 3 displays the AUROC, NNE at 40% sensitivity, and sensitivity at NNE = 10, while Fig. 2 displays the relationship between sensitivity (x-axis) and NNE (y-axis) when the sensitivity is ≤60% and the NNE is ≤40. With respect to discrimination, AUROC was higher in the 2-week models than in the 1-week models and was higher for the Sepsis-3 hospitalization outcome than in the non-elective hospitalization outcome. The AUROC was lowest in the hospital models and highest in the clinic models. Thus, the lowest AUROC (0.687) was for NEH within 1 week of a hospitalization discharge, while the highest (0.904) was for a Sepsis-3 hospitalization within 2 weeks of a clinic visit. In contrast, the models with the lowest NNE (i.e., those models with the fewest false positives) at a sensitivity of 40% were the hospital models for an NEH within 1 week (7.3) and 2 weeks (4.3) of the T₀, whereas those with the highest NNE at that sensitivity were the clinic models for a sepsis hospitalization (NNE of 44.5 and 25.5 at 1 and 2 weeks, respectively).

Table 3.

Model performance in temporal validation cohort^a

	Outcomeb
Time of predictionc Metrics	Non-elective hospitalization within 1 week	Non-elective hospitalization within 2 weeks	Sepsis hospitalization within 1 week	Sepsis hospitalization within 2 weeks
At hospital discharge	Rated: 6.38%	Rate: 10.07%	Rate: 1.99%	Rate: 3.21%
AUROCe	0.687	0.711	0.797	0.803
NNEf at 40% sensitivity	7.3	4.3	13.0	7.8
Sensitivity at NNE = 10	69.0%	100%	22.9%	60.1%
At discharge from EDTRg	Rate: 2.5%	Rate: 3.5%	Rate: 0.5%	Rate: 0.8%
AUROC	0.811	0.821	0.889	0.893
NNE at 40% sensitivity	8.7	5.7	17.5	11.2
Sensitivity at NNE = 10	50.4%	72.7%	10.9%	33.1%
At end of outpatient visit	Rate: 0.60%	Rate: 0.94%	Rate: 0.12%	Rate: 0.20%
AUROC	0.824	0.827	0.896	0.904
NNE at 40% sensitivity	21.9	13.6	44.5	25.5
Sensitivity at NNE = 10	14.3%	28.9%	1.5%	9.1%

^aCoefficients from the derivation cohorts (patient encounters from 2012–2018) were applied to the holdout (temporal validation) cohort (patient encounters in 2019). See the text for details on the modeling process.

^bSee text for definitions of non-elective and sepsis hospitalizations that occurred within 1 and 2 weeks of the prediction T₀. The rate listed under each outcome, provided for comparison purposes, is the incident rate of the specified outcome within 1 or 2 weeks following the T₀, which differs slightly from the incident rate shown in Fig. 1.

^cTime of prediction refers to when the prediction was made (i.e., a physical locus where a prediction would be issued at the T₀).

^dRate of the outcome of interest (non-elective or sepsis hospitalization) within the indicated time frame (1 or 2 weeks from the T₀).

^eAUROC area under the receiver operator characteristic curve. Supplementary Information provides confidence intervals and also areas under the precision-recall curve for the different models.

^fNNE number needed to evaluate (1/positive predictive value).

^gEDTR treat-and-release emergency department encounter.

Fig. 2. Relationship between sensitivity (X-axis) and NNE (number needed to evaluate, Y-axis) in the temporal validation cohort for the twelve models we evaluated.

Top label indicates the type of encounter where the prediction would be made: Hospital discharge, ED treat and release (EDTR), and clinic visits. The NEH outcome models are shown in blue, and the sepsis hospitalization outcomes in red. Lower NNE and higher sensitivity indicate better performance.

The difference in NNE at 40% sensitivity across race groups was lower for the non-elective hospitalization models compared to the sepsis models. For this outcome, the highest difference in NNE between a race group and the average was 1.0 among all three cohorts. For the sepsis outcome, the largest difference in NNE between a race group and the mean in the clinic cohort was 8.0 for Hispanics (37.0 vs. 45.0), 3.0 for Blacks and Asians (15.0 vs. 18.0) in the EDTR cohort, and 1.0 for all race groups in the hospital cohort. Additional data on model performance by race is in the Supplementary Information.

Discussion

We have demonstrated the feasibility of extracting and formatting new predictors from a commercially available EHR. Combining these new variables with previously defined predictors and indices, we used them to predict patient risk of non-elective and sepsis hospitalization within a short time frame—1 or 2 weeks from a clinical encounter. Despite low rates of hospitalizations within this time frame, the models we developed have NNEs that fall within ranges that are potentially clinically actionable. They could serve as an initial screening step in a clinical program designed to prevent short-term hospitalizations and sepsis.

Our findings are consistent with our previous work on sepsis hospitalizations, where we anticipated, on theoretical grounds, that NNEs for prediction of future hospitalization risk could be as low as 15 to 25 following hospital or EDTR discharge⁸. With respect to NEHs in general, they are also in broad agreement with previous work done by us and others, as well as clinical experience, in that the Shapley relative contribution analyses consistently showed that previous NEHs, comorbidity burden, and acute physiology were strong predictors across all models. Nonetheless, given that the new predictors did enhance statistical performance, they highlight the potential use—in real-time—of data elements (e.g., prescriptions in the preceding year) that have been primarily employed for retrospective analyses. Significantly, the data used in our work all came from a commercially available EHR in widespread use in the United States.

Certain important limitations to our findings must be stressed. Our work was conducted within an insured population receiving care from a capitated system in which financial incentives favoring hospitalization (e.g., following an EDTR visit) are absent. Further, the system has an unusually high degree of integration. This integration is manifest in the availability of services aimed at preventing hospitalization, such as chronic condition management programs, a central call center, and electronic patient portals. Thus, both the prevalence of predictors and outcomes may be very different from those found in other healthcare settings. In addition, KPNC has an extensive track record of employing automated indices for inpatient and outpatient use such as the ones described in this report as well as others^9,11,12. Deployment of the models herein described would only require partial modifications to existing KPNC programs, which might not be the case in many other settings.

Thus, organizations without similar pre-existing analytic capabilities and pre-existing infrastructure would not necessarily be able to implement our models or gain as much value. Other factors that could impact models, including ours, are the effects of the coronavirus pandemic, which impacted both population health as well as how care was delivered. For example, in this study we did not include telephone or video visits, whose frequency has increased substantially in KPNC.

A hypothetical example can be used to illustrate the potential benefits of our models in the context of relatively low incidence (e.g., sepsis following an EDTR visit) as well as the impact of adding new predictors. In KPNC, approximately 760,000 EDTR encounters take place each year, with an expected 2-week sepsis hospitalization rate of 0.8%. Assume that a sepsis hospitalization costs $50,000 and that an intervention involving a post-EDTR outpatient visit costs $200 and would have a 10% success rate in preventing such sepsis hospitalizations. Using our base model with an NNE of 12.7 at a sensitivity of 40% would detect 2432 hospitalizations at risk and require 31,000 patients to receive the intervention (for a total cost of $ 6.2 million), preventing 243 hospitalizations, for a net savings of $ 6 million per year. Switching to our enhanced models, with a decrease of 11.8% in the NNE, would yield an additional $730,000 in annual savings. Obviously, this is a simplified example, but our recent experience with predictive model deployment^10,12 suggests that it is within the realm of possibility. Even greater savings might be possible with respect to non-elective hospitalizations.

Methods

Setting

Our study is set in Kaiser Permanente Northern California (KPNC). Under a mutual exclusivity agreement, salaried physicians of The Permanente Medical Group, Inc., care for 4.5 million Kaiser Foundation Health Plan, Inc. (KFHP) members at 21 hospitals owned by Kaiser Foundation Hospitals, Inc., as well as over 250 medical office buildings. The Epic EHR (www.epicsystems.com) was in use in all KPNC facilities during the study period. The KPNC Epic build employs a common medical record number system for all patient encounters and administrative transactions, including those involving KFHP members who get care outside the system and nonmembers cared for at KPNC emergency departments (EDs) and hospitals. Prior work by our team describes the comprehensive nature of KPNC’s integrated information systems and our ability to extract and use clinical, diagnostic, laboratory, and health services data for multiple uses, including prediction of in-hospital events within very narrow time frames^8,11–14. Compliance with the TRIPOD guidelines (www.tripod-statement.org) is provided in the Supplementary Information.

Unit of analysis and eligible population

The unit of analysis for our models was an index encounter that occurred between January 1, 2012, and December 31, 2019, at any KPNC facility (data collection for predictors included the 2011 calendar year, and outcomes for 2019 encounters were tracked into the 2020 calendar year). Eligible encounters were outpatient clinic visits, EDTRs, or overnight hospitalizations where the patient survived to discharge home. The end of the index encounter served as the T₀ for the predictive models quantifying the risk of a subsequent incident hospitalization within 7 and 14 days of that T₀. Eligible outpatient clinic visits were those scheduled, and procedure-only ambulatory office visits attended by a physician or nurse practitioner. Eligible EDTRs were those that began and ended in the ED with no transition to hospitalization or transfer to another facility. We did not include obstetric hospitalizations, same-day surgery hospitalizations not resulting in an overnight stay, or psychiatric hospitalizations; all other hospitalizations were eligible. Eligible patients were adults ≥18 years of age with an index encounter (whether or not they were KFHP members at the time of the encounter). We included multiple encounters per patient because each encounter is a potential opportunity to risk-stratify a patient. Each encounter also affords an opportunity to update a patient’s risk factors. Thus, a patient with eight outpatient visits, three EDTRs, and four hospitalizations would contribute eight observations to the clinic models, three to the EDTR models, and four to the hospitalization models. Since our study cohort included nonmembers, and since we did not have access to such individuals’ non-KPNC data, our model predictions for these individuals are only for future hospitalizations within KPNC.

Model development

Consistent with our previous work^11,13, we included all encounters experienced by the patients in our cohort, ignoring within-patient clustering (random) effects. Given the rich data available to us, we have previously found that within-patient clustering showed minimal effects on our results. All indicator features such as medications, labs, membership, and diagnosis categories were set to 0 when missing, indicating the absence of that characteristic. Utilization features were set to zero if there was no utilization. Age and vital signs were not imputed, and missing values were handled by the H2O GBM package, which treats missing values as a separate category. Missing values for sex (0.03% in the hospital, 0.1% in EDTR, and 0.03% in the clinic) were set to male. Age was missing in less than 0.1% of encounters. The missingness rate for vitals was lowest for blood pressure, temperature, heart rate, respiratory rate, O₂ saturation, weight, and neurological status. The missing rate for these vitals was close to 0% in the hospital, 1% in the ED, and 50–90% in the clinic. BMI was missing for about 20% of hospital, 80% of ED, and 98% of clinic encounters. For vital features that were largely missing, the missingness itself contributed to the predictive ability of that feature.

Initial variable selection (Table 2)

Our intent was to employ variables routinely available in commercial EHRs such as Epic and to use them within a standardized schema to facilitate generalizability to other EHRs. Table 2 summarizes the variable groupings (dimensions) of the variables in our models. We included demographics, membership status at the time of the encounter (KFHP member or not), and utilization variables (encounter counts, length of stay for hospital encounters; encounter counts were those in the 12 months, 3 months, 4 weeks, and 2 weeks preceding the T₀). We also included laboratory test orders in the 30 days prior to the T₀. We grouped prescriptions into 50 therapeutic classes based on EHR medication tables; lookback intervals for these were within the past year and the 2 weeks preceding the T₀. In addition, we also added a category of “new” prescriptions (those that occurred in the month preceding the T₀ but not in the prior 11 months). Diagnoses incurred in the year preceding the T₀, which were grouped into 70 binary (present/absent) hierarchical condition categories¹⁵, were also included. We included vital signs from outpatient and EDTR visits. Lastly, we also included three previously described automated indices: the Comorbidity Point Score (COPS2), which is based on diagnoses incurred the 12 months preceding the T₀¹¹, the hospital Laboratory-based Acute Physiology Score, version 2 (LAPS2), which is based on vital signs and laboratory test results in the 72 h preceding hospital entry¹¹, and the outpatient abbreviated Laboratory Acute Physiology Score (abLAPS)¹⁶, which is based on selected laboratory tests obtained in the month preceding the T₀. A detailed listing of all variables we evaluated is provided in the Supplementary Information.

Outcomes

We evaluated predictive models for four outcomes: non-elective hospitalizations and sepsis hospitalizations that occurred within 1 or 2 weeks from the T₀. We defined a non-elective hospitalization as one that began in the ED and a sepsis hospitalization as a non-elective hospitalization that met the Sepsis-3 definition^17–19. This definition is based on physiologic criteria, presumptive treatment with antibiotics, and an order for blood culture (see Supplementary Information for details on how we implemented the Sepsis-3 definition).

Modeling approach

We first built and evaluated 12 base models (four outcomes × three eligible encounters, as described above). These base models included utilization in the 12 months preceding the T₀, demographics, age, sex, COPS2, and acuity measures (LAPS2, which is only available for hospitalized patients, and abLAPS, available for all patients). We then evaluated the incremental change in performance of models that added medication orders, laboratory orders, vital signs, and diagnoses¹⁵. The final models included all dimensions shown in Table 2 except for the diagnoses. The machine learning algorithm we employed retained all variables within a given dimension when variables in that dimension were added to the model.

To develop our models, we split 2012–2018 encounters into three datasets (training, 50% of observations; validation, 25%; and test, 25%), and we held out 2019 encounters for temporal validation for each setting separately. Because we did not observe substantial differences in model performance in the training, validation, and test datasets, this report focuses on the result of applying coefficients from the training dataset to the 2019 (temporal validation) cohort. Details on all cohorts used for the models are provided in the Supplementary Information.

We used the gradient boosting machine (GBM) algorithm^20,21 to fit the hospital, EDTR, and clinic predictive models for each outcome separately, as GBM models have shown strong performance across many algorithms using EHR data. The GBM models were trained in a grid search across a range of hyperparameters with learning rates of {0.001, 0.005, 0.01}, a maximum depth of each tree of {3, 5}, and number of trees of {300, 500, 700} for the clinic cohort. For the EDTR and hospital cohorts the grid search was over learning rates of {0.001, 0.01, 0.1}, a maximum depth of each tree of {5, 7}, and number of trees of {75, 300, 500} (see Supplementary Information for additional detail). Models were trained using H2O version 3.36.1.3 in a pySpark version 3.2 environment. We evaluated the models’ performance via the area under the receiver operator characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), calibration curves, and sensitivity/specificity at varying numbers needed to evaluate (NNE, or 1/positive predictive value) to find a true positive. We measured variable importance (relative contribution of predictors) using Shapley values^22,23 calculated in the same H2O environment. We compared the NNE and sensitivity distributions across racial/ethnic group categories (Asian, Black, Hispanic, White, and all others). For reporting our findings succinctly in this paper, we focus on the NNE at a sensitivity of 40% (i.e., in the context of a hypothetical clinical intervention, how many patients would need to be evaluated for each score above a threshold that correctly identified 40% of the hospitalizations among patients experiencing a given clinical encounter). This also facilitates comparing changes in model performance across encounter types and with different types of predictors included or not in a given model. Additional details are provided in the Supplementary Information.

Future directions

We are currently exploring several potential uses for these models. At the operational level, we have developed the electronic “pipelines” needed to extract data, calculate scores, and display results in real-time. We have started feasibility discussions with clinical leaders with the goal of testing at least one of these models at an initial pilot site. As was the case with the in-hospital deterioration model now running in all KPNC hospitals^12,24, such testing would be preceded by passive assignment of risk scores (i.e., not displaying results to actual caregivers) to some patients. These patients’ records would then be reviewed by selected clinicians to begin the development of discrete intervention protocols. Knowing that some of the models still have a high workup-to-detection ratio (i.e., outpatient models), additional risk stratification methods beyond just EHR predictive models could be extremely valuable, including the use of sensors/wearables to detect physiologic signals of impending disease as well as other diagnostic or biomolecular tests that could indicate pathophysiologic responses to early infection or inflammation.

Author contributions

V.X.L. obtained funding, oversaw the study from beginning to finish, directed the interpretation of study findings, and worked on the manuscript draft. G.J.E. assisted in the interpretation of study findings and worked on the manuscript draft. P.K. designed and supervised all statistical analyses and worked on the manuscript draft. L.C.M. assisted with the interpretation of study findings and preliminary analyses. L.O., K.K.T., D.S., J.D.G., F.B., and L.D.G. assisted with data acquisition, data formatting/cleaning, and conduct of the analyses and reviewed the manuscript draft.

Data availability

The datasets generated during and/or analyzed in this report are the property of Kaiser Foundation Health Plan, Inc. and are not publicly available due to their potentially identifiable information and Kaiser Permanente Northern California privacy regulations.

Code availability

The underlying code for this study, as well as specific variable coefficients, are not publicly available but may be made available to qualified researchers on reasonable request from the corresponding author.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-025-01574-6.