An externally validated machine learning model to predict cesarean surgical site infections

Abstract

Objective

To develop a comprehensive risk stratification model to identify patients at high risk of cesarean surgical site infection (cSSI) based on information available at the time of hospital discharge.

Methods

This is a retrospective cohort study based on datasets from two independent level IV academic medical centers to improve generalizability. The first study includes 2336 cesarean deliveries for analysis with 109 instances of cSSI (4.7%). The second study includes 2936 cesarean deliveries with 72 instances of cSSI (2.5%). The first study cohort was utilized for both model training and internal validation, using the 15 shared features present in both datasets. Random search with cross‐validation was used to choose the type of machine learning (ML) model and its hyperparameters, with area under the receiver operating characteristic curve (AUC) as the validation metric. The second study cohort was then used to evaluate testing accuracy (external validation) and generalization capability of this model. The predictive importance of each feature was evaluated using permutation importance applied to the second cohort. Risk stratification cutoff scores were defined based on centiles over the first patient cohort and evaluated on the second cohort.

Results

The ML model with the highest validation AUC was an XGBoost classifier. This trained model achieved an AUC of 0.78 on the training data and a 0.72 cross‐validated AUC. When tested on data from the second study cohort, it achieved an AUC of 0.69. The features showing the strongest predictive generalization, as measured by permutation importance, were length of hospital stay, primary indication for cesarean delivery, and estimated blood loss. Using the second cohort to test our model, the model shows a 22.7% positive predictive value (PPV) of cSSI among women in the highest risk group, and a 0.8% PPV among women in the bottom risk group, compared with 2.5% cSSI incidence for the study population at large.

Conclusion

We developed and validated a predictive ML model for cSSI with information available to healthcare professionals at the time of hospital discharge. This model suggests that the most predictive risk factors for cSSI include length of hospital stay, primary indication for cesarean delivery, and estimated blood loss. Additionally, the model provides reasonable risk stratifications for individual patients. This tool may be used for identifying subgroups at higher risk of cSSI and identifying those who may require closer surveillance or clinical intervention.

1. INTRODUCTION

Cesarean delivery is the most common surgical procedure performed worldwide and 1.2 million cesarean deliveries (32.1% of all recorded births) were completed in the United States in 2022 [1]. Despite advances in perioperative preventative measures, infection control practices, and evidence‐based interventions, cesarean surgical site infections (cSSI) remain one of the most common postoperative complications with a varying incidence of up to 8% of cesarean deliveries [2, 3]. This wide range is likely a product of variation in population characteristics, available clinical resources, and/or perioperative practices [4]. cSSI results in increased maternal morbidity, mortality, and increased costs related to prolonged hospital stays, readmissions, and cost of treatment. Identifying those at the highest risk of cSSI to intervene early and mitigate disease burden may ultimately improve utilization of healthcare resources and costs to patients and practices alike [5].

Several antepartum and peripartum factors associated with an increased risk of cSSI have been identified: obesity, hypertensive disorders, diabetes, ruptured membranes, the presence of labor, the absence of prophylactic antibiotics, chorioamnionitis, emergency delivery, and more [3, 6, 7]. At present, however, limited tools exist to accurately predict via a comprehensive review of a multitude of contributing factors, those at the highest risk for cSSI [2, 8, 9]. Additionally, there are no standardized protocols or guidelines for the identification and management of patients deemed to be at increased risk. Implementation of various surveillance methods for development of cSSI after discharge (mobile applications to remotely document photographs and measurements of the surgical site, timed postoperative surveys aimed at identifying cSSI signs and symptoms, frequent postoperative check‐ins, etc.), however, had limited or inconsistent success [10, 11, 12, 13]. Such studies posited a lack of accurate models for identifying high‐risk patients as one factor inhibiting effective surveillance programs [14].

Machine learning (ML) models can identify and provide insight into important features associated with clinical outcomes. In recent years, there has been a rise in the development of effective ML models to detect and predict various clinical outcomes based on patterns in large, complex datasets [15, 16, 17, 18]. A limitation to many currently published ML models in healthcare is the lack of external validation on independent datasets. Devising an ML model to determine those at the highest risk of cSSI may aid in obstetric decision‐making and the development of a more tailored and effective postpartum care plan for those undergoing cesarean deliveries. The primary aim of this study was to develop and validate a risk‐stratification tool for cSSI after the delivery hospitalization based on information available in the electronic medical record (EMR) at the time of hospital discharge.

2. METHODS

This study utilizes datasets from two retrospective cohort studies assessing risk of postoperative infections with varying clinical interventions [19, 20]. This project (WIH 22‐0059) was deemed “Not Human Subject Research” by the Institutional Review Board (IRB) on August 5, 2022 and was excluded from IRB review. The studies utilized were conducted at two independent level IV academic medical centers to improve the generalizability of our results. The first study (Cohort 1) was a before‐and‐after implementation cohort study of women delivered via cesarean section that assessed the effectiveness of adding azithromycin to standard antimicrobial prophylaxis with a primary outcome of composite postpartum infections [19]. The second study (Cohort 2) was a retrospective cohort study that assessed the association of a change in surgical closing protocol with cSSI rate [20]. Both studies abstracted data directly from the EMR, consistent with the primary aim of this study.

Methodologies of both studies have been previously described and demonstrated no significant difference in the primary outcome of cSSI rate; thus, all participants were analyzed. cSSI classification in each study was consistent with the Centers for Disease Control's (CDC) 2024 surgical site infection (SSI) criteria, including superficial incisional, deep incisional, and organ/space SSI; however, Cohort 1 observed infections until 6 weeks post‐operation rather than the 30‐day surveillance window outlined by the CDC and observed in Cohort 2 [21]. Temporal data regarding the timing of postoperative infection in Cohort 1 were unavailable; therefore, all SSIs up to 6 weeks post‐operation were included in the training model rather than solely those that occurred within the standard 30 days. Both datasets included a comprehensive record of antepartum, peripartum, surgical, and SSI characteristics for primary analysis that were obtained by manual chart extraction. Cohort 1 included 2336 cesarean deliveries from March 1, 2016 to August 31, 2018 after initial exclusion of 253 participants due to maternal age (not 18‐49 years), fetal anomaly, or fetal death, with a total of 109 instances of cSSI (4.7%). Cohort 2 analyzed 2936 cesarean deliveries from July 1, 2013 to April 30, 2018 with no exclusions and 72 reported instances of cSSI (2.5%).

The primary aim of our study was to develop and assess the performance of an ML model trained to predict the risk of cSSI within 6 weeks postpartum. Secondary aims were to measure the contribution of various factors in their association with cSSI as well as a risk stratification for cSSI based on centile groups. cSSI was defined according to the CDC's SSI criteria for superficial, deep, and organ/space SSI; however, we did not specifically restrict the time constraints to 30 days given the deviation to six weeks postpartum in Cohort 1.

As the two studies collected overlapping but distinct feature sets, we selected the available features present in both datasets to be used by our predictive model. Given the restricted number overlapping features, we did not perform any additional analysis to further reduce the number of features utilized in the predictive model. While each study represented a diverse cohort of participants, racial and ethnic demographics were intentionally excluded from development of the predictive model to avoid serving as an intrinsic marker of health differences. We then used these overlapping features to train our predictive model.

We utilized three classes of ML models well‐suited to tabular data. Random Forest (RF) and XGBoost (XGB) are decision‐tree‐based ensemble methods that can capture complex, nonlinear relationships between predictive features and outcomes. Regularized logistic regression (RLR) is a linear model that incorporates penalties to reduce overfitting and improve generalizability. The dataset obtained from the first study was used for both training and model selection. Hyperparameter tuning for each model class was performed using random search. For all tree‐based methods, we use ensembles of size 100.

Each model was trained using repeated stratified 10‐fold cross‐validation. In 10‐fold cross‐validation, the dataset is repeatedly partitioned such that the model is trained on 90% of the data and evaluated on the remaining held‐out 10% of the data. We measured the quality of each model using the area under the receiver operating characteristic curve (AUC), computed from the scores on the held‐out data in each fold. To address missing data, for all features, we imputed missing data with the median (or modal for categorical features) value from the training dataset. Outlier values were capped at ±2 standard deviations from the observed mean to reduce the influence of extreme outliers, as determined from the training dataset.

We used Cohort 1 to train and select our ML model, and Cohort 2 was used to test the resulting model's performance on data collected from a separate institution. After data processing and training, we determined the best‐performing ML model class and hyperparameters as measured by the cross‐validation AUC. We then trained a model with these hyperparameters on the entirety of the training dataset.

To verify that the overlapping features used to develop the model adequately represented risk stratification, we trained a separate expanded‐feature XGB model using the additional nonoverlapping features available in Cohort 1 (preeclampsia and severe preeclampsia, maternal immunocompromise, intrauterine growth restriction, Group B Streptococcus status, use of cervical ripening agents, use of epidural, intrauterine resuscitation measures, and use of intrapartum antibiotics) as Cohort 2 had fewer reported nonoverlapping features. Additionally, we included race and ethnicity data in this expanded‐feature XGB model to assess if the omission of the nonoverlapping features or demographic data altered the model's predictive performance.

Next, we calibrated the overlapping feature model to output individualized cSSI risk probabilities via Platt scaling over the training scores. Using these risk probabilities, we stratified the training population into groups by risk percentile and analyzed the model's performance on each partition. To assess the ML model's expected performance on prospective patient populations, we utilized the second patient cohort as a testing dataset. We measured the model's overall performance on this testing dataset via AUC. We similarly partitioned this cohort using the previously determined quartile cutoffs to examine the degree to which the risk probabilities produced by the model generalize to the new cohort. To quantify the contribution of each feature to the model's prediction, we applied the methodology of permutation importance over Cohort 2's dataset. For each feature, we measured the decrease in AUC that occurs when the values for that feature are randomly shuffled across the dataset. This indicates how well a feature's impact generalizes from the training to the testing dataset. We performed all statistical analyses using PYTHON version 3.9.6. All ML models were built and trained using scikit‐learn package version 1.2.2, and XGB package version 2.2.1.

3. RESULTS

The combined studies consisted of a total of 5272 cesarean deliveries, of which 181 (3.4%) resulted in a cSSI. The 15 overlapping cohort characteristics between datasets are presented in Table 1. There were significant differences among multiple variables between those with and without a cSSI in each cohort. Observed differences, including rupture of membranes, BMI, chorioamnionitis, and reason for cesarean delivery, are consistent with prior literature.

TABLE 1.

Overlapping feature characteristics of the study cohorts.

Variable	Missing data n (%)	cSSI Cohort 1 (n = 109) Cohort 2 (n = 72)	No cSSI Cohort 1 (n = 2227) Cohort 2 (n = 2864)	Unadjusted odds ratio	p value Cohort 1 Cohort 2
Preoperative antibiotics	2 (0.09) 1 (0.03)	1.00 0.86	1.00 0.84	23.49 (0.23–75.75) 1.18 (0.65–2.79)	0.03 0.27
Membrane rupture > 4 h pre‐CS	10 (0.43) 20 (0.68)	0.40 0.56	0.25 0.30	1.99 (1.33–2.89) 2.94 (1.91–4.79)	<0.01 <0.01
Blood transfusion	6 (0.26) 0	0.17 0.08	0.07 0.02	3.03 (1.62–4.82) 4.22 (1.15–9.16)	<0.01 0.02
Chorio‐amnionitis	0 0	0.08 0.15	0.03 0.04	2.84 (1.17–5.37) 4.01 (1.80–7.59)	0.02 <0.01
Chronic hypertension	0 0	0.17 0.03	0.15 0.01	1.17 (0.66–1.90) 3.94 (0.01–12.41)	0.30 0.17
Gestational hypertension	1 (0.04) 0	0.17 0.00	0.13 0.04	1.42 (0.79–2.22) 0.01 (0.00–0.09)	0.12 <0.01
Diabetes (pre‐ and gestational)	0 0	0.22 0.04	0.15 0.03	1.55 (0.92–2.44) 1.54 (0.01–3.86)	0.05 0.34
Multifetal gestation	0 4 (0.14)	0.06 0.08	0.06 0.06	1.02 (0.45–1.98) 1.46 (0.44–2.86)	0.51 0.23
Prior CS	0 0	0.37 0.35	0.57 0.46	0.43 (0.27–0.64) 0.64 (0.36–1.02)	<0.01 0.03
BMI at time of delivery	27 (1.16) 57 (1.94)	35.9 [31.4–40.9] 34.0 [29.0–40.0]	33.9 [29.1–39.1] 32.30 [28.1–37.0]	1.02 (1.00–1.04) 1.04 (1.01–1.07)	0.01 0.01
Gestational age (weeks)	8 (0.34) 0	37.0 [35.0–39.0] 38.0 [36.0–39.0]	38.0 [36.0–39.0] 39.0 [37.0–39.0]	0.93 (0.89–0.99) 0.95 (0.91–1.02)	0.01 0.07
Length of hospital stay (days)	0 0	5.0 [3.0–7.0] 3.0 [3.0–3.0]	3.0 [2.0–4.0] 3.0 [3.0–3.0]	1.50 (1.36–1.65) 1.73 (1.37–2.10)	<0.01 <0.01
Maternal age (years)	0 0	26.0 [23.0–33.0] 29.0 [25.8–35.0]	29.0 [25.0–33.0] 31.0 [27.0–34.0]	0.97 (0.93–1.01) 0.97 (0.92–1.03)	0.05 0.16
Estimated blood loss (mL)	8 (0.34) 3 (0.10)	900.0 [700.0–1050.0] 900.0 [800.0–1000.0]	800.0 [700.0–905.0] 800.0 [800.0–900.0]	1.00 (1.00–1.00) 1.00 (1.00–1.00)	<0.01 <0.01
Reason for CS	0
Arrest of dilation		0.13 0.17	0.05 0.08	2.77 (1.37–4.65) 2.17 (0.96–3.93)	<0.01 0.03
Arrest of descent		0.07 0.22	0.03 0.09	2.66 (0.92–5.31) 3.07 (1.57–5.34)	0.03 <0.01
NRFHT		0.28 0.18	0.19 0.21	1.66 (1.06–2.48) 0.85 (0.41–1.45)	0.02 0.28
Mal‐presentation		0.10 0.08	0.11 0.13	0.88 (0.40–1.54) 0.60 (0.19–1.20)	0.32 0.09
Planned repeat		0.20 0.22	0.43 0.36	0.33 (0.20–0.52) 0.60 (0.19–1.20)	<0.01 <0.01
Other		0.22 0.12	0.19 0.14	1.21 (0.71–1.87) 0.90 (0.37–1.64)	0.20 0.38

Note: Categorical data are presented as fraction positive. Continuous data are presented as median [interquartile range]. Odds ratios are presented along with 95% confidence intervals. Bold values represent statistical significance of p < 0.05.

Abbreviations: BMI, body mass index; cSSI, cesarean surgical site infection; CS, cesarean section; NRFHT, non‐reassuring fetal heart tones.

3.1. cSSI prediction model

We compared the training performance of the three previously defined classes of ML models (RLR, XGB, and RF) as measured by cross‐validation AUC scores (Figure 1). The XGB model had the highest validation performance (AUC = 0.72 (95% confidence interval [CI] [0.66–0.76])). This XGB model was then evaluated using data from Cohort 2 as a testing dataset and was found to have a similar performance (AUC = 0.69 (95% CI [0.63–0.75])). We also compared the overlapping feature XGB model to the performance of the XGB model developed using the 25 available features and race and ethnicity data in Cohort 1 (Appendix 1). We found a similar validation performance (AUC = 0.72 (95% CI [0.66–0.75])) compared to the overlapping feature model.

FIGURE 1.

Training performance of the overlapping feature machine learning models by receiver operating characteristic curves. AUC, area under the receiver operating characteristic curve; RFC, Random Forest classifier; RLR, regularized logistic regression; XGB, XGBoost.

We then investigated the degree to which the XGB model can identify high‐risk patients in an external patient population. We first converted the model's outputted scores to risk probabilities through Platt scaling over Cohort 1's dataset. Using these probabilities, we determined risk thresholds that divided the first cohort into five groups based on centile of risk (Table 2). In this population (on which the model was trained), those stratified into the 91–100 percentile had a 3.85 times higher likelihood of a cSSI compared to the overall cohort population.

TABLE 2.

Stratification of the model results on Cohort 1 (training data).

Per group					Cumulative
Risk groups (centiles)	Predicted risk range	Percent of population	cSSI incidence	Lift	Percent of population	cSSI incidence	Sensitivity/specificity	Lift
91%–100	11.2%–62.0%	10%	17.9%	3.85	10%	17.9%	0.39/0.91	3.85
76%–90	5.2%–11.1%	15%	6.6%	1.41	25%	11.1%	0.60/0.77	2.39
51%–75	2.6%–5.2%	25%	5.3%	1.14	50%	8.2%	0.88/0.52	1.76
26%–50	1.4%–2.6%	25%	1.7%	0.37	75%	6.1%	0.97/0.26	1.30
0%–25	0.8%–1.4%	25%	0.5%	0.11	100%	4.7%	1.00/0.00	1.00

Abbreviation: cSSI, cesarean surgical site infection.

The calibrated XGB model was then applied to Cohort 2 using the estimated risk ranges determined from the above centile stratification (Table 3). In Cohort 2 (which had an overall cSSI incidence of 2.5%), the model identified a small group of the highest risk patients. Among them, 22.7% developed a cSSI. By expanding to the top two risk range groups, the model identified 4.5% of the population at a four times higher risk than the overall cohort, comprising 18.1% of the total cSSI cases. Conversely, those grouped in the lowest risk range had a cSSI incidence of 0.8%, three times lower than the population average. A calibration plot of the second cohort's data is presented in Appendix 2. This demonstrates the degree to which the XGB model's calculated probability of cSSI is in agreement with the observed frequency of cSSI in the second cohort based on the stratification described above.

TABLE 3.

Stratification of the model results on Cohort 2 (testing dataset).

Per group				Cumulative
Predicted risk range	Percent of population	cSSI incidence	Lift	Percent of population	cSSI incidence	Sensitivity/specificity	Lift
11.2%–28.8%	0.7%	22.7%	9.27	0.7%	22.7%	0.07/0.99	9.27
5.2%–10.9%	3.7%	7.3%	2.97	4.5%	9.8%	0.18/0.96	4.02
2.6%–5.2%	35.9%	2.9%	1.20	40.4%	3.7%	0.61/0.60	1.51
1.4%–2.6%	39.2%	2.0%	0.81	79.6%	2.9%	0.93/0.21	1.17
0.8%–1.4%	20.4%	0.8%	0.34	100.0%	2.5%	1.00/0.00	1.00

Abbreviation: cSSI, cesarean surgical site infection.

3.2. Feature importance

We then quantified the importance of each included variable to the trained XGB model using permutation importance over the Cohort 2 dataset (Figure 2). The height of each feature represents the average decrease in AUC resulting from shuffling a given feature's values across the dataset. We determined that the length of hospital stay, reason for cesarean delivery, and estimated blood loss are the best generalizing features for predicting cSSI risk. Of note, a low feature importance was not a definitive sign of irrelevance to cSSI, only that it was not deemed to be an important feature for the trained XGB model. Given the importance of the three features listed above, we trained an XGB model on Cohort 1 using only these three highest performing features and tested this model on Cohort 2 (Appendix 1). The validation and testing performance of this selected‐feature model (AUC = 0.71 (95% CI [0.65–0.75]) and AUC = 0.69 (95% CI [0.63–0.76]), respectively) was similar to the overlapping feature XGB model performance.

FIGURE 2.

Feature importance using the permutation of the overlapping feature model. Black bars represent one standard deviation.

4. DISCUSSION

This study developed a novel ML model to predict cSSI that stratifies those at the highest risk for cSSI with information available to healthcare professionals at the time of hospital discharge after delivery. Importantly, the model was validated on an independent patient cohort, thus suggesting its accuracy and generalizability if applied to outside institutions. In this second patient cohort, the ML model successfully identified 4.5% of the population at high risk for cSSI, of which 9.8% developed cSSI, which accounted for 18% of the overall cases of cSSI. The model additionally identified a small subset of this group with a 22.7% risk of cSSI. Furthermore, the model's risk estimates remain well‐calibrated over this second cohort despite different rates of cSSI between the two studies (4.7% vs. 2.5%), though the significance of the highest risk group's calibration is limited due to sample size. The development of a predictive model that utilizes objective data available at the time of discharge allows for a standardized approach to the identification and surveillance of those at the highest risk of infection, thus addressing potential for bias and disparities in care.

Our methodology also identifies the most important features for predicting cSSI. Permutation importance over the second cohort identified the length of hospital stay, reason for cesarean delivery, and estimated blood loss as the features that best inform the prediction model. This may be useful as a surrogate predictor if all 15 features are not available in the EMR at the time of discharge, given the similar selected‐feature performance to the overlapping feature model. Unlike traditional statistical methods such as odds ratios that assume linear relationships between features and response, permutation importance implicitly accounts for nonlinear responses and complex interactions between features inherent to more powerful ML models.

Our study was not without limitations. First, this model assesses the strength of associations rather than the cause of cSSI; therefore, it may be difficult to determine the most effective interventions to improve rates of cSSI. Second, the evaluation of various factors contributing to the risk of cSSI was restricted to the available variables included in each dataset, specifically those overlapping between studies. Efforts to mitigate this limitation were demonstrated in the relatively comparable AUCs between the overlapping feature model and the model trained using the 25 additional features and race/ethnicity data of Cohort 1. The similar performance of these models, as well as the selected‐feature model, suggests that their predictive power is limited more by the size of the datasets than the richness of features. Third, Cohort 1 notably excluded 253 participants from the initial study due to age less than 18 or greater than 50 years, known fetal anomalies, or fetal death. The selection bias in excluding these participants restricts our ability to assess if these factors contributed to the overall risk of cSSI. Lastly, both original studies were conducted at large, academic, level IV maternity care centers. Though validated on an external cohort, the predictive model may not be as generalizable to different‐level care centers or lower risk populations. Furthermore, Cohort 1 included diagnosis of cSSI within 6 weeks postpartum rather than the 30‐day criteria defined by the CDC. This may contribute to more false positives in the model as well as limit comparability with existing literature.

Our study had several strengths. First, our prediction model of cSSI takes a comprehensive approach at including patient and system‐level factors to create a predictive model for cSSI. Second, the utilization of two datasets to validate this model allowed for insight into how this model may perform when applied across various institutions. Third, this model was specifically designed to be used by healthcare providers at the time of delivery hospital discharge. This approach allows for feasibility in actual practice by evaluating readily available information within the medical record during hospitalization. Potential use may include an automated warning within the EMR if a patient is deemed higher risk or an easily accessible calculator to determine risk. Future studies are needed to determine effective post‐discharge surveillance methods or interventions to reduce infections or re‐admissions in this high‐risk group.

This study developed an externally validated ML model for stratifying patient risk of cSSI. Implementation of ML models for cSSI has the potential to improve healthcare resource utilization, optimize patient surveillance, and reduce maternal morbidity.

FUNDING INFORMATION

The authors received no specific funding for this work.

CONFLICT OF INTEREST STATEMENT

Stephen M. Wagner is a consultant for Elythea. All other authors declare no conflicts of interest.

APPENDIX 1. Training of XGB Using Expanded‐ and Selected‐Feature Sets, by Receiver Operating Characteristic Curves.

1.1.

AUC, area under the receiver operating characteristic curve; XGB, XGBoost.

APPENDIX 2. CALIBRATION PLOT

2.1.

Black bars represent one standard deviation. cSSI, cesarean surgical site infection.