Abstract
Introduction
Postpartum hemorrhage (PPH) is a major cause of maternal morbidity and mortality. Timely prediction may prevent adverse maternal outcomes, and efforts are needed to develop accurate predictive tools. A high‐performing machine learning model to predict PPH using data from the US Consortium for Safe Labor (CSL) remains to be widely validated in contemporary clinical settings using electronic health record (EHR) data. Our goal was to evaluate the performance of the CSL PPH predictive model using EHR data across a large healthcare system in the Northeastern United States.
Methods
We conducted a retrospective cohort study across eight hospitals in the Northeastern United States between May 2015 and May 2024. We used the same sociodemographic, clinical diagnoses, family history, laboratory, and vital signs available on labor and delivery admission in the EHR that were used to train the original CSL model. The binary outcome was PPH, defined as estimated blood loss of 1000 mL or more at delivery or blood transfusion within 24 h postpartum. We then refit a new model using the original features to assess whether model performance could be further improved in our study population using the best‐performing machine learning approach (extreme gradient boosting [XGBoost]) from the original CSL model. We evaluated model discrimination as measured using the area under the curve (AUC), feature importance, calibration, and decision analysis curves of both the original CSL model with external validation and the further refit model.
Results
Among 87,662 deliveries, the incidence of PPH was 7.7%. The original CSL model demonstrated modest discrimination for predicting PPH with an AUC of 0.60 (95% confidence interval [CI], 0.58–0.61). Refitting a new model with XGBoost resulted in improved discrimination with an AUC of 0.75 (95% CI, 0.74–0.76). Calibration analyses demonstrated that the refit model overestimated PPH risk across a range of predicted probabilities.
Conclusion
A previously developed PPH predictive model had substantially reduced performance with external validation using contemporary EHR data across an eight‐hospital health system in the Northeastern United States, underscoring limited generalizability. These findings highlight the importance of external validation, local adaptation, and ongoing surveillance for assessing model performance before implementing predictive models clinically.
Keywords: electronic health records, external validation, machine learning, postpartum hemorrhage, risk prediction, XGBoost
1. INTRODUCTION
Postpartum hemorrhage (PPH) is defined as estimated blood loss of at least 1000 mL or blood loss accompanied by signs of hypovolemia within 24 h after delivery [1] and represents a leading cause of severe maternal morbidity and mortality [2]. The frequency of PPH in the United States increased from 2.7% to 4.3% during 2000–2019, which was associated with an increase in the proportion of deliveries with at least one PPH risk factor from 18.6% to 26.9% [3]. Early recognition and management of PPH can mitigate adverse maternal outcomes [4]. Predictive models using clinical data available at hospital admission—such as demographics, medical history, vital signs, and laboratory results—could help clinicians optimize staffing, proactively prepare blood products, and enhance vigilance for high‐risk patients.
The most widely used tool to predict PPH, recommended by the American College of Obstetricians and Gynecologists and the California Maternal Quality Care Collaborative, is based on expert opinion and often misclassifies risk [1]. For example, only 7% of those classified as “high risk” develop PPH, while 40% who experience PPH were initially considered “low risk” [5]. These data illustrate the challenges in accurately identifying PPH risk and have led to the development of multiple alternative tools, such as data‐driven predictive models [6, 7]. Incorporating a high‐performing tool into the electronic health record (EHR) could lead to correctly identifying the most at‐risk individuals for PPH.
Recent advancements in predictive analytics have generated increasing interest in predicting adverse pregnancy outcomes using EHR [8, 9]. Machine learning methods, especially ensemble models, can analyze many complex clinical features simultaneously and can outperform traditional risk scores and regression techniques [8, 9]. One widely used machine learning approach is extreme gradient boosting (XGBoost), which combines multiple predictive models to capture complex, nonlinear relationships among clinical factors. Unlike logistic regression, XGBoost does not assume linear relationships among variables. In contrast to random forest methods, XGBoost sequentially develops models during training to correct errors from prior models, combining these into a single integrated prediction to enhance accuracy. Importantly, evaluating a predictive model also involves assessing its calibration, which describes how closely the model's predicted probabilities match the actual observed event rates. A well‑calibrated model provides predicted risks that accurately reflect the true likelihood of outcomes such as PPH. A promising machine learning PPH model was proposed using data from the US Consortium for Safe Labor (CSL), which included deliveries from >152,000 individuals across 19 clinical sites in the United States from 2002 to 2008 [6, 10]. This machine learning model was highly performing and well‐calibrated. A subsequent single‐center study using contemporary EHR data and quantitative blood loss (QBL) rather than estimated blood loss (EBL) was unable to replicate the model's original performance [11]. Thus, a major challenge in implementing advanced machine learning methods has been their generalizability [6, 7]. Models trained on data from one cohort nearly always demonstrate lower performance when applied to an external cohort, commonly due to differences in data quality, patient characteristics, or changes in medical practice [11, 12].
The objective of this study was to examine the performance of the original CSL machine learning PPH prediction model on labor and delivery admission using contemporary EHR data across a large healthcare system in the Northeastern United States. We hypothesized that the original CSL model would demonstrate lower performance in our cohort compared with the original CSL dataset and that model refitting would improve its predictive performance.
2. MATERIALS AND METHODS
2.1. Study setting and population
We leveraged EHR data from eight hospitals, two tertiary academic centers, and six community hospitals from the Mass General Brigham (MGB), a large healthcare system in the New England region. We included deliveries between May 31, 2015 (when our institution implemented single‐vendor EHR across all outpatient offices and inpatient sites) and May 31, 2024. All data were processed using our machine learning platform [13], which extracts and harmonizes data from the EHR. All demographic, family history, medical history, laboratory studies, medications, vital signs, and procedural information data were extracted. Patient race and ethnicity as social constructs were self‐reported. We identified the features used in the original manuscript and mapped them in our data. The definitions and sources of each feature are listed in Table S1. This study was approved by the MGB's Institutional Review Board, protocol # 2020P002859, with a waiver of patient consent. We used the TRIPOD reporting guidelines [14].
Deliveries were included based on documented pregnancies greater than 20 weeks’ gestation with billing codes for cesarean or vaginal delivery, analyzing each delivery independently [15]. We excluded deliveries with missing blood loss data or >30% missing features.
2.2. Study outcome
For consistency with the original CSL study, our primary outcome (PPH) was pragmatically defined as estimated blood loss ≥1000 mL or blood transfusion within 24 h postpartum. Due to >10% missing structured blood loss entries, we supplemented data by extracting blood loss information from operative and delivery notes using regular expressions [16]. Deliveries missing documented blood loss (n = 1490) were manually reviewed and excluded.
2.3. Predictive model validation
First, we externally validated the original CSL model using all our study sites. To remain consistent with the original methodology, we imputed missing numerical features using the mean and missing binary features as absence. We identified the original 55 features on which the CSL model was trained [6].
Next, we refitted a new model using the same 55 variables with the best‐performing machine learning approach from the original CSL study (XGBoost), utilizing data from all MGB hospital sites. We trained this XGBoost model on 65% of the MGB data (development) and subsequently evaluated the performance in the remaining 35% (test). We employed the same set of hyperparameters that the pre‐trained model was developed with, employing a 10‐fold cross‐validation approach for the model to only determine in each fold the number of estimators considered. During each fold, we applied early stopping after 100 iterations without improvement on fold‐wise validation data, with a maximum of 1000 estimators, keeping exclusively the best‐performing model checkpoint in terms of area under the curve (AUC). To train the final model, we fixed the number of estimators to that of the model in which the AUC score was the highest. We also evaluated model performance separately across self‐reported racial and ethnic groups (White, Black, Asian, Hispanic, and other).
We analyzed the feature importance to better characterize the source of the differences observed between the models using gain visualization and SHapley Additive exPlanations (SHAP) value estimation. Whereas gain represents the improvement in accuracy contributed by a feature to the branches it influences in the model, serving as a straightforward measure of the contribution of a feature during training, SHAP values provide a comprehensive estimation of feature importance by considering the average contribution of each feature across all possible coalitions of features [17].
2.4. Statistical analysis
Categorical variables were reported as counts and proportions, and continuous variables were reported with medians and interquartile ranges. The AUC was defined as the ability of the model to discriminate between the positive and negative classes by plotting the sensitivity (true positive rate) and 1‐specificity (false positive rate). Bootstrapping was applied to evaluate the statistical significance of differences in model performance. We evaluated model performance using metrics such as accuracy, precision, recall, F1‐score, and AUC. Additionally, we calculated calibration and decision curves to evaluate how well these models would perform in clinical practice. We fitted a Platt scaling regressor or a non‐parametric isotonic regressor to improve the calibration of our refitted model [18]. Python 3.10 and R 4.4 were used to conduct all analyses. Learning curves, results, and interpretability measurements were visualized using Matplotlib (v3.8.4). Model validation, calibration, and comparison were performed with bootstrapping using the scipy (v1.11.1) package. The pre‐trained model was evaluated using the R programming language in which it was initially developed. Several R packages were employed: XGBoost (v1.7.7.1) to manage the pre‐trained model and pROC (1.18.5), rms (v6.8.2), and CalibrationCurves (v2.0.3) to carry out statistical analyses.
3. RESULTS
3.1. Baseline characteristics and outcomes
In this cohort of 137,340 pregnancies at MGB from May 2015 to May 2024, 135,850 had documented blood loss using either estimated or quantitative methods at delivery. We included in the study all 87,662 pregnancies in which blood loss was visually estimated to be consistent with the methods used in the development of the original CSL model (Figure 1).
FIGURE 1.
Patient flow diagram. EHR, electronic health records; MGB, Mass General Brigham; PPH, postpartum hemorrhage.
In the included MGB cohort (n = 87,662), the mean maternal age was 32.4 years, and the mean body mass index (BMI) was 26.2 kg/m2. Self‐reported race distribution was 60.3% White (n = 52,878), 8.0% Black (n = 7052), 10.0% Asian (n = 8799), and 5.0% identifying as other races (n = 4378). Additionally, 16.6% of individuals (n = 14,555) reported Hispanic ethnicity. Cesarean delivery occurred in 31.9% (n = 27,937) of cases. Compared to the individuals in the CSL cohort, those in the MGB cohort were older and had more comorbidities (Table 1).
TABLE 1.
Patient characteristics by cohort.
| Characteristic | Consortium for Safe Labor (n = 228,438) | Mass General Brigham (n = 87,662) | ||
|---|---|---|---|---|
| Missing | Missing | |||
| Maternal age at delivery (years) | 27.6 ± 6.2 | 323 (0.1) | 32.4 ± 5.07 | 0 (0.0) |
| Self‐reported race and ethnicity | ||||
| White | 113,224 (51.7) | 9361 (4.0) | 52,878 (60.3) | 0 (0.0) |
| Black | 51,392 (23.5) | 9361 (4.0) | 7052 (8.0) | 0 (0.0) |
| Asian | 9345 (4.3) | 9361 (4.0) | 8799 (10.0) | 0 (0.0) |
| Other | 5400 (2.5) | 9361 (4.0) | 4378 (5.0) | 0 (0.0) |
| Hispanic | 39,716 (18.1) | 9361 (4.0) | 14,555 (16.6) | 0 (0.0) |
| Academic hospital | 130,205 (57.0) | 0 (0.0) | 59,396 (67.8) | 0 (0.0) |
| Pre‐pregnancy BMI (kg/m2) | 25.4 ± 6.25 | 76,818 (33.6) | 26.2 ± 5.99 | 26,046 (29.7) |
| Pre‐pregnancy weight (kg) | 68.0 ± 17.6 | 67,294 (29.4) | 70.0 ± 16.6 | 23,576 (26.9) |
| Admission weight (kg) | 82.5 ± 17.9 | 35,984(15.7) | 81.7 ± 16.9 | 934 (1.1) |
| Maternal temperature (F) | 98.0 ± 0.81 | 47,977 (21.0) | 99.1 ± 0.85 | 167 (0.2) |
| Systolic blood pressure (mmHg) | 124 ± 14.9 | 52,766 (23) | 139 ± 17.1 | 811 (0.9) |
| Diastolic blood pressure (mmHg) | 74 ± 11.6 | 52,800 (23.1) | 86.2 ± 12.0 | 811 (0.9) |
| Tobacco use | 15,247 (6.6) | 0 (0.0) | 181 (0.2) | 0 (0.0) |
| Illicit drug use | 4725 (2.0) | 22,544 (9.8) | 1785 (2.0) | 0 (0.0) |
| Parity | 1 (0, 2) | 0 (0.0) | 1 (1, 2) | 33,325 (38.0) |
| Gestational age at delivery (weeks) | 39 (38, 40) | 0 (0.0) | 38.9 (2.12) | 0 (0.0) |
| Anemia | 23,057 (10.4) | 7877 (3.4) | 12,671 (14.5) | 0 (0.0) |
| Assisted reproductive technology | 1101 (0.9) | 107,479 (47.0) | 10,197 (11.6) | 173 (0.2) |
| Multiple gestation | 5053 (2.2) | 0 (0.0) | 2821 (3.2) | 0 (0.0) |
| Breech presentation | 11,647 (5.1) | 0 (0.0) | 4946 (5.6) | 0 (0.0) |
| History of preterm birth | 17,017 (7.6) | 5436 (2.3) | 1252 (1.4) | 0 (0.0) |
| Polyhydramnios | 885 (0.4) | 14,169 (6.2) | 2318 (2.6) | 173 (0.2) |
| Fetal macrosomia | 1858 (1.5) | 102,247 (44.7) | 7431 (8.5) | 807 (0.9) |
| Cesarean delivery | 65,990 (28.8) | 0 (0.0) | 27,937 (31.9) | 532 (0.6) |
| Prior cesarean delivery | 31,321 (14.5) | 13,219 (5.7) | 1626 (1.9) | 32,772 (37.4) |
| Spontaneous labor | 122,673 (53.7) | 0 (0.0) | 15,064 (17.2) | 0 (0.0) |
| Trial of labor | 192,074 (84) | 0 (0.0) | 76,359 (87.1) | 532 (0.6) |
| Placenta previa | 1647 (0.7) | 0 (0.0) | 3124 (3.6) | 0 (0.0) |
| Placental abruption | 3794 (1.6) | 0 (0.0) | 473 (0.5) | 0 (0.0) |
| Placenta accreta | 138 (0.0) | 82,335 (36.0) | 7056 (8.0) | 0 (0.0) |
| Fetal growth restriction | 2678 (1.2) | 15,626 (6.8) | 4904 (5.6) | 173 (0.2) |
| Large for gestational age | 2031 (1.4) | 84,338 (36.9) | 1474 (1.7) | 173 (0.2) |
| Antenatal steroids | 5945 (4.0) | 82,507 (36.1) | 3634 (4.1) | 13,647 (15.6) |
| Antepartum bleeding in third trimester | 3088 (1.7) | 49,410 (21.6) | 15,427 (17.6) | 173 (0.2) |
| Threatened preterm birth antepartum | 7880 (3.4) | 0 (0.0) | 2794 (3.2) | 173 (0.2) |
| Premature rupture of membranes | 16,219 (7.1) | 0 (0.0) | 1158 (1.3) | 0 (0.0) |
| Chorioamnionitis on admission | 1527 (1.0) | 83,009 (36.3) | 41 (0.0) | 0 (0.0) |
| Antepartum hospital admission | 17,500 (7.6) | 84,603 (37.0) | 17,484 (19.9) | 0 (0.0) |
| Gestational diabetes | 11999 (5.2) | 0 (0.0) | 12,269 (14.0) | 0 (0.0) |
| Gestational hypertension | 6286 (2.7) | 0 (0.0) | 6022 (6.9) | 173 (0.2) |
| Preeclampsia without severe features | 7457 (3.2) | 0 (0.0) | 2020 (2.3) | 173 (0.2) |
| Preeclampsia with severe features | 3772 (1.6) | 0 (0.0) | 1236 (1.4) | 0 (0.0) |
| Superimposed preeclampsia | 1971 (0.8) | 0 (0.0) | 129 (0.1) | 173 (0.2) |
| Eclampsia | 326 (0.1) | 0 (0.0) | 19 (0.0) | 173 (0.2) |
| Fetal death | 1145 (0.5) | 0 (0.0) | 263 (0.3) | 0 (0.0) |
| Group B Streptococcus colonization | 40,207 (17.6) | 0 (0.0) | 6856 (7.8) | 0 (0.0) |
| Intrapartum magnesium sulfate | 8763 (4.3) | 26,136 (11.4) | 4101 (4.7) | 0 (0.0) |
| Pregestational diabetes | 5305 (2.3) | 0 (0.0) | 2446 (2.8) | 0 (0.0) |
| Chronic hypertension | 7690 (3.3) | 0 (0.0) | 4391 (5.0) | 0 (0.0) |
| Heart disease | 3481 (1.5) | 0 (0.0) | 1837 (2.1) | 173 (0.2) |
| Thyroid disease | 5583 (2.5) | 7878 (3.4) | 8682 (9.9) | 173 (0.2) |
| Renal disease | 1856 (0.8) | 0 (0.0) | 1574 (1.8) | 0 (0.0) |
| Asthma | 17,490 (7.6) | 0 (0.0) | 7059 (8.1) | 0 (0.0) |
| Depression | 9850 (4.3) | 0 (0.0) | 8433 (9.6) | 173 (0.2) |
| Gastrointestinal disease | 2592 (1.1) | 0 (0.0) | 29,156 (33.3) | 0 (0.0) |
| Active seizure disorder | 210 (0.1) | 24,679 (10.8) | 567 (0.6) | 0 (0.0) |
| History of seizure disorder | 1528 (0.7) | 22,544 (9.8) | 383 (0.4) | 32,772 (37.4) |
| Outcome: postpartum hemorrhage | 7279 (4.8) (data available for 152,279) | 0 (0.0) | 6789 (7.7) | 0 (0.0) |
Note: Mean ± standard deviation for continuous variables; n (%) for categorical variables. Direct statistical comparisons between cohorts could not be performed due to a lack of access to raw CSL data.
Abbreviations: BMI, body mass index; CSL, Consortium for Safe Labor.
The rate of PPH, defined as blood loss of at least 1000 mL in the first 24 h after delivery, was 7.7% (n = 6789) in the MGB cohort, which was higher than 4.8% (n = 7279) in the CSL cohort. The maternal characteristics of the individuals from MGB based on estimated blood loss are summarized in Table S3. Similar to the individuals in the CSL cohort, those in the MGB cohort who developed PPH had higher rates of comorbidities such as preeclampsia, gestational diabetes, and anemia.
3.2. CSL model performance in MGB cohort data
We evaluated the ability of the original CSL using the machine learning approach of XGBoost to predict the risk of PPH in MGB data. The overall performance in predicting PPH risk, as measured by the AUC, was 0.60 (95% confidence interval [CI], 0.58–0.61), Figure 2A, which was lower compared with the performance in the original dataset, AUC 0.93 (95% CI, 0.92–0.93). The model also achieved different performance based on hospital and year. Specifically, the model performed slightly better on data by hospital site from HOSPITAL 2 (Massachusetts General Hospital) than on HOSPITAL 1 (Brigham and Women's Hospital) data, with an AUC of 0.64 (95% CI, 0.63–0.65) and 0.56 (95% CI, 0.55–0.57), respectively. Additionally, performance improved modestly in deliveries after 2021, with an AUC of 0.61–0.64 (95% CI, 0.58–0.67). The model demonstrated slightly lower predictive performance among individuals who self‐identified as Black (AUC 0.56; 95% CI, 0.54–0.58) compared with individuals from other self‐reported racial and ethnic groups (AUC 0.61–0.62; 95% CI, 0.59–0.65). Additional performance metrics are reported in Tables S4–S6.
FIGURE 2.
Performance of the models in the Mass General Brigham (MGB) data. (A) Area under the receiver–operator characteristics curve of the original CSL model on the entire MGB data and (B) the newly refit XGBoost models evaluated on the held‐out MGB test dataset. AUC, area under the curve; CSL, Consortium for Safe Labor.
3.3. Refitting a new model in MGB data
We subsequently selected the features used in the CSL model and refitted a new model using MGB data, which led to an improvement in the AUC to 0.75 (95% CI, 0.73–0.76) when considering the prediction over the held‐out test set (Figure 2B). The remaining performance metrics are shown in Table 2. To evaluate the influence of hospital‐level differences, we conducted two additional analyses. Including hospital site as a predictor resulted in a minimal change in predictive accuracy (AUC 0.74 [95% CI, 0.73–0.75] without vs. 0.75 [95% CI, 0.73–0.76] with site). Additionally, when training the model exclusively on data from the largest hospital and testing on all other sites, model discrimination decreased to a mean AUC of 0.58 (95% CI, 0.56–0.59).
TABLE 2.
Performance metrics of the CSL model applied to the MGB data and the newly refit model applied on held‐out MGB data.
| Model | Accuracy | Precision | Recall | F1‐score | AUC (95% CI) |
|---|---|---|---|---|---|
| CSL model on MGB data | 0.10 | 0.07 | 0.92 | 0.14 | 0.60 (0.60, 0.61) |
| Refit MGB model on held‐out MGB data. | 0.71 | 0.16 | 0.66 | 0.26 | 0.75 (0.74, 0.76) |
Abbreviations: AUC, area under the curve; CI, confidence interval; CSL, Consortium for Safe Labor; MGB, Mass General Brigham.
Given that models from the extreme gradient boosting family (such as XGBoost) may be prone to overfitting, we also trained a logistic regression model and a Lasso model. However, these models had worse performance than XGBoost across all metrics, such as AUC, accuracy, precision, recall, and F1 score.
3.4. Calibration of the original CSL and refit models
In contrast to the excellent calibration of the CSL model in the original CSL cohort, the model demonstrated poor calibration in the MGB cohort. Among individuals classified as high risk (defined as having a predicted probability of PPH greater than 80%), the model consistently overestimated the incidence of PPH compared with observed outcomes. Conversely, among low‐risk individuals (defined as having a predicted probability below 20%), the model consistently underestimated the actual incidence of PPH (Figure 3A). The calibration curve for the refit model in the MGB test cohort overestimated PPH risk, demonstrating that individuals predicted at high PPH risk by the model generally did not develop PPH (Figure 3B). Post hoc fitting using either Platt scaling or a non‐parametric isotonic regression in our refit model did not improve model calibration.
FIGURE 3.
(A) Calibration plot of the original CSL model (held‐out Mass General Brigham [MGB] test set) and (B) calibration of the locally refit XGBoost model (held‐out MGB test set). The red 45° line indicates perfect calibration (predicted = observed). The black line shows the estimated calibration curve; dotted lines indicate 95% CIs. Black triangles are bin‐averaged points (deciles of predicted risk). CI, confidence interval; CSL, Consortium for Safe Labor; PPH, postpartum hemorrhage.
3.5. Feature importance
We compared the rankings of feature importance between the original CSL model and the refitted MGB model using gain (Figure 4A) and Shapley values methods (Figure 4B). Spontaneous labor, temperature, trial of labor, and weight remained consistently important in both models. In the refitted model, hospital site, pre‐pregnancy BMI, and admission blood pressure increased in relative importance, while history of preterm birth and fetal macrosomia decreased.
FIGURE 4.
(A) Top features ranked by mean feature gain, which measures how frequently and how effectively each variable improved the model's predictive power during training. (B) Corresponding mean absolute SHAP values, representing each feature's average contribution to model predictions across all patients. SHAP, SHapley Additive exPlanations.
3.6. Clinical usefulness of the models using decision‐curve analysis
Decision‐curve analysis evaluating net benefit at various threshold probabilities showed that both the original CSL model and the locally refit model provided minimal clinical advantage over the strategies of classifying all individuals as low risk (“treat none”) or as high risk (“treat all”). Furthermore, the net benefit was very similar between the original CSL and refit models, indicating limited clinical utility for both approaches in the MGB cohort.
4. DISCUSSION
We performed an external validation of a recently developed PPH predictive model from the US CSL cohort using contemporary EHR data across eight hospitals in a large Northeastern healthcare system. We found that the CSL model demonstrated substantially reduced discrimination and poor calibration to predict PPH accurately using contemporary EHR data. Although the CSL model performed extremely well in its original derivation cohort, we observed a marked decrease in performance (AUC of 0.60 in MGB vs. 0.93 originally reported) and inferior calibration when performing external validation. Despite improved model discrimination with refitting (AUC 0.75), the refit model did not result in enhanced clinical utility based on the decision‐curve analysis. Calibration and net benefit remained low. These results demonstrate the importance of external validation before considering prospective implementation, as the generalizability of models beyond the development cohort cannot be assumed.
Several factors may explain the observed decrease in performance, including differences in patient populations, methods, and evolving clinical practice. Importantly, the CSL data collection protocol was not designed for a machine learning study, potentially allowing inclusion of some features not available at admission [10]. Predictive models frequently demonstrate lower performance in external datasets due to overfitting, bias, and poor reporting quality [12]. To overcome these limitations, we collaborated with the authors of the original model and followed their methodology closely. Similar replication challenges occurred previously, attributed to differences in outcome ascertainment [11]. In one single‐center study, the CSL model performed with an AUC of 0.57 on local data [11]. The rate of blood loss was measured using quantitative methods, and the frequency of PPH was higher at 25% compared with 5% in the original CSL cohort and 8% in the current analysis. Implementation of quantitative blood loss measurement methods leads to increased PPH detection and blood transfusion rates [19, 20]. Although we excluded quantitative blood loss data, evolving practices at HOSPITAL 1 (Brigham and Women's Hospital) since 2018 may have introduced variability in data consistency [21].
In the current study, refitting the model using MGB data led to improved discrimination (AUC of 0.75), suggesting valuable predictive information in the original model features. However, even the locally refit model showed poor calibration, demonstrating systematic bias, similar to others [11, 22]. These calibration failures highlight that strong discrimination alone is insufficient; models that are not well‐calibrated may offer little value for guiding clinical decision‐making. Poor calibration can lead to either over‐ or undertreatment of individuals. One explanation for calibration failures is that significant improvements in clinical practice over the past decade have altered the relationship between traditional PPH risk factors and outcomes. For example, widespread PPH screening on admission [23], protocolized management of uterine atony [24, 25], novel use of tranexamic acid [26, 27], and new intrauterine devices have been introduced [28]. As a result, new types of models and additional risk factors may have to be considered.
The net benefit analyses had limited clinical utility, further emphasizing the importance of additional analyses beyond standard performance metrics. Despite improved discrimination, neither the original nor the refitted model demonstrated sufficient net benefit for clinical implementation. Machine learning models for PPH may need to incorporate more granular and dynamic features, such as detailed obstetric management, temporal factors, or improved measures of blood loss, to enhance clinical utility. These considerations extend beyond machine learning models, as logistic regression and rule‐based tools for PPH risk have also demonstrated limited clinical utility [29]. The accurate prediction of PPH risk has been fundamentally challenging as unexpected hemorrhage may develop in individuals without known risk factors, and retrospective data may lack sufficient depth for a comprehensive analysis [29].
Our results highlight an important consideration of predictive model portability in medicine: even high‐performing models trained on data from select populations often exhibit reduced predictive accuracy and require local adaptation when applied across different institutions, practice patterns, and patient populations. Refitting existing predictive models to local populations has improved accuracy and generalizability in diverse clinical contexts [30, 31]. External validation, while important, does not by itself guarantee readiness for immediate clinical use [32]. Establishing consistent definitions, documentation, and protocolized care should help achieve better patient outcomes [1], though whether or not this will be sufficient is uncertain. It may be that models should be developed that learn on their own at individual sites or sites within networks. In addition, the integration of predictive analytics tools into clinical workflows requires ongoing recalibration and validation to ensure that predictions remain accurate and clinically relevant. As such, local adaptation, continuous monitoring of model performance, and periodic recalibration should be routine steps in the life cycle of a clinical prediction model [33]. Moreover, hospitals without extensive research infrastructures could pursue local model refitting through collaboration within healthcare networks or partnerships, leveraging centralized analytic teams or clinical informatics resources to periodically recalibrate models using routinely collected EHR data.
Future research should focus on creating robust frameworks for ongoing model development, recalibration, and adaptation of predictive models. In particular, incorporating more granular and time‐linked features may help close the gap between predicted and observed outcomes. Models need to be tested across different community and academic settings and patient populations to identify the full range of relationships between risk and outcome. Moreover, the dynamic nature of obstetric care—where treatment practices, interventions, and protocols evolve—necessitates models capable of “adapting” over time. Prospective validation studies and pragmatic clinical trials could also generate evidence of whether the real‐world use of machine learning tools for PPH can truly improve maternal outcomes.
4.1. Strengths and limitations
This study was based on a large, multisite cohort that captured a broad and diverse patient population in a real‐world clinical practice. Despite its strengths, our study has several limitations. First, we used only the features available in the EHR. Some individuals lacked documented blood loss, and we partially mitigated this by extracting data from clinical notes. Second, we included only deliveries with EBL to remain consistent with the original CSL model methodology. Since our institution later transitioned to routine QBL measurement, our approach may introduce temporal bias due to evolving practices, potentially limiting model generalizability and accuracy. Third, the time frame of data collection and the specific patient population at the MGB may not be representative of all obstetric care settings in other institutions and countries. Future research should explore including additional features, refining blood loss measurement methods, standardizing clinical practice, and extending validation efforts to other practice environments. Given the growing preference for QBL as a more accurate measure of PPH, future models specifically targeting QBL may enhance predictive accuracy and clinical applicability.
5. CONCLUSION
The CSL PPH predictive model exhibited substantially reduced predictive performance in contemporary EHR data across an eight‐hospital health system in the Northeastern United States. Specifically, even after refitting with local data, persistent issues with calibration and limited clinical benefit remained. These findings highlight the importance of external validation, local adaptation, and ongoing surveillance for assessing the performance of such models in light of evolving prevention, management, and treatment strategies. Achieving reliable, generalizable, and clinically meaningful PPH predictive models will require a concerted effort to incorporate accurate data sources, refine modeling methods, establish best practices for continuous model evaluation and updating with evolving obstetric practice, and underscore the importance of evaluating the local performance of such models.
CONFLICT OF INTEREST STATEMENT
V.P.K. reports funding from the NIH/NHLBI grants 1K08HL161326‐01A1, Anesthesia Patient Safety Foundation (APSF), and BWH IGNITE Award. V.P.K. also reports consulting fees from Avania CRO unrelated to the current work and patent #WO2021119593A1 for the control of a therapeutic delivery system assigned to Mass General Brigham. D.W.B. reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from ValeraHealth, equity from Clew, equity from MDClone, personal fees and equity from AESOP, personal fees and equity from FeelBetter, personal fees and equity from Guided Clinical Solutions, outside the submitted work. K.J.G. reports consulting fees for BillionToOne, Aetion, Roche, and Janssen Global, outside the submitted work. E.J. reports institutional funding from NICHD, NIDDK, and FDA, honoraria from UpToDate. R.K., N.W., and K.K.V. have not reported any conflicts of interest.
ETHICS STATEMENT
This study was approved by the Mass General Brigham Institutional Review Board.
Supporting information
Supporting Information
ACKNOWLEDGEMENTS
This study was supported by NIH/NHLBI grants 1K08HL161326‐01A1 and the Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women's Hospital.
This study was presented at AMIA Informatics Summit 2024, Boston, MA, USA.
DATA AVAILABILITY STATEMENT
The data supporting the findings of this study are not publicly available due to privacy and ethical restrictions imposed by the Institutional Review Board (IRB). Data sharing requests may be directed to the corresponding author but will require review and approval from the IRB.
REFERENCES
- 1. Committee on Practice BO . 2017. “Practice Bulletin No. 183: Postpartum Hemorrhage.” Obstetrics and Gynecology 130(4): e168–e86. [DOI] [PubMed] [Google Scholar]
- 2. Bienstock, J. L. , Eke A. C., and Hueppchen N. A.. 2021. “Postpartum Hemorrhage.” New England Journal of Medicine 384(17): 1635–45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Corbetta‐Rastelli, C. M. , Friedman A. M., Sobhani N. C., Arditi B., Goffman D., and Wen T.. 2023. “Postpartum Hemorrhage Trends and Outcomes in the United States, 2000–2019.” Obstetrics and Gynecology 141(1): 152–61. [DOI] [PubMed] [Google Scholar]
- 4. Bonnet, M.‐P. , de Vries P. L. M., Levet S., Saucedo M., Morau E., Dreyfus M., Goffinet F., and Deneux‐Tharaux C., 2024. “Trends in Maternal Mortality From Obstetric Hemorrhage in France: 15 Years of Confidential Enquiry Into Maternal Deaths.” Anesthesia and Analgesia 139(6): 1170–80. [DOI] [PubMed] [Google Scholar]
- 5. Dilla, A. J. , Waters J. H., and Yazer M. H.. 2013. “Clinical Validation of Risk Stratification Criteria for Peripartum Hemorrhage.” Obstetrics and Gynecology 122(1): 120–6. [DOI] [PubMed] [Google Scholar]
- 6. Venkatesh, K. K. , Strauss R. A., Grotegut C. A., Heine R. P., Chescheir N. C., Stringer J. S. A., Stamilio D. M., Menard K. M., and Jelovsek J. E., 2020. “Machine Learning and Statistical Models to Predict Postpartum Hemorrhage.” Obstetrics and Gynecology 135(4): 935–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Ende, H. B. , French B., Shi Y., Kertai M. D., Osmundson S. S., and Bauchat J. R.. 2022. “Development of Clinical Risk‐Prediction Models for Uterine Atony Following Vaginal and Cesarean Delivery.” International Journal of Obstetric Anesthesia 51: 103550. [DOI] [PubMed] [Google Scholar]
- 8. Schmidt, L. J. , Rieger O., Neznansky M., Hackelöer M., Dröge L. A., Henrich W., Higgins D., and Verlohren S., 2022. “A Machine‐Learning‐Based Algorithm Improves Prediction of Preeclampsia‐Associated Adverse Outcomes.” American Journal of Obstetrics and Gynecology 227(1): 77.e1–e30. [DOI] [PubMed] [Google Scholar]
- 9. Belaghi, R. A. . 2024. “Prediction of Preterm Birth in Multiparous Women Using Logistic Regression and Machine Learning Approaches.” Scientific Reports 14(1): 21967. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Zhang, J. , Troendle J., Reddy U. M., Laughon S. K., Branch D. W., Burkman R., Landy H. J., et al. 2010. “Contemporary Cesarean Delivery Practice in the United States.” American Journal of Obstetrics and Gynecology 203(4): 326.e1–326.e10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Meyer, S. R. , Carver A., Joo H., Venkatesh K. K., Jelovsek J. E., Klumpner T. T., and Singh K.. 2024. “External Validation of Postpartum Hemorrhage Prediction Models Using Electronic Health Record Data.” American Journal of Perinatology 41(5): 598–605. [DOI] [PubMed] [Google Scholar]
- 12. For the IPPIC Collaborative Network , Snell, K. I. E. , Allotey J., Smuk M., Hooper R., Chan C., Ahmed A., et al. 2020. “External Validation of Prognostic Models Predicting Pre‐Eclampsia: Individual Participant Data Meta‐Analysis.” BMC Medicine 18(1): 302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Cohen, R. Y. and Kovacheva V. P.. 2023. “A Methodology for a Scalable, Collaborative, and Resource‐Efficient Platform, MERLIN, to Facilitate Healthcare AI Research.” IEEE Journal of Biomedical And Health Informatics 27(6): 3014–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Collins, G. S. , Reitsma J. B., Altman D. G., and Moons K. G. M.. 2015. “Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement.” BMJ 350: g7594. [DOI] [PubMed] [Google Scholar]
- 15. Kuklina, E. V. , Whiteman M. K., Hillis S. D., Jamieson D. J., Meikle S. F., Posner S. F., and Marchbanks P. A., 2008. “An Enhanced Method for Identifying Obstetric Deliveries: Implications for Estimating Maternal Morbidity.” Maternal and Child Health Journal 12(4): 469–77. [DOI] [PubMed] [Google Scholar]
- 16. Alsentzer, E. , Rasmussen M. J., Fontoura R., Cull A. L., Beaulieu‐Jones B., Gray K. J., Bates D. W., and Kovacheva V. P.. 2023. “Zero‐Shot Interpretable Phenotyping of Postpartum Hemorrhage Using Large Language Models.” npj Digital Medicine 6(1): 212. Available From: https://www.medrxiv.org/content/medrxiv/early/2023/06/01/2023.05.31.23290753.full.pdf. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Strumbelj, E. and Kononenko I.. 2010. “An Efficient Explanation of Individual Classifications Using Game Theory.” Journal of Machine Learning Research 11: 1–18. [Google Scholar]
- 18. Jiang, X. , Osl M., Kim J., and Ohno‐Machado L.. 2012. “Calibrating Predictive Model Estimates to Support Personalized Medicine.” Journal of the American Medical Informatics Association 19(2): 263–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Saeed, H. , Coviello E., Amgalan A., Thomas A., and Kawakita T.. 2022. “Evaluation of Blood Transfusion Rates by Blood Loss Estimation Techniques.” Journal of Maternal‐Fetal & Neonatal Medicine 35(25): 6961–6. [DOI] [PubMed] [Google Scholar]
- 20. Coviello, E. , Iqbal S., Kawakita T., Chornock R., Cheney M., Desale S., and Fries M.. 2019. “Effect of Implementing Quantitative Blood Loss Assessment at the Time of Delivery.” American Journal of Perinatology 36(13): 1332–6. [DOI] [PubMed] [Google Scholar]
- 21. Lumbreras‐Marquez, M. I. , Reale S. C., Carusi D. A., Robinson J. N., Scharf N., Fields K. G., and Farber M. K., 2020. “Introduction of a Novel System for Quantitating Blood Loss After Vaginal Delivery: A Retrospective Interrupted Time Series Analysis With Concurrent Control Group.” Anesthesia and Analgesia 130(4): 857–68. [DOI] [PubMed] [Google Scholar]
- 22. Ahmadzia, H. K. , Dzienny A. C., Bopf M., Phillips J. M., Federspiel J. J., Amdur R., Rice M. M., and Rodriguez L., 2024. “Machine Learning Models for Prediction of Maternal Hemorrhage and Transfusion: Model Development Study.” JMIR Bioinformatics and Biotechnology 5: e52059. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. The Joint Commission . R3 Report: Provision of Care, Treatment, and Services Standards for Maternal Safety. Issue 24. [Internet]. 2019. Available From: https://digitalassets.jointcommission.org/api/public/content/ed4a0ea369c44adca8bc48ba05b76b61?v=a0747800. [Google Scholar]
- 24. Doyle, J. L. , Kenny T. H., Gothard M. D., Seagraves E., McCarroll M., and Silber A.. 2019. “A Standardized Oxytocin Administration Protocol After Delivery to Reduce the Treatment of Postpartum Hemorrhage.” Joint Commission Journal on Quality and Patient Safety 45(2): 131–43. [DOI] [PubMed] [Google Scholar]
- 25. Kovacheva, V. P. , Soens M. A., and Tsen L. C.. 2015. “A Randomized, Double‐Blinded Trial of a “Rule of Threes” Algorithm Versus Continuous Infusion of Oxytocin During Elective Cesarean Delivery.” Anesthesiology 123(1): 92–100. [DOI] [PubMed] [Google Scholar]
- 26. Ker, K. , Sentilhes L., Shakur‐Still H., Madar H., Deneux‐Tharaux C., Saade G., Pacheco L. D., et al. 2024. “Tranexamic Acid for Postpartum Bleeding: A Systematic Review and Individual Patient Data Meta‐Analysis of Randomised Controlled Trials.” Lancet 404(10463): 1657–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Shakur, H. , Roberts I., Fawole B., Chaudhri R., El‐Sheikh M., Akintan A., Qureshi Z., et al. 2017. “Effect of Early Tranexamic Acid Administration on Mortality, Hysterectomy, and Other Morbidities in Women With Post‐Partum Haemorrhage (WOMAN): An International, Randomised, Double‐Blind, Placebo‐Controlled Trial.” Lancet 389(10084): 2105–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Overton, E. , D'Alton M., and Goffman D.. 2024. “Intrauterine Devices in the Management of Postpartum Hemorrhage.” American Journal of Obstetrics and Gynecology 230(3S): S1076–88. [DOI] [PubMed] [Google Scholar]
- 29. Pressly, M. A. , Parker R. S., Waters J. H., Beck S. L., Jeyabalan A., and Clermont G.. 2021. “Improvements and Limitations in Developing Multivariate Models of Hemorrhage and Transfusion Risk for the Obstetric Population.” Transfusion 61(2): 423–34. [DOI] [PubMed] [Google Scholar]
- 30. Meijerink, L. M. , Dunias Z. S., Leeuwenberg A. M., de Hond A. A. H., Jenkins D. A., Martin G. P., Sperrin M., et al. 2025. “Updating Methods for Artificial Intelligence‐Based Clinical Prediction Models: A Scoping Review.” Journal of Clinical Epidemiology 178: 111636. [DOI] [PubMed] [Google Scholar]
- 31. Subasri, V. , Krishnan A., Kore A., Dhalla A., Pandya D., Wang B., Malkin D., et al. 2025. “Detecting and Remediating Harmful Data Shifts for the Responsible Deployment of Clinical AI Models.” JAMA Network Open 8(6): e2513685. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. la Roi‐Teeuw, H. M. , van Royen F. S., de Hond A., Zahra A., de Vries S., Bartels R., Carriero A. J., et al. 2024. “Don't Be Misled: 3 Misconceptions About External Validation of Clinical Prediction Models.” Journal of Clinical Epidemiology 172: 111387. [DOI] [PubMed] [Google Scholar]
- 33. Labkoff, S. , Oladimeji B., Kannry J., Solomonides A., Leftwich R., Koski E., Joseph A. L., et al. 2024. “Toward a Responsible Future: Recommendations for AI‐Enabled Clinical Decision Support.” Journal of the American Medical Informatics Association 31(11): 2730–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supporting Information
Data Availability Statement
The data supporting the findings of this study are not publicly available due to privacy and ethical restrictions imposed by the Institutional Review Board (IRB). Data sharing requests may be directed to the corresponding author but will require review and approval from the IRB.