Key Points
Question
Can prenatal and postdelivery variables accurately predict the risk of stillbirth and neonatal deaths in resource-limited settings of low- and middle-income countries?
Findings
Using advanced machine learning–based modeling techniques on a large multicountry prospective maternal and neonatal database, this cohort study found that the prediction accuracy of models for risk of stillbirth and neonatal death using variables before delivery is low, but the prediction accuracy for neonatal death can be improved by including postdelivery variables. Birth weight was the most important predictor of neonatal mortality.
Meaning
Models that include postdelivery variables have good prediction accuracy for neonatal deaths.
This cohort study evaluates the accuracy of a machine learning–based model to predict risk of stillbirth and neonatal mortality in low- and middle-income countries.
Abstract
Importance
The overwhelming majority of fetal and neonatal deaths occur in low- and middle-income countries. Fetal and neonatal risk assessment tools may be useful to predict the risk of death.
Objective
To develop risk prediction models for intrapartum stillbirth and neonatal death.
Design, Setting, and Participants
This cohort study used data from the Eunice Kennedy Shriver National Institute of Child Health and Human Development Global Network for Women’s and Children’s Health Research population-based vital registry, including clinical sites in South Asia (India and Pakistan), Africa (Democratic Republic of Congo, Zambia, and Kenya), and Latin America (Guatemala). A total of 502 648 pregnancies were prospectively enrolled in the registry.
Exposures
Risk factors were added sequentially into the data set in 4 scenarios: (1) prenatal, (2) predelivery, (3) delivery and day 1, and (4) postdelivery through day 2.
Main Outcomes and Measures
Data sets were randomly divided into 10 groups of 3 analysis data sets including training (60%), test (20%), and validation (20%). Conventional and advanced machine learning modeling techniques were applied to assess predictive abilities using area under the curve (AUC) for intrapartum stillbirth and neonatal mortality.
Results
All prenatal and predelivery models had predictive accuracy for both intrapartum stillbirth and neonatal mortality with AUC values 0.71 or less. Five of 6 models for neonatal mortality based on delivery/day 1 and postdelivery/day 2 had increased predictive accuracy with AUC values greater than 0.80. Birth weight was the most important predictor for neonatal death in both postdelivery scenarios with independent predictive ability with AUC values of 0.78 and 0.76, respectively. The addition of 4 other top predictors increased AUC to 0.83 and 0.87 for the postdelivery scenarios, respectively.
Conclusions and Relevance
Models based on prenatal or predelivery data had predictive accuracy for intrapartum stillbirths and neonatal mortality of AUC values 0.71 or less. Models that incorporated delivery data had good predictive accuracy for risk of neonatal mortality. Birth weight was the most important predictor for neonatal mortality.
Introduction
The neonatal period is the period in life with the highest risk for mortality.1 Annually, 2.5 million neonatal deaths and 2.6 million stillbirths occur globally, of which 1.3 million are intrapartum stillbirths.2 It is estimated that approximately 98% of all neonatal and perinatal deaths occur in low- and middle-income countries.3,4,5 However, almost all the published literature on identifying predictors of fetal and neonatal mortality and risk scoring tools are based on data from high-income countries. Data from low- and middle-income countries are limited to small sample size studies that lack validation with an independent sample. Additionally, machine learning prediction models may perform better than conventional models when applied to large data sets given their ability to delineate complex relationships and identify novel interactions between variables.6,7,8,9,10,11 Although machine learning–based prediction models are expected to perform better with large data sets, this hypothesis has not been convincingly tested with a good quality prospectively collected population database.6,7,8,11
We aimed to develop a risk assessment tool for intrapartum stillbirth and neonatal mortality that would include maternal and neonatal variables from a prospective multicountry maternal and neonatal database. We compared various conventional and advanced machine learning–based, analytical modeling methods at specific time points to establish individual predictive accuracies of the models. We tested the hypothesis that intrapartum stillbirth and neonatal mortality risk prediction models that include antenatal and delivery variables provide a high accuracy. Additionally, we also tested whether advanced machine learning–based models have higher predictive accuracy than a conventional logistic regression model.
Methods
Study Design and Participants
The study was conducted in the Eunice Kennedy Shriver National Institute of Child Health and Human Development Global Network for Women’s and Children’s Health Research, which includes clinical sites in resource-limited settings in South Asia (India and Pakistan), Africa (Democratic Republic of Congo, Zambia, and Kenya), and Latin America (Guatemala). A population-based Global Network Maternal Newborn Health Registry (GN-MNHR) vital registry was established in 2009.12 Pregnant women and their 502 648 offspring participating in the GN-MNHR database from January 1, 2010, to December 31, 2018, were included. The description of the sites and the cluster settings has been reported.13 The GN-MNHR database includes data starting with the initial prenatal visit and up to 42 days after delivery of study participants. The GN-MNHR data processes include close quality monitoring and quality improvement interventions both at local and central levels to ensure data completeness and quality.12 The GN-MNHR definitions were used to define variables and outcomes as reported previously.13 The GN-MNHR database has been reviewed and approved by all sites’ ethics review committees and the institutional review boards at each US partner university and at the data coordinating center (RTI International). All women provided written informed consent for participation in the GN-MNHR database, including data collection and the follow-up visits. The study is reported as per the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline for multivariable prediction modeling reporting.14 The NICHD Global Network Maternal Newborn Health Registry is registered with ClinicalTrials.gov (NCT01073475).
Outcomes
The outcome variables were intrapartum stillbirth and neonatal mortality. Intrapartum stillbirth was defined as nonmacerated stillbirth presumably occurring during labor. Neonatal mortality was defined as death up to 28 days after birth. Potential risk factors (variables) for the outcome were selected from the database based on the existing literature and relevancy to the outcomes.
Statistical Analysis
The risk factors were added sequentially into 4 scenario data sets: (1) prenatal (variables until first prenatal care visit), (2) predelivery (variables until just before delivery), (3) delivery and day 1 (delivery/day 1), and (4) postdelivery through day 2 (postdelivery/day 2). Day 0 was defined as the calendar day of birth. Day 1 and day 2 were defined as the subsequent calendar days. We evaluated mortality outcomes using potential risk factor sets with sequentially additional variables to determine whether additional potential risk factors improved outcome predictive accuracy.15 The first 2 scenario models (prenatal and predelivery) evaluated the outcome of intrapartum stillbirth and neonatal mortality. The third scenario model (delivery/day 1) evaluated the outcome of neonatal mortality on days 2 through 27. The fourth scenario model (postdelivery/day 2) evaluated the outcome of neonatal mortality on days 3 through 27. The delivery and postdelivery data sets were censored for deaths occurring prior to grouping time points and missing data so that only surviving neonates with complete data were included.
To build and validate predictive models, split sampling was used to set aside sections of the data for uncertainty estimation and model validation of predictive accuracy. The models considered were logistic regression and 5 machine learning models (SVM [support vector machine with radial basis function kernel], EN [logistic elastic net], NN [neural network], GBE [gradient boosted ensemble], and RF [random forest]). Data management was completed using SAS 9.4 software (SAS Institute Inc), and the model building was completed using the scikit-learn Python module. Graphics were completed using R 4.0.2 (R Project for Statistical Computing). All models except logistic regression were tuned using 10-fold cross-validation on training data, and then each tuned model was applied to the test data for a predictive accuracy assessment. For assessment of the consistency of accuracy, the tuning was repeated on training plus test data, and tuned models were applied to the validation data. The predictive accuracy was assessed using the area under the curve (AUC) of the receiver operating characteristic (ROC) curves. Because the result from any accuracy assessment using randomly split data is random, the entire analysis was repeated within each of 10 mutually exclusive data subsets for each scenario. This enabled us to have 10 assessments of accuracy for each model within each scenario, allowing an assessment of the uncertainty in the estimated accuracy for all models. Paired t tests were used on these 10 estimates of accuracy to compare models in order to descriptively assess whether or not the models within a scenario were discernably different in light of the uncertainty. The process of building and validating the best predictive machine learning model and using the results to build a modified logistic regression model for mortality risk scoring is described in the eAppendix in the Supplement.
Results
After the removal of missing data and deaths before grouping time points, the prenatal data set contained 487 642 neonates, the predelivery data set contained 487 537 neonates, the delivery/day 1 data set contained 469 516 neonates, and the postdelivery/day 2 data set contained 468 356 neonates (Figure 1). The sex distribution of the neonates was 51.6% male and 48.4% female in the prenatal data set. Baseline maternal and neonatal variables vary slightly for each of the 4 data sets owing to censoring (Table 1). The sample sizes for each subset before splitting into training, test, and validation data sets also vary slightly, which reflects variation owing to missing data (eTable 1 in the Supplement).
Figure 1. Participant Flow Diagram.
The flow diagram denotes the number of participants for each data set prior to the removal of missing covariate data. The data sets were censored for both deaths occurring before the time points and missing data. Ineligible participants included those who were enrolled early and later found not to be pregnant and those who were residing outside the study clusters.
502 648/578 633 deliveries had outcome data, 487 642/502 648 deliveries had complete predictor data for prenatal scenario, 487 537/502 648 deliveries had complete predictor data for predelivery scenario, 469 516/487 326 neonates alive on day 2 had complete predictor data for delivery/day 1 scenario, and 468 356/485 966 neonates alive on day 3 had complete predictor data for postdelivery/day 2 scenario.
Table 1. Baseline Maternal and Neonatal Variables.
| Variable | Prenatal (n = 487 642) | Predelivery (n = 487 537) | Delivery/day 1 (n = 469 516) | Postdelivery/day 2 (n = 468 356) | Scenario | ||
|---|---|---|---|---|---|---|---|
| Continuous variables, mean (SD) | |||||||
| Gestational age at enrollment, wk | 20.2 (9.2) | 20.2 (9.2) | 20.2 (9.3) | 20.2 (9.3) | Prenatal | ||
| Parity, No. | 1.8 (2.1) | 1.8 (2.1) | 1.7 (2.1) | 1.7 (2.1) | Prenatal | ||
| Cluster mortality, ratea | 0.04 (0.02) | 0.04 (0.02) | 0.04 (0.02) | 0.04 (0.02) | Prenatal | ||
| Maternal age, y | 24.8 (5.3) | 24.8 (5.3) | 24.8 (5.3) | 24.8 (5.3) | Prenatal | ||
| Gestational age, wk | 38.6 (3.5)b | 38.6 (3.5)b | 38.7 (3.3) | 38.7 (3.3) | Delivery/day 1 | ||
| Birth weight, g | 2899 (505)c | 2899 (505)c | 2914 (480) | 2916 (478) | Delivery/day 1 | ||
| Categorical variables, No. (%) | |||||||
| Site | Prenatal | ||||||
| DRC | 31 141 (6.4) | 31 138 (6.4) | 29 595 (6.3) | 29 553 (6.3) | |||
| Zambia | 60 159 (12.3) | 60 157 (12.3) | 58 556 (12.5) | 58 489 (12.5) | |||
| Guatemala | 73 901 (15.2) | 73 901 (15.2) | 72 295 (15.4) | 72 119 (15.4) | |||
| Belagavi, India | 101 385 (20.8) | 101 350 (20.8) | 97 913 (20.9) | 97 671 (20.9) | |||
| Pakistan | 76 273 (15.6) | 76 218 (15.6) | 71 102 (15.1) | 70 793 (15.1) | |||
| Nagpur, India | 77 774 (15.9) | 77 769 (16.0) | 75 612 (16.1) | 75 354 (16.1) | |||
| Kenya | 67 009 (13.7) | 67 004 (13.7) | 64 443 (13.7) | 64 377 (13.7) | |||
| Maternal variables | |||||||
| Age (categorical), y | Prenatal | ||||||
| <20 | 63 976 (13.1) | 63 963 (13.1) | 61 695 (13.1) | 61 562 (13.1) | |||
| 20-35 | 401 172 (82.3) | 401 084 (82.3) | 386 461 (82.3) | 385 491 (82.3) | |||
| >35 | 22 494 (4.6) | 22 490 (4.6) | 21 360 (4.5) | 21 303 (4.5) | |||
| Education (categorical) | Prenatal | ||||||
| None | 111 608 (22.9) | 111 558 (22.9) | 105 057 (22.4) | 104 716 (22.4) | |||
| Primary | 143 875 (29.5) | 143 853 (29.5) | 138 867 (29.6) | 138 549 (29.6) | |||
| Secondary | 197 542 (40.5) | 197 510 (40.5) | 191 767 (40.8) | 191 354 (40.9) | |||
| University | 34 617 (7.1) | 34 616 (7.1) | 33 825 (7.2) | 33 737 (7.2) | |||
| Birth order | Prenatal | ||||||
| First | 482 984 (99.0) | 482 885 (99.0) | 465 612 (99.2) | 464516 (99.2) | |||
| Second | 4615 (0.9) | 4609 (0.9) | 3875 (0.8) | 3811 (0.8) | |||
| Third | 42 (0.0) | 42 (0.0) | 29 (0.0) | 29 (0.0) | |||
| Fourth | 1 (0.0) | 1 (0.0) | 0 (0.0) | 0 (0.0) | |||
| ≥1 prenatal visit | 474 530 (97.4) | 474 433 (97.4) | 457 467 (97.5) | 456 338 (97.5) | Predelivery | ||
| Antepartum hemorrhage | 5625 (1.2) | 5621 (1.2) | 3856 (0.8) | 3797 (0.8) | Predelivery | ||
| Hypertension | 12 798 (2.6) | 12 793 (2.6) | 11 501 (2.5) | 11 407 (2.4) | Predelivery | ||
| Suspected sepsis | 2142 (0.5) | 2141 (0.5) | 1902 (0.4) | 1892 (0.4) | Predelivery | ||
| Eclampsia | 222 (0.0) | 222 (0.0) | 192 (0.0) | 191 (0.0) | Predelivery | ||
| Antenatal corticosteroids | 9228 (2.6) | 9227 (2.6) | 8544 (2.5) | 8463 (2.5) | Predelivery | ||
| Hospitalization | 3790 (0.8) | 3790 (0.8) | 3502 (0.8) | 3492 (0.8) | Postdelivery/day 2 | ||
| Antibiotics | 195 383 (48.6) | 195 368 (48.6) | 187 268 (48.3) | 186 604 (48.3) | Postdelivery/day 2 | ||
| Delivery variables | |||||||
| Attendant | Predelivery | ||||||
| Physician | 177 644 (36.4) | 177 605 (36.4) | 170 942 (36.4) | 170 362 (36.4) | |||
| Nurse/midwife | 180 191 (37.0) | 180 181 (37.0) | 174 874 (37.2) | 174 562 (37.3) | |||
| TBA | 105 025 (21.5) | 105 016 (21.5) | 100 981 (21.5) | 100 775 (21.5) | |||
| Family/self/other | 24 735 (5.1) | 24 735 (5.1) | 22 719 (4.8) | 22 657 (4.8) | |||
| Location | Predelivery | ||||||
| Hospital | 219 777 (45.1) | 219 767 (45.1) | 211 746 (45.1) | 211 101 (45.1) | |||
| Clinic/health center | 146 058 (30.0) | 146 034 (30.0) | 141 705 (30.2) | 141 417 (30.2) | |||
| Home | 121 743 (25.0) | 121 736 (25.0) | 116 065 (24.7) | 115 838 (24.7) | |||
| Lie | Predelivery | ||||||
| Transverse | 306 (0.5) | 306 (0.5) | 289 (0.5) | 286 (0.5) | |||
| Oblique | 129 (0.2) | 129 (0.2) | 115 (0.2) | 115 (0.2) | |||
| Breech | 823 (1.3) | 823 (1.3) | 717 (1.1) | 715 (1.1) | |||
| Vertex | 64 073 (98.1) | 64 071 (98.1) | 62 169 (98.2) | 62 026 (98.2) | |||
| Multiple birth | 9251 (1.9) | 9243 (1.9) | 7957 (1.7) | 7846 (1.7) | Delivery/day 1 | ||
| Mode | Delivery/day 1 | ||||||
| Vaginal | 413 453 (84.8) | 413 364 (84.8) | 396 606 (84.7) | 396 704 (84.7) | |||
| Vaginal, assisted | 4753 (1.0) | 4750 (1.0) | 4311 (0.9) | 4284 (0.9) | |||
| CD | 69 425 (14.2) | 69 418 (14.2) | 67 599 (14.4) | 67 368 (14.4) | |||
| Obstructed labor | 40 540 (8.3) | 40 527 (8.3) | 36 938 (7.9) | 36 760 (7.9) | Delivery/day 1 | ||
| Neonatal variables | |||||||
| Sex | Delivery/day 1 | ||||||
| Male | 251 197 (51.6) | 251 148 (51.6) | 241 240 (51.4) | 240 561 (51.4) | |||
| Female | 236 080 (48.4) | 236 026 (48.4) | 228 240 (48.6) | 227 759 (48.6) | |||
| Bag and mask resuscitation | 19 669 (4.1) | 19 649 (4.1) | 16 692 (3.6) | 16 254 (3.5) | Delivery/day 1 | ||
| Congenital anomalies | 1084 (0.5) | 1084 (0.5) | 677 (0.3) | 658 (0.3) | Delivery/day 1 | ||
| Hospitalization | 9650 (2.2) | 9648 (2.2) | 9474 (2.1) | 9315 (2.1) | Postdelivery/day 2 | ||
| Antibiotics | 20 740 (5.8) | 20 736 (5.7) | 19 178 (5.5) | 18 726 (5.4) | Postdelivery/day 2 | ||
| Cord care | 83 172 (77.2) | 83 172 (77.2) | 81 661 (78.8) | 81 416 (78.8) | Postdelivery/day 2 | ||
| Medicinal cord care | 52 076 (20.6) | 52 076 (20.6) | 51 110 (20.9) | 50 950 (20.9) | Postdelivery/day 2 | ||
Abbreviations: CD, cesarean delivery; DRC, Democratic Republic of Congo; TBA, traditional birth attendant.
Cluster perinatal mortality is the rate of perinatal mortality within each distinct geographical area (cluster) of the sites as defined by the Global Network.
Missing 0.40% of data.
Missing 1.13% of data.
Models using either only prenatal or prenatal and predelivery variables had predictive accuracy for intrapartum stillbirth and neonatal mortality of AUC values 0.71 or less (Figure 2). The analysis of models for intrapartum stillbirth showed that all prenatal models had AUC values of 0.63 or less and all predelivery models had AUC values of 0.72 or less (eFigure 1 in the Supplement). Cluster perinatal mortality was the most important predictor of intrapartum stillbirth in the prenatal data set (AUC, 0.60) and antepartum hemorrhage was the most important predictor in the predelivery data set (AUC, (0.56) (eTable 2 in the Supplement). Other important predictors of intrapartum stillbirth in the prenatal data set were gestational age at enrollment, maternal age, birth order, and parity. Other important predictors of intrapartum stillbirth in the predelivery data set were cluster perinatal mortality; gestational age at enrollment; hypertension, severe pre-eclampsia, or eclampsia; and maternal age.
Figure 2. Mean (95% CI) for Validation AUC by Scenario for Outcomes of Intrapartum Stillbirth and Neonatal Mortality.
EN indicates logistic elastic net; GBE, gradient boosted ensemble; NN, neural network; RF, random forest; and SVM, support vector machine with radial basis function kernel.
The predictive models based on the data sets that included delivery/day 1 and postdelivery/day 2 variables had good predictive accuracy for neonatal mortality, with 5 of the 6 models having AUCs above 0.80. Birth weight was the most important predictor in both the delivery/day 1 and postdelivery/day 2 scenarios, with independent predictive ability of AUC 0.78 and 0.76, respectively. The increase in probability of mortality with decreasing in birth weight occurred in both the delivery/day 1 and postdelivery/day 2 scenarios (Figure 3; eFigure 2 in the Supplement). Bag and mask resuscitation, gestational age, cluster perinatal mortality rate, and maternal age were the other top predictors for the delivery/day 1 scenario. Conditions requiring hospitalization, antibiotics, gestational age, and bag and mask resuscitation were the other top predictors for the postdelivery/day 2 scenario. The addition of these other top predictors resulted in increases in the AUCs (0.83 and 0.87 for the delivery/day 1 and postdelivery/day 2 scenarios, respectively) relative to birth weight alone.
Figure 3. Probability of Mortality as a Function of Birth Weight, Delivery/Day 1 Scenario.
For models assessing the outcomes of stillbirth and neonatal mortality, the pairwise paired t test showed that there were statistically insignificant differences between AUC values illustrated by considerable overlap of confidence intervals (Figure 2). However, gradient boosted ensemble and random forest models were consistently among the best-performing models. Even though the logistic regression model was not the best-performing model in any scenario, the AUC of the logistic regression model was not significantly different than the top-performing models.
The sequential addition of variables was done to identify the individual relative contribution of each variable to AUC for combined outcomes of intrapartum stillbirth and neonatal mortality based on validation data set (Table 2) and to develop a risk scoring system for stillbirth and neonatal mortality. Table 2 lists the changes in AUC as predictors are added to the model using the ordering of predictor importance (see eAppendix in the Supplement for more details about predictor importance). A modified logistic regression model based on findings from the best machine learning model as well as results from a variable selection study using logistic regression and the least absolute shrinkage and selection operator (LASSO) method for identification of potential interactions was fit to create a risk scoring system for the delivery/day 1 and postdelivery/day 2 data (eTable 3 in the Supplement). Using risk scores calculated from this risk scoring system, a logistic regression model was fit to the total risk scores in order to derive a formula for predicting the probability of mortality given the calculated risk score. The logistic model for neonatal mortality risk scoring had an AUC value on validation data equal to 0.809 (97% of the best model) for the delivery/day 1 scenario and 0.845 (97% of the best model) for the postdelivery/day 2 scenario (eTable 4 in the Supplement).
Table 2. Top Predictors by Scenario (Intrapartum Stillbirth and Neonatal Mortality Outcome)a.
| Rank | Predictor | AUC | AUC increase |
|---|---|---|---|
| Delivery/day 1 | |||
| 1 | Birth weight | 0.776 | NA |
| 2 | Bag and mask resuscitation | 0.814 | +0.039 |
| 3 | Gestational age | 0.817 | +0.003 |
| 4 | Cluster perinatal mortality | 0.816 | −0.001 |
| 5 | Maternal age | 0.820 | +0.004 |
| 6 | Parity | 0.822 | +0.002 |
| 7 | Gestational age at enrollment | 0.822 | 0 |
| 8 | Antepartum hemorrhage | 0.823 | +0.001 |
| 9 | Multiple birth | 0.824 | +0.001 |
| 10 | Hypertension/severe pre-eclampsia/eclampsiab | 0.822 | −0.002 |
| 11 | Antenatal corticosteroids | 0.823 | +0.001 |
| 12 | Birth order | 0.822 | −0.001 |
| 13 | Suspected maternal sepsisc | 0.823 | +0.001 |
| 14 | Neonatal sex | 0.827 | +0.004 |
| 15 | Obstructed labor | 0.829 | +0.002 |
| Postdelivery/day 2 | |||
| 1 | Birth weight | 0.763 | NA |
| 2 | Neonatal hospitalization | 0.813 | +0.050 |
| 3 | Neonatal antibiotics | 0.845 | +0.032 |
| 4 | Gestational age | 0.842 | −0.003 |
| 5 | Bag and mask resuscitation | 0.852 | +0.010 |
| 6 | Cluster perinatal mortality | 0.865 | +0.013 |
| 7 | Gestational age at enrollment | 0.867 | +0.002 |
| 8 | Maternal age | 0.870 | +0.003 |
| 9 | Parity | 0.870 | 0 |
| 10 | Multiple birth | 0.870 | 0 |
| 11 | Antenatal corticosteroids | 0.871 | +0.001 |
| 12 | Maternal education: none | 0.871 | 0 |
| 13 | Hospital delivery | 0.872 | +0.001 |
| 14 | Maternal antibiotics | 0.871 | −0.0001 |
| 15 | Delivery by physician | 0.872 | +0.001 |
Abbreviations: AUC, area under the curve; NA, not applicable.
Predictors are added consecutively using gradient boosted ensemble model; then AUC calculated. The order used in this table was the order of importance assigned by the predictive model assessment of important predictors.
Hypertensive disease/severe pre-eclampsia/eclampsia defined as blood pressure >140/90 mm Hg, proteinuria, and seizures.
Suspected maternal sepsis defined as fever with pelvic pain and abnormal vaginal discharge (foul smelling or presence of pus).
Discussion
This cohort study found that predictive models using only prenatal or prenatal and predelivery variables had predictive accuracy for intrapartum stillbirth and neonatal mortality of AUC values of 0.71 or less. We identified that a better neonatal mortality risk prediction could be made when variables obtained immediate postdelivery and up to 2 days after birth were included in the models, with AUCs increasing up to 0.87. Birth weight was identified as the most important predictor for neonatal mortality among all variables considered. The contribution of other predictors to the AUC increase was relatively minor.
Many studies have analyzed variables associated with increased risk for stillbirths or neonatal mortality, but only 1 study16 used a relatively large group of neonates to develop predictive models. As found in the current study, low predictive accuracy (AUC = 0.58, 95% CI = 0.56–0.59) using prenatal variables was reported in a study based on 10 sites from 3 countries in South Asia (N = 49 632).16 Similar to the findings of the current study, the predictive accuracy of the model improved when postdelivery variables were included (AUC = 0.83, 95% CI = 0.79–0.86), but only logistic regression modeling was applied to develop the model, and the results were not validated with an independent sample. The current results are consistent with the data from a large prospective neonatal database of participants living in a high-income country, which showed that inclusion of delivery variables (especially birth weight) results in better accuracy for predicting neonatal mortality than models with only prenatal variables.15
Some of the important predictors in the current study have been reported to be associated with an increased risk for neonatal mortality in simple association analyses. In a pooled analysis of data from low- and middle-income countries, small for gestational age and prematurity were found to be associated with increased risk of neonatal mortality in association (bivariate) analysis, although birth weight was not specifically analyzed.17 Inferences from pooled analyses can raise questions of quality, accuracy, and generalizability.18 Additionally, vital registries from low- and middle-income countries are of questionable accuracy.19 The novel finding of the current study that birth weight is the most important variable for predicting the risk of neonatal mortality provides the strongest evidence based on a high-quality large prospective population-based database from resource-limited settings. To our knowledge, the current study is the largest study that uses a quantitative approach for risk prediction of stillbirth and neonatal mortality of data from low- and middle-income countries. This study identifies the contribution of birth weight as a continuous variable to arrive at its independent predictive ability for risk of neonatal mortality. In studies of association with increased risk of all-cause stillbirth or neonatal mortality, many variables have been identified, but these analyses were limited to descriptive analyses. Intrapartum stillbirth and early neonatal mortality have overlapping causes (in contrast to early stillbirth20), so we have included only intrapartum stillbirth in the present study. Adjusted analysis based on an earlier cohort of the GN-MNHR database indicated associations between maternal age younger than 20 or older than 35 years, lower maternal education, 0 parity or 3 parity greater than 3, and no prenatal care with all-cause stillbirth.21,22 Using other databases, other factors associated with all-cause stillbirth were poverty, parity of 5 or more, prematurity, low birth weight, and previous stillbirth.20 Factors associated with neonatal mortality also included maternal age, education, parity, multiple gestations, birth order, suspected maternal sepsis, antepartum hemorrhage, eclampsia, and obstructed labor.23,24,25 Also, in a large pooled analysis of cross-sectional data from 57 low- and middle-income countries (N = 464 728), antenatal care was associated with decrease in neonatal mortality in univariate analyses.26
In large modeling studies from high-income countries of extremely preterm15,27 or less than 2000 g neonates28 admitted to neonatal intensive care units, birth weight was among the most important common factors associated with hospital mortality. With the model derived from a high-income country, birth weight was also found to be a factors associated with hospital mortality when it was applied to a sample of 550 neonates from a single center in The Gambia.28 In another study from a developed country database of extreme preterm neonates, birth weight and gestational age were found to be equally associated with 2-year mortality.29 Gestational age was also among the top factors associated with mortality in the earlier developed country database studies.15,27 However, in the current study, birth weight was found to be a better predictor of neonatal mortality than gestational age. This could be related to estimation errors of gestational age based on the last menstrual period and limited availability of first-trimester ultrasonography-based dating confirmation.30 Small for gestational age (not specifically birth weight) has been associated with higher risk of perinatal and neonatal mortality and morbidity in data from high-income countries in bivariate analysis.31 Similar to the current study, neonatal sex, resuscitation, and antenatal corticosteroids have been factors associated with neonatal mortality in data from high-income countries.32,33,34
Limitations
This study used an established database with data that are population-based and prospectively collected with multiple quality assurance checks. The study was hypothesis-driven with only prespecified analyses performed. This large sample size study was adequately powered for prediction of the risk for intrapartum stillbirth and neonatal mortality in the represented resource-limited settings in low- and middle-income countries. Because of the large sample size, we were able to evaluate rigorously the machine learning predictive models. Predictive accuracy was assessed on test and validation data sets to check for consistency in predictive accuracy estimates, and the modeling building process was completed 10 times to quantify uncertainty in predicted model accuracy. Although generalizability of the results could be questioned, the results are likely to be pertinent to many communities in resource-limited regions similar to those of the study settings. There is a possibility of potential confounders like health status, availability of care, interventions, and health policy that were not captured. Nonetheless, as the study results are based on a large number of participants over 9 years, the potential impact of confounders on the study result should be low. Additionally, the use of the database did not include high-definition data, including stratification of individual risk factors as per illness severity, extensive laboratory test results, or details of treatments received and clinical response. As the risk score is intended to be useful for all health care professionals, adding more complexity to the scoring method by including variables that need a higher level of training and resources might have reduced the ease of application and usability of the score. However, incorporating additional variables could have improved the predictive accuracy, especially for the prenatal models. The gestational age variable would be prone to estimation errors depending on the mother’s accounting of the last menstrual period and the availability of more accurate assessments such as first-trimester ultrasonography. Intrapartum and antepartum stillbirth differentiation can be associated with identification errors, but training of the health care professionals and several quality checks were made to minimize this error.
Conclusions
In the current study, prediction of the risk of intrapartum stillbirth alone or in combination with neonatal mortality based on prenatal or predelivery data had predictive accuracy of AUC values of only 0.72 or less. The best risk prediction for neonatal death was only achieved after including delivery and early neonatal variables, which can be used to identify neonates at the highest risk for mortality who may need specialized care. Birth weight was by far the most important predictor for neonatal mortality, while the contribution of other variables was relatively minor. Mortality risk–based triage and referral could be tested as a strategy to reduce the burden of neonatal deaths in resource-limited settings. Given these findings, prenatal and predelivery data are not sufficient to develop strategies to identify those who are at a high risk of perinatal mortality and require advanced care at birth and referral. Birth weight could be prioritized in the identification of neonates at risk for dying. Predelivery estimation of birth weight could be evaluated as a strategy for predelivery triage and referral.
eMethods.
eTable 1. Subset Sample Sizes
eTable 2. Top Predictors by Scenario (Fresh Stillbirth Outcome)
eTable 3. Risk Score Model Coefficients and Calculation (Neonatal Mortality Outcome)
eTable 4. Mortality Probability Functions for the Total Risk Score (Neonatal Mortality Outcome)
eFigure 1. Mean (95%CI) for Validation AUC by Scenario for Outcomes of Fresh Stillbirth
eFigure 2. Probability of Mortality as a Function of Birthweight, Post-delivery/Day-2 Scenario
eReferences.
References
- 1.Levels and trends in child mortality. UN Inter-agency Group for Child Mortality Estimation. Published 2018. Accessed September 1, 2020. https://data.unicef.org/wp-content/uploads/2018/10/Child-Mortality-Report-2018.pdf
- 2.Liu L, Oza S, Hogan D, et al. Global, regional, and national causes of child mortality in 2000-13, with projections to inform post-2015 priorities: an updated systematic analysis. Lancet. 2015;385(9966):430-440. doi: 10.1016/S0140-6736(14)61698-6 [DOI] [PubMed] [Google Scholar]
- 3.Bhutta ZA, Black RE. Global maternal, newborn, and child health—so near and yet so far. N Engl J Med. 2013;369(23):2226-2235. doi: 10.1056/NEJMra1111853 [DOI] [PubMed] [Google Scholar]
- 4.Lawn JE, Cousens S, Zupan J; Lancet Neonatal Survival Steering Team . 4 million neonatal deaths: when? where? why? Lancet. 2005;365(9462):891-900. doi: 10.1016/S0140-6736(05)71048-5 [DOI] [PubMed] [Google Scholar]
- 5.Wall SN, Lee AC, Niermeyer S, et al. Neonatal resuscitation in low-resource settings: what, who, and how to overcome challenges to scale up? Int J Gynaecol Obstet. 2009;107(suppl 1):S47-S62, S63-S64. doi: 10.1016/j.ijgo.2009.07.013 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Peterson ED. Machine learning, predictive analytics, and clinical practice: can the past inform the present? JAMA. 2019. doi: 10.1001/jama.2019.17831 [DOI] [PubMed] [Google Scholar]
- 7.Maddox TM, Rumsfeld JS, Payne PRO. Questions for artificial intelligence in health care. JAMA. 2019;321(1):31-32. doi: 10.1001/jama.2018.18932 [DOI] [PubMed] [Google Scholar]
- 8.Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317-1318. doi: 10.1001/jama.2017.18391 [DOI] [PubMed] [Google Scholar]
- 9.Shah ND, Steyerberg EW, Kent DM. Big data and predictive analytics: recalibrating expectations. JAMA. 2018;320(1):27-28. doi: 10.1001/jama.2018.5602 [DOI] [PubMed] [Google Scholar]
- 10.Darcy AM, Louie AK, Roberts LW. Machine learning and the profession of medicine. JAMA. 2016;315(6):551-552. doi: 10.1001/jama.2015.18421 [DOI] [PubMed] [Google Scholar]
- 11.Hinton G. Deep learning: a technology with the potential to transform health care. JAMA. 2018;320(11):1101-1102. doi: 10.1001/jama.2018.11100 [DOI] [PubMed] [Google Scholar]
- 12.Goudar SS, Carlo WA, McClure EM, et al. The Maternal and Newborn Health Registry Study of the Global Network for Women’s and Children’s Health Research. Int J Gynaecol Obstet. 2012;118(3):190-193. doi: 10.1016/j.ijgo.2012.04.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bose CL, Bauserman M, Goldenberg RL, et al. The Global Network Maternal Newborn Health Registry: a multi-national, community-based registry of pregnancy outcomes. Reprod Health. 2015;12(suppl 2):S1. doi: 10.1186/1742-4755-12-S2-S1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Collins GS, Reitsma JB, Altman DG, Moons KG; TRIPOD Group . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Circulation. 2015;131(2):211-219. doi: 10.1161/CIRCULATIONAHA.114.014508 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Ambalavanan N, Carlo WA, Bobashev G, et al. ; National Institute of Child Health and Human Development Neonatal Research Network . Prediction of death for extremely low birth weight neonates. Pediatrics. 2005;116(6):1367-1373. doi: 10.1542/peds.2004-2099 [DOI] [PubMed] [Google Scholar]
- 16.Houweling TAJ, van Klaveren D, Das S, et al. A prediction model for neonatal mortality in low- and middle-income countries: an analysis of data from population surveillance sites in India, Nepal and Bangladesh. Int J Epidemiol. 2019;48(1):186-198. doi: 10.1093/ije/dyy194 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Katz J, Lee AC, Kozuki N, et al. ; CHERG Small-for-Gestational-Age-Preterm Birth Working Group . Mortality risk in preterm and small-for-gestational-age infants in low-income and middle-income countries: a pooled country analysis. Lancet. 2013;382(9890):417-425. doi: 10.1016/S0140-6736(13)60993-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ioannidis JP, Lau J. Pooling research results: benefits and limitations of meta-analysis. Jt Comm J Qual Improv. 1999;25(9):462-469. doi: 10.1016/S1070-3241(16)30460-6 [DOI] [PubMed] [Google Scholar]
- 19.Black RE, Cousens S, Johnson HL, et al. ; Child Health Epidemiology Reference Group of WHO and UNICEF . Global, regional, and national causes of child mortality in 2008: a systematic analysis. Lancet. 2010;375(9730):1969-1987. doi: 10.1016/S0140-6736(10)60549-1 [DOI] [PubMed] [Google Scholar]
- 20.Lawn JE, Blencowe H, Pattinson R, et al. ; Lancet’s Stillbirths Series steering committee . Stillbirths: where? when? why? how to make the data count? Lancet. 2011;377(9775):1448-1463. doi: 10.1016/S0140-6736(10)62187-3 [DOI] [PubMed] [Google Scholar]
- 21.Saleem S, Tikmani SS, McClure EM, et al. Trends and determinants of stillbirth in developing countries: results from the Global Network’s Population-Based Birth Registry. Reprod Health. 2018;15(suppl 1):100. doi: 10.1186/s12978-018-0526-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.McClure EM, Pasha O, Goudar SS, et al. ; Global Network Investigators . Epidemiology of stillbirth in low-middle income countries: a Global Network Study. Acta Obstet Gynecol Scand. 2011;90(12):1379-1385. doi: 10.1111/j.1600-0412.2011.01275.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Chomba E, Carlo WA, Goudar SS, et al. ; First Breath Study Group . Effects of essential newborn care training on fresh stillbirths and early neonatal deaths by maternal education. Neonatology. 2017;111(1):61-67. doi: 10.1159/000447421 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Kozuki N, Lee AC, Silveira MF, et al. ; Child Health Epidemiology Reference Group Small-for-Gestational-Age-Preterm Birth Working Group . The associations of parity and maternal age with small-for-gestational-age, preterm, and neonatal and infant mortality: a meta-analysis. BMC Public Health. 2013;13(suppl 3):S2. doi: 10.1186/1471-2458-13-S3-S2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Bellizzi S, Sobel H, Betran AP, Temmerman M. Early neonatal mortality in twin pregnancy: Findings from 60 low- and middle-income countries. J Glob Health. 2018;8(1):010404. doi: 10.7189/jogh.08.010404 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Doku DT, Neupane S. Survival analysis of the association between antenatal care attendance and neonatal mortality in 57 low- and middle-income countries. Int J Epidemiol. 2017;46(5):1668-1677. doi: 10.1093/ije/dyx125 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Tyson JE, Parikh NA, Langer J, Green C, Higgins RD; National Institute of Child Health and Human Development Neonatal Research Network . Intensive care for extreme prematurity—moving beyond gestational age. N Engl J Med. 2008;358(16):1672-1681. doi: 10.1056/NEJMoa073059 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Medvedev MM, Brotherton H, Gai A, et al. Development and validation of a simplified score to predict neonatal mortality risk among neonates weighing 2000 g or less (NMR-2000): an analysis using data from the UK and The Gambia. Lancet Child Adolesc Health. 2020;4(4):299-311. doi: 10.1016/S2352-4642(20)30021-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Salas AA, Carlo WA, Ambalavanan N, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Gestational age and birthweight for risk assessment of neurodevelopmental impairment or death in extremely preterm infants. Arch Dis Child Fetal Neonatal Ed. 2016;101(6):F494-F501. doi: 10.1136/archdischild-2015-309670 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Goldenberg RL, Nathan RO, Swanson D, et al. Routine antenatal ultrasound in low- and middle-income countries: first look: a cluster randomised trial. BJOG. 2018;125(12):1591-1599. doi: 10.1111/1471-0528.15287 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Mendez-Figueroa H, Truong VT, Pedroza C, Khan AM, Chauhan SP. Small-for-gestational-age infants among uncomplicated pregnancies at term: a secondary analysis of 9 Maternal-Fetal Medicine Units Network studies. Am J Obstet Gynecol. 2016;215(5):628.e1-628.e7. doi: 10.1016/j.ajog.2016.06.043 [DOI] [PubMed] [Google Scholar]
- 32.Vu HD, Dickinson C, Kandasamy Y. Sex difference in mortality for premature and low birth weight neonates: a systematic review. Am J Perinatol. 2018;35(8):707-715. doi: 10.1055/s-0037-1608876 [DOI] [PubMed] [Google Scholar]
- 33.Bajaj M, Natarajan G, Shankaran S, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Delivery room resuscitation and short-term outcomes in moderately preterm infants. J Pediatr. 2018;195:33-38.e2. doi: 10.1016/j.jpeds.2017.11.039 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Roberts D, Brown J, Medley N, Dalziel SR. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. Cochrane Database Syst Rev. 2017;3:CD004454. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eMethods.
eTable 1. Subset Sample Sizes
eTable 2. Top Predictors by Scenario (Fresh Stillbirth Outcome)
eTable 3. Risk Score Model Coefficients and Calculation (Neonatal Mortality Outcome)
eTable 4. Mortality Probability Functions for the Total Risk Score (Neonatal Mortality Outcome)
eFigure 1. Mean (95%CI) for Validation AUC by Scenario for Outcomes of Fresh Stillbirth
eFigure 2. Probability of Mortality as a Function of Birthweight, Post-delivery/Day-2 Scenario
eReferences.