Risk factors for pediatric acute chest syndrome using machine learning techniques

Key Points

• •This is the first machine learning model, to our knowledge, to predict risk factors of ACS in children with sickle cell disease.
• •Abnormal vital signs, laboratory values, and use of pain medications were important variables in the model to predict ACS.

Visual Abstract

Abstract

Acute chest syndrome (ACS) causes significant morbidity and mortality in both adult and pediatric patients with sickle cell disease. Despite known risk factors for developing ACS, this complication is often unexpected. We hypothesized that a machine learning model could predict progression to ACS in children. A random forest model was used using the Scikit-Learn Python library. Prediction performance was assessed using the C-statistic, sensitivity, and specificity. The features included are demographics, vital signs, laboratory values, medication orders, and additional comorbidities. The positive cohort developed ACS on diagnosis or during admission, and the negative cohort did not. A total of 3223 encounters were included in the model for training and testing. Up to 65% of the positive cohort developed ACS 24 hours after admission. The model had a sensitivity of 65.1% and specificity of 77.4% at the cutoff point of 0.6. For diagnosing ACS, the negative predictive value was 89.8%, and the positive predictive value was 42.3%. For the correct positive cases, the model predicted ACS a median of 18 hours before the diagnosis was determined in the electronic health record. The features that were important included pulse, respiratory rate, presence of hypoxia, temperature, neutrophil count, and use of pain medications. Despite the model’s limitations, it is, to our knowledge, the first model using machine learning techniques to predict ACS in children. Detecting patients who are at higher risk of ACS could affect the approach to acute care management.

Introduction

Acute chest syndrome (ACS) is a complication of sickle cell disease (SCD) in both adult and pediatric patients. ACS has a high mortality rate^1,2 and potential long-term pulmonary sequelae.^3,4 In addition, ACS prolongs hospital stays,⁵ and severe cases require intensive care unit management and mechanical ventilation.^2,6 Progression to ACS can occur during a vaso-occlusive event¹ due to fat embolism from necrotic bone marrow, hypoventilation from splinting, or sedating side effects from opiate management.⁷ Although there are known risk factors for developing ACS in children (asthma,⁵ after anesthesia,8, 9, 10 and obstructive sleep apnea⁵), the development of ACS during a hospitalization period is often unpredictable. Clinical predictors of pediatric ACS in other studies^11,12 have included fever, hypoxia, anemia, and leukocytosis, but these studies could not concurrently assess whether receipt of a wide array of medications, laboratory values, or vital signs is associated with the development of ACS during a hospitalization for vaso-occlusive event. Identifying new risk factors could lead to the development of novel strategies for the prevention of ACS in children.

Machine learning (ML) is a subset of artificial intelligence in which statistical models and algorithms enable computers to learn and make predictions based on data.¹³ ML techniques, such as random forest models, analyze data and create inferences based on the information provided. These models can integrate an extensive list of variables compared with other statistical methods and, hence, potentially could lead to the discovery of novel risk factors.¹⁴ ML is able to analyze an extensive number of variables that could investigate patterns in data that are challenging to conduct with traditional variable selection methods, such as logistic regression.¹⁵ Random forest models have outperformed other statistical models,¹⁶ particularly when there is a complex and nonlinear data set and can also be flexible in handling missing data elements. ML is increasingly being explored in the field of medicine, but only a few studies have used it in understanding risk factors for managing patients with SCD.17, 18, 19, 20, 21, 22

We developed a random forest model to predict ACS in a retrospective cohort of pediatric patients with SCD. The objective of the model was to identify ACS before the diagnosis of ACS was documented in the electronic health record (EHR). In addition, we identified other risk factors that could contribute to the development of ACS that have not been previously reported.

Methods

Study design and setting

We report a retrospective cohort study from a single, urban, tertiary-care institution. Data were extracted from the Epic EHR into the Observational Medical Outcomes Partnership Common Data Model, Clarity software, and the Epic system for all admissions of pediatric patients with SCD aged 0 to 21 years during the period from 1 June 2016 to 31 December 2022. The study protocol was reviewed and approved by Montefiore’s Institutional Review Board. Informed consent was waived.

Study population

The model was trained on data from both positive and negative cohorts. Encounters in the positive cohort had ACS and met the following criteria: (1) inpatient admission encounter was associated with an SCD-related International Classification of Diseases-10 (ICD-10) code or with an ICD-10 code for ACS; and (2) a chest X-ray and >2 doses of azithromycin were ordered during that encounter (typical diagnostic and treatment choices for patients with ACS). Encounters in the negative cohort without ACS met the following criteria: (1) inpatient admission encounter was associated with an SCD-related ICD-10 code; (2) there was not an ICD-10 code for ACS or pneumonia associated with the encounter; and (3) there were no combined orders of a chest X-ray and at least 2 doses of azithromycin. All positive cohort encounters were verified by chart review, and a random sample of 20% of negative cohort encounters were verified by chart review. Patients were excluded from either cohort if they had diagnosis codes of sickle cell trait (D57.3) or a history of bone marrow transplant (BMT; Z94.81) in that encounter. A list of patients who had a known BMT was cross-referenced, and all encounters before their BMT date were included in the model, whereas all encounters after the BMT date were excluded.

Data collection

Selected variables that are known to affect the clinical care of patients with SCD were included in the model. These included baseline demographics, vital signs, laboratory data, comorbidities, and procedure codes. A data dictionary was created for vital signs, laboratory values, and ICD-10 and Current Procedural Terminology (CPT) codes for comorbidities and exposures (supplemental Tables 1-4). Each data element in the dictionary was mapped to an existing field in the EHR, and abstraction and validation procedures were developed as described in the description of variable inputs.

Description of variable inputs

Vital signs included were heart rate, oxygen saturation, respiratory rate, and systolic and diastolic blood pressures. Normal vital sign values were determined based on age (supplemental Table 1).²³ Systolic and diastolic blood pressure parameters were adjusted based on the study of Pegelow et al²⁴ (supplemental Table 1). The Pasero sedation score²⁵ was included to determine arousability, with abnormal values <0 and >4. Body mass index parameters were determined using the Centers for Disease Control (CDC) growth charts (www.cdc.gov/growthcharts). Parameters for male and female sex were analyzed separately. Lower parameters (<3% for females and <5 % for males) or higher parameters (>85%) were considered to be abnormal between the ages of 2 and 18 years. Weight-for-age percentiles were used for children aged from birth to 24 months for both male and female growth charts using CDC guidance as above. The minimum normal value was below the 2nd percentile, and the maximum normal value was above the 98th percentile. The determination of those age ranges was based on the CDC curves (supplemental Table 2).

Normal ranges for laboratory parameters were based on the institutional laboratory value ranges, except those that differ by age: hemoglobin, hematocrit, and mean corpuscular volume.26, 27, 28 Laboratory variables included hemolysis laboratory tests, viral panels, bacterial cultures (blood and urine, etc), and inflammatory markers. The complete list of laboratory variables is in supplemental Table 3.

CPT codes were included for potential comorbid conditions associated with ACS, including asthma,²⁹ wheezing, stridor, obstructive sleep apnea,⁵ food allergies, seasonal allergies, eczema, and nocturnal hypoxia. Other comorbidities that could lead to surgical intervention requiring anesthesia (cholangitis, liver disease, and cholecystitis) were also included if they were coded for that encounter. Medications inputted in the model were disease-modifying therapies available at that time (hydroxyurea, glutamine, and voxeletor), opioids (morphine and oxycodone, etc), nonopioid pain medications (pregabalin and nortriptyline, etc), nonsteroidal medications (ketorolac and ibuprofen, etc), and antibiotics. The complete list of included ICD-10 codes and medications in the model is in supplemental Tables 4 and 6.

Model building

The random forest model was developed using the Scikit-Learn Python library.³⁰ The retrospective positive and negative cohorts were input into the model, and all the variables described above were included in the model. The output of ACS was a continuous dependent variable, and a continuous likelihood estimation score was developed from 0 to 1. The time stamp of ACS diagnosis was when the first dose of azithromycin was ordered in the EHR. Eighty percent of the data were used for training and 20% for testing. We experimented with different depths of trees ranging from 1 to 24. We chose the minimum tree depth that did not show further improvement in test data performance. The inputs were calculated as the minimum and maximum values of the 260 variables as described above: vital signs, respiratory support, demographics, laboratory data, comorbidities, and medications. Vital signs and laboratory values were included in the model every 6 hours from the start of the hospital encounter, and only the latest time point was included in the model. Minimum and maximum values were calculated within variables derived within predefined 6-hour windows (00:00-06:00 and 06:00-12:00, etc) to ensure that only data available up to each prediction time point were included and to preserve temporal structure. The forward fill method was used to fill the null values of all the features in an encounter if it was a binary variable. Missing values in the vital signs were imputed using a k-nearest neighbors algorithm (k = 3), leveraging similarity across patients’ physiological profiles. Variables with >25% missingness per encounter were excluded. We applied a last observation carried forward (LOCF) strategy within each patient encounter, updating values every 6 hours. Prediction for ACS was calculated every 6 hours from the start of the encounter. If a patient developed ACS, laboratory values, vital signs, newly inputted CPT codes, or ICD-10 codes after the ACS diagnosis were not included in the analysis. Every encounter was included as an independent data point. To calculate the important features of the model, we used the Gini impurity method as the concept for the variables of importance of the mean decrease in impurity importance. Prediction performance was assessed using the C-statistic, sensitivity, and specificity. We performed group-based crossvalidation, ensuring that all observations from the same encounter were kept within a single fold. Calibration was assessed using calibration plots and the Brier score. We accounted for class imbalance by applying class weights during model training, using the class_weight = "balanced" option. This approach assigns higher weights to the minority class inversely proportional to class frequencies, ensuring that misclassification of minority class observations is penalized more heavily. This weighting scheme was incorporated during model fitting. We also reported the most important predictors as the features of importance from our best model. Feature importance for the random forest model was calculated using impurity-based importance (mean decrease in Gini), which quantifies the contribution of each feature to reducing classification error across all trees. To mitigate the effects of multicollinearity on importance estimates, highly correlated features (eg, Pearson r > 0.8) were identified and removed before model development, retaining the variable with the stronger association with the outcome. Calculation of scores and analyses were performed using Python version 3.9.2 software. The libraries used in Python were Numpy, Pandas, and Scikit-Learn.

Statistical analysis of variable inputs

For nonnormally distributed continuous variables, the study used the median and interquartile range (IQR) for description. Categorical variables were described using numbers and percentages.

Results

A total of 2694 encounters were used to train the model (2189 in the negative cohort and 505 in the positive cohort), and 539 encounters were used to test the model (430 encounters in the negative cohort and 109 encounters in the positive cohort). There were 720 unique patients in the training cohort and 320 unique patients in the testing cohort. The uncalibrated model overestimated risk. Platt scaling using group-based out-of-fold predictions improved calibration, reducing the Brier score from 0.128 to 0.069 and aligning predicted probabilities with observed outcomes, as shown in supplemental Figure 1 and supplemental Table 5. The demographics of the training and testing cohorts are in Table 1. The median length of stay was 2.86 days (IQR, 1.04-4.6) in the training cohort and 2.9 days (IQR, 1.97-4.78) in the testing cohort. The variable inputs included in the model were 180:12 demographic/physical examination findings (Table 1), 46 laboratory tests (Table 2), 31 comorbidities (supplemental Table 6), and 91 medications (supplemental Table 4). Missingness rates for selected features included in the model are in supplemental Table 7.

Table 1.

Results of demographic and vital sign inputs in the full cohort (3233 encounters) for the random forest ML model

	Training	Validation
	N = 2694	N = 539
Age, median (IQR), y	11.6 ± 6.6	11.7 ± 6.5
Sex, n (%)
Female	1414 (52.5)	260 (48.2)
Male	1280 (47.5)	279 (51.8)
Body mass index, mean (SD), kg/m2	18.1 (15.9-21.2)	17.8 (15.7-21.2)
Height, mean (SD), m	1.5 (1.2-1.6)	1.5 (1.2-1.7)
Weight, median (IQR), kg	40.1 (18.9-56.7)	40.0 (20.9-56.2)
Blood pressure, mean (SD), mm Hg
Systolic	107.7 (102.3-113.7)	107.6 (103.2-113.4)
Diastolic	60.7 (57.0-64.6)	61.0 (57.2-65.0)
Oxygen saturation, %	98.8 (97.7-99.4)	98.8 (97.6-99.4)
Pulse, mean (SD), bpm	93.3 (82.1-113.5)	91.7 (81.3-110.1)
Respiratory rate, mean (SD), breaths per min	20.9 (19.5-25.1)	20.8 (19.6-24.3)
Temperature, mean (SD), F	98.4 (98.2-98.9)	98.4 (98.1-98.9)

Data are presented as mean ± SD or median (IQR) for continuous measures and n (%) for categorical measures.

bpm, beats per minute; F, Fahrenheit; mm Hg, mm of mercury; SD, standard deviation.

Table 2.

Mean laboratory variable value inputs in the full cohort (3233 encounters) for the random forest ML model

	Development	Validation
	n = 2694	n = 539
Laboratory value
Blood urea nitrogen, mg/dL	8.1 ± 3.6	7.9 ± 2.9
Albumin	4.3 ± 0.4	4.3 ± 0.4
Alanine aminotransferase, U/L	27.8 ± 36.8	26.6 ± 37.7
Aspartate aminotransferase, U/L	50.6 ± 34.9	49.3 ± 29.7
Band, %	2.5 ± 4.7	1.8 ± 2.6
Amylase, U/L	74.1 ± 78.5	83.7 ± 157.0
Anion gap, meq/L	13.8 ± 3.5	13.8 ± 3.4
Bicarbonate, meq/L	24.0 ± 2.5	23.6 ± 2.3
Chloride, meq/L	102.6 ± 3.2	102.7 ± 3.1
Direct bilirubin, mg/dL	0.7 ± 2.0	0.6 ± 1.4
Eosinophil, 103/μL	0.3 ± 0.4	0.2 ± 0.3
Erythrocyte sedimentation rate, mm/h	25.1 ± 24.5	25.3 ± 28.2
Ferritin, ng/mL	3339.2 ± 7224.1	2586.9 ± 3101.9
Fibrinogen, mg/dL	335.3 ± 130.6	327.0 ± 83.0
Glucose, mg/dL	96.5 ± 22.3	96.0 ± 18.2
Hematocrit, %	25.1 ± 4.5	25.6 ± 4.9
Hemoglobin, g/dL	8.7 ± 1.6	8.9 ± 1.7
Immature granulocyte, 103/μL	0.2 ± 0.3	0.2 ± 0.4
Lipase, U/L	43.0 ± 152.2	76.0 ± 227.9
Lymphocytes, 103/μL	3.8 ± 2.3	3.9 ± 2.1
Magnesium, mg/dL	2.0 ± 0.2	2.0 ± 0.2
Mean corpuscular volume, fL	85.0 ± 11.1	83.2 ± 11.5
Monocytes, 103/μL	8.5 ± 3.8	8.2 ± 3.6
Neutrophils, 103/μL	8.0 ± 4.9	8.0 ± 4.6
Platelets, 103/μL	346.0 ± 146.4	339.6 ± 147.3
Potassium, M/vol	4.4 ± 0.5	4.4 ± 0.5
Prothrombin time, s	19.7 ± 7.6	20.0 ± 5.9
Reticulocyte count, 103/μL	278.8 ± 148.1	270.0 ± 143.6
Reticulocyte count, %	10.4 ± 6.8	10.0 ± 6.6
Sodium, M/vol	138.0 ± 2.8	138.2 ± 3.0
Total bilirubin, mg/dL	3.2 ± 4.0	3.0 ± 3.4
Total protein, g/dL	7.2 ± 0.7	7.2 ± 0.6
Urine protein, mg/dL	1.0 ± 0.0	1.0 ± 0.0
Urine WBC, per HPF	6.3 ± 11.6	5.1 ± 7.6
WBC, 103/μL	13.5 ± 6.1	13.5 ± 5.7

Data are presented as mean ± standard deviation.

HPF, high powered field; meq, milliequivalent; U, units; vol, volume; WBC, white blood cell count.

Sixty-five percent of the positive cohort developed ACS within 24 hours and 35 % between 12 and 24 hours after the start of the encounter. The model predicted ACS a median of 18 hours before the diagnosis was determined in the EHR. In the test cohort, the model had a sensitivity of 65.1% and a specificity of 77.4% at the cutoff point of 0.6 (based on a maximum F score) for diagnosing ACS in the model (Figure 1). The area under the curve was 82%. The model’s negative predictive value was 89.8%, and the positive predictive value (PPV) was 42.3% for diagnosing ACS in the model. The maximum depth of the tree was 10. The important features for model prediction included the presence of hypoxia, low and elevated temperature, pulse below age-based normal ranges, abnormal neutrophil count (low), patient age, medication exposure (penicillin, diphenhydramine, methadone, hydromorphone, oxycodone, acetaminophen, and albuterol), respiratory rate above and below age-based normal ranges, elevated blood urea nitrogen, low glucose, elevated total protein, and history of chronic pain (Figure 2).

Figure 1.

Performance metrics for an ML model predicting ACS before diagnosis in the EHR. Receiver operating characteristic (ROC) curves and the corresponding area under the curve are shown. The area under the ROC curve was 82%, using a random forest model with the Scikit-Learn Python library.

Figure 2.

Important variables for the prediction of ACS in the random forest model. Absolute importance is then scaled to give relative importance, with a maximum importance of 100. bpm, beats per minute; F, Fahrenheit.

Discussion

This is, to our knowledge, the first study to use ML methods to analyze predictors of ACS in pediatric patients with SCD and to develop a model to predict ACS. Using a random forest model, we can identify patients who will develop ACS 18 hours before the initiation of ACS treatment in a retrospective cohort. The majority of the variables of importance in the model have been described before as risk factors of impending ACS^5,11,12 and support our model’s validity.

The model has an excellent negative predictive value of 89.8%, with a sensitivity of 65.1 % and a specificity of 77.4%. At this cutoff, patients with predicted risk ≥0.6 were classified as high risk for ACS, which aligns with clinical prioritization of early identification of at-risk patients while limiting false positives. The PPV of the model is only 42.3%, but this was comparable to an adult emergency department study³¹ assessing risk factors for developing ACS using a logistic regression model.³¹ Their logistic regression model³¹ demonstrated that reticulocyte count, hemoglobin, leukocyte count, and spine and pelvis pain were independently associated with ACS. This predictive score has been recently validated in an international prospective observational study.³² Our study also showed that leukocytes are important components of predicting ACS in our pediatric cohort. Additional study is necessary to investigate the unique components that differentiate the prediction of pediatric and adult ACS.

In random forest models, importance reflects a variable’s significance for prediction in the model, with a maximum importance of 100%¹⁸ and reflects complex relations of the model. Each feature has a normalized importance score that reflects how much the feature contributes to improving the accuracy of the model. Similar to other studies, fever¹¹ and hypoxia^11,12 are predictors of developing ACS in this model. Abnormal vital signs are the most important features in predicting ACS (abnormal heart rate, respiratory rate, and oxygen saturation) in our model, reflecting that triage vital signs could be used in the future for predicting high-risk patients. Markers of hemolysis (bilirubin¹²) have been associated with impending ACS; but in our model, a low neutrophil count is an important feature. One could postulate that a low neutrophil count is related to bone marrow suppression from viral illnesses, which frequently are an underlying trigger for developing ACS in pediatrics,³³ but that needs to be further investigated. Albuterol use was an important predictor of developing ACS in our model, and this is consistent with the significant literature on asthma being a risk factor for ACS.^5,34,35 In addition, use of pain medications (acetaminophen, hydromorphone, and/or oxycodone) before developing ACS is a variable of importance in the model, which aligns with the underlying pathophysiology of ACS precipitated by a vaso occlusive episode (VOE).⁵

The design of the data set and determination of the cohort variables are worth noting because there are many unique factors to consider, which will be important for future model building. Defining ACS in this context was challenging. ICD-10 codes are often inaccurate,^36,37 and because the development of ACS can be an evolution during hospital admission, providers may not add additional codes after admission. Azithromycin is used in treating other infections besides ACS, so it was necessary to ensure the encounters had multiple days of the antibiotic ordered to avoid false positives. Because individual patients with SCD have varying degrees of hemolysis, we decided to use normal value ranges based on age or the standard normal ranges per test. Hence, the model is unable to define more detailed or refined ranges of laboratory abnormalities that could be helpful in detecting development of ACS and will need future investigation.

There are several limitations. This is a single-institution retrospective study that only included pediatric patients with ACS. There could be bias in our model due to the unique social determinants of health in Bronx, where our study was conducted. This model may not be generalizable in predicting ACS in pediatric patients in other hospital systems and needs to be studied further.

For missing data, we applied an LOCF strategy within each patient encounter, updating values every 6 hours. This approach reflects real-world clinical practice, in which laboratory and physiological parameters are assumed to remain stable until a new measurement is obtained, particularly for variables that are not measured every 6 hours (eg, laboratory values typically obtained once daily). Importantly, our method does not assign “normal” values to all missing data; instead, it preserves the most recent observed value, thereby maintaining patient-specific trajectories. This reduces the risk of artificially normalizing abnormal values, which could bias predictions toward lower risk. We acknowledge that LOCF may introduce bias if a patient’s condition changes rapidly between measurements. However, many laboratory parameters in inpatient settings are measured at relatively low frequency by design, and LOCF is commonly used in longitudinal clinical data sets. Binary variables were forward filled, which may have introduced misclassification by propagating previous positive values beyond their true duration. However, this approach was chosen to reflect clinical practice, in which measurements, particularly laboratory values, are obtained intermittently rather than at fixed intervals. In this context, the absence of a new measurement may itself carry information, often indicating a lack of clinical urgency or no perceived need for reassessment.

In addition, we defined ACS encounters that had both an order for a chest X-ray and 2 orders of azithromycin at different time points. Although treatment of ACS typically includes 2 antibiotics to treat both Streptococcus pneumoniae and atypical organisms, we chose to only use azithromycin in the algorithm due to variation in the choice of antibiotics for S pneumoniae. Although we chart verified each ACS encounter included in the model, there is a potential for misclassification risk when expanding to additional hospital systems that will need to be considered. Adult patients with SCD have higher rates of morbidity and mortality³⁸ due to ACS, and studying this cohort would provide important insight, and the approach using azithromycin ordering in adults could be less accurate. Although we did include many clinical features, we did not include additional information such as echocardiogram reports, previous pulmonary function test results, notes including physical examination findings, or patient symptoms. Other limitations were that we did not input doses of medications (ie, cumulative opioid dose), total amount of IV fluids, or fraction of inspired oxygen into the model. There are a few laboratory values for which data could not be collected (calcium, creatinine, and partial thromboplastin time) or were not interpretable (lactate dehydrogenase) and should be included in future models. Although we used the sickle cell genotype ICD-10 codes to identify the patients, we did not categorize it in our analysis, a limitation we will correct in future models. Voxeletor was removed from the market after this study was complete, and future models should consider removing it from the list of variables. Using other ML techniques in the future, such as deep learning, large language models, and artificial neural networks,³⁹ could improve the accuracy of future models by automatically extracting and analyzing information from raw data, such as providers’ notes in the emergency department. In addition, developing a random forest model to predict ACS in a prospective cohort will be an important next step.

This study provides a key initial step to conceptualize how to design and build ML models to predict ACS in pediatric patients with SCD. To improve the robustness and applicability of this research, future models will need to be trained on significantly more data with diverse data sets in both pediatric and adult cohorts. Future models will need a higher PPV to enable integration into clinical workflows effectively; otherwise, providers may be overtreating patients who do not have ACS. The data set development took multiple iterations and a significant amount of time for validation. Close collaboration between data scientists and hematologists enabled this project to move forward, but more research is needed to effectively use big data. Given that ACS has significant morbidity and mortality, it is imperative that we focus on identifying patients who could progress to this complication.

Conflict-of-interest disclosure: K.M. is a consultant for X4 Pharmaceuticals and was in a focus group for Novo Nordisk. The remaining authors declare no competing financial interests.