Construction and evaluation of prognostic models of ECMO in elderly patients with cardiogenic shock based on BP neural network, random forest, and decision tree

Abstract

Objective: To analyze the predictive effect of a back propagation (BP) neural network, random forest (RF) and decision tree model on the prognosis of elderly patients with cardiogenic shock after extracorporeal membrane oxygenation (ECMO). Methods: This is a retrospective analysis of the clinical data of elderly patients with cardiogenic shock (258 cases) who underwent ECMO in People’s Hospital of Guangxi Zhuang Autonomous Region from January 2016 to January 2022. All patients were followed up for 6 months after ECMO treatment. The prognosis was evaluated, and the prognostic factors were analyzed. BP neural network, RF and decision tree were used to establish predictive models, and the predictive performance of the models was evaluated. Results: Among the 258 elderly patients with cardiogenic shock, 52 (20.16%) died 6 months after the ECMO treatment. Based on BP neural network, RF, and decision tree, predictive models for the prognosis and death of elderly patients with cardiogenic shock were constructed. A test set was used to predict the performance of the three models. The results showed that the predictive performances of the three models were all more than 80.00%. The accuracy, sensitivity, and specificity of the RF model were 0.987, 1.000, and 0.929 respectively, which were higher than those of the decision tree model. The area under the receiver operating characteristic curve (AUC) of the RF model was 1.000, which was higher than 0.916 for the decision tree model. DeLong test showed that there was a significant difference in the AUC of the RF model compared to the decision tree test set (D=-2.063, P=0.042 < 0.05). Conclusion: The predictive performance is good in all the three models, which have a high application value for prognosis of ECMO in elderly patients with cardiogenic shock. In clinical practice, predictive models should be selected according to the actual situation, so clinicians and patients can make decisions.

Introduction

Cardiogenic shock is usually caused by the failure of cardiac output function. It tends to occur in the elderly population and should be treated as soon as possible after diagnosis [1]. Extracorporeal membrane oxygenation (ECMO) is an effective way of providing extracorporeal life support for refractory heart failure, especially for patients with cardiac shock. It can provide patients with 4-6 L circulating blood oxygenation, and partially or completely replace cardiac function, promoting fast recovery in patients with major visceral perfusion, and maintaining stable hemodynamics [2,3]. However, the mortality and prognosis are still not ideal in some elderly patients after ECMO treatment. Therefore, for elderly patients with cardiogenic shock treated with ECMO, the collection of prognostic clinical data is of great significance for assessing the severity of the disease, guiding clinical treatment decisions, and predicting the prognosis. Regarding electronic health records (EHRs), a large database covering emergency, hospitalization, and even community medical information has been commonly established [4]. Through EHRs, medical staff can quickly obtain a large amount of clinical data, which makes it possible to predict the prognosis of ECMO in patients with cardiogenic shock. The related risk factors and probable prognosis can be evaluated by traditional logistic regression and Cox proportional hazard model [5,6]. However, the above methods all face some challenges. First, the predictive accuracy is low, and there is a risk of false positives. In the clinical environment, accuracy is the most important indicator to evaluate the quality of the algorithm. A high incidence of false positives will lead to a waste of medical resources and even overtreatment. Secondly, most of the above algorithms can predict only at a certain time, rather than predict in real time according to the changes of the disease, which greatly limits the clinical application of the predictive model. Therefore, how to mine valuable information from a database, accurately predict the prognosis of patients and formulate individualized treatment plans is still a challenge in this field.

In recent years, data mining technology has been well promoted in the medical field, which is related to the high guiding value of data mining technology in disease diagnosis, risk assessment, and prognosis judgment [7]. For instance, random Forest (RF) algorithm has high classification accuracy and fault tolerance; back propagation (BP) neural network method can automatically identify signal characteristics based on input signal conditions; decision tree is suitable for processing data from a small sample [8-10]. They can quickly and effectively explore the factors that have a significant impact on the occurrence of diseases, and scientifically assess the risk factors. At present, there are few reports on the prognosis of ECMO treatment in elderly patients with cardiogenic shock based on clinical data and machine learning algorithms. This study aims to use a BP neural network, RF, and decision tree to construct prognostic models for evaluating ECMO treatment in elderly patients with cardiogenic shock, and compare the predictive efficacy of the three models.

Materials and methods

Basic Information

This is a retrospective analysis of the clinical data of elderly patients with cardiogenic shock (258 cases) who underwent ECMO in People’s Hospital of Guangxi Zhuang Autonomous Region from January 2016 to January 2022. Inclusion criteria: 1) patients who met one of the diagnostic criteria for cardiogenic shock [11]: high-dose positive inotropic drugs [epinephrine > 0.2 μg/(kg·min), dopamine or dobutamine > 20 μg/(kg·min)] were unable to improve the cardiac function and cardiac index < 2 L/(min·m²); mean arterial pressure < 60 mmHg (1 mmHg = 0.133 kPa), and urine volume < 0.5 mL/(k·min); 2) patients who were having the first onset at admission; 3) patients with good cognitive ability; 4) patients who were treated with ECMO, and the treatment assistance time was less than 48 h; 5) patients with well-recorded clinical data. Exclusion criteria: 1) patients with cardiogenic shock caused by low blood volume; 2) patients with respiratory insufficiency and requiring combined ECMO treatment; 3) patients with severe organ dysfunction (liver, kidney); 4) patients with infection or immune diseases; 5) patients with malignant tumor; 6) patients with irreversible brain injury or irreversible myocardial injury; 7) patients with duration of cardiopulmonary resuscitation ≥ 30 min; 8) patients who were in need of thoracotomy surgery; 9) patients who failed to complete the follow-up. This study has been approved by the Ethics Committee of People’s Hospital of Guangxi Zhuang Autonomous Region.

Methods

Clinical data statistics

The clinical sample data and the laboratory indicators of the enrolled patients were analyzed. The clinical data included sex, age, body mass index, hypertension, hyperlipidemia, diabetes, smoking history, use of intra-aortic balloon pump, history of beta-blocker use, history of cardiac arrest, invasive mechanical ventilation, central venous pressure, central venous pressure oxygen saturation, pre-ECMO heart rate, and Acute Physiology and Chronic Health Rating Scale (APACHE II) score. The laboratory indicators before ECMO treatment included lymphocyte count, platelet count, serum creatinine, urea nitrogen, troponin I, and lactic acid.

Evaluation of prognosis after ECMO treatment

All the enrolled patients received ECMO adjuvant therapy. After the treatment, the patients were followed up for 6 months. For prognosis analysis, patients who died were included in the death group, and the survivors were included in the survival group.

Model construction

BP neural network model: The 9 variables obtained through independent variable screening were used as the input variables in the BP neural network model, and the prognosis of ECMO treatment in elderly patients with cardiogenic shock (survival = 2, death = 1) was used as the “dependent variable” to rank and establish the predictive model.

RF model: Random forest in R 4.1.2 was used to construct a RF model. In the process of establishing a random Forest model, the main parameters involved were the number of decision trees in the random Forest (ntree) and the number of variables setting the branches of the decision tree (mtry). Parameter mtry is the value obtained by taking the square root of the number of variables in the data set. Ntree specifies the number of decision trees in the RF. According to the relationship between the error rate and the number of the model, an appropriate number of decision trees was chosen to build the model.

Decision tree model: The decision tree model included the 9 variables screened by univariate analysis as ‘independent variables’ and the prognosis (survival, death) of the patients as the ‘dependent variable’. Cross-validation was used to prune and fold, and a decision tree analysis model was generated.

The diagnostic effects of the three models were evaluated and compared by calculating the accuracy, sensitivity, specificity, recall, F1 value, and area under the receiver operating characteristic (ROC) curve (AUC).

Data processing methods

SPSS 25.0 and R 4.1.2 were used for data analysis in this study. Continuous variables (consistent with normal distribution) were described as mean ± standard deviation (mean ± sd) and compared by t-test. Categorical variables were described in the form of N (%) and compared by χ² test. Univariate analysis was used to screen out the factors with significant differences, and these factors were respectively incorporated into the BP neural network model, RF model, and decision tree model. P < 0.05 was used to indicate statistical significance. RF, decision tree, and BP neural network models were set according to machine learning (training set:test set = 7:3). The model was trained with the training set to determine the model parameters and verified by the test set to evaluate the predictive value.

Results

Comparison of clinical characteristics between the two groups

Among the 258 elderly patients with cardiogenic shock, 52 (20.16%) died 6 months after the ECMO treatment. The age, central venous pressure oxygen saturation, APACHE II score, lymphocyte count, platelet count, serum creatinine, urea nitrogen, troponin I, and lactic acid were significantly different between the death group and the survival group (P < 0.05). No significant difference was observed in other indices between the two groups (P > 0.05). See Table 1.

Table 1.

Comparison of clinical characteristics between the two groups

Item	Death group (n=52)	Survival group (n=206)	t/χ2	P
Sex (Male/Female)	29/23	131/75	1.079	0.299
(age)	74.10±3.59	65.91±4.09	13.207	< 0.001
Body mass index (Kg/m2)	21.59±2.16	21.49±2.10	0.318	0.750
Hypertension	15 (28.85)	35 (16.99)	3.735	0.053
diabetes	17 (15.94)	39 (26.72)	3.148	0.076
Hyperlipidemia	9 (17.31)	26 (12.21)	0.778	0.378
Smoking history	16 (30.77)	41 (19.90)	2.848	0.091
Use of intra-aortic balloon pump	19 (36.54)	58 (28.16)	1.394	0.238
History of beta-blocker use	21 (40.38)	68 (33.01)	0.999	0.317
History of cardiac arrest	9 (17.31)	31 (15.05)	0.162	0.688
Invasive mechanical ventilation	8 (15.34)	29 (14.08)	0.058	0.810
Central venous pressure oxygen saturation (cmH2O)	14.25±1.28	13.95±1.24	1.564	0.119
Central venous pressure oxygen saturation (%)	50.15±17.76	61.45±12.65	-5.256	< 0.001
Heart rate (beats/min)	99.67±6.41	101.69±7.45	-1.792	0.074
APACHE II score (points)	53.52±4.34	44.10±3.85	15.258	< 0.001
Lymphocyte count (×109/L)	1.47±0.38	0.90±0.11	18.734	< 0.001
PLT (×109/L)	184.27±18.47	189.12±18.23	-2.451	0.015
Serum creatinine (μmol/L)	127.34±40.18	105.26±33.58	4.067	< 0.001
Urea nitrogen (mmol/L)	7.86±2.23	6.21±2.03	5.120	< 0.001
Troponin I (μg/L)	33.61±11.49	25.39±7.95	6.035	< 0.001
Lactic acid (mmol/L)	11.72±3.44	9.27±2.47	5.861	< 0.001

Notes: APACHE II score: Acute Physiology and Chronic Health Evaluation score; PLT: platelet count.

Construction of BP neural network, RF, and decision tree predictivemodels

The 9 characteristic variables screened by the above table analysis were taken as the “independent variables”, and the outcome of the patients (survival = 2, death = 1) was taken as the “dependent variable”. The screened variables were included in the BP neural network, RF, and decision tree models, respectively.

BP neural network

Nine independent variables were selected as the input variables of the BP neural network model, and the outcome of the patients (survival = 2, death = 1) was taken as the output variable. The number of input layers of the neural network model was the same as the number of input variables, that is, there were 9 nodes in the input layer of the neural model. The BP neural network model for the prognosis of ECMO treatment in elderly patients with cardiogenic shock is shown in Figure 1. According to the degree of influence of input factors on the network, a ranking map was plotted (Figure 2). The variables in descending order of their influence, from high to low, were as follows: lymphocyte count, APACHE II score, urea nitrogen, troponin I, age, lactic acid, serum creatinine, diabetes, and central venous oxygen saturation.

Figure 1.

BP neural network model; APACHE II score: Acute Physiology and Chronic Health Evaluation score; I: Input layer; H: Hidden layer; B: Hidden layer; O: Output layer. The process of the BP neural network is mainly divided into two stages. The first stage is the forward propagation of signals from the input layer through the hidden layer and finally to the output layer. The second stage is the back propagation of errors. From the output layer to the hidden layer and finally to the input layer, the weight and bias from the hidden layer to the output layer are adjusted successively, and the weight and bias from the input layer to the hidden layer are also adjusted successively.

Figure 2.

Sequence diagram; APACHE II score: Acute Physiology and Chronic Health Evaluation score. The larger the ordinate in the figure, the greater the influence of this factor.

RF model

Mtry specifies the number of variables used for the binary tree in nodes. By default, it was the quadratic root of the number of variables in the dataset (classification model) or one third (predictive model). In this study, the number of input variables in the RF model was 9, which makes mtry = 3. Ntree specifies the number of decision trees in the RF, which determines the number of votes and accuracy of the RF. RF modeling was carried out with mtry = 3, and the relationship between model error rate and decision tree was visualized (Figure 3). It can be seen from the figure that when Ntree > 360, the variation range of model error rate began to decrease and tended to be stable. RF was also used to indicate Mean Decrease in Accuracy after variable substitution and Mean Decrease in Gini after variable substitution, as shown in Figure 4. Combining the two classification methods of average drop accuracy and average drop gini, the lymphocyte count score was the highest, suggesting that lymphocyte count played the greatest role in the model. After that, the order of average decreasing accuracy was age, APACHE II score, central venous oxygen saturation, troponin I, urea nitrogen, lactic acid, diabetes, and serum creatinine, respectively.

Figure 3.

Relationship between RF model error rate and tree order. Green lines: death group; Black lines: Out of Bag (OOB); Red line: survival group. This shows that the predictive error rate decreases with an increase in the number of living trees. When the Ntree > 360, the variation range of model error rate decreases and tends to be stable.

Figure 4.

Importance measure of input variables; A: Mean Decrease Accuracy map; B: Mean Decrease Gini map. Greater the Mean Decrease Accuracy value is associated with the small circle in the figure and greater influence of this factor.

Decision tree

The model derived from the decision tree included 2 layers and a total of 5 nodes, including 3 terminal nodes. Two explanatory variables were screened out, lymphocyte count and age. The results showed that lymphocyte count was the most important prognostic factor. The incidences of death were 13% and 8%, respectively, in the patients with lymphocyte count ≥ 1.3 and age ≥ 73, as shown in Figure 5.

Figure 5.

Decision tree predictive model for ECMO prognosis in elderly patients with cardiogenic shock; death.group: death group; survival.group: survival group.

Comparison of predictive value of BP neural network, RF, and decision tree model in test set

According to a ratio of 3:1, the training data set and the test data set were randomly divided for model construction and validation. The same training set was used for all the three models, so as the same test data set. The predictive performances of the three models were all found to be over 80.00%. The accuracy, sensitivity and specificity of the RF model were 0.987, 1.000 and 0.929, respectively, which were higher than those of the decision tree model. The AUC of the RF model was 1.000, which was higher than 0.916 of the decision tree model. The DeLong test showed a significant difference in the AUC between the RF model and the decision tree test set (D=-2.063, P=0.042 < 0.05). See Table 2.

Table 2.

Comparison of predictive effect

Model	BP neural network		RF		Decision tree
	Training set	Test set	Training set	Test set	Training set	Test set
Accuracy	1.000	0.935	1.000	0.987	0.959	0.907
Sensitivity	1.000	0.833	1.000	1.000	0.968	0.864
Specificity	1.000	0.967	1.000	0.929	0.957	0.922
Recall	1.000	0.833	1.000	1.000	0.968	0.864
Accuracy	1.000	0.882	1.000	0.985	0.833	0.792
F1	1.000	0.857	1.000	0.991	0.896	0.826
AUC	-	-	1.000	1.000	0.960	0.916

Note: ROC: receiver operating characteristic.

Discussion

In this study, 9 variables (age, history of diabetes, central venous pressure oxygen saturation, APACHE II score, lymphocyte count, serum creatinine, urea nitrogen, troponin I, and lactic acid) were screened out by univariate analysis and associated with the prognosis of ECMO treatment. These influencing factors were consistent with reported clinical studies [12,13]. 1) With the increase of age, the organ function and immune function of the body are gradually weakening, and the organs are mostly in a compensatory stage. At the same time, the hematocrit level of older patients with cardiogenic shock is also higher, and the cardiovascular endothelium is damaged, which can cause myocardial ischemic injury and reperfusion injury after ECMO treatment, leading to possible aggravated organ injury [14]. 2) As diabetes progresses, patients may develop myocardial interstitial fibrosis and diffuse small vessel disease in the myocardial wall. These conditions can aggravate myocardial injury, hinder the cardiac blood flow, reduce cardiac output, and induce cardiogenic shock. However, perfusion was restored after ECMO treatment, but due to the severe degree of diabetic stenosis and the diffuse and compound arterial stenosis, there may be more severe bleeding, calcification and embolization [15]. 3) Oxygen saturation of central venous pressure can reflect the balance between systemic oxygen metabolism, as well as the balance between oxygen supply and demand of the body. Early detection of tissue hypoxia can guide perioperative fluid therapy, enabling monitoring and evaluation of the balance between oxygen supply and demand, as well as tissue oxygenation changes [16]. When oxygen delivery is reduced or oxygen demand is greater than oxygen supply and exceeds the compensatory capacity of the body, oxygen saturation of central venous pressure decreases, affecting the prognosis of patients. 4) APACHE II score can predict the mortality of critically ill patients. In clinical practice, the severity of illness is often evaluated based on age, acute physiology and chronic health abnormalities [17]. Before ECMO treatment, high APACHE II score indicates that patients with severe cardiogenic shock, decreased cardiac output, insufficient blood supply to tissues and vital organs, are the patients are prone to recurrent malignant arrhythmias or cardiac arrest, thus affecting the prognosis. 5) As an important part of human cellular immunity, lymphocytes can participate in systemic inflammatory response and cellular immunity. If the lymphocyte count decreases continuously, the body can be in a state of immune suppression. After implantation of an ECMO pipeline into the body, the non-physiological surface between the blood and the artificial surface can trigger an inflammatory reaction, and the reperfusion injury caused by ECMO treatment can also aggravate the systemic inflammatory response. Therefore, if the lymphocyte count decreases before ECMO treatment, it is difficult for patients to cope with the inflammatory reaction, which increases the risk of organ damage and ultimately affects the prognosis [18]. 6) Serum creatinine is a product of muscle metabolism in the human body. It is primarily excreted by glomeruli, rarely reabsorbed by renal tubules, and excreted with urine. Clinically, renal function injury can be assessed by measuring serum creatinine level. Studies have shown that serum creatinine is significantly correlated with the risk of cardiogenic shock. An increased serum creatinine level indicates poor prognosis patients [19]. 7) Some studies have shown that serum creatinine and urea nitrogen are sensitive indicators of renal dysfunction. Renal insufficiency can lead to electrolyte disturbance, acid-base imbalance, and water and sodium storage. This insufficiency also increases cardiac load, damages the internal environment of cardiomyocytes, increase the risk of cardiogenic shock, and produces reperfusion injury during ECMO treatment [20]. 8) Troponin T or troponin I are usually used clinically to assist in the diagnosis of acute coronary syndrome. In patients with myocardial infarction or fulminant myocarditis, especially those also with cardiogenic shock, there is often a complete metabolic disorder, severe acidosis, obvious tissue hypoxia, and an imbalance between myocardial oxygen consumption and oxygen demand. These conditions contribute to hypoxic injury of cardiomyocytes, increased membrane permeability, and subsequent troponin leakage [21]. As a marker of myocardial injury, a sustained increase in troponin often indicates myocardial cell damage, and a significant decrease often indicates the recovery of cardiac function. If troponin continues to be in a high state, it may indicate that the myocardium is still damaged, indicating a poor prognosis [22]. 9) Lactic acid is an intermediate product generated during glycolysis under hypoxic conditions. Increased level of blood lactic acid can indicate weakened clearance ability of the body, accumulation of acid substances, and potential hyperlactatemia, which affects the prognosis of patients [23]. At the same time, lactic acid can directly reflect the balance of systemic oxygen metabolism, so a high level of lactic acid can aggravate the damage to liver, kidney and other organs, and further increase the mortality of elderly patients with cardiogenic shock [24].

Aleman et al. [25] showed that the use of machine learning predictive models may provide accurate and necessary feedback for early detection and appropriate management of cardiogenic shock. Ayers et al. [26] used machine learning to predict the survival after ECMO and reported a good predictive value of machine learning model in increasing the clinical decision-making of VA-ECMO patients. The results of this study exhibited that the predictive performances of all the three models were over 80.00%. The accuracy of the RF model was 0.987, the sensitivity was 1.000, and the specificity was 0.929, showing a high predictive performance. The reasons may be as follows: RF model, as a common machine learning algorithm, takes multiple decision trees as the learner, capturing the complex interactive information in the data with a low deviation. RF can overcome the overfitting phenomenon and analyze the data with complex nonlinear relationships without limiting the number of predictor variables. Compared to BP neural network, where the weight coefficients of multiple hidden layers are of real significance, and decision tree, which has difficulty in processing continuous data and can easily limit the number of predictor variables, RF can not only automatically identify the most important input variables, but also process categorical variables and continuous variables at the same time. It can also analyze the importance of each independent variable for model predictive [27,28]. In this study, the RF model demonstrated a AUC of 1.000, higher than 0.916 for the decision tree model. The DeLong test showed that there was a significant difference in the AUC of the RF model compared to that of the decision tree test set (D=-2.063, P=0.042 < 0.05). Thus, the predictive performance of the RF model was higher than that of the decision tree model. Since the ROC curve of the BP neural network was not established in this study, the related comparison cannot be conducted, which may cause some controversy on the prognostic predictive effect. However, it can be seen that the predictive effect of the RF model, BP neural network, and decision tree model were comparable, so the predictive model can be selected according to the actual situations in clinical practice.

There are some shortcomings in this study. First, this is a retrospective study, with a small sample size. Second, the patients were followed up for only 6 months in this study, and the long-term prognosis was not observed, so the results may have a certain information bias. Therefore, subsequent research should use a larger and more comprehensive sample set and improve the model validation to further establish a more robust and accurate machine learning model.

Conclusion

All three models we constructed exhibited good predictive performance and high application value in assessing the prognosis of ECMO treatment in elderly patients with cardiogenic shock. In clinical practice, predictive models should be selected according to the actual situations, to help clinicians and patients make decisions.

Disclosure of conflict of interest

None.