Neural network prediction of severe lower intestinal bleeding and the need for surgical intervention

Abstract

Background

The prognosis for patients with severe acute lower intestinal bleeding (ALIB) may be assessed by complex artificial neural networks (ANNs) or user-friendly regression-based models. Comparisons between these modalities are limited, and predicting the need for surgical intervention remains elusive. We hypothesized that ANNs would outperform the Strate rule to predict severe bleeding and would also predict the need for surgical intervention.

Methods

We performed a 4-y retrospective analysis of 147 adult patients who underwent endoscopy, angiography, or surgery for ALIB. Baseline characteristics, Strate risk factors, management parameters, and outcomes were analyzed. The primary outcomes were severe bleeding and surgical intervention. ANNs were created in SPSS. Models were compared by area under the receiver operating characteristic curve (AUROC) with 95% confidence intervals.

Results

The number of Strate risk factors for each patient correlated significantly with the outcome of severe bleeding (r = 0.29, P < 0.001). However, the Strate model was less accurate than an ANN (AUROC 0.66 [0.57–0.75] versus 0.98 [0.95–1.00], respectively) which incorporated six variables present on admission: hemoglobin, systolic blood pressure, outpatient prescription for Aspirin 325 mg daily, Charlson comorbidity index, base deficit ≥5 mEq/L, and international normalized ratio ≥1.5. A similar ANN including hemoglobin nadir and the occurrence of a 20% decrease in hematocrit was effective in predicting the need for surgery (AUROC 0.95 [0.90–1.00]).

Conclusions

The Strate prediction rule effectively stratified risk for severe ALIB, but was less accurate than an ANN. A separate ANN accurately predicted the need for surgery by combining risk factors for severe bleeding with parameters quantifying blood loss. Optimal prognostication may be achieved by integrating pragmatic regression-based calculators for quick decisions at the bedside and highly accurate ANNs when time and resources permit.

Introduction

Acute lower intestinal bleeding (ALIB) resolves spontaneously in about 80%of all cases.¹ For the other 20%, surgical management may become necessary in cases of severe bleeding when endoscopic and angiographic interventions fail to diagnose and treat the source. Artificial neural network (ANN) models have outperformed scoring systems based on regression models in predicting severe bleeding.^2–7 The accuracy of ANNs is attributable to their flexible, nonlinear structure, and large parameter space. The machine-learning technology that drives ANNs is based on complex and chaotic relationships among individual variables, which may be best expressed by nonlinear statistical processing.^8,9 This allows the network to incorporate the intricate associations among variables into algorithms.¹⁰ ANNs have been used in the medical field to perform a variety of difficult predictions: long-term functional recovery after spinal cord injury,¹¹ the incidence of dangerous viral infections,¹² and the toxicity of thrombolytic nanoparticles.¹³

ANNs have not been compared directly with the Strate prediction rule, which appears to be the most robust regression-based model for ALIB.^4,5,7 The Strate prediction rule assesses seven risk factors for severe bleeding and then classifies each patient as low, moderate, or high risk for severe bleeding based on the number of risk factors present (0 risk factors = low risk [9%], 1–3 factors = moderate risk [43%], ≥4 factors = high risk [84%]).^3,4 Although efforts to predict severe bleeding have been effective, predicting the need for surgical management of ALIB remains elusive.

The purposes of this study were to compare an ANN to the Strate model in predicting severe bleeding and to evaluate the ability of an ANN to predict the need for surgery. We hypothesized that an ANN would have greater accuracy than the Strate prediction rule for predicting the incidence of severe ALIB and that an ANN would also predict the need for operative management.

Methods

We performed a retrospective analysis of 147 patients who underwent inpatient endoscopy, angiography, or surgery for ALIB at our tertiary care center from June 2011 to September 2015. Institutional Review Board approval was obtained. Patients were identified by the International Classification of Diseases, Ninth Revision, codes for LIB and encounter codes for the Gastroenterology Endoscopy laboratory, Interventional Radiology procedure room, and operating room. All consecutive adult patients (aged ≥ 18 y) with confirmed or suspected LIB were included. This heterogeneous population was selected to improve the generalizability of the prediction models. Patients were classified as suspected LIB if they presented with bleeding per rectum in the absence of hematemesis and identifiable sources of bleeding on upper endoscopy or if they had lower endoscopy with an identifiable lesion but no active bleeding. Patients who did not require a procedural intervention were excluded because this population would not warrant surgical consultation. Patients managed for >12 h at an outside facility were excluded in order to ensure that records at the time of presentation were as complete and accurate as possible. A protocol standardizing communication practices for multidisciplinary management of acute gastrointestinal bleeding was implemented at our institution in August 2013. Apart from this protocol, there were no structured clinical practice changes pertinent to ALIB patients during the study period.

Variables describing patient characteristics, management, and outcomes were collected by query of our institutional research database and supplemented by electronic medical record review. Hemoglobin and creatinine levels prior to admission (PTA) represented the most recently recorded value for each variable before the date of admission. PTA hemoglobin levels were available for 117 patients (80% of the study population) and PTA creatinine levels were available for 114 patients (78% of the study population). The Strate prediction rule assesses seven risk factors for severe bleeding and then classifies each patient as low, moderate, or high risk for severe bleeding based on the number of risk factors present (0 risk factors = low risk [9%], 1–3 factors = moderate risk [43%], ≥4 factors = high risk [84%]).^3,4 Strate prediction rule inputs for syncope, rectal bleeding within 4 h of initial evaluation, and nontender abdominal examination were determined from physician-authored admission notes in the electronic medical record. Aspirin use was defined as an outpatient prescription for ≥81 mg aspirin per day. The definition of severe bleeding was also derived from Strate et al.^3,4 and included patients with any of the following: decrease in hematocrit by at least 20%, transfusion of at least two units of packed red blood cells, and readmission with LIB within 1 wk of discharge.

Statistical analysis was performed using SPSS (version 23; IBM, Armonk, NY). The patient population was described by mean (95% confidence interval) for continuous variables and n (percentage) for discrete variables. Efficacy of the Strate prediction rule was assessed by enumerating risk factors for each patient, grouping patients by low, moderate, or high risk for severe bleeding, correlating predicted risk to observed severe bleeding, and then comparing groups by Fisher’s exact test. Correlation between the number of risk factors and severe bleeding was assessed by Pearson’s r. The number of risk factors for each patient was entered into a multiple logistic regression equation to generate the predicted probability of severe bleeding for each patient. Predicted probabilities were then used to generate a receiver operating characteristic curve so that the area under the receiver operating characteristic curve (AUC) for the Strate prediction rule could be compared with AUC for ANN.

Multilayer perceptron is a type of ANN in which a small number of parameters may be used to predict an output variable. Multilayer perceptron ANN models were built using normalized rescaling of covariates, one hidden layer, and the hyperbolic tangent function to predict single binary outcomes (severe bleeding and surgical intervention) with the Softmax activation function. Therefore, there was an input layer of nodes containing information about the risk factors, followed by a second layer of nodes which interacted with the input variables to predict the status of the outcome node, which represented severe bleeding or the need for surgery. A random number generator (Microsoft Excel 2010; Redmond, WA) was used to allocate 103 patients (70% of the study population) to the training sample and 44 patients (30%of the study population) to the testing sample. Model strength was assessed by AUC. Relative contributions of each independent variable were reported as normalized importance. Importance represents degree to which the predicted value will be affected by changing the independent variable. Importance values were normalized by dividing each importance value by the largest importance value and expressing the quotient as a percentage.

Results

One hundred and forty-seven patients were included (Table 1). The study population was advanced in age (mean age 64 years), chronically ill (mean Charlson comorbidity index 3.1), and had baseline mild anemia (mean hemoglobin 11.4 g/dL) by World Health Organization criteria.¹⁴ Fifteen etiologies of ALIB were represented, and the source of bleeding was not identified in one of five patients. All patients had one or more risk factors for severe bleeding per Strate criteria^3,4 (Table 2). Sixty-three percent were at moderate risk for severe bleeding, and 37% were at high risk.

Table 1.

Study population characteristics.

Patient characteristics	n = 147
Age (y)	64 (61–66)
Male, n (%)	80 (54)
Diagnosis, n (%)
Diverticular disease	33 (22)
Colorectal cancer	19 (13)
Colitis	17 (12)
Small bowel arteriovenous malformation	11 (7)
Crohn’s disease	8 (5)
Colonic arteriovenous malformation	6 (4)
Colon polyp	6 (4)
Proctitis/rectal ulcer	5 (3)
Small bowel tumor	3 (2)
Enteritis	2 (1)
Small bowel ischemia	2 (1)
Ulcerative colitis	2 (1)
Cecal volvulus	1 (1)
Meckel’s diverticulum	1 (1)
Rectal varices	1 (1)
Unknown	30 (20)
Charlson comorbidity index	3.1 (2.7–3.5)
Outpatient medications, n (%)
Aspirin 81 mg daily	50 (34)
Aspirin 325 mg daily	12 (8)
Dual antiplatelet therapy	11 (7)
Anticoagulant therapy	29 (20)
PTA
Hemoglobin (g/dL)	11.4 (11.0–11.8)
Creatinine (mg/dL)	1.2 (1.0–1.5)
On admission
Heart rate	89 (81–97)
Systolic blood pressure (mm Hg)	124 (115–134)
Base deficit (mEq/L)	2.5 (1.1–3.9)
International normalized ratio	1.6 (1.2–2.2)
Hemoglobin (g/dL)	9.8 (9.3–10.4)
Creatinine (mg/dL)	1.4 (1.2–1.6)

Data are presented as mean (95% confidence interval) or n (%).

Table 2.

Risk factors for severe bleeding, as described by Strate et al.^3,4

Risk factors	n = 147
Strate predictors, n (%)
Heart rate ≥100	44 (30)
Systolic blood pressure ≤115 mm Hg	43 (29)
Syncope	10 (7)
Rectal bleeding within 4 h of admission	78 (53)
Nontender abdomen	116 (79)
Aspirin use	62 (42)
More than two comorbidities	129 (88)
Low risk for severe bleeding (0 points)	0
Moderate risk for severe bleeding, n (%)	92 (63)
1 point	6 (4)
2 points	27 (18)
3 points	59 (40)
High risk for severe bleeding, n (%)	55 (37)
4 points	33 (22)
5 points	19 (13)
6 points	3 (2)
7 points	0

Data are presented as n (%).

Ninety-five percent of all patients underwent endoscopy, 10% had angiography, and 9% required surgical intervention (n = 13) (Table 3). Of the 13 patients who had surgery, bowel resections were performed in 11. The two patients who did not undergo bowel resection had intraoperative surgeon-assisted enteroscopy in which no source of bleeding was identified. None of the surgery patients required reoperation. The average decrease in hemoglobin during admission was 1.8 g/dL, and about half of all patients received a blood transfusion. Forty-one percent of all patients had severe bleeding by Strate criteria.^3,4 The inpatient mortality rate was 4%. Management and outcomes are stratified by risk for severe bleeding in Supplementary Table 1. Patients at high risk for severe bleeding were more likely to require multiple interventions, although this difference did not reach statistical significance (36% versus 21%, P = 0.053). High-risk patients had significantly higher rates of red blood cell transfusion during admission, (56% versus 36%, P = 0.017) and were also more likely to require multiple transfusions (55% versus 28%, P = 0.003).

Table 3.

Management and outcomes for all patients.

Management and outcomes	n = 147
Endoscopy	140 (95%)
Angiography	14 (10%)
Surgery	13 (9%)
Bowel resection	11 (7%)
Number of interventions	1.5 (1.3–1.7)
Two or more interventions	39 (27%)
Interval between interventions (h)	62 (50–78)
Patients who received a PRBC transfusion	64 (44%)
PRBC transfusions per patient	2.6 (1.6–4.2)
Hemoglobin (g/dL)
Δ admission to nadir	−1.8 (−2.1 to −1.5)
Δ admission to discharge	0.1 (−0.3 to 0.5)
Creatinine (mg/dL)
Δ admission to peak	0.2 (0.1 to 0.3)
Δ admission to discharge	−0.2 (−0.4 to −0.1)
Severe bleeding	61 (41%)
Hematocrit decrease by 20% within 24 h	2 (1%)
≥2 units PRBC transfused	56 (38%)
Readmission with LIB within 1 wk	9 (6%)
Hospital length of stay (d)	8.2 (7.1–9.6)
Intensive care unit length of stay (d)	1.4 (0.7–2.4)
Inpatient mortality	6 (4%)

PRBC = packed red blood cell.

Data are presented as n (%) or mean (95% confidence interval).

The relationship between Strate risk factors and observed severe bleeding is illustrated in Figure 1. Severe bleeding occurred in 33% of all patients at moderate risk and 56% of all patients at high risk for severe bleeding (P = 0.006). There was a weak but statistically significant linear correlation between the number of risk factors and severe bleeding (r = 0.290, P < 0.001). Based on predicted probabilities for each patient, enumeration of Strate risk factors produced a model with AUC 0.656 (0.566–0.745). The ANN model was significantly more accurate (AUC 0.979 [0.953–1.00]; Fig. 2). This ANN included six predictors, each of which were assessed at the time of admission (Table 4). Admission hemoglobin level had the strongest impact on model predictions.

Fig. 1.

Enumeration of risk factors featured in the Strate prediction rule^3,4 correlated with severe bleeding.

Fig. 2.

A multilayer perceptron artificial neural network was more accurate in predicting severe bleeding than the Strate prediction rule.^3,4 Neural network AUC: 0.979 (0.953–1.000), Strate score AUC: 0.656 (0.566–0.745).

Table 4.

Normalized importance of components in multilayer perceptron networks predicting severe bleeding and the need for surgical intervention.

Outcome predictors	Normalized importance, %
Severe bleeding
Hemoglobin (g/dL) on admission	100
Systolic blood pressure on admission (mm Hg)	83
Outpatient prescription for aspirin 325 mg daily	60
Charlson comorbidity index	30
Base deficit ≥5 mEq/L on admission	24
International normalized ratio ≥ 1.5 on admission	13
Need for surgical intervention
Nadir hemoglobin (g/dL)	100
Systolic blood pressure on admission (mm Hg)	86
Decrease in hematocrit by 20% within 24 h	28
Outpatient prescription for aspirin 325 mg daily	17
Charlson comorbidity index	10

A separate ANN predicted the need for surgical intervention (AUC 0.954 [0.898–1.000]; Table 4). This model combined three predictors from the severe bleeding ANN with two additional factors: hemoglobin nadir and 20% decrease in hematocrit. Hemoglobin nadir had the strongest impact on model predictions. On subgroup analysis, there were no significant differences in the predictive capacity of Strate risk factors or ANNs after implementation of an acute gastrointestinal bleeding multidisciplinary communication protocol at our institution during the study period.

Discussion

Our data indicate that the Strate prediction rule was useful in stratifying risk for severe bleeding among patients with ALIB. Although enumeration of Strate risk factors is pragmatic and efficient, a logistically cumbersome ANN was significantly more accurate in predicting severe bleeding. These findings imply synergy between these methods in predicting outcomes for patients with ALIB. In addition, predicting outcomes that have previously been difficult to forecast may be within the reach of ANNs. In our study, an ANN accurately predicted the need for surgical management of LIB. Although several authors have identified risk factors associated with the need for operative management, viable prediction models have yet not been reported.^2,15–17 Notably, the ANN predicting the need for surgery was created by combining risk factors for severe bleeding with two parameters that quantified blood loss: nadir hemoglobin and percent decrease in hematocrit. These findings are consistent with previous reports suggesting that patients with ongoing severe bleeding and large volume blood loss are most likely to require surgical intervention.^2,16–19

The machine-learning technology that drives ANNs is based on complex and chaotic relationships among individual variables which may be best expressed by nonlinear statistical processing.^8,9 This allows the program to incorporate the intricate associations among variables into algorithms.¹⁰ Barriers to widespread acceptance and adoption of machine-learning decision models include insufficient technological infrastructure, difficulty integrating ANNs into routine work flow, and reluctance to substitute computational analysis for clinical judgement.^20,21 Although machine-learning techniques produce excellent prediction models, they are difficult to disseminate in the same manner as an online risk calculator. An ideal approach to evidence-based clinical decision-making may incorporate the pragmatism of regression-based prediction rules like the one developed by Strate et al.^3,4 and the superior accuracy of ANNs.

This study was limited by its retrospective design, small sample size (n = 147), selection bias, and heterogeneous study population. The sample size could have been expanded by including patients who did not require procedural management, an approach that would also reduce selection bias, but this method would have included patients for whom surgical consultation would be unnecessary. Although the wide variety of etiologies for LIB likely created variability in diagnostic and therapeutic management decisions, it also allows these findings to be generalized to a broader population of patients with acute LIB. In addition, it seems unlikely that many surgeons currently use prediction models when making the decision to operate for LIB. This may begin to change when regression and neural network prediction models become widely available on mobile devices. Finally, an acute gastrointestinal bleeding multidisciplinary communication protocol was implemented during the study period, but did not significantly affect the predictive capacity of Strate risk factors or the ANNs. Future investigations should seek to validate these ANNs in a large, prospective cohort of patients managed in diverse settings to improve the statistical power and generalizability of these findings.

Conclusions

The Strate prediction rule was effective in stratifying risk for severe LIB. However, the Strate model was less accurate than an ANN featuring six variables present on admission: hemoglobin, systolic blood pressure, outpatient prescription for Aspirin 325 mg daily, Charlson comorbidity index, base deficit ≥5 mEq/L, and international normalized ratio ≥1.5. A similar ANN predicted the need for surgery by integrating two additional parameters: hemoglobin nadir and the occurrence of a 20% decrease in hematocrit. The optimal approach to clinical prognostication may incorporate the efficiency and pragmatism regression-based risk calculators and the accuracy of ANNs.