Comparison of six outcome prediction models in an adult burn population in a developing country

Summary

There are two types of prognostic model - burn-specific and general - to predict mortality risk in burn patients. Most prediction models were devised in developed countries. The aim of this study was to compare the performance of six outcome models in a developing country. In a retrospective cohort study, data of all thermal burned adult patients (age ≥ 18 years) admitted to the Burn Intensive Care Unit (BICU) were collected and then the following six prediction models were used to assess each patient: Acute Physiology and Chronic Health Evaluation (APACHE II), Abbreviated Burn Severity Index (ABSI), Belgian Outcome in Burn Injury (BOBI), the Ryan model, revised Baux and FLAMES model. Discriminative ability and goodness-of-fit of the prediction models were determined by receiver operating characteristic curve analysis and Hosmer–Lemeshow tests. We included 238 patients (mean age: 38.3 ± 18.39 years, average TBSA: 58.27% ± 24.55) in our study; 172 (72.3%) of them were diagnosed with inhalation injury and 178 (72.4%) were intubated. Mortality rate was 69.7%. Deceased patients had significantly higher mean age, %TBSA and number of inhalation injury. The area under the curve of the models was between 64.5 (APACHE II) and 85.9 (ABSI). The best estimation of predicted mortality was obtained with the ABSI model (67.2%).

Introduction

Over the last few decades, the survival rate of burn-injured patients has increased significantly due to advancements in burn care. However, mortality rate for severe burn injuries is still high.^1,2,3 Multiple predictive factors including patient characteristics (age, gender, comorbid diseases …) and burn features (total body surface area, inhalation injury, burn depth, associated trauma …) influence the prognosis. Consequently, if a prediction model considers all of the factors involved, it will be reliable and accurate.^4,5 Many prognostic scoring models have been devised to predict mortality risk in burn patients. Mortality prediction models in burn injury have existed since the mid-20th century.^6,7 Generally, there are two types of model. The first type is specific for burn-injured patients and applies the features of burn injuries to predicate prognosis. The second type is a general model which focuses on the patient’s acute physiological status and associated comorbidities to evaluate outcome.^7,8,9

In addition to assessing the probability of burn patient mortality, accurate and reliable prediction models as a standardized tool can measure burn service performance.¹⁰ To evaluate burn service performance, standardized mortality ratio (SMR = observed mortality/predicted mortality) is a suitable index.⁸ A scoring model must have good predictive value to determine realistic predictions of mortality, as well as repeatability and generalizability potential in different burn populations and hospitals. ¹¹

High mortality rate due to burn injury is still an important and challenging problem in developing countries. The World Health Organization (WHO) estimates that the vast majority of post-burn deaths occur in low- and middle-income countries.¹² However, most prediction models were designed in developed countries, and all claim to be the most accurate in different burn populations.^8,13,14 There are a limited number of studies that compare well-known prediction models in the same burn population group (cohort).^9,13,15 We therefore conducted this study to assess the performance of the six bestknown models for predicting hospital mortality among severely burned adult patients admitted to a burn intensive care unit in a developing country.

Methods

Patient selection

This retrospective population-based cohort study was carried out at the 18-bed Burns Intensive Care Unit (BICU) of Mothahary Hospital in Tehran. This tertiary burn hospital provides burn care services for over 12 million people and is the biggest referral burn center in the capital of Iran. The study was approved by the ethical committee of Iran University of Medical Sciences. Inclusion criteria were all thermal burned adult patients (age ≥ 18 years) admitted to the BICU. Those with multiple traumas, chemical and electrical burns were excluded.

Variable collection

Medical files of eligible burned patients admitted to the BICU over a period of 14 months (Jan 2015 - Mar 2016) were the source of demographic, clinical, physiologic and laboratory data. The worst point for each physiological variable on the first day of a patient’s admission to the ICU was as mentioned in the Acute Physiology and Chronic Health Evaluation (APACHE II)^16,17,18 and FLAMES prediction model.¹⁹ Percentage of total body surface area (%TBSA), age, gender, clinical signs of inhalational injury and presence of third-degree burn were recorded to calculate Ryan Score,²⁰ Revised Baux Score,^21,22 Belgian Outcome in Burn Injury (BOBI) Score²³ and Abbreviated Burn Severity Index (ABSI).²⁴ Data on mortality during hospital stay was also collected.

In this study, the following six most well-known and routinely-used prediction models were compared:

1. Acute physiology and chronic health evaluation (APACHE II) is a physiology-based scoring system and is applicable on the first day of a patient’s admission to the intensive care unit (ICU) for prognosis assessment. The original APACHE was designed in 1981.¹⁶ The APACHE II was revised in 1985, eliminating infrequent variables.^17,18 The new version includes 12 variables, each of them rated from 0 to 4. The worst point for each physiological variable on the first day of a patient’s admission to the ICU is recorded. Total score ranges from 0 to 71; a higher score is associated with higher mortality risk

2. FLAMES was developed by Gomez et al to predict outcome in severe burns.¹⁹ The probability of death in burn patients, percentage of partial and full thickness burns, age and gender are computed based on the APACHE II score on day 1. Inhalation injury is not considered in this model.

3. Ryan et al. published their mortality prediction model for burn injuries in 1998.²⁰ This simple method uses 3 parameters - age, total burn surface area and inhalation injury - to assess the risk of mortality. An increasing number of risk factors (0-3) is associated with an increasing mortality rate. When there is no risk factor, mortality rate is 0.3%, one risk factor 3%, two risk factors 33% and 3 risk factors give a probability of death of 90%.

4. The main Baux Score was modified by Osler et al in 2010 (Revised Baux Score) and in inhalational injury cases, the number 17 is added to the sum of the patient’s age and TBSA.²¹ Risk of mortality goes from 0 to 100%. Introducing a new nomogram to calculate the Revised Baux Score has simplified its usage.²²

5. Belgian outcome in burn injury (BOBI) was developed in 2009.²³ In this model increased age, TBSA and presence of inhalation injury are the 3 risk factors. Age is divided into four groups (0–3 points) and TBSA into 5 groups (0– 4 points). Presence of inhalation injury takes 3 points. Based on total score (0-10 points), predicted mortality ranges between 0.1% and 99%.

6. The abbreviated burn severity index (ABSI) was published by Tobiasen et al in 1982.²⁴ This system uses five variables to predict prognosis: increased TBSA (1–10 points), increased age (1–5 points), female gender (1 point), presence of inhalation injury (1 point) and presence of full-thickness burns (1 point).¹³ The sum of these values ranges from 2 to 18 points, and survival probability percentage decreases as the score increases (≤ 10% and ≥ 99%).

Statistical analysis

Statistical analysis was accomplished using the Statistical Package for Social Sciences (SPSS 16, SPSS Inc., Chicago, US). Comparison of demographic and burn characteristics between survivors and deceased patients was performed. Comparison of quantitative continuous variables was carried out using the sample t-test, and categorical variables were compared using the 𝜒2 test. Data are expressed as number (%) or mean ± standard deviation (SD). A P value <0•05 is considered statistically significant.

Binary logistic regression analyses were used to calculate LA50.

The Hosmer and Lemeshow Chi-square statistic test was used to measure accuracy and goodness-of-fit of the prediction models. A Receiver Operator Characteristics (ROC) graph was plotted to assess the discriminative ability of these six prediction models. The area under the curve (AUC) was used to detect which model was more accurate at distinguishing between survivors (false positives) and non-survivors (true positives). An area over 0.9 indicates high accuracy, 0.7- 0.9 moderate accuracy, 0.5-0.7 low accuracy and 0.5 indicates chance discrimination.²⁵

Results

During the study period, 238 eligible patients (59/24.8% female and 179/75.2% male) were admitted to the BICU. Mean age was 38.3 ± 18.39 years, ranging from 18 to 98 years, with a mean total body surface area of 58.27% (± 24.55), ranging from 10% to 100%. A total of 172 (72.3%) patients had inhala tion injury and 178 (72.4%) were mechanically ventilated. Mortality rate was 69.7% (N=166).

In a comparison between survivors and deceased victims (Table I), the survivors (N=72) had a significantly lower mean age, %TBSA and number of inhalation injuries (respectively P=.021, P=.000 and p=.027). Total LA50 was 49.49%, which was correlated with age (Table II).

Table I. Comparison of demographic and burn characteristics between survivors and deceased patients.

Table II. Lethal area fifty percent (LA50) in all burn patients.

The probability of death predicted by BOBI, rBaux, ABSI, FLAMES, Ryan and APACHE II was respectively 181 (76.0%), 185 (77.7%), 160 (67.2%), 138 (57.9%), 223 (93.6%) and 200 (84.0%), while true mortality was 166 (69.7%) (Table III). The best prediction of mortality percentage was estimated by the ABSI model (67.2%), which had the highest area under the curve of 85.9 (80.5-91.3). Apart from FLAME, the other models over-predicted the number of deaths. The APACHE II area under the ROC curve of 64.5 (56.6-72.3) was lower than the other models, indicating low accuracy at discriminating between survivors and non-survivors (Fig. 1).

Table III. The area under the curve (AUC) analysis and predicted mortality by six models.

Fig. 1. Receiver operator characteristics (ROC) curve of predicting models. An area under the curve (AUC) of more than 0.9 indicates high accuracy, while an area of 0.5 indicates chance discrimination. The area under the curve (AUC) for the all the scoring models except APACHE II was 0.7–0.9, considered moderate accuracy at distinguishing between survivors and non-survivors. The area under the ROC curve of APACHE II was 64.5(56.6-72.3), indicating low accuracy at discriminating between mortality and survival in patients. The AUC of the ABSI model was higher than the other models 85.9(80.5- 91.3).

Discussion

Despite the large number of burn prognostic models, none can claim to be the most accurate across the entire burn population. ⁸ The sheer number of different models suggests that none accurately predicts outcome in every population.²⁶ The models have mostly been validated on populations in developed countries, and the number of studies on external validation of the models on populations in developing countries are sparse. There are several studies with varying results regarding the performance of the same outcome prediction models.^{7,8,13,18,21,26-28} These different results may be due to variations in the population or differences in standards of care in various regions.²⁸ Internally validated prediction models may degrade in other populations.^29,30 Therefore, it is very important to evaluate the independent or external validation of the prediction models before using them in a new population or case mix.^28,31

In this cohort study, five specific outcome models for burn injured patients and one general prognostic model were evaluated for predicting mortality.

Compared with survivors, deceased patients had a significantly higher mean age, %TBSA and number of inhalation injuries (respectively P =.021, .000, .027). These three important factors are the foundation of nearly all specific burn prediction models.⁹ Consequently, the importance of these prognostic factors in burn injuries was confirmed by our study.

The area under the curve (AUC) for the five specific outcome models was 0.7–0.9, demonstrating moderate accuracy at distinguishing between survivors and non-survivors, so the majority of burn injured patients who died were expected to die. In this study, true mortality was 166 (69.7%). The predicted mortality estimated by ABSI (160/67.2%) showed the least difference with observed mortality and had the highest area under the curve 85.9 (80.5-91.3). Our finding was similar to other recent studies.^13,27,32 Based on one new study, the ABSI scoring system is still an accurate and useful tool to predict mortality in burn patients.³²

Percent of inhalation injury in our study was 72.3%, which was higher than previous studies.^33,34 Inhalation injury was mostly diagnosed by history taking of exposure to smoke in closed areas, burns due to explosion and physical examination findings. Fiberoptic bronchoscopy had been performed in only a few cases, therefore overestimation of inhalation was possible. This may explain why the Ryan model predicted the highest mortality rate (93%). Because of the importance of inhalation injury in the Ryan model, an increase in inhalation injury percentage increases its score more so than in the other models. The diagnosis of inhalation injury needs to be standardized before using the Ryan model.

The area under the curve (AUC) for APACHE II as a general prognostic model was 64.5, meaning low accuracy at distinguishing between survivors and non-survivors and weak compatibility between observed and predicted deaths. APACHE II is a useful tool to predict mortality in a general intensive care unit, but its validation for burn victims is not clear. The original investigators of the APACHE scoring system did not include burn patients in their study.^16,17 As we know that the influence of comorbidities is important in the APACHE II model and since burn patients before injury compared with those admitted to the general ICU are usually healthy, there are fewer comorbidities to increase their APACHE II score.³⁴

Gomez et al. incorporated specific burn risk factors (percentages of partial and full thickness burns, age and gender) in APACHE II and FLAMES.¹⁹ In our analysis the FLAMES model under-predicted the number of deaths (57.9 vs. 69.7). Inhalation injury was not considered in this model, and this is probably the reason for the underestimation of predicted mortality in our analysis.

In this study, unlike in that of Brusselaers et al.,¹³ these three models (BOBI, Ryan and rBaux) overestimated the number of deaths. This difference might be due to the higher number of diagnosed inhalation injuries in our study.

Compared to similar studies,^{13,15,27,32,35} observed mortality (69.7%) in the present study was higher. Also, lethal area fifty percent (LA50) (49.49%) was lower than in a recent study conducted in our center.³⁶ It is important to take into account that the present study was carried out on severely burned patients with a higher incidence of inhalation injuries admitted to the burn ICU. The higher mortality observed and lower LA50 in our study may be due to a high TBSA mean and high rate of inhalation injury (58.27 ± 24.55% and 72.3% respectively). However, as mentioned by Heng et al., the selection of extensively burn-injured patients had a relative strength due to the high mortality rate that enabled us to reach adequate statistical power, especially when we had a small sample size.⁹

Limitations

The first limitation of our study was the lack of fiberoptic bronchoscopy to diagnose inhalation injury in all the cases. In most patients, the diagnosis of inhalation injury had been made based on clinical findings. Despite this limitation, the high number of patients receiving mechanical ventilation in the first few days after admission probably confirms the diagnosis of inhalation injury. Moreover, the study was conducted in only one center with a small sample size.

Finally, this research was not a prospective study and the authors could not control confounding factors. This is a preliminary study, and in order to validate our results we need a prospective, multicenter study with a large study population.

Conclusion

In the present study, the six prediction models (five specific burn outcome models and one general prognostic model) were validated, with the ABSI scoring system showing the best performance in predicting mortality and APACHE II the worst.

Our findings confirmed that age, %TBSA and inhalation injury are the most important prognostic factors in burn patients. The variations between our results and other previous studies may be due to differences in population and standards of burn care in various places. Therefore, it is better to verify the validity of prediction models before applying them in a new population.