Performance Assessment of the SOFA, APACHE II Scoring System, and SAPS II in Intensive Care Unit Organophosphate Poisoned Patients

Abstract

This study assessed the ability of the Sequential Organ Failure Assessment (SOFA) and Acute Physiology, Chronic Health Evaluation (APACHE) II scoring systems, as well as the Simplified Acute Physiology Score (SAPS) II method to predict group mortality in intensive care unit (ICU) patients who were poisoned with organophosphate. The medical records of 149 organophosphate poisoned patients admitted to the ICU from September 2006 to December 2012 were retrospectively examined. The SOFA, APACHE II, and SAPS II were calculated based on initial laboratory data in the Emergency Department, and during the first 24 hr of ICU admission. The probability of death was calculated for each patient based on the SOFA score, APACHE II score, and SAPS II equations. The ability to predict group mortality by the SOFA score, APACHE II score, and SAPS II method was assessed using two by two decision matrices and receiver operating characteristic (ROC) curve analysis. A total of 131 patients (mean age, 61 yr) were enrolled. The sensitivities, specificities, and accuracies were 86.2%, 82.4%, and 83.2% for the SOFA score, respectively; 65.5%, 68.6%, and 67.9% for the APACHE II scoring system, respectively; and 86.2%, 77.5%, and 79.4% for the SAPS II, respectively. The areas under the curve in the ROC curve analysis for the SOFA score, APACHE II scoring system, and SAPS II were 0.896, 0.716, and 0.852, respectively. In conclusion, the SOFA, APACHE II, and SAPS II have different capability to discriminate and estimate early in-hospital mortality of organophosphate poisoned patients. The SOFA score is more useful in predicting mortality, and easier and simpler than the APACHE II and SAPS II.

INTRODUCTION

Organophosphate (OP) poisoning is a major medical problem worldwide. The World Health Organization (WHO) has estimated that > 300 million case of acute pesticide poisoning occur worldwide each year, and that most of these cases are due to OP intoxication (1). The mortality rate from acute OP poisoning is 10%-20% (2-5), and WHO has estimated that 200,000 people die each year of pesticide poisoning. OP compounds increase the accumulation of acetylcholine in the synaptic cleft by inhibiting acetylcholinesterase and decreasing acetylcholine degradation; thus, leading to excessively increased cholinergic activity and cholinergic symptoms. Poisoning can occur through ingestion, inhalation, or cutaneous exposure. Nerve gas agents are similar in structure and mechanism of action to OP insecticides but have a higher potency (5).

OP insecticide ingestion is still a widely employed means of suicide with a high mortality rate despite great developments in intensive care (4). Although new advances in treating OP poisoned patients in the intensive care unit (ICU) have resulted in increased survival, such measures prolong patient ICU stay and increase hospital expenses. Informing patients of the severity of the illness at the time of ICU admission helps determine continuation of expensive treatment and avoids unnecessary procedures.

Use of scoring systems particularly developed for patient assessment at the time of ICU admission has reduced many problems and facilitated treatment planning. The Sequential Organ Failure Assessment (SOFA) and Acute Physiology and Chronic Health Evaluation II (APACHE II) scoring systems, as well as the Simplified Acute Physiology Score II (SAPS II) are three tools widely used by most ICUs to predict clinical outcome. These scoring systems have been evaluated and validated in many centers and can be adjusted according to needs (7-10).

The purpose of the this study was to evaluate the performance of SOFA, APACHE II, and SAPS II scoring systems for predicting illness severity and the mortality of patients poisoned with OP who were admitted to the ICU.

MATERIALS AND METHODS

Study design

This was a retrospective cohort study conducted from September 2006 to December 2012. This study was conducted at a regional emergency center affiliated with an academic university hospital in Changwon, Korea.

Data collection

The medical records of OP poisoned patients admitted to the emergency center were carefully examined. Acute OP poisoning was defined based on history of exposure, characteristic clinical features, and decreased serum cholinesterase activity. Patients with any of the following conditions were excluded: 1) an uncertain history of exposure; 2) combined drug exposures; 3) exposure to OP for > 24 hr prior to presentation; 4) discharge against medical advice or transfer to another hospital; 5) pre-hospital cardiac arrest; 6) < 15 yr of age; 7) severe chronic illness; or 8) missing data. Severe chronic illness included liver cirrhosis with portal hypertension, New York Heart Association class IV congestive heart failure, chronic respiratory disease, end-stage renal disease, or an immune-compromised state (e.g., leukemia, lymphoma, or AIDS).

Medical records were carefully examined, and two investigators collected the following parameters, age, gender, smoking status, presence of muscarinic and nicotinic symptoms, systolic and mean arterial blood pressure (mmHg), heart rate, respiratory rate, body temperature, initial Glasgow Coma Scale score, arterial blood gas analysis (pH, PaO₂, PaCO₂), FiO₂, laboratory data (white blood cell count, hematocrit, platelet count, and serum levels of cholinesterase, sodium, potassium, creatinine, albumin, glucose, bilirubin, and C-reactive protein), use of mechanical ventilation, seizure, pneumonia, pancreatitis, mortality, SOFA score, APACHE II score, and SAPS II. The investigators who collected the data were blinded to the study objectives, and they both collected the same data and compared it for accuracy.

The outcome parameter was 60-day mortality. The most abnormal values for each parameter are typically observed within the first 24 hr after ICU admission. The SOFA score, APACHE II score, and SAPS II were calculated as described previously (7, 8, 11).

Statistical analysis

Data were analyzed using SPSS version 21.0 (SPSS Inc., Chicago, IL, USA) and MedCalc version 12.2 (MedCalc Inc., Mariakerke, Belgium). Data are presented as mean ± standard deviation, median with interquartile range, or frequency. Differences between the two groups were tested using the independent two-sample t-test or the Mann-Whitney U-test for continuous variables and the chi-square test for categorical variables All variables found to be significant in a univariate analysis underwent multivariate logistic regression analysis. The area under the receiver operating characteristic (ROC) curve was calculated to determine the ability of the scores to discriminate using mortality as an independent variable. The differences between the sensitivities, specificities, and accuracies of the scores were determined using the McNemar test. Type I error was corrected by the Boneferroni's method for comparisons of three or more variables. Calibration or the agreement between predicted mortality of the study population was examined with the Hosmer-Lemeshow goodness-of-fit test and standardized mortality ratio (SMR) (defined as the actual population mortality divided by individual patient predicted mortality) (12). A P < 0.05 was considered significant.

Ethics statement

The study protocol was reviewed and approved by the institutional review board of Samsung Changwon Hospital (IRB No. 2013-SCMC-010-00). Informed consent was waived by the board.

RESULTS

Among the 149 patients, 131 were included from September 2006 to December 2012, and 18 patients with the following conditions were excluded (Fig. 1). The average age was 61 yr (range, 21-85 yr) and 82 (80.4%) patients were men. The general characteristics and laboratory data related to demographic and laboratory findings in both the survivor and non-survivor groups are summarized in Table 1.

Fig. 1.

Organophosphate poisoned patients between September 2008 and February 2013.

Table 1.

Characteristics of survivors and non-survivors

Values are expressed as mean ± standard deviation or frequency. MAP, mean arterial pressure; GCS, Glasgow Coma Scale; WBC, white blood cell count; RR, respiratory rate; ChE, cholinesterase; CRP, C-reactive protein (mg/dL); APACHE II, Acute Physiology and Chronic Health Evaluation II; SAPS II, Simplified Acute Physiology Score II; SOFA, Sequential Organ Failure Assessment.

The mortality rate was 22.1% (29 of 131). Twenty-four patients died in the hospital (82.8%), and five were discharged with impending death (17.2%). In the non-survivor group, the direct cause of death in 10 patients was complications of OP poisoning (6 patients, pneumonia and sepsis; 4 patients, multiple organ failure).

The scores of the non-surviving patients were significantly higher than those of the surviving patients. That is, the APACHE II scores were 13.3 ± 5.0 in the non-survivor group and 9.2 ± 4.9 in the survivor group (P < 0.001). The SAPS II scores were 62.1 ± 17.4 and 38.2 ± 15.5 in the non-survivor and survivor groups (P < 0.001), and the SOFA scores were 6.7 ± 2.2 and 3.0 ± 1.9 in the non-survivor and survivor groups, respectively (P < 0.001) (Table 1).

The area under the ROC curve was calculated to evaluate the predictive value of the scoring systems. The APACHE II scoring system had an area of 0.716 and a cut-off value of 11, the SAPS II score had an area of 0.852 and a cut-off of 45, and the SOFA score had an area of 0.896 and a cut-off value of 4 (Fig. 2).

Fig. 2.

Receiver operating curves for predicting death according the Sequential Organ Failure Assessment (SOFA), Acute Physiology and Chronic Health Evaluation (APACHE) II, and Simplified Acute Physiology Score (SAPS) II scoring systems. The areas under the curve and 95% confidence intervals for these indicators were 0.896 (0.839-0.954) for SOFA, 0.716 (0.615-0.817) for APACHE II, and 0.852 (0.780-0.923) for SAPS II, respectively.

The probability of death estimates for the SOFA, APACHE II, and SAPS II are shown in Table 2 for the survivor and non-survivor groups. The overall methods were significantly associated between survivors and non-survivors with good probability of death. The predictive power of the mortality estimates for the overall study population according to the SOFA, APACHE II, and SAPS II is shown in Table 3. The sensitivity of SOFA was remarkably higher than that of APACHE II (P < 0.001 vs SOFA), but was not significantly different from that of SAPS II (P = 0.200). The specificity of SOFA was also higher than that of APACHE II (P < 0.001) and SAPS II (P = 0.024). The accuracy rate of SOFA was higher than that of APACHE II (P < 0.001) and SAPS II (P = 0.021).

Table 2.

Mean and standard deviation for the SOFA, APACHE II, and SAPS II values

SOFA, Sequential Organ Failure Assessment; APACHE, Acute Physiology and Chronic Health Evaluation; SAPS II, Simplified Acute Physiology Score II.

Table 3.

Sensitivities, specificities and accuracy rates of the SOFA, APACHE II, and SAPS II scoring systems for predicting mortality

^*,†,‡P < 0.05 vs SOFA (McNemar's test with Boneferroni's correction) SOFA, Sequential Organ Failure Assessment; APACHE, Acute Physiology and Chronic Health Evaluation; SAPS II, Simplified Acute Physiology Score II.

The SMR values for SOFA, APACHE II, and SAPS II were 0.70 (95% confidence interval, 0.08-1.41), 0.76 (0.46-1.82), and 8.25 (1.53-16.62), respectively. The chi-square values for the Lemeshow-Hosmer goodness-of-fit statistic were 3.79 (P = 0.63), 7.71 (P = 0.47), and 8.62 (P = 0.26), respectively. The Hosmer-Lemeshow goodness-of-fit test revealed no significant differences between actual mortality and predicted mortality for each of the scoring systems.

DISCUSSION

The SOFA score was developed during a consensus conference organized by the European Society of Intensive Care and Emergency Medicine (8). The SOFA score calculates a summary value for the degree of dysfunction of six sets of organs (respiratory, coagulation, liver, cardiovascular, central nervous system, and renal). Four levels of dysfunction are identified for each of the organ systems in the SOFA score. Organ dysfunction is associated with high rates of ICU morbidity and mortality, and, as such, treatment for these disorders accounts for a high proportion of the ICU budget (13, 14). In this study, although SOFA scores were calculated on the day of admission, SOFA scores were useful for predicting the outcomes of OP poisoned patients who were admitted to ICU. Furthermore, this methodology is simple compared to the other scoring systems.

The APACHE II uses a point score based on initial values of 12 routine physiological measurements, patient age, and medical history to provide a general measure of disease severity in a patient. This system stratifies a wide variety of patients according to prognosis because of the strong and consistent relationship between acute physiological dysfunction and the risk of death due to acute illness (15-17). The APACHE II system serves as a useful index for evaluating the severity of poisoning due to multiple organ system involvement (10). In this study, APACHE II scores were useful for predicting the outcomes of OP poisoned patients; however, the APACHE II system is highly complex.

SAPS II was designed to measure disease severity in patients admitted to the ICU who are < 15 yr of age (7). A point score is calculated from 12 routine physiological measurements during the first 24 hr, information about health status, and some information obtained at admission. This calculation method results in a predicted mortality. Although SAPS II was useful for predicting the outcomes of OP poisoned patients in our study, it is relatively complex, similar to the APACHE II scoring system.

The two most popular systems introduced to date are APACHE II and SAPS II, which are based on multiple logistic regression equations describing abnormalities in physiological variables during the first 24 hr of ICU admission (7, 10). According to our results, we revealed strong calibration and discrimination power despite the correlation between APACHE II and SAPS II. In addition, studying the correlation between APACHE II/SAPS II revealed the actual death rate. Kim et al. (18) reported that APACHE II and SAPS could indicate the predictor of mortality in OP poisoned patients. However, SOFA scores were easy to calculate and useful for predicting the outcomes of ICU OP poisoned patients. The actual death rate revealed that a stronger relationship existed between SOFA scores and actual death rate, although a moderate correlation also existed for APACHE II and SAPS II.

The severity of OP poisoning differs depending upon the nature of the compound and its quantity, the way through which the exposure has occurred and the time at which treatment is initiated. As long as OP poisoning patients are treated appropriately, their mortality reduced. There have been reported on methods in the severity and prognosis of OP toxicity. In our study, SOFA scores were easy to calculate and useful for predicting the outcomes of ICU OP poisoned patients.

Our conclusions are limited by the single-center retrospective nature of the study, and the results may lack wider applicability due to missing data and the small sample size. In addition, we could not determine the influence of the time difference between the arrival of patients to the ED and the time they ingested the poison because it was a retrospective study. Therefore, we suggest that a multi-center or a randomized trial be conducted in the future to avoid these limitations and confirm our results.

In conclusion, the SOFA score is significantly associated with mortality and has strong discriminative power for predicting mortality. Considering the difficulty and complexity of measuring the APACHE II and SAPS II, the superiority of the SOFA score as a mortality predictor is understandable. The SOFA score is more useful than the APACHE II and SAPS II for predicting the outcomes of ICU OP poisoned patients, as the method for calculating SOFA scores is easier and simpler than that for APACHE II and SAPS II.