Comparison of different risk stratification systems for prediction of acute pancreatitis severity in patients referred to the emergency department of a tertiary care hospital

ABSTRACT

BACKGROUND:

Prognostic prediction and estimation of severity at early stages of acute pancreatitis (AP) are crucial to reduce the complication rates and mortality. The objective of the present study is to evaluate the predicting ability of different clinical and radiological scores in AP.

METHODS:

We retrospectively collected demographic and clinical data from 159 patients diagnosed with AP admitted to Canakkale Onsekiz Mart University Hospital between January 2017 and December 2019. Bedside index for severity AP (BISAP), and acute physiology and chronic health evaluation II (APACHE II) score at admission, Ranson and modified Glasgow Prognostic Score (mGPS) score at 48 h after admission were calculated. Modified computed tomography severity index (CTSI) was also calculated for each patient. Area under the curve (AUC) was calculated for each scoring system for predicting severe AP, pancreatic necrosis, length of hospital stay, and mortality by determining optimal cutoff points from the (ROC) curves.

RESULTS:

mGPS and APACHE II had the highest AUC (0.929 and 0.823, respectively) to predict severe AP on admission with the best specificity and sensitivity. In predicting mortality BISAP (with a sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV) of 75.0%, 70.9%, 98.2%, and 12.0%, respectively, [AUC: 0.793]) and APACHE II (with a sensitivity, specificity, NPV and PPV of 87.5%, 86.1%, 99.2%, and 25.0%, respectively, [AUC: 0.840]).

CONCLUSION:

mGPS can be a valuable tool in predicting the patients more likely to develop severe AP and maybe somewhat better than BISAP score, APACHE II Ranson score, and mCTSI.

INTRODUCTION

Acute pancreatitis (AP), which refers to the inflammation of pancreatic tissue with increased rates of morbidity and mortality, is one of the major causes of emergency department (ED) admissions worldwide.[1,2] The incidence of AP has increased dramatically in recent years in many Asian and European countries and throughout the USA, with an incidence of 36.4 in 2010 compared to 27.6 per 100,000 in 1999.[3] Local and systemic complications including pleural effusions, pancreatic pseudocyst, pancreatic abscess, and necrosis are the major causes of negative outcomes related to AP. These complications are well defined in several large level studies and mostly linked to severe inflammatory response causing systemic inflammatory response syndrome (SIRS), multi-organ failure (MOF), and pancreatic tissue necrosis. This inflammatory response is an active and complex process in which activation of pancreatic digestive enzymes leads to auto digestion of the normal pancreatic parenchyma with an unpredictable patient response.[4] In the majority of cases, this response is mild with a self-limiting pancreatic and systemic inflammation, but 10–25% of the cases develop severe disease progressing to SIRS, MOF, and death.[5] Although detection of severe cases in the early stages significantly reduces morbidity and mortality in AP patients in ED settings, the greatest challenge seems to be the difficulty in recognizing mechanisms that induce the switch from mild to severe AP and at what point it occurs.[1] For this reason, multiple radiologic and clinical scoring systems have been proposed for evaluating the severity and prediction of complications related to AP.

In emergency clinics, it is important to identify patients at high risk of developing severe AP in the initial stages to begin early treatment, optimize medical therapy, and improve patient outcomes. To achieve these goals, a number of severity scoring systems, including the modified Glasgow prognostic score (mGPS), bedside index for severity in AP (BISAP) score, Ranson’s score, sequential organ failure assessment score, acute physiology and chronic health evaluation II (APACHE II) scoring system, simplified acute physiology score II, mortality probability model II, and computed tomography severity index (CTSI), have been proposed to predict the severity and prognosis of AP.[6–8] However, potential drawbacks of these scoring systems limit their general use in clinical practice. For instance, the Ranson and mGPS scoring systems can only be calculated after 48 h of inpatient follow-up, while BISAP is reported to have a lower sensitivity for prediction of mortality and severity of AP. The APACHE II scoring system is very complex to use because it has a large number of variables that clinicians find difficult to remember.[9–12] CTSI is regarded as a valid and usable tool for staging the severity of AP. It was first introduced by Balthazar[6] in 1990 and was found to correlate with the clinical course and prediction of mortality in AP patients. The modified CTSI is an extension of the original CTSI and was developed to surpass the potential limitations of the original CTSI. A study by Bollen et al.[13] demonstrated that modified CTSI was superior to the original CTSI for evaluating AP severity.

This retrospective study was designed with the aim of evaluating and comparing the early predictive potential of various risk stratification systems in AP such as BISAP, mGPS, APACHE II, Ranson, and modified CTSI.

MATERIALS AND METHODS

This retrospective cross-sectional study was performed after obtaining Institutional Review Board approval (No: 2011 KAEK-27/2020-E.2000070223) for analyzing hospital medical records through Canakkale Onsekiz Mart University (COMU) Hospital’s Information and Management System (HIMS). The ED of COMU Training and Research Hospital’s records between January 2017 and December 2019 were included in this study. The diagnosis of AP was made according to the revised definition of Atlanta 2012 criteria.[14] Based on these criteria, AP diagnosis was made if a minimum two of the following three criteria were present: (i) Severe abdominal/epigastric pain often radiating to the back, (ii) serum lipase/amylase level 3 times normal, and (iii) characteristic findings of AP on CT or magnetic resonance imaging. Based on the revised Atlanta 2012 classification, patients were categorized as having mild, moderately severe, and severe AP at the time of discharge or in-hospital death. Mild AP is defined as the nonexistence of organ failure and local/systemic complications, whereas severe AP is characterized by the existence of organ dysfunction after 48 h. Moderately severe AP is defined as the presence of transient organ dysfunction or local or systemic complications without persistent organ failure. The definition of organ failure was based on the modified Marshall scoring system.[15] Patients excluded from the study included patients with other causes of hyperamylasemia, use of immunosuppressive drugs, patients with a malignancy, incomplete records or those with a doubtful diagnosis, uremia, cardiac failure, and patients with chronic pancreatitis.

Demographic and laboratory information, including length of stay (LOS) in hospital, of all enrolled patients were collected from HIMS and recorded. Clinical, vital and laboratory parameters, including complete blood cell counts, renal and liver function tests, serum electrolytes, and arterial blood gas analysis, were also recorded. BISAP and APACHE II score at admission, and Ranson and mGPS score at 48 h after admission were calculated according to the international guidelines. Modified CTSI was calculated from the extent of pancreatic inflammation, existence of pancreatic necrosis and extra-pancreatic complications.

Statistical Analysis

The Statistical Package for the Social Sciences 20.0 for Windows was used to analyze the data. Categorical variables were expressed as numbers and percentages (%), whereas continuous variables were summarized as mean±standard deviation. The diagnostic accuracy of scoring systems was assessed for each outcome variable using empirical receiver operating characteristic (ROC) analysis. Sensitivity, specificity, area under the curve (AUC) with 95% confidence intervals (CI), positive/negative predictive values (PPV/NPV), and optimal cutoff values were calculated for each scoring system according to mortality, complications, and severe pancreatitis. The optimal cutoff point was determined by maximizing the sum of sensitivity and specificity. All statistical tests had a statistical significance level of p<0.05 (two-tailed).

RESULTS

For the present study, 159 patients with AP and a mean age of 68.6±15.9 years (M/F: 62/97) were enrolled. Of these patients, 24 (15.1%) were classified as severe AP according to the revised Atlanta classification and eight (5%) patients died after a 1-month follow-up. Table 1 lists the number of patients with standardized cutoffs of APACHE II, BISAP, mGPS, Ranson, and modified CTSI to predict AP severity. CT findings revealed a pancreatic necrosis rate of 6.9%, pancreatic fluid collection rate of 4.4%, and abscess rate of 2.5%. Demographic, clinical, and laboratory characteristics of study participants are also presented in Table 1. The most common etiology of AP was biliary (78%) origin followed by alcoholic (12%) and others (10%).

Table 1.

Demographic, clinical and laboratory characteristics of study participants (n=159)

Characteristics	Value
Age (years)	68.6±15.9
Male/female, n(%)	62 (39)/97 (61)
Initial laboratory
Hemoglobin (g/dl)	12.67±1.98
WBC (/mm3x103)	12.49±5.13
Plt (/mm3x103)	243.55±93.90
ALT (U/l)	147.91±164.33
AST (U/l)	177.07±191.12
BUN (mg/dl)	38.04±19.79
Creatinin (mg/dl)	0.98±0.73
Calcium (mg/dl)	9.11±0.79
Albumin (g/dl)	3.83±0.56
CRP (mg/l)	7.70±7.95
Atlanta 2012, n (%)
Mild	83 (52.2)
Moderately severe	52 (32.7)
Severe	24 (15.1)
APACHE II (within 72 hours), n (%)
≥8	46 (28.9)
<8	113 (71.1)
BISAP score, n (%)
≥3	17 (10.7)
<3	142 (89.3)
Imrie score, n (%)
≥3	36 (22.6)
<3	123 (77.4)
Ranson score, n (%)
≥3	60 (37.7)
<3	99 (62.3)
Modified CTSI
>4	9 (5.7)
≤4	150 (94.3)
Complications, n (%)
Necrosis	11 (6.9)
Pleural effusion	14 (8.8)
Pancreatic fluid collection	7 (4.4)
Pancreatic abscess	4 (2.5)
2 and more complications	3 (1.9)
Median Length of stay (days) (IQR)	5 (4–7)
Mortality, n (%)	8 (5)

WBC: White blood cells; Plt: Platelet; ALT: Alanine aminotransferase; AST: Aspartate aminotransferase; BUN: Blood urea nitrogen; CRP: C-reactive protein; APACHE II: Acute Physiology And Chronic Health Evaluation II; BISAP: Bedside index for severity acute pancreatitis; CTSI: Computed tomography severity index; IQR: Inter Quantile Range.

The area under the receiver operating curves (AUROCs) of BISAP, RANSON, mGPS, APACHE II, and modified CTSI for predicting severe AP are outlined in Table 2. The predictive value of each of these scoring systems for adverse complications and mortality according to ROC analysis is presented in Table 3. To achieve the highest performance of the predicting variables, optimal cutoff values were obtained from the ROC curves. Graphical representation of ROC curves with the respective AUC to compare severity, complications, and mortality is provided in Figure 1.

Table 2.

Receiver Operating Characteristics (ROC) of different scoring systems predicting severe acute pancreatitis according to revised Atlanta criteria

Scoring system	Cut-off	AUC (95% CI)	Sensitivity (%)	Specificity (%)	NPV (%)	PPV (%)	OA (%)
BISAP	2	0.759 (0.655–0.863)	75.0	61.5	93.3	25.7	63.5
mGPS	3	0.929 (0.884–0.974)	87.5	88.9	97.6	58.3	88.7
Ranson	2	0.766 (0.664–0.868)	91.7	38.5	96.3	21.0	46.5
APACHE II	8	0.823 (0.733–0.912)	70.8	78.5	93.8	37.0	77.4
Modified CTSI	2	0.818 (0.719–0.918)	75.0	74.1	34.0	94.3	74.2

AUC: Area under curve; CI: Confidence interval; NPV: Negative predictive values; PPV: Positive predictive values; OA: Overall accuracy; BISAP: Bedside index for severity acute pancreatitis; mGPS: Modified Glasgow Prognostic Score; APACHE II: Acute Physiology And Chronic Health Evaluation II; CTSI: Computed tomography severity index.

Table 3.

Predictive value of each scoring systems for adverse outcomes and mortality according to Receiver Operating Characteristics (ROC) analysis

	Cut-off	AUC (95% CI)	Sensitivity (%)	Specificity (%)	NPV (%)	PPV (%)	OA (%)
Adverse outcomes
BISAP	2	0.689 (0.588–0.789)	60.6	60.3	85.4	28.6	60.4
mGPS	2	0.755 (0.663–0.847)	81.8	56.4	92.2	32.9	61.6
Ranson	3	0.661 (0.559–0.763)	64.6	66.7	84.9	30.0	64.2
APACHE II	7	0.793 (0.713–0.873)	84.6	59.5	93.8	35.4	64.8
Modified CTSI	2	0.957 (0.927–0.988)	97.0	83.3	99.1	60.4	86.2
Mortality
BISAP	2	0.793 (0.907–0.979)	75.0	70.9	98.2	12.0	71.1
mGPS	2	0.737 (0.576–0.898)	87.5	63.6	99.0	11.3	64.8
Ranson	3	0.591 (0.395–0.788)	60.0	76.2	96.6	10.0	74.8
APACHE II	9	0.840 (0.654–1.000)	87.5	86.1	99.2	25.0	86.2
Modified CTSI	2	0.730 (0.607–0.942)	87.5	82.8	99.2	21.2	83.0

AUC: Area under curve; CI: Confidence interval; NPV: Negative predictive values; PPV: Positive predictive values; OA: Overall accuracy; BISAP: Bedside index for severity acute pancreatitis; mGPS: Modified Glasgow Prognostic Score; APACHE II: Acute Physiology And Chronic Health Evaluation II; CTSI: Computed tomography severity index.

Figure 1.

Graphical representation of ROC curves with the respective area under the curve to compare (a) severity, (b) adverse outcomes, and (c) mortality.

With regard to the prediction of severe AP on admission (Table 2), mGPS and APACHE II had the highest AUC (0.929 and 0.823 respectively) with the best specificity and sensitivity. The overall accuracy of mGPS and APACHE II in determination of severe AP was 88.7% and 77.4%, respectively.

In predicting mortality (Table 3) BISAP (with a sensitivity, specificity, NPV, and PPV of 75.0%, 70.9%, 98.2%, and 12.0%, respectively, [AUC: 0.793]) and APACHE II (with a sensitivity, specificity, NPV and PPV of 87.5%, 86.1%, 99.2%, and 25.0%, respectively, [AUC: 0.840]) were again the best predictive parameters. Due to the low mortality rate in this study, NPV values were high, as were the rest of the variables, but PPV was considerably lower. The outcomes of complications on admission in terms of sensitivity and specificity with AUC=0.957, reaching a NPV and PPV of 99.1% and 60.4%, respectively.

DISCUSSION

In this study, severe AP is defined according to the revised Atlanta classification proposed in 2012 and severity, mortality, and LOS in a large AP patient cohort admitted to the ED is evaluated. The results of this study demonstrate that the scoring systems, including mGPS, APACHE II, BISAP, Ranson, and CTSI, are reliable predictors of severity and mortality in AP patients. Based on the AUROC (0.929), mGPS was the most reliable and effective scoring system to predict severity and mortality in AP patients. Moreover, using a cutoff of 9 (AUROC: 0.840), the results indicate that the APACHE II score is the most significant and feasible parameter to predict LOS in hospital.

AP is a frequent diagnosis encountered in EDs all over the world by physicians of several specialties. It comprises a heterogeneous clinical manifestation ranging from minimal pancreatic tissue inflammation, which usually resolves spontaneously in 2–6 days without developing organ dysfunction, to extensive pancreatic necrosis with increased mortality rates in 10–25% of patients.[5] It is, therefore, crucial to identify those patients who might progress to severe AP so that aggressive drug therapies can be applied or triaging in appropriate categories as per internationally accepted protocols can be undertaken.[16]

The BISAP score is a simple, convenient, prognostic, and multi-factorial scoring system that provides valuable data that can prioritize or modify a patient’s clinical care or triaging to a regular ward or ICU. Although several studies have evaluated the predictive value of BISAP, the majority are of limited size and suffer from methodological shortcomings.[8,17,18] Moreover, some of these studies have not used the revised Atlanta classification for severity estimation.[19] Therefore, this study of great value as is demonstrates the importance of the BISAP score in estimating not only the severity of AP but also LOS, pancreatic necrosis and mortality according to the revised definition of Atlanta criteria. A recent study from Spain demonstrated similar results to those reported here that BISAP is the best predictor on admission for severe AP, mortality, and ICU admission with an AUC of 0.9 (95% CI 0.83–0.97), 0.97 (95% CI 0.95–0.99), and 0.89 (95% CI 0.79–0.99), respectively.[5]

In a study by Arif et al.,[20] accuracy of the BISAP score in comparison with Ranson’s score in detection of AP severity was investigated. The authors demonstrated that the accuracy of the BISAP score with a cutoff value of ≥3 for predicting severe AP was 76.2% with a kappa value of 0.34. Kim et al.[21] reported a higher sensitivity of 84% for BISAP with a lower cut-off value of 2 in predicting severe AP.

In this study, of 159 patients, 24 (15.1%) were found to have severe AP according to the revised Atlanta criteria. According to BISAP, 17 of 159 patients had severe AP. Therefore, with a cutoff value of 2, the sensitivity of BISAP to predict severe AP is 75%.

The AUROC for modified CTSI score was one of the highest for each of the three variables considered as indicators for severity of AP, namely, severe AP (0.818), complications (0.957) and death (0.850). As modified CTSI is a more recently introduced scoring system compared to original CTSI, it is not surprising to see more study designs including CTSI rather than modified CTSI. In this context, Kumar et al.’s[22] study adds valuable new information demonstrating that modified CTSI is consistently the highest for predicting severe AP (AUC=0.919), pancreatic necrosis (AUC=0.993), organ dysfunction (AUC=0.893), and intensive care admission (AUC=0.993) compared to APACHE II. However, a study by Yang et al.[23] reported outstanding performance of modified CTSI in predicting local complications, poor prediction of AP severity and mortality, suggesting that modified CTSI needs to be improved for different methods.

APACHE II is a reliable and effective scoring system to predict the adverse outcomes and mortality rate in patients with severe AP.[24,25] It is a composite risk stratification score with an extensive range of clinical and laboratory parameters that often provides insight into the ongoing pathophysiology and upholds accurate predictions of adverse outcomes.[26]

The Ranson system is also a commonly used scoring tool designed for the risk stratification of AP but unlike BISAP and APACHE II, which can be calculated at any time point after admission, the Ranson is a two-step risk stratification system and can accurately be calculated at 48 h after hospital admission. This study demonstrates that APACHE II and Ranson have a high sensitivity and NPV for predicting adverse complications including pancreatic necrosis (84.6% and 93.8% for APACHE II; and 64.6% and 84.9% for Ranson) and death (87.5% and 99.2% for APACHE II; and 60.0% and 96.6% for Ranson), which makes them both ideal scoring tools for decision-making regarding referral to bigger centers.

Moreover, based on the AUROC comparisons in this study, APACHE II (0.823) and Ranson (0.766) scores were comparable to BISAP and modified CTSI in terms of predicting the severity of AP. In one of the few similar studies using APACHE II and Ranson in predicting severity of AP, Kumar et al.[22] found that APACHE II was comparable to Ranson and modified CTSI in terms of severity of AP. Conversely, based on the AUROC, APACHE II was found to be significantly higher than the BISAP score in predicting the severity of AP. Similar comparable results between APACHE II and Ranson scores are demonstrated by Zhang et al.[8] and Venkatesh et al.,[27] and more completely in a study by Mounzer et al.[28] in which the authors compare the accuracy of several scoring systems in predicting persistent organ dysfunction and further develop rules that combine these scores to optimize predictive accuracy.

The findings of this study have some limitations, such as the fact that it is a single-center study with a relatively limited sample size. In addition, the retrospective, nonrandomized, and single-center design of this study may result in selection bias. Furthermore, the study population only consists of biliary pancreatitis; therefore, no meaningful comparisons can be made among the diverse prognostic scoring systems for different AP etiologies.

Conclusion

Although all scoring systems perform well in predicting severe AP, it can be safely said that mGPS can be a valuable tool in predicting the patients who are more likely to develop severe AP and possibly somewhat better than other scoring systems.