Evaluation of APACHE-IV Predictive Scoring in Surgical Abdominal Sepsis: A Retrospective Cohort Study

Abstract

Introduction

Evaluation of the effectiveness of care and clinical outcomes in critically ill patients is dependent on predictive scoring models that calculate measures of disease severity and an associated likelihood of mortality. The APACHE scoring system is a logistic regression model incorporating physiologic and laboratory parameters. APACHE-IV is the most updated scoring system for ICU mortality prediction. However, APACHE scores may not accurately predict mortality in patients who require surgery for abdominal sepsis, whose trajectory is modulated by source control procedures.

Aim

To evaluate the accuracy of APACHE-IV mortality prediction in a cohort of ICU patients with surgical abdominal sepsis (SABS) requiring emergent laparotomy for source control.

Materials and Methods

The study was conducted in a combined medical and surgical intensive care unit in a large urban Canadian tertiary care hospital. Retrospective review of 211 consecutive adult ICU admissions that fulfilled the 2012 ACCP/SCCM criteria for severe sepsis/septic shock due to abdominal source was performed. APACHE-IV score and predicted mortality rate (PMR) were calculated and evaluated using area under the ROC curve (AUROC).

Results

Overall in-hospital mortality was 28.4%. There was overestimation of PMR by the APACHE-IV model in the overall cohort with an absolute difference of 16.6% (relative difference 36.9%). APACHE-IV crudely distinguished between survivors and non-survivors, with a PMR of 40% vs. 59% (p<0.001). AUROC of the APACHE-IV score was 0.67, 95% CI (0.58, 0.76) while the AUROC for the PMR was 0.72, 95% CI (0.64, 0.80), indicating poor performance in this cohort.

Conclusion

APACHE-IV has poor discrimination in SABS. Future research should explore disease-specific prediction models.

Introduction

Sepsis accounts for approximately 20% of Intensive Care Unit (ICU) admissions [1]. Despite critical care advances [2,3] the mortality rate for sepsis in general surgery patients has remained around 30% over the last two decades [4]. Trials in sepsis tend to combine medical and surgical patient populations, with multiple anatomic sites of infection [5], despite evidence that anatomic site is strongly and independently associated with death even after adjusting for disease severity [6]. While such trials contribute to our understanding of overall goals of resuscitation, there is a need for more diagnosis-specific sepsis research, particularly as the clinical trajectory of surgical patients is modulated by source control procedures.

Evaluation of the effectiveness of care and clinical outcomes in critically ill patients is dependent on predictive scoring models that calculate measures of disease severity and an associated likelihood of mortality. The APACHE scoring system is a logistic regression model incorporating physiologic and laboratory parameters. It is a widely used ICU stratification tool that is considered a highly accurate predictor of mortality. Model accuracy degrades over time and periodically requires updating. A 2012 study deemed APACHE-III performance inadequate even with a predicted mortality of only 2% higher than the observed mortality rate (16% vs. 14%) [7]. Most recently, APACHE-IV was introduced in 2006 and was derived from 110,558 American ICU admissions [8].

The APACHE system is used to benchmark ICU performance, compare disease-specific mortality ratios, and predict individual patient mortality. No prior studies have specifically evaluated the accuracy of APACHE-IV in a subset of patients undergoing surgery for management of abdominal sepsis. We hypothesize that the APACHE-IV prediction model is an inadequate predictor of mortality in the surgical abdominal sepsis (SABS) population.

Materials and Methods

This study was conducted in accordance with institutional ethical standards and with the Helsinki Declaration of 1975 and its subsequent revision in 2000.

Study design

Retrospective review of consecutive adult ICU admissions with the diagnosis of “open abdomen” or “abdominal sepsis” occurring between January 1, 2006 and December 31, 2010 at a provincial referral hospital’s combined medical and surgical ICU. Data collection was performed between July 2011-December 2013. All admissions were screened by trained study investigators. Admissions that met inclusion criteria were reviewed in full. No follow up data were obtained after hospital discharge.

APACHE-IV scores and predicted mortality rates (PMR) were independently calculated using the Cerner protocol [9]. Score generation based on this protocol utilizes parameters recorded within the ICU during the first 24 hours of a patient’s admission, after a minimum 4-hour stay in ICU.

Inclusion criteria

SABS cases were defined as patients with evidence of preoperative severe sepsis or septic shock with a suspected or known abdominal source of infection requiring laparotomy for source control (SC) surgery. Severe sepsis was defined as meeting at least one criterion of systemic inflammatory response syndrome (WBC <4 or>12x10³/μL, Temperature<36 or >38.2°C, HR >90bpm, RR>20/min) along with evidence of organ dysfunction (altered mental status, arterial hypoxaemia with Pa0₂/Fi0₂<300, urine output <0.5 ml/kg/hr, creatinine increase >0.5mg/dL, ileus, platelets<100,000/μL), hypo-perfusion (lactate>1mmol/L) or hypotension (systolic BP<90mmHg) responsive to fluid resuscitation. Septic shock was defined as hypotension non-responsive to fluid challenge (30mL/kg bolus), MAP<60mmHg, or use of vasopressors (2012 ACCP/SCCM criteria). Diagnosis of an abdominal source was based on a combination of imaging, clinical history and examination, and assessment by a general surgery consultant. ICU admission was mandatory in order to generate an APACHE score. We included all patients who were admitted to the ICU as a result of SABS, secondarily developed SABS while admitted for another primary indication (e.g. respiratory failure), or had elective abdominal surgery that subsequently resulted in a septic complication requiring surgery.

Exclusion criteria

Patients less than 18 years of age, laparoscopy without conversion to laparotomy, sepsis secondary to trauma, laparotomies for non-septic indications, and abdominal sepsis without laparotomy. Cases deemed non-survivable during or immediately after the initial SC laparotomy (e.g. global bowel ischemia) based on physician notes were also excluded.

Statistical Analysis

Descriptive statistics were performed on demographic parameters. Continuous variables were expressed as means with 95% confidence intervals or standard deviations where appropriate. APACHE scores and outcome variables were compared between survivors and non-survivors using Student’s t-test, with a significance threshold of p<0.05. Accuracy of APACHE-IV was assessed using the Area Under the Receiver Operator characteristic Curve (AUROC) with a designation of ‘good’ being an AUROC of >0.80. All statistical analyses were carried out using SPSS [10].

Results

Demographics

A total of 211 patients fulfilled inclusion criteria. Demographic data and aetiologies of sepsis for the cohort are summarized in [Table/Fig-1]. There was a significant difference in age between survivors (60.7 years, 95% CI 58.3, 63.2) and non-survivors (67.9 years, 95% CI 64.1, 71.6). Most common aetiologies of SABS included large bowel perforation (15.6%), small bowel perforation (14.7%), and bowel ischemia (14.7%).

[Table/Fig-1]:

Demographics of SABS Cohort.

	Overall (n=211)	Survivors (n=151)	Non-survivors (n=60)
Age (mean ± SD)	62.8 ± 15.2	60.7 ± 15.0	67.9 ± 14.5
Female (%)	47.9	50.3	41.2
Underlying metastatic cancer, leukaemia, or lymphoma (n)	28	17	11
Aetiology n (% within category)
Large bowel perforation	33 (15.6)	24 (15.9)	9 (14.8)
Small bowel perforation	31 (14.7)	21 (13.9)	10 (16.4)
Ischemia/infarction	31 (14.7)	20 (13.2)	11 (18)
Anastamotic failure	30 (14.2)	22 (14.6)	8 (13.1)
C. difficile Colitis	14 (6.6)	8 (5.3)	6 (9.8)
Abscess (intra-abdominal)	13 (6.2)	12 (7.9)	1 (1.6)
Other (multiple sources, gynecologic, fecal impaction, iatrogenic)	12 (5.7)	10 (6.6)	2 (3.3)
Small bowel obstruction	9 (4.3)	6 (4.0)	3 (4.9)
Necrotizing Pancreatitis	9 (4.3)	7 (4.6)	2 (3.3)
Gastric perforation	7 (3.3)	6 (4.0)	1 (1.6)
Abdominal wall	6 (2.8)	5 (3.3)	1 (1.6)
Biliary complication	6 (2.8)	2 (1.3)	4 (6.6)
Large bowel obstruction	4 (1.9)	3 (2.0)	1 (1.6)
Typhlitis	2 (0.9)	2 (1.3)	0
Fistula	3 (1.4)	6 (4.0)	1 (1.6)
Ulcerative colitis	1 (0.5)	1 (0.7)	0

Evaluation of APACHE-IV

Outcome data are summarized in [Table/Fig-2]. Overall in-hospital mortality was 28.4%. There was overestimation of PMR by the APACHE-IV model in the overall cohort with an absolute difference of 16.6% (relative difference 36.9%). APACHE-IV crudely distinguished between survivors and non-survivors, with mean survivor score of 82 compared to 100 for non-survivors, with a PMR of 40% vs. 59% (p<0.001). ROC curves for the APACHE-IV score and PMR are depicted in [Table/Fig-3]. AUROC of the APACHE-IV score was 0.67, 95% CI (0.58, 0.76) while the AUROC for the PMR was 0.72, 95% CI (0.64, 0.80). While both of these values are significantly better than chance, the overall performance of the APACHE-IV score and PMR was poor, below the 0.80 threshold for a “good” discriminatory test.

[Table/Fig-2]:

Outcome measures for SABS cohort.

	Overall (n=211)	Survivors (n=151)	Non-survivors (n=60)	p-value
APACHE-IV score (mean ± SD)	87.7 ± 24.1	82.7 ± 19	100.4 ± 30.3	<0.001
APACHE-IV PMR (mean % ± SD)	45 ± 23.6	39.6 ± 20.5	58.6 ± 25.6	<0.001
ICU LOS (mean days ± SD)	13.5 ± 14.7	12.8 ± 12.4	15.2 ± 19.5	NS
Hospital LOS (mean days)	54.3 ± 56.9	60.4 ± 61.3	39.0 ± 40.3	0.01 NS
≤ 28 Day OM n (%)	40 (19.0)	-	-	-
In-hospital OM n (%)	60 (28.4)	-	-	-

[Table/Fig-3]:

ROC Curve for APACHE-IV PMR and APACHE-IV score.

ROC curve for APACHE-IV PMR (dashed line) AUROC=0.72, 95% CI [0.64, 0.80]; and APACHE-IV score (solid black line), AUROC=0.67, 95% CI [0.58, 0.76].

Discussion

Multiple factors may contribute to APACHE-IV’s poor performance in SABS. In the derivation cohort, APACHE-IV had an AUROC of 0.88, indicating good model performance. However, only 2% of the derivation cohort was categorized as a postoperative gastrointestinal surgery patient, which included non-sepsis indications such as transplantation and malignancy [8]. The small proportion of SABS patients may limit model applicability to a SABS cohort.

Only 1290 of the 110, 000 patients in the derivation study were categorized upon ICU admission into operative diagnostic groups with a sepsis indication (e.g. perforation, obstruction, vascular ischemia). Non-operative diagnostic groups also included obstruction, perforation, and ischemia, though it is not within the ability of the APACHE model to specify whether or not these patients went on to require operative management. Therefore, physiologic response to surgical management is not incorporated in mortality prediction.

Since the PMR is calculated based on values from the initial 24 hours of ICU admission, the patient’s physiologic state captured by the APACHE score may vary depending on the extent to which the patient has been treated for their surgical disease. Patients who undergo surgery before ICU admission or were admitted to the ICU for less than four hours prior to surgery do not generate APACHE variables until after their operation. The PMR in this scenario would be representative of a patient’s postoperative status. Their physiologic state and resultant PMR thus reflect substantial treatment interventions. In contrast, a patient admitted to the ICU for at least four hours prior to surgery will begin generating their APACHE variables. The PMR in this latter scenario is reflective of a preoperative state. This may result in greater physiologic derangements due to their uncontrolled septic source. Since parameters measured outside of the ICU are not incorporated into the score, patients who undergo surgery during this initial window of score generation may have important information omitted.

Accurately quantifying a patient’s response to SC also requires dynamic evaluation of the patient, while APACHE-IV is not designed for sequential scoring. Dynamic scoring systems based on APACHE-II and APACHE-III have been evaluated by multiple groups, though the accuracy of prediction has been plagued by low sensitivity ranging from 15-43% [11]. A recent prospective study with 541 patients compared the PIRO, APACHE-II, and MODS score in severe sepsis, with an AUROC of 0.71, 0.71 and 0.63 respectively [12]. In patients with septic shock secondary to abdominal sepsis, the SOFA, SAPS-II, and MODS on day one of ICU admission have been found to be non-predictive of mortality [13].

Limitations

The retrospective design is prone to selection bias due to potential miscoding of the ICU database and missing data. However, we took steps to ensure that our selection criteria were strict in terms of defining the physiologic state of SABS patients according to consensus definitions for severe sepsis/septic shock. We recognize that “abdominal sepsis” is a heterogeneous label for a collection of conditions that may have differential implications on mortality. However, our cohort of 211 patients is the largest reported study of patients with severe sepsis/septic shock due to abdominal sepsis managed surgically, which provides a level of uniformity that has not been previously documented.

Conclusion

Our results suggest that APACHE-IV is a poor predictor of mortality in the SABS population. The relationship between the pathophysiology of SABS, response to surgical therapy, and mortality are not reflected by the APACHE-IV model. SABS-specific models are likely required to improve accuracy of outcome predictions.

Financial or Other Competing Interests

None.