Evaluation and calibration of SAPS 3 in patients with COVID-19 admitted to intensive care units

Dear Editor,

Scoring systems are invaluable tools for research, quality assurance, and performance comparison in intensive care medicine. They allow for adjustment for underlying risk of unwanted outcomes, primarily mortality, and make comparisons between interventions, units, and systems possible. The Simplified Acute Physiology Score 3 (SAPS 3) is a well-validated scoring system used worldwide for the prediction of hospital mortality based on variables of acute physiologic derangements, current conditions and interventions, and previous health status [1, 2].

Coronavirus disease 2019 (COVID-19) put pressure on intensive care units (ICU), health care systems, nations, and societies worldwide. Measurement of health care performance is therefore more important than ever. Previous studies put the validity of other well-established intensive care scoring systems in COVID-19 cases in question [3], while others claimed good predictive capabilities [4]. Several outcome prognostication models for patient groups affected by COVID-19 have also been proposed. For these, a living systematic review and critical appraisal of the available literature finds C-index estimates for the prognostication of mortality to range from 0.68 to 0.98 [5]. In this study, we seek to evaluate the performance of SAPS 3 in the prediction of hospital mortality in COVID-19 patients admitted to ICUs.

Anonymous data from the Austrian Centre for Statistics and Documentation in Intensive Care (ASDI) database on patients with documented SARS-CoV-2 infection admitted to participating ICUs from January 1st, 2020, to January 31st, 2021, were retrieved and used for retrospective analyses. The anonymous fashion of the dataset precluded the need for ethical approval.

Discriminative performance of SAPS 3 for hospital mortality was evaluated by calculation of the area under the receiver operating curve (AUC) and 95% confidence intervals (95% CI) based on the DeLong approach [6]. Hosmer–Lemeshow test in deciles of observed-to-expected ratios (O/E ratios) and the calibration belt method [7] were used to assess goodness of fit of the formula to calculate predicted hospital mortality. Recalibration was conducted using the structure of the formula for regional customisations in ref. [2]; parameters were estimated by maximum likelihood method. Weights of individual SAPS 3 items remained unchanged. Analyses were conducted using R version 4.0.0 with packages pROC and givitiR.

1464 patients with COVID-19 admitted to 90 participating ICUs were identified (electronic supplement Table 1). Of these, 501 (34%) died during their hospital stay. AUC for discrimination of hospital mortality was 0.745 (95% CI 0.719–0.770). Standard calibration [2] led to under-estimation of hospital mortality [O/E ratio (95% CI) 1.20 (1.12–1.27), Ĥ = 41.10 (p < 0.001), Ĉ = 40.92 (p < 0.001)], especially in lower risk groups (Fig. 1, electronic supplement Fig. 2). Dedicated calibration for COVID-19 using the formula Probability of death = e^logit/(1 + e^logit), where logit = − 14.451 + 3.666 * ln(SAPS3 + − 12.092) led to improved goodness of fit [Ĥ = 7.15 (p = 0.71), Ĉ = 5.01 (p = 0.89)]] (Fig. 1, electronic supplement Fig. 3).

Fig. 1.

Binned plots of hospital mortality predicted by SAPS 3 in deciles (x-axis), observed mortality (primary y-axis) for COVID-19 calibration (left) and standard calibration (right) [dot-and-line graphs, unadjusted confidence intervals], number of patients (secondary y-axis) [bar graphs]

We find the SAPS 3 to be of satisfactory performance in the prognostication of hospital mortality in patients with COVID-19 admitted to intensive care units. Use of a general prediction model such as the SAPS 3 allows for the evaluation of outcomes in patient cohort, units, and systems irrespective of the underlying disease and is, therefore, preferable. Recalibration of the SAPS 3 can be used to allow for more precise performance evaluation in COVID-19 cohorts.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Conflicts of interest

MP reports institutional grants from the Austrian Center for Documentation and Quality Assurance in Intensive Care Medicine. TF reports his employment at the Center for Medical Statistics, Informatics, and Intelligent Systems to be funded by the Austrian Center for Documentation and Quality Assurance in Intensive Care Medicine.