SAPS-3 performance for hospital mortality prediction in 30,571 patients with COVID-19 admitted to ICUs in Brazil

Dear editor,

Severity of illness scores are used for benchmarking and assessment of adjusted mortality in intensive care units (ICU). The Acute Physiology and Chronic Health Evaluation (APACHE) and the Simplified Acute Physiology Score (SAPS) 3 scores were recently evaluated in ICU patients with coronavirus disease 2019 (COVID-19) with conflicting findings [1–4]. While in the cohorts from United Kingdom [1] and United States [2] ICU scores underestimated the actual mortality and poorly stratified disease severity, analyses from Austria using the SAPS-3 with first-level customization suggested satisfactory performance [3]. In this study, we aimed to evaluate the performance of SAPS-3 to predict hospital mortality in a large cohort of COVID-19 patients admitted to ICUs in Brazil.

We included all adult patients (> 16 years) with RT-PCR-confirmed SARS-CoV-2 infection admitted to 188 ICUs of 45 hospitals (Rede D’Or São Luiz) from February 26th, 2020, to April 30th, 2021. Anonymized information was obtained from an electronic system, which contains prospectively collected structured data for all ICU admissions (Epimed Monitor^®, Rio de Janeiro, Brazil).

In addition to the standard equation (SE), we obtained recalibrated probabilities for COVID-19 patients after performing a first-level customization of the SAPS-3 equation (Supplementary Methods). We assessed the discrimination for hospital mortality using the area under the receiver operating curve (AUROC) and Brier’s Score with 95% confidence intervals (95% CI). Calibration was evaluated using the Hosmer–Lemeshow goodness-of-fit (GOF) test and the calibration belt method [5]. R 4.1 was used for all analyses.

A total of 30,571 COVID-19 patients had complete hospital outcomes and were analyzed (Supplementary Table 1). Median age was 55 (interquartile range 42–69) years and 42% required advanced respiratory support. Overall, 4581 (15%) patients died in the hospital. Using the SAPS-3-SE, the predicted mortality was 15.7% and the standardized mortality ratio (SMR) was 0.95 (95% CI 0.93–0.98). Model`s discrimination was satisfactory (AUROC = 0.835 [95% CI 0.828–0.841]; Brier`s score = 0.097 [0.095–0.100]). However, the calibration was inappropriate for both SAPS-3-SE and the COVID-19-customized equation. Calibration belts and curves demonstrated underestimation of mortality in lower to intermediate risk groups and overestimation in higher risk group, which was unaffected by customization (Fig. 1; Supplementary sTable 2). These results were consistent when stratifying patients in three consecutive periods (Supplementary sTable 3 and sFigure 1).

Fig. 1.

Calibration assessment of SAPS-3 standard equation and COVID-19 customized equation. Panel a calibration belts for predicted probabilities; Panel b calibration curves with predicted (blue line) and observed (red line) mortalities

We compared the main results of the present study with previous studies (Supplementary sTable 4). Our results did not confirm that first-level customization improves the performance of SAPS-3 in predicting hospital mortality in ICU patients with COVID-19 as shown by others. Differences in models’ performance may be caused by differing admission policies and the timing of patient ICU admission. Our study may also not reflect the entirety of the Brazilian healthcare system as we included data from a private healthcare network with almost unrestricted access to ICU care. Regardless, our findings reinforce that standard severity of illness scores should be used with caution for mortality prognostication or benchmarking of ICU performance in COVID-19 patients. Moreover, our results highlights the need for proper calibration of these scores to estimate risk-adjusted metrics such as the SMR in this population. Further work is warranted to improve current severity scores or develop COVID-19-specific prognostic measures.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

PK and LSLB contributed equally. FAB, PK, LSLB, MS, JIFS contributed to the study conception and design, and data interpretation. LSLB, PK, and MS performed data processing and statistical analysis. LSLB, PK and MS drafted the first version of the manuscript. FAB and MS supervised the study. All authors had full access to data, participated in data interpretation, revised the manuscript, and approved the final version of the manuscript.

Funding

This study was supported by the National Council for Scientific and Technological Development (CNPq), the Coordination for the Improvement of Higher Education Personnel (CAPES)—Finance Code 001, Carlos Chagas Filho Foundation for Research Support of the State of Rio de Janeiro (FAPERJ), the Pontifical Catholic University of Rio de Janeiro, and by departmental funds from the D’Or Institute for Research and Education.

Availability of data and material

Limited data that support the findings of this study are available from the corresponding author, upon reasonable request.

Declarations

Conflict of interest

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. MS and JIF Salluh are founders and equity shareholders of Epimed Solutions^®, which commercializes the Epimed Monitor System^®, a cloud-based software for ICU management and benchmarking. The other authors declare that they have no conflict of interest.