Validation of ISARIC 4C Mortality and Deterioration Scores in a Mixed Vaccination Status Cohort of Hospitalized Coronavirus Disease 2019 (COVID-19) Patients in Singapore

Abstract

In this cross-sectional study, we studied performance of the International Severe Acute Respiratory and Emerging Infections Consortium mortality and deterioration scores in a cohort of 410 hospitalized patients (51.2% fully vaccinated). area under the receiver

operating characteristic curves were 0.778 and 0.764, respectively, comparable to originally published validation cohorts. Subgroup analysis showed equally good performance in vaccinated and partially or unvaccinated patients.

Coronavirus disease 2019 (COVID-19) resulting from infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has placed a significant burden on healthcare systems worldwide. To guide clinical management and resource allocation, numerous clinical prognostic models and risk-prediction scores have been developed to varying success. The International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) Coronavirus Clinical Characterization Consortium (4C) mortality score utilizes readily available clinical and laboratory parameters to easily calculate a numerical score at hospital presentation that stratifies patients to various mortality risk levels [1]. The group further extended this work to develop the 4C Deterioration model to prognosticate in-hospital clinical deterioration (requirement for ventilatory support and/or critical care or death) [2]. These scores (a full list of variables and scoring is shown in Supplementary Table 1) have been well validated in several other settings and widely adopted in clinical practice, including in Singapore.

However, since these scores were developed before widespread implementation of SARS-CoV-2 vaccination, their derivation and validation cohorts were based on only unvaccinated patients. Other published external validation studies also did not specifically look at vaccination status of the patients in their cohorts, as these were conducted prior to or at the beginning of vaccination programs [3].

With the increasing uptake of COVID-19 vaccination globally, there is a need to validate these risk scores in this population. COVID-19 vaccine breakthrough patients have a distinct phenotype, with a decreased risk of having severe disease, as well as altered viral kinetics [4]. Furthermore, the Delta variant of concern has overtaken almost all other variants and is associated with increased severity and viral loads [5]. Whether the ISARIC risk scores still perform well in this evolving era of vaccine breakthrough and Delta variant infections remains unknown as currently published studies were done prior to the Delta emergence. In this cross-sectional study, we examined a cohort of vaccinated and unvaccinated hospitalized patients with COVID-19 to externally validate the ISARIC 4C mortality and deterioration scores.

METHODS

We performed a single-time-point cross-sectional study of all admitted patients with COVID-19 on 27 October 2021 in the National Centre for Infectious Diseases (NCID), the national outbreak management center in Singapore. The inclusion criterion was confirmed COVID-19 infection as defined by positive SARS-CoV-2–specific polymerase chain reaction (PCR). There were no exclusion criteria. Baseline clinical and demographic data including laboratory and radiologic results on admission were collected using a standardized data-collection form to calculate baseline ISARIC mortality and deterioration scores. Vaccination status was defined as follows—(1) full: 2 doses at least 2 weeks before illness onset; (2) partial: 1 dose, or 2 doses less than 2 weeks before illness onset; or (3) no: zero doses. Outcomes were mortality or intensive care unit (ICU) admission. Data were censored upon discharge or at 4 weeks after admission. Data collection was conducted by appointed public health officers and approved by the Singapore Ministry of Health under the Infectious Diseases Act with waiver of informed consent, as part of a cross-sectional audit of hospitalized patients with COVID-19.

Patients were categorized based on worst outcomes of mortality, ICU admission, or neither. Baseline characteristics were compared using Fisher’s exact test for categorical variables and 1-way analysis of variance (ANOVA) or Kruskal-Wallis test for continuous variables, as appropriate. Performance of the ISARIC mortality and deterioration scores was evaluated by calculating the area under the receiver operating characteristic curve (AUC) using outcomes of mortality and mortality/ICU admission respectively. In the subgroup analysis, partially vaccinated patients were grouped together with unvaccinated patients. P values of less than .05 were considered significant, and all statistical tests were 2-tailed. Data analysis was done using STATA version 13.0 (StataCorp, College Station, TX) and plots were graphed using GraphPad Prism version 9.2 (GraphPad Software, San Diego, CA).

RESULTS

A total of 410 patients were admitted to the NCID on the date of the cross-sectional sampling. The mean age was 70.1 years (standard deviation [SD] ± 13.5 years), and 155 (37.8%) were female. A total of 210 (51.2%) were fully vaccinated, 45 (11.0%) partially vaccinated, and 155 (37.8%) not vaccinated. Seventy-one patients (17.3%) were admitted to the ICU during their admission, and 41 (10%) died. Baseline demographics, comorbidities, presenting symptoms, investigations, and parameters on admission of the entire cohort, and stratified by outcomes, are summarized in Supplementary Table 2.

The median ISARIC 4C mortality score was 8.5 (interquartile range [IQR]: 6–11), while mean ISARIC deterioration score was 356 (SD: ±160) for the entire cohort. ISARIC mortality and deterioration scores differed significantly among the 3 groups (P = .0001 for both; Supplementary Table 1). The ISARIC 4C mortality score performed well in this validation cohort, with an AUC of .778 (95% confidence interval [CI]: .698–.858; P < .0001) for all patients. Subgroup analysis of unvaccinated and vaccinated patients showed equally good performance with AUCs of .783 (95% CI: .688–.878; P < .0001) and .794 (95% CI: .655–.933; P = .0011), respectively (Figure 1A), with no significant difference between the 2 subgroups (AUC difference: .0108; standard error [SE] of difference: .0986; P = .91).

Figure 1.

Receiver operating characteristic curves of ISARIC 4C mortality (A) and deterioration (B) scores. ∗For all patients. Abbreviations: AUC, area under the receiver operating characteristic curve; CI, confidence interval; ICU, intensive care unit; ISARIC, International Severe Acute Respiratory and Emerging Infections Consortium; ROC, receiver operating characteristic curve; 4C, Coronavirus Clinical Characterization Consortium.

The ISARIC deterioration score performed equally well, with an AUC of .764 (95% CI: .711–.818; P < .0001) for the composite outcome of in-hospital deterioration (ICU admission or mortality) for all patients (Figure 1B). Similar subgroup analysis showed AUCs of .723 (95% CI: .649–.798; P < .0001) and .805 (95% CI: .722–.888; P < .0001) for unvaccinated and vaccinated subgroups, respectively, with no significant difference between the 2 subgroups (AUC difference: .0816; SE of difference: .0597; P = .17).

DISCUSSION

In this study, we validated published ISARIC 4C mortality and deterioration scores in a large cohort of hospitalized patients with mixed vaccination status, and showed equivalent performance in vaccinated and unvaccinated subgroups. Our reported AUC figures are comparable to those of the originally published models (.77 and .76 in the initial validation cohorts for mortality and deterioration scores, respectively). These scores have been well validated in multiple cohorts in diverse settings since their original publication, and have been found to consistently outperform other risk predictive scores in various study populations in Europe [6–8], North America [9], and Asia [10, 11]. The comparison of our results with these other validation studies is summarized in Supplementary Table 3. However, these were assessed in largely unvaccinated cohorts, and the applicability of these scores to vaccinated individuals remains questionable. While vaccinated individuals have a significantly lower risk of developing severe disease, high-risk individuals are still at risk of deterioration and mortality. As vaccine coverage continues to increase, risk scores that can accurately risk-stratify and identify these patients will be important. We provide data that the well-validated ISARIC scores can be similarly used to triage patients with vaccine breakthrough infection to guide patient disposition.

Another important variable of the pandemic that has changed since the original development and validation of the ISARIC risk models is the emergence of SARS-CoV-2 Delta variant, which subsequently became the predominant lineage worldwide. Delta variant infections are associated with higher viral loads and potentially greater disease severity [12]. Altered virulence may similarly alter the clinical course and risk factors of patients, with a previous study showing individuals infected with the Delta variant were younger and less likely to have comorbid conditions [13]. Therefore, validating existing risk scores to ensure they perform well in Delta variant infection is similarly important. During the study period, the Delta variant was the predominant circulating variant in Singapore, accounting for almost all confirmed infections [14]. Our study thus further validates the use of the ISARIC scores in a cohort of majority Delta variant infections, which may have been underrepresented in earlier validation studies.

There are several limitations to our study. First, the cohort comprised only hospitalized patients at a single center. Our findings may thus not be applicable to other outpatient or community settings. Second, healthcare services are readily available in Singapore and public health policy during the study period included active case finding with early referral to tertiary hospitals for evaluation of COVID-19 cases. In our cohort, the median day of illness at presentation was 3 days (IQR: 2–6 days) and this may have affected risk score calculations since patients were evaluated very early in their disease course, unlike in other settings where presentation to medical care may occur later. Nonetheless, we expect ISARIC scores to only increase with time (eg, through progression of inflammatory markers or development of chest radiographic opacities), and hence our finding that they perform well in early illness is reassuring. Last, we did not consider timing of vaccination and analyzed the entire vaccinated group as one. Recent data suggest that protective immunity from COVID-19 vaccine may wane after a period of time, with booster doses becoming increasingly deployed in many countries [15]. Only 4% of the vaccinated patients in our study received a third vaccine booster dose. Further studies to validate 4C in vaccinated cohorts, stratified by time from vaccination, and booster dose status, may be warranted.

Conclusions

We evaluated ISARIC 4C mortality and deterioration scores in a hospitalized cohort of mixed vaccination status in Singapore and showed equally good performance in both vaccinated and unvaccinated subgroups. Our data support the continued use of these scores in the era of increasing vaccine coverage and vaccine breakthrough infections.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Author contributions. S. W. X. O. has full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Potential conflicts of interest. B. E. Y. reports personal fees from Roche and Sanofi, outside the submitted work. All other authors report no potential conflicts. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.