External validation of the PRIORITY model in predicting COVID-19 critical illness in vaccinated and unvaccinated patients

Abstract

Objectives

Predictive scores are important tools for the triage of patients with coronavirus disease 2019. The PRIORITY score is advantageous because it does not require laboratory and radiologic information. However, the original development and validation cohorts studied only unvaccinated patients in early 2020. We aimed to externally validate the PRIORITY score in a cohort of patients with the novel delta and omicron variants of coronavirus disease 2019 and mixed vaccination status.

Methods

A total of 410 patients were included in a cross-sectional sampling of all patients admitted to the National Centre of Infectious Diseases on October 27, 2021. A further 102 and 136 patients with vaccine-breakthrough Delta and Omicron variant infection from April to August and December 2021, respectively, were also included. Variables at the time of admission were collected retrospectively from medical records and used to calculate the probability of deterioration using the PRIORITY model.

Results

Of the total 648 included patients, 447 (69.0%) were vaccinated. The mean age was 61.6 years (standard deviation ± 19.0 years), and 268 patients (41.4%) were female. A total of 112 patients (17.3%) met the primary outcome of developing critical illness or mortality. The performance of the score in this cohort was comparable with the original cohorts, with an area under the receiver operating characteristic curve for all patients of 0.794 (95% CI, 0.752–0.835; p < 0.001), regression coefficient of 1.069, and intercept of 0.04. Subgroup analysis of unvaccinated and vaccinated patients showed that performance was superior in vaccinated individuals, with an area under the receiver operating characteristic curve of 0.684 (95% CI, 0.608–0.760; p < 0.0001) and 0.831 (95% CI, 0.772–0.891; p < 0.0001), respectively.

Discussion

Our data support the continued use of the PRIORITY score in this era of novel variants and increased vaccination uptake.

Introduction

We read with interest the article, “Predicting critical illness on initial diagnosis of COVID-19 based on easily obtained clinical variables: Development and validation of the PRIORITY model” by Martínez-Lacalzada et al., published in Clinical Microbiology and Infection [1]. Risk scores are important tools for the triage of patients with coronavirus disease 2019 (COVID-19), and the PRIORITY score has advantages over numerous others in that it eliminates the requirement for laboratory and radiologic variables, permitting easy administration in a community or ambulatory setting. However, the development and validation cohorts comprised only patients from March to May 2020, during the first wave of the COVID-19 pandemic.

Since then, many variables of the pandemic have changed. Widespread vaccination uptake has significantly affected the incidence of severe COVID-19, and patients with vaccine-breakthrough infections have markedly different clinical features and virologic kinetics compared with unvaccinated individuals [2]. Furthermore, emerging variants of concern (VOCs) have differing clinical presentations and spectra of severity. For example, the Delta variant is associated with potentially greater disease severity [3,4], whereas the Omicron variant appears to be associated with milder illness and less pulmonary involvement [5,6]. Reassessment of existing risk-prediction tools in the setting of currently circulating VOCs and in vaccinated individuals is thus important before extending their applicability to other settings.

Methods

For these reasons, we aimed to externally validate the PRIORITY model in a cohort of patients with COVID-19 from the National Centre for Infectious Diseases. For this study, we combined two separate study databases for a cohort of 648 patients. First, we included all 410 patients with COVID-19 who were admitted to the National Centre for Infectious Diseases in Singapore on October 27, 2021. The inclusion criterion was PCR-confirmed COVID-19. There were no exclusion criteria. To further augment this dataset, we included a cohort of 102 patients with vaccine-breakthrough Omicron variant infection and 136 patients with vaccine-breakthrough Delta variant infection, from a separate study that compared the clinical course and outcomes of Delta and Omicron infection [7].

The inclusion criteria for this second cohort were admission to the National Centre for Infectious Diseases during the study period (April 27–August 11, 2021 for Delta, December 1–18 2021 for Omicron), confirmed infection with the Delta or Omicron variant as confirmed by whole genome sequencing, and illness onset >14 days after completion of any primary COVID-19 vaccine series. During these study periods, all patients with COVID-19 in Singapore with a PCR cycle threshold (Ct) value of <30 had whole genome sequencing done by the National Public Health Laboratory, and all those with confirmed Delta or Omicron infection were admitted for isolation and evaluation regardless of disease severity.

Data were collected retrospectively from the medical record by study investigators using a standardized data collection form. Comorbidities (e.g. cardiovascular and chronic kidney disease) were defined based on previous records of these diagnoses in the medical record and excluded admission serum creatinine level or new diagnoses during the COVID-19–related hospitalization. A waiver of retrospective data collection was approved by the institutional ethics committee (National Healthcare Group Domain Specific Review Board reference number 2020/01122).

We calculated the probability of deterioration for each patient using the same PRIORITY risk formula outlined by Martínez-Lacalzada et al. in their paper. Scores were calculated retrospectively using data from the medical record as recorded by managing physicians upon hospital admission. We used the same primary outcome of critical illness as defined by intensive care unit admission or mortality and evaluated the performance of the risk score by calculating the area under the receiver operating characteristic curve (AUC) in the entire cohort and in subgroups of vaccinated and unvaccinated patients. Vaccinated was defined as receipt of two COVID-19 vaccine doses ≥2 weeks before illness onset. All other patients were considered unvaccinated. Outcome data were censored upon discharge or at 28 days after illness onset.

We also calculated the minimum sample size required for validation of the existing PRIORITY model using methods and STATA code provided by Riley et al. [8]. With an event outcome proportion of 0.25 and a target C-statistic of 0.8, the minimum sample size required was 402 (minimum number of observed events: 101). Model calibration was assessed using a calibration plot and calculation of the regression coefficient, intercept, and Hosmer–Lemeshow goodness-of-fit test [9].

Data analysis was done with STATA, version 13.0 (StataCorp, College Station, TX), and plots were graphed using GraphPad Prism, version 9.2 (GraphPad Software, San Diego, CA). A p-value of <0.05 was considered significant.

Results

The mean age was 61.6 years (standard deviation ± 19.0 years), and 268 patients (41.4%) were female. A total of 447 patients (69.0%) were fully vaccinated. Although we did not have detailed data on the type of vaccine received, the majority (>85%) of vaccinated individuals in Singapore received mRNA vaccines [10]. 122 patients (18.8%) needed assistance or were dependent on others for activities of daily life, and 159 (24.5%) and 78 (12.0%) patients had cardiovascular disease and chronic kidney disease respectively. In addition, 101 patients (15.6%) presented with shortness of breath at presentation, with 56 (8.6%) having a respiratory rate of >20 breaths/min and 112 (17.3%) having an oxygen saturation of ≤93% or requiring supplemental oxygen. Seventy-eight patients (12.0%) showed confusion on admission. The mean systolic blood pressure on admission was 135 mmHg (standard deviation ± 20.6 mmHg). Finally, 112 patients (17.3%) met the primary outcome of developing critical illness, of whom 36 (32.1%) were fully vaccinated. Twenty-nine patients (4.5%) died within 28 days. There were no missing data.

After calculation of the PRIORITY risk scores, the median predicted risk of critical illness was 6.9% (interquartile range, 3.2%–14.4%). The performance of the PRIORITY score in this validation cohort was comparable to that in the original development and validation cohorts, with an AUC for all patients of 0.794 (95% CI, 0.752–0.835; p < 0.001) and a Brier score of 0.123 (scaled Brier score: 0.144, Fig. 1 ). The model was well calibrated, with a regression coefficient of 1.069 and intercept of 0.04. The Hosmer–Lemeshow goodness-of-fit test p-value was <0.001. Subgroup analysis of unvaccinated and vaccinated patients showed that performance was superior in vaccinated individuals, with an AUC of 0.684 (95% CI, 0.608–0.760; p < 0.0001) and 0.831 (95% CI, 0.772–0.891; p < 0.0001), respectively. Performance in vaccinated patients was significantly better than in unvaccinated patients (absolute difference in AUC: 0.147; standard error of difference: 0.0583; p = 0.012).

Fig. 1.

Receiver operator characteristic curves of the PRIORITY score for prediction of the development of critical illness. AUC, area under the curve; ICU, intensive care unit; ROC, receiver operator characteristic.

Discussion

In this study, we validated the published PRIORITY risk score in a cohort of hospitalized patients with COVID-19 and demonstrated comparable performance between vaccinated and unvaccinated patients. Model performance was also comparable with that in the cohorts of Martínez-Lacalzada et al.‘s original paper (C-statistic: 0.823 and 0.794 in the development and validation cohorts, respectively, compared with 0.794 in our cohort) [1], despite the smaller sample size, differences in demographics between study cohorts, and changes in viral variants. In our study cohorts, almost all cases were due to the Delta or Omicron variants [11].

The PRIORITY score remains a promising risk prediction tool, because its ease of administration permits use on a mass scale in the community without the requirement for radiographic or laboratory parameters, which is especially important during periods of disease surges and stretched health care capacity. The authors of the original paper have provided a freely available online calculator (https://www.evidencio.com/models/show/2344) to facilitate adoption in clinical practice. Our data support the continued use of the PRIORITY score in this era of novel VOCs and increased vaccination uptake.

Transparency declaration

BEY reports personal fees from Roche and Sanofi outside of the submitted work. All other authors declare no competing interests. No additional funding was required for this study.

Author contributions

SWXO has full access to all the data in the study, and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Editor: L. Leibovici