Abstract
The ongoing coronavirus disease 2019 (COVID-19) pandemic presents serious public health threats. Omicron, the current most prevalent strain of COVID-19, has a low fatality rate and very high transmissibility, so the number of patients with mild symptoms of COVID-19 is rapidly increasing. This change of pandemic challenges medical systems worldwide in many aspects, including sharp increases in demands for hospital infrastructure, critical shortages in medical equipment, and medical staff. Predicting deterioration in mild patients could alleviate these problems. A novel scoring system was proposed for predicting the deterioration of patients whose condition may worsen rapidly and those who all still mild or asymptomatic. Retrospective cohorts of 954 and 2,035 patients that quarantined in the Residential Treatment Center were assembled for derivation and external validation of mild COVID-19, respectively. Deterioration was defined as transfer to a local hospital due to worsening condition of the patients during the 2-week isolation period. A total of 15 variables: sex, age, seven pre-existing conditions (diabetes, hypertension, cardiovascular disease, respiratory disease, liver disease, kidney disease, and organ transplant), and five vital signs (systolic blood pressure (SBP), diastolic blood pressure (DBP), heart rate (HR), body temperature, and oxygen saturation (SpO2)) were collected. A scoring system was developed using seven variables (age, pulse rate, SpO2, SBP, DBP, temperature, and hypertension) with significant differences between the transfer and not transfer groups in logistic regression. The proposed system was compared with existing scoring systems that assess the severity of patient conditions. The performance of the proposed scoring system to predict deterioration in patients with mild COVID-19 showed an area under the receiver operating characteristic (AUC) of 0.868. This is a statistically significant improvement compared to the performance of the previous patient condition assessment scoring systems. During external validation, the proposed system showed the best and most robust predictive performance (AUC = 0.768; accuracy = 0.899). In conclusion, we proposed a novel scoring system for predicting patients with mild COVID-19 who will experience deterioration which could predict the deterioration of the patient’s condition early with high predictive performance. Furthermore, because the scoring system does not require special calculations, it can be easily measured to predict the deterioration of a patients’ condition. This system can be used as effective tool for early detection of deterioration in mild COVID-19 patients.
Keywords: Clinical deterioration, COVID-19, Early warning score, Infection control, Pandemics
Subject terms: Infectious diseases, Public health
Introduction
The ongoing coronavirus disease 2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), presents serious public health threats, with 5.5 million people deaths worldwide and 272 million reported cases, as of December 2021. Since 2019, when the pandemic began, pneumonia, acute respiratory distress syndrome, pulmonary thrombosis, clinical phenomena, and management strategies have been broadly investigated1–5. Reducing the disease fatality and incidence remains challenging despite increasing vaccination rates due to uncertainty from recently identified COVID-19 variants6. The Omicron B.1.1.529 variant (hereinafter “Omicron”) was identified by the Technical Advisory Group on SARS-CoV-2 Evolution, and the World Health Organization considers it a variant of concern7.
Despite its recency, studies on the transmissibility and fatality of Omicron have been conducted8–11. Omicron has a shorter incubation period and similar or milder clinical symptoms than previous variants12. It has a lower fatality rate but a higher transmission power than the existing variants. Most patients with this variant have mild symptoms and a good prognosis; however, some progress rapidly to serious conditions, including septic shock and multiple organ and acute respiratory failure13. The COVID-19 pandemic has induced difficulties for the global healthcare system, particularly insufficient hospital beds, medical equipment shortages, and medical and nursing staff shortages14. Since March 2020, the Korean government has operated a Residential Treatment Center (RTC), in addition to existing medical services, to quarantine, regularly screen, and monitor asymptomatic and mild COVID-19 patients15. Omicron has a lower fatality rate; therefore, the number of mild patients is expected to increase, and patient surveillance is consequently expected to become more important in future pandemics.
Many approaches to assessing the risks of COVID-19 existed, even before the Omicron outbreak. Kostakis et al. used the National Early Warning Score (NEWS) 1 and 2, widely used in emergency medicine, to predict mortality and intensive care unit admission in patients with COVID-1916. The NEWS has been widely used in patients with COVID-19, including for predicting in-hospital mortality17,18, predicting poor outcomes after hospitalization19, and identifying deterioration20,21. Other emergency score systems, including the rapid emergency medicine score (REMS)22,23 and quick sepsis-related organ failure assessment (qSOFA)24,25, have also been used to predict mortality. Liu et al.26 compared these emergency score systems in terms of effectiveness for risk prediction in COVID-19 patients. There have been attempts for early detection of COVID-19 using mathematical modeling27, knowledge based expert system28, deep learning29,30, and cloud-based image analysis31. However, most studies have focused on determining the current patient severity and mortality, and few predicted changes in the patients’ condition. There was even less research focused on mild COVID-19 patients.
We propose a novel scoring system for patients with mild or asymptomatic COVID-19 to predict rapid deterioration and requirement for transfer to tertiary hospitals. As in previously proposed scoring systems, the values measured on the day of admission were used. Our proposed scoring system was compared with previous emergency scoring systems. Here, we attempted to identify patients with mild or asymptomatic COVID-19 who needed special care because of the high likelihood that their condition could worsen rapidly. Although our study covered a single racial or regional group, factors for society and variants were differed between the cohort because there was a difference in the collection period and virus mutation occurred rapidly. The data were collected during a period of strong social distancing policies, and this study was based on community transmission not in-hospital infection and intra-facility infection. In our knowledge this is the inaugural study to employ a scoring system for predicting hospital admission requirements among patients with mild COVID-19. The proposed system will assist governments in devising differentiated strategies for high- and low-risk cohorts, thereby optimizing the management of limited healthcare resources and mitigating patient anxiety.
Results
Demographics and clinical characteristics
In the derivation cohort, the average ages of the transfer and non-transfer patients were 50.9 and 38.8 years, respectively. The average temperature of transfer patients was 37 °C, which is higher than normal. All continuous variables except respiratory rate showed significant differences between the transfer and non-transfer groups. Among the categorical variables, only hypertension showed a significant between-group difference. Patients who were transferred were older, had lower SpO2, a high pulse rate, and showed higher levels of SBP, DBP, fever, and hypertension. There was no significant difference between SBP and DBP in the external validation cohort, unlike in the derivation cohort; however, there was a significant difference in diabetes and hypertension in the external validation cohort. The triage center’s patient demographics and clinical characteristics are summarized in Table 1.
Table 1.
Demographics and clinical characteristics of patients in derivation and external validation cohorts.
| Derivation cohort | ||||
|---|---|---|---|---|
| Variable | Total (n = 954) | Transfer (n = 82) | Not transfer (n = 872) | p-value |
| Continuous variable, mean (SD) | ||||
| Age (years) | 39.7 (15.4) | 50.5 (10.5) | 38.7 (15.4) | < 0.001a |
| Pulse rate (bpm) | 78.7 (10.0) | 85.3 (11.5) | 78.1 (9.6) | < 0.001a |
| SpO2 (%) | 97.3 (1.2) | 95.8 (2.4) | 97.4 (0.9) | < 0.001a |
| Systolic blood pressure (mmHg) | 120.4 (14.1) | 124.9 (14.8) | 119.9 (14.0) | 0.001a |
| Diastolic blood pressure (mmHg) | 79.5 (9.8) | 82.3 (9.8) | 79.2 (9.7) | 0.003a |
| Temperature (℃) | 36.5 (0.4) | 37.0 (0.7) | 36.5 (0.4) | < 0.001a |
| Respiratory rate (min-1) | 17.3 (3.0) | 18.0 (3.0) | 17.2 (3.0) | 0.016a |
| Categorical variable, n (% total) | ||||
| Male | 462 (48.4%) | 31 (37.8%) | 431 (49.4%) | 0.214b |
| Diabetes | 42 (4.4%) | 6 (7.3%) | 36 (4.1%) | 0.165c |
| Hypertension | 111 (11.6%) | 23 (28.0%) | 88 (10.1%) | 0.004b |
| Cardiovascular disease | 21 (2.2%) | 4 (4.9%) | 17 (1.9%) | 0.098c |
| Respiratory disease | 26 (2.7%) | 5 (6.1%) | 21 (2.4%) | 0.065c |
| Liver disease | 18 (1.9%) | 3 (3.7%) | 15 (1.7%) | 0.196c |
| Kidney disease | 13 (1.3%) | 3 (3.7%) | 10 (1.1%) | 0.093c |
| Organ transplant | 2 (0.2%) | 0 (0.0%) | 2 (0.2%) | > 0.999c |
| External validation cohort | ||||
|---|---|---|---|---|
| Variable | Total (n = 2035) | Transfer (n = 73) | Not transfer (n = 1962) | p-value |
| Continuous variable, mean (SD) | ||||
| Age (years) | 43.2 (15.1) | 48.0 (14.2) | 38.7 (15.4) | 0.005a |
| Pulse rate (bpm) | 79.8 (9.7) | 89.1 (12.7) | 79.5 (9.4) | < 0.001a |
| SpO2 (%) | 97.0 (1.5) | 96.0 (2.6) | 97.1 (1.4) | 0.001a |
| Systolic blood pressure (mmHg) | 125.9 (14.3) | 126.6 (14.9) | 125.8 (14.3) | 0.692a |
| Diastolic blood pressure (mmHg) | 82.9 (9.6) | 83.4 (10.2) | 82.8 (9.6) | 0.614a |
| Temperature (℃) | 36.2 (0.6) | 36.8 (0.7) | 36.2 (0.6) | < 0.001a |
| Respiratory rate (min−1) | 18.3 (3.6) | 19.1 (3.5) | 18.3 (3.6) | 0.047a |
| Categorical variable, n (% total) | ||||
| Gender, Men | 1098 (54.0%) | 44 (60.3%) | 1054 (53.7%) | 0.536b |
| Diabetes | 127 (6.2%) | 12 (16.4%) | 115 (5.9%) | 0.001c |
| Hypertension | 340 (16.7%) | 25 (34.2%) | 315 (16.1%) | < 0.001b |
| Cardiovascular disease | 79 (3.9%) | 9 (12.3%) | 70 (3.6%) | 0.002c |
| Respiratory disease | 30 (1.5%) | 2 (2.7%) | 28 (1.4%) | 0.293c |
| Liver disease | 33 (1.6%) | 3 (4.1%) | 30 (1.5%) | 0.112c |
| Kidney disease | 14 (0.7%) | 2 (2.7%) | 12 (0.6%) | 0.087c |
| Organ transplant | 2 (0.1%) | 0 (0.0%) | 2 (0.1%) | > 0.999c |
Statistically significant values are in bold.
aMann–Whitney U test.
bPearson’s chi-squared test.
cFisher’s exact test.
Clinical indicators associated with the transfer
The simple and multiple logistic regression results using the factors associated with the transfer are shown in Table 2. In the simple logistic regression analysis, most parameters except male sex, liver disease, kidney disease, and organ transplant were significantly associated with transfer (p < 0.01). Of the ten variables included, the four candidate variables of age, pulse rate, SpO2, and temperature were significantly related to transfer through multivariable regression analysis.
Table 2.
Logistic regression analyses to determine each parameter’s association with the transfer.
| Variable | Univariable | Multivariable | ||
|---|---|---|---|---|
| Odds ratio (95%) | p-value | Odds ratio (95% CI) = 0.317 | p-value | |
| Age (years) | 0.951 (0.946–0.956) | < 0.001 | 1.052 (1.028–1.076) | < 0.001 |
| Pulse rate (bpm) | 0.972 (0.969–0.974) | < 0.001 | 1.052 (1.022–1.083) | 0.001 |
| SpO2 (%) | 0.976 (0.974–0.978) | < 0.001 | 0.63 (0.553–0.718) | < 0.001 |
| Systolic blood pressure (mmHg) | 0.981 (0.979–0.983) | < 0.001 | 1.003 (0.972–1.034) | 0.871 |
| Diastolic blood pressure (mmHg) | 0.971 (0.968–0.974) | < 0.001 | 0.993 (0.948–1.041) | 0.780 |
| Respiratory rate (min-1) | 0.875 (0.863–0.886) | < 0.001 | 0.953 (0.876–1.037) | 0.260 |
| Temperature (℃) | 0.938 (0.932–0.943) | < 0.001 | 2.745 (1.914–3.938) | < 0.001 |
| Gender, Men | – | N/A | – | N/A |
| Diabetes | 0.167 (0.07–0.396) | < 0.001 | 2.911 (1.283–6.606) | 0.011 |
| Hypertension | 0.261 (0.165–0.414) | < 0.001 | 1.3 (0.663–2.549) | 0.446 |
| Cardiovascular disease | 0.235 (0.079–0.699) | 0.009 | 0.426 (0.058–3.145) | 0.403 |
| Respiratory disease | 0.238 (0.09–0.631) | 0.004 | 0.431 (0.05–3.708) | 0.444 |
| Liver disease | 0.2 (0.058–0.691) | 0.011 | 0.44 (0.059–3.263) | 0.422 |
| Kidney disease | 0.3 (0.083–1.09) | 0.067 | 0.793 (0.079–7.907) | 0.843 |
| Organ transplant | – | 1 | 1.141 (0.008–155.47) | 0.958 |
Statistically significant values are in bold.
CI confidence interval, N/A not applicable.
Predictive performance of our scoring system
Each scoring system’s discriminatory ability for predicting transfer was compared using receiver operating characteristic (ROC) curves (Fig. 1A). Our scoring system had an AUC of 0.868, which was higher than that for the NEWS, REMS, and qSOFA (0.646, 0.612, and 0.509, respectively; (Fig. 1B)). The optimized thresholds were determined using the ROC curve as a trade-off between sensitivity and specificity to evaluate the prediction’s accuracy. Our scoring system had the best predictive performance for identifying whether patients were transferred when the threshold was 4.
Fig. 1.
Comparison of the performance of National Early Score (NEWS), Rapid Emergency Medicine Score (REMS), quick Sepsis-related Organ Failure Assessment (qSOFA), and our scoring system in predicting transfer. (A) Receiver operating characteristic (ROC) curves of four methods for prediction of transfer. (B) Performance was compared by calculating area under the ROC curve (AUC) for NEWS, REMS, qSOFA, and the proposed scoring system with DeLong statistic test. The asterisk (*) indicates a p-value lower than 0.05, and double asterisk (**) indicates a p-value lower than 0.01.
The accuracy, sensitivity, specificity, positive predictive value, and negative predictive value at the optimized threshold values for each model are shown in Table 3. Our system showed the best performance in terms of accuracy, followed by the qSOFA, NEWS, and REMS. Our system showed the best performance in all indicators except for sensitivity. Concerning the REMS, which had the highest sensitivity, the accuracy was < 50%, indicating poor performance in predicting negative cases. Although the positive predictive values were relatively low in all systems owing to data imbalance, our system had a value of 0.356, which showed a higher value than NEWS.
Table 3.
Performance comparison of the ability of four different scoring systems in the optimal cut-off.
| System | ACC | SEN | SPE | PPV | NPV |
|---|---|---|---|---|---|
| Proposed | 0.868 | 0.659 | 0.888 | 0.356 | 0.965 |
| NEWS | 0.733 | 0.500 | 0.755 | 0.161 | 0.941 |
| REMS | 0.473 | 0.732 | 0.448 | 0.111 | 0.947 |
| qSOFA | 0.799 | 0.159 | 0.860 | 0.096 | 0.916 |
ACC accuracy, SEN sensitivity, SPE specificity, PPV positive predictive value, NPV negative predictive value, NEWS National Early Score, REMS Rapid Emergency Medicine Score, qSOFA quick sepsis-related organ failure assessment.
External validation
The predictive performance of applying our scoring system with significant variables in the derivation to the external validation cohort was compared with that of other models, as shown in Table 4. The external validation cohort had a more severe data imbalance problem than the derivation cohort did; therefore, the overall predictive performance, especially the sensitivity and positive predictive value, was low. All models except our system had AUC values < 0.6 and positive predictive values < 0.1. Our system showed a relatively stable performance for all indicators except for sensitivity. Sensitivity and specificity based on different threshold are shown in Supplementary Table 1. Regarding accuracy, all systems showed higher performance than in the derivation cohort, possibly resulting from a higher proportion of negative cases in this cohort.
Table 4.
Performance comparison of the four scoring systems with external validation cohort.
| System | AUC | ACC | SEN | SPE | PPV | NPV |
|---|---|---|---|---|---|---|
| Proposed | 0.768 | 0.899 | 0.397 | 0.918 | 0.153 | 0.976 |
| NEWS | 0.604 | 0.819 | 0.301 | 0.838 | 0.065 | 0.970 |
| REMS | 0.577 | 0.644 | 0.521 | 0.648 | 0.052 | 0.973 |
| qSOFA | 0.527 | 0.786 | 0.247 | 0.806 | 0.045 | 0.966 |
AUC area under the receiver operating curve, ACC accuracy, SEN sensitivity, SPE specificity, PPV positive predictive value, NPV negative predictive value, NEWS National Early Score, REMS Rapid Emergency Medicine Score, qSOFA quick sepsis-related organ failure assessment.
Discussion
Identifying mild COVID-19 is essential for effectively controlling the pandemic, owing to its wide prevalence, which was 80% in a study of 72,314 patients32. As COVID-19 is often mild during diagnosis, preemptively responding to worsening of mild disease is important32. Additionally, since most patients with mild symptoms are recommended to isolate at home, or they are managed in RTCs with insufficient medical facilities and medical staff33, it is difficult to respond quickly to serious conditions. Therefore, it is necessary to enable easy screening of those with a high probability of deterioration. Development of a scoring system that can be easily measured using simple vital signs and clinical indicators is important. We have proposed a novel scoring system that could help predict whether patients with mild COVID-19 will be transferred based on their characteristics and clinical parameters, collected on the day of admission. Our scoring system will enable proactive determination of whether a patient requires close observation. There exist several scoring systems for the early prediction of patients requiring emergency treatment34–36. However, we believe this is the first study to use a scoring system to predict the need for hospital admission among patients with mild COVID-19. This will help governments to formulate different strategies for high- and low-risk groups, which will allow better management of the limited health care resources, and will be useful for decreasing anxiety in patients.
The variables used in our system included age, pulse rate, SpO2, systolic and diastolic blood pressures, temperature, and hypertension. Compared with previous systems, the only additional variables were DBP and hypertension. Regarding age, an important variable in the other systems, our system suggests that classification at a lower age is necessary (threshold of our system = 39, REMS = 45). Because most patients had mild disease, we needed to set the threshold SpO2 value relatively high compared with that suggested by other systems (threshold of our system = 98, REMS = 89, and NEWS = 96). Our scoring system has lower pulse rate and temperature thresholds for scoring the results as abnormal than other scoring systems, possibly because existing systems focus on distinguishing severely ill patients. Our results confirm that the system proposed a cut-off for a sensitive variable for determining milder patients’ conditions.
Consequently, our system had superior performance in predicting whether the patient would be transferred compared with the previous systems. It showed a statistically significant improvement in AUC compared with previous systems, and high values in almost all performance evaluation parameters. However, data bias may have occurred because the occurrence frequency was low, and the transfer of mild patients with COVID-19 to the hospital, determined by the subjective judgment of the medical staff, was considered as an outcome. More sensitive staff may decide to transfer patients to the hospital before their condition becomes serious. This would have had a greater effect on the transfer group, which had a smaller sample size, than on the non-transfer group. To exclude the effects of these biases, we verified our system in an external validation cohort from another institution and confirmed that it still showed promising results (Table 4). Furthermore, all patients tested had indications for the test; therefore, we consider that there was no referral bias for most subjects14.
According to the transfer prediction results of previous systems, the classification performance was in the following order: NEWS, REMS, and qSOFA. The NEWS is a clinical evaluation tool developed by the Royal College of Physicians in the UK, and can be calculated using the respiratory rate, SpO2, temperature, SBP, heart rate, and consciousness level34. The NEWS showed better mortality predictive power in patients with severe COVID-19 than other scoring systems26. In one study26, the authors reported that NEWS2’s predictive performance37, which added two SpO2 scales, was as good as that of the NEWS; however, we did not compare it with our system because the added SpO2 scale was not useful when analyzing patients with mild COVID-19. Our study confirmed that the existing risk prediction scoring systems, including NEWS, are somewhat useful in predicting the deterioration in mild COVID-19 patients at an early stage. However, the proposed system performed significantly better than the existing systems because it was developed with sensitive threshold adjustments to predict deterioration in patients with mild COVID-19 different with the existing systems. Recent studies have suggested applying the bacterial co-infection, hypo-lymphocytosis, multilobular infiltration, and the MuLBSTA score for predicting COVID-19 infection severity38–40. However, because our system uses only relatively easy-to-measure variables, the evaluation can be performed quickly and easily. Additionally, it can be performed even by non-professionals; therefore, it can be applied in poor environments where there are few medical staff with special training.
Despite using only variables that are relatively easy to measure, our scoring system has better performance in predicting the worsening condition of mild COVID-19 patients than other indicators, and has shown strong predictive power in external validation, so it can be applied in a variety of ways in actual clinical environments. It's possible. First, medical institutions that diagnose COVID-19 can apply a scoring system to confirmed patients to indicate whether the patient's condition is expected to worsen. Patients whose condition is expected to worsen require closer observation of their condition. Patients whose condition is not expected to worsen may only be considered for isolation to prevent transmission. In addition, by applying the developing wearable device technology, it is possible to predict the deterioration of the patient's condition in real time by fixing age and hypertension, which are variables that do not easily change, and measuring only the remaining four indicators with a wearable device. Nevertheless, this study has some limitations. First, it was retrospective, and all data were collected as part of usual care and not for research. Therefore, it was impossible to use new variables to predict which patients with mild COVID-19 could worsen.
In conclusion, this study proposed a novel scoring system was proposed for predicting patients with mild COVID-19 who will experience deterioration using simple measurable variables. The proposed system achieved 0.868 in derivation dataset and 0.899 accuracy in external validation dataset, outperforming existing scoring system that evaluated patient severity, such as NEWS, REMS, and qSOFA. This system can be used as effective tool for early screening of deterioration in mild COVID-19 patients. In particular, it can help to efficiently manage limited infrastructure in areas lacking medical infrastructure where the number of confirmed patients is explosively increasing. In the future studies, effectiveness of the proposed system could be confirmed for early screening of patients who are expected to deteriorate in medical resource management.
Methods
Sample and data
This study retrospectively utilized data generated from patients who received treatment at the RTCs operated by Seoul National University Hospital (SNUH). All patients who admitted at the RTC had been diagnosed with COVID-19, with laboratory confirmation using real-time reverse transcription-polymerase chain reaction testing in local health centers. Upon diagnosis, a group of public health experts triaged the patients and determined the treatment policy based on symptom severity. Patients with severe symptoms were hospitalized in a negative-pressure isolation room, while mild and asymptomatic patients were admitted to the RTC. The government entrusted several hospitals with the operation of RTCs to manage patients with mild COVID-19. The RTC is not a hospital facility and was originally operated with its existing accommodations41. Using the SNUH information and communication technology–based remote patient monitoring system, all patients admitted to the RTC operated by SNUH used a mobile app to self-report their past medical history and subjective acute COVID-19 symptoms. Patients admitted to the RTC received non-face-to-face treatment from medical staff at least once daily and were transferred to a local hospital if their condition worsened.
From March 6, 2020, to January 12, 2021, during the early stages of the COVID-19 pandemic, RTCs were operated in Mungyeong and Seongnam regions. The data from patients admitted during this period were used as the derivation cohort. From July 15, 2021, to January 12, 2022, the RTC in Seongnam was reopened, and the data from patients admitted during this period were used as the external validation cohort. This period coincided with the prevalence of mutant viruses. The alpha mutation detection rate was on decline from about 13%, and the delta mutation detection rate increased from 33 to 100% and then decreased to 73%. The Omicron variant began to be detected in December, 2021 and were 26% by the end of recruitment. Over time, the percentage of severe patients had declined. The transfer rate in the external validation cohort was reduced from 8.6% to 3.6%, compared to the derivation cohort. Reports were made once on admission and discharge, and twice a day during the quarantine period. The data recorded from patients were stored in the hospital information system in SNUH. These electronic health records were extracted using a clinical data warehouse at the SNUH, the SNUH Patient Research Environment. Through the health records left by the medical staffs, fifteen variables were collected: sex, age, pre-existing conditions (diabetes, hypertension, cardiovascular disease, respiratory disease, liver disease, kidney disease, organ transplant), and vital signs (systolic blood pressure (SBP), diastolic blood pressure (DBP), heart rate (HR), body temperature, and oxygen saturation (SpO2)). Blood pressure was measured using the Blood Pressure Monitor (OMRON Healthcare, Kyoto, Japan), and body temperature was measured using the Axillary Thermometer (TERUMO Medical Corporation, Tokyo, Japan), and oxygen saturation was measured using the Pulse Oximeter Monitor (OMRON Healthcare, Kyoto, Japan). Furthermore, to enhance the accuracy of patient vital sign measurements, patients were instructed to watch educational videos on the method of vital sign measurement by skilled nurses upon admission, allowing them to measure their vital signs. Each time vital signs were measured, the responsible nurse double-checked to ensure proper measurement and entry of values. All collected variables were measured at the time of admission the RTC. The (primary) outcome was an all-cause transfer to the hospital while staying at the center.
Parameters and measures of parameters
Using logistic regression analysis, a scoring system for transfer prediction was developed using selected variables known to be statistically related to COVID-19 severity. Variables for which there were significant differences between the demographics and clinical characteristics of transfer and non-transfer patients were selected. A scoring system was developed using seven variables (age, pulse rate, SpO2, SBP, DBP, temperature, and hypertension) with significant differences in the transfer group and not transfer group (Table 1). Among these, four variables, age, pulse rate, SpO2, and temperature, had high odds ratio and significant p-values in multivariate logistic regression, so they were considered more important variables and were allocated 2 points and 1 point to each of the remaining 3 variables. A score ranging from 0 to 11 was allocated to each patient by adding the points. The cut-off value for each variable was determined by comparing the transfer and non-transfer patients’ characteristics. Patients aged > 39 years with a pulse rate > 86 bpm and SpO2 < 98% were assigned a score of 2; those with an SBP > 118 mmHg and DBP < 77 mmHg were assigned a score of 1; those with a temperature level > 37 °C were assigned a score of 2; and hypertensive patients were assigned a score of 1. Each threshold was set at the point where the difference between the transfer group and not transfer group was greatest. Finally, a scoring system for detecting transfer and non-transfer patients was designed (from 0 to 11) and each patient’s score was calculated. Patients with scores of 0–4 were classified as non-transfer, and scores of 5–11 as transfer (Table 5).
Table 5.
Scoring system for predicting transfer to a hospital of a patient with COVID-19.
| Variables | Score | ||
|---|---|---|---|
| 0 | + 1 | + 2 | |
| Age | < 39 | – | ≥ 39 |
| Pulse rate | < 86 | – | ≥ 86 |
| SpO2 | > 98 | – | ≤ 98 |
| Systolic blood pressure | < 118 | ≥ 118 | – |
| Diastolic blood pressure | < 77 | ≥ 77 | – |
| Temperature | < 37 | – | ≥ 37 |
| Hypertension | No | Yes | – |
Proposed scoring systems and data analysis
Categorical variables were presented as numbers (percentages) and analyzed using Pearson’s chi-square test. Fisher’s exact test was used when the expected frequency was < 5 because the chi-square approximation might not hold for the relatively small sample size. Continuous variables were presented as means ± standard deviations. Since they were non-regularly distributed data, they were compared using the Mann–Whitney U test. Simple and multivariable logistic regression analyses were used to evaluate candidates for constituent variables in the scoring system for predicting patient transfer. In the logistic regression analysis, R-squared was calculated by McFadden’s Pseudo R-squared.The area under the receiver operating curve (AUC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were compared to evaluate the discriminatory power of our scoring system, as well as that of the NEWS, REMS, and qSOFA, in predicting patient transfer. The AUC was used to evaluate the predictive ability to distinguish between patients with and without interest outcomes. The DeLong test42 was performed to compare the predictive ability of our system with those of other systems. The other values were measured using the optimal dividing Youden’s index cut-off value43. The developed scoring system was applied to an independent external validation cohort. All tests were two-sided, and p < 0.01 was considered statistically significant. All statistical analyses were performed on computer with an Intel(R) Core(TM) i5 with 16 GB of RAM memory using Python v. 3.8.8 and SciPy v1.5.2.
Ethical declarations
This study was approved by the SNUH’s institutional review board (H-2105-158-1221) and was conducted in accordance with the relevant guidelines and regulations. The ethics committee waived the need for informed consent considering the study design and adherence to the relevant guideline. In particular, the study data were deidentified to protect privacy and preserve the confidentiality of the study participants.
Supplementary Information
Acknowledgements
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-00312, development of non-face-to-face patient infection activity prediction and protection management SW technology at home and community treatment centers for effective response to infectious disease); and in part by the Korea Health Promotion Research and Development Project through the Ministry of Health and Welfare, Republic of Korea, under Grant HS21C0016.
Author contributions
S.B.L., Y.S.B contributed to design the study. S.B.L performed data analysis, model development and drafted the manuscript. J.Y.K performed data pre-processing, analysis and model evaluation. E.K.C and Y.S.B collected the data. All authors contributed to the review of the manuscript.
Data availability
The data sets generated or analyzed during the current study are not publicly available in accordance with the hospital’s regulations, adhering to the National Privacy Act and relevant guidelines. However, if there is a reasonable request after review and approval of the institutional review board and the institutional data steering committee, it may be available from the corresponding author.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
The online version contains supplementary material available at 10.1038/s41598-024-71033-x.
References
- 1.She, J. et al. 2019 novel coronavirus of pneumonia in Wuhan, China: Emerging attack and management strategies. Clin. Transl. Med.9, 1–7 (2020). 10.1186/s40169-020-00271-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Li, L., Huang, Q., Wang, D. C., Ingbar, D. H. & Wang, X. Acute lung injury in patients with COVID-19 infection. Clin. Transl. Med.10, 20–27 (2020). 10.1002/ctm2.16 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Agostino, D. A. et al. Brief psychotic disorder during the national lockdown in Italy: An emerging clinical phenomenon of the COVID-19 pandemic. Schizophr. Bull.47, 15–22 (2021). 10.1093/schbul/sbaa112 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Azizi, M. R., Atlasi, R., Ziapour, A., Abbas, J. & Naemi, R. Innovative human resource management strategies during the COVID-19 pandemic: A systematic narrative review approach. Heliyon7, e07233 (2021). 10.1016/j.heliyon.2021.e07233 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Hani, C. et al. COVID-19 pneumonia: A review of typical CT findings and differential diagnosis. Diagn. Interv. Imaging101, 263–268 (2020). 10.1016/j.diii.2020.03.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wang, X. & Powell, C. A. How to translate the knowledge of COVID-19 into the prevention of Omicron variants. Clin. Transl. Med.11, e680 (2021). 10.1002/ctm2.680 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Dejnirattisai, W. et al. Reduced neutralisation of SARS-CoV-2 omicron B.1.1.529 variant by post-immunisation serum. Lancet399, 234–236 (2022). 10.1016/S0140-6736(21)02844-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Jassat, W. et al. Clinical severity of COVID-19 patients admitted to hospitals during the Omicron wave in South Africa: A retrospective observational study. Lancet Glob. Health10, e961–e969 (2022). 10.1016/S2214-109X(22)00114-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Bhattacharyya, R. P. & Hanage, W. P. Challenges in inferring intrinsic severity of the SARS-CoV-2 Omicron variant. N. Engl. J. Med.386, e14 (2022). 10.1056/NEJMp2119682 [DOI] [PubMed] [Google Scholar]
- 10.Liu, Y., Yu, Y., Zhao, Y. & He, D. Reduction in the infection fatality rate of Omicron variant compared with previous variants in South Africa. Int. J. Infect. Dis.120, 146–149 (2022). 10.1016/j.ijid.2022.04.029 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Lee, M. et al. Impact of COVID-19 on case fatality rate of patients with cancer during the Omicron wave. Cancer Cell40, 343–345 (2022). 10.1016/j.ccell.2022.02.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jansen, L. et al. Investigation of a SARS-CoV-2 B.1.1.529 (Omicron) variant cluster—Nebraska, November–December 2021. MMWR Morb. Mortal. Wkly. Rep.70, 1782–1784 (2021). 10.15585/mmwr.mm705152e3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chen, N. et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: A descriptive study. Lancet395, 507–513 (2020). 10.1016/S0140-6736(20)30211-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Zoabi, Y., Deri-Rozov, S. & Shomron, N. Machine learning-based prediction of COVID-19 diagnosis based on symptoms. NPJ Digit. Med.4, 3 (2021). 10.1038/s41746-020-00372-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kang, E. et al. Operating protocols of a community treatment center for isolation of patients with coronavirus disease, South Korea. Emerg. Infect. Dis.26, 2329–2337 (2020). 10.3201/eid2610.201460 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kostakis, I. et al. The performance of the National Early Warning Score and National Early Warning Score 2 in hospitalised patients infected by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Resuscitation159, 150–157 (2021). 10.1016/j.resuscitation.2020.10.039 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Myrstad, M. et al. National Early Warning Score 2 (NEWS2) on admission predicts severe disease and in-hospital mortality from Covid-19—A prospective cohort study. Scand. J. Trauma Resusc. Emerg. Med.28, 66 (2020). 10.1186/s13049-020-00764-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Wibisono, E. et al. National early warning score (NEWS) 2 predicts hospital mortality from COVID-19 patients. Ann. Med. Surg. (Lond.)76, 103462 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Cr, P., Vanidassane, I., Pownraj, D., Kandasamy, R. & Basheer, A. National Early Warning Score 2 (NEWS2) to predict poor outcome in hospitalised COVID-19 patients in India. PLoS One16, e0261376 (2021). 10.1371/journal.pone.0261376 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Brand, J. W., Morrice, D. J., Patteril, M. V. & Mackay, J. H. National Early Warning Score 2 (NEWS2) to identify inpatient COVID-19 deterioration: The importance of pO2:FiO2 ratio. Clin. Med. (Lond.)21, e315–e316 (2021). 10.7861/clinmed.Let.21.3.2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Jahandideh, S. et al. Evaluation of machine learning-based models for prediction of clinical deterioration: A systematic literature review. Int. J. Med. Inform.175, 105084 (2023). 10.1016/j.ijmedinf.2023.105084 [DOI] [PubMed] [Google Scholar]
- 22.Bourn, S. S. et al. Initial prehospital Rapid Emergency Medicine Score (REMS) to predict outcomes for COVID-19 patients. J. Am. Coll. Emerg. Phys. Open2, e12483 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Hu, H., Yao, N. & Qiu, Y. The comparison of REMS and MEWS for COVID-19 patients less than 65 years of age. Acad. Emerg. Med.27, 1219 (2020). 10.1111/acem.14127 [DOI] [PubMed] [Google Scholar]
- 24.Churpek, M. M. et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am. J. Respir. Crit. Care Med.195, 906–911 (2017). 10.1164/rccm.201604-0854OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Ramos-Rincón, J. M. et al. The quick sepsis-related organ failure assessment (qSOFA) is a good predictor of in-hospital mortality in very elderly patients with bloodstream infections: A retrospective observational study. Sci. Rep.9, 15075 (2019). 10.1038/s41598-019-51439-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Liu, F. Y. et al. Evaluation of the risk prediction tools for patients with coronavirus disease 2019 in Wuhan, China: A single-centered, retrospective, observational study. Crit. Care Med.48, e1004–e1011 (2020). 10.1097/CCM.0000000000004549 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Frimpong, S. A. et al. Innovative IoT-based wristlet for early COVID-19 detection and monitoring among students. Math. Model Eng. Probl.9, 1557–1564 (2022). 10.18280/mmep.090615 [DOI] [Google Scholar]
- 28.Wubineh, B. Z., Salau, A. O. & Braide, S. L. Knowledge based expert system for diagnosis of COVID-19. J. Pharm. Negat. Results14, 1242–1249 (2023). [Google Scholar]
- 29.Misra, P. et al. PETLFC: Parallel ensemble transfer learning based framework for COVID-19 differentiation and prediction using deep convolutional neural network models. Multimed. Tools Appl.83(5), 14211–14233 (2024). 10.1007/s11042-023-16084-4 [DOI] [Google Scholar]
- 30.Ayalew, A. M. et al. X-Ray image-based COVID-19 detection using deep learning. Multimed. Tools Appl.82(28), 44507–44525 (2023). 10.1007/s11042-023-15389-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Prasad, V. K. et al. CIA-CVD: Cloud based image analysis for COVID-19 vaccination distribution. J. Cloud Comput.12(1), 163 (2023). 10.1186/s13677-023-00539-y [DOI] [Google Scholar]
- 32.Wu, Z. & McGoogan, J. M. Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: Summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention. JAMA323, 1239–1242 (2020). 10.1001/jama.2020.2648 [DOI] [PubMed] [Google Scholar]
- 33.World Health Organization.
- 34.World Health Organization. Coronavirus disease 2019 (COVID-19) Situation Report – 46. [accessed June 17, 2020]. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200306-sitrep-46-covid-19.pdf?sfvrsn=96b04adf_4 (2020).
- 35.Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E. & Featherstone, P. I. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation84, 465–470 (2013). 10.1016/j.resuscitation.2012.12.016 [DOI] [PubMed] [Google Scholar]
- 36.Olsson, T., Terent, A. & Lind, L. Rapid emergency medicine score: A new prognostic tool for in-hospital mortality in nonsurgical emergency department patients. J. Intern. Med.255, 579–587 (2004). 10.1111/j.1365-2796.2004.01321.x [DOI] [PubMed] [Google Scholar]
- 37.Singer, M. et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA315, 801–810 (2016). 10.1001/jama.2016.0287 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Smith, G. B. et al. The National Early Warning Score 2 (NEWS2). Clin. Med. (Lond.)19, 260 (2019). 10.7861/clinmedicine.19-3-260 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Guo, L. et al. Clinical features predicting mortality risk in patients with viral pneumonia: The MuLBSTA score. Front. Microbiol.10, 2752 (2019). 10.3389/fmicb.2019.02752 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Erden, A. et al. Evaluation of 17 patients with COVID-19 pneumonia treated with anakinra according to Hscore, SOFA, MuLBSTA, and Brescia-COVID respiratory severity scale (BCRSS) scoring systems. J. Med. Virol.93, 1532–1537 (2021). 10.1002/jmv.26473 [DOI] [PubMed] [Google Scholar]
- 41.Bae, Y. S. et al. Information technology-based management of clinically healthy COVID-19 patients: Lessons from a living and treatment support center operated by Seoul National University Hospital. J. Med. Internet Res.22, e19938 (2020). 10.2196/19938 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics44, 837–845 (1988). 10.2307/2531595 [DOI] [PubMed] [Google Scholar]
- 43.Fluss, R., Faraggi, D. & Reiser, B. Estimation of the Youden Index and its associated cutoff point. Biometr. J.47, 458–472 (2005). 10.1002/bimj.200410135 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data sets generated or analyzed during the current study are not publicly available in accordance with the hospital’s regulations, adhering to the National Privacy Act and relevant guidelines. However, if there is a reasonable request after review and approval of the institutional review board and the institutional data steering committee, it may be available from the corresponding author.