Abstract
Background
Early identification and timely therapeutic strategies for potential critical patients with coronavirus disease 2019 (COVID-19) are of crucial importance to reduce mortality. We aimed to develop and validate a prediction tool for 30-day mortality for these patients on admission.
Methods
Consecutive hospitalized patients admitted to Tongji Hospital and Hubei Xinhua Hospital from January 1 to March 10, 2020, were retrospective analyzed. They were grouped as derivation and external validation set. Multivariate Cox regression was applied to identify the risk factors associated with death, and a nomogram was developed and externally validated by calibration plots, C-index, Kaplan-Meier curves and decision curve.
Results
Data from 1,717 patients at the Tongji Hospital and 188 cases at the Hubei Xinhua Hospital were included in our study. Using multivariate Cox regression with backward stepwise selection of variables in the derivation cohort, age, sex, chronic obstructive pulmonary disease (COPD), as well as seven biomarkers (aspartate aminotransferase, high-sensitivity C-reactive protein, high-sensitivity troponin I, white blood cell count, lymphocyte count, D-dimer, and procalcitonin) were incorporated in the model. An age, biomarkers, clinical history, sex (ABCS)-mortality score was developed, which yielded a higher C-index than the conventional CURB-65 score for predicting 30-day mortality in both the derivation cohort {0.888 [95% confidence interval (CI), 0.869–0.907] vs. 0.696 (95% CI, 0.660–0.731)} and validation cohort [0.838 (95% CI, 0.777–0.899) vs. 0.619 (95% CI, 0.519–0.720)], respectively. Furthermore, risk stratified Kaplan-Meier curves showed good discriminatory capacity of the model for classifying patients into distinct mortality risk groups for both derivation and validation cohorts.
Conclusions
The ABCS-mortality score might be offered to clinicians to strengthen the prognosis-based clinical decision-making, which would be helpful for reducing mortality of COVID-19 patients.
Keywords: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), coronavirus disease 2019 (COVID-19), mortality, nomogram
Introduction
Coronavirus disease 2019 (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a newly recognized viral pneumonia that has been detected since December 2019 (1-4). Currently, COVID-19 is prevalent globally, and it is characterized as a pandemic by the World Health Organization (WHO). As of July 15, 2020, the total number of patients has risen sharply to 13,287,651 in 188 countries, areas or territories worldwide, with 577,954 (4.3%) deceased (5). There was a wide clinical spectrum of COVID-19, which ranged from mild to critically ill cases. Several studies have described the general epidemic characteristics, clinical manifestations, and clinical outcomes of patients with COVID-19 (4,6-13). Although severe and critical cases only accounted for about 19% of all patients (8), the mortality of critically ill cases was reported to be as high as 61.5% (6).
A surge of patients entering the hospital in a short period will result in the medical resources being overwhelmed, and it is therefore crucial to identify those potential patients with a high mortality risk, as such patients are most likely to benefit from immediate intensive care treatment. Previous studies have indicated that older age, comorbidities (e.g., cardiovascular disease, diabetes mellitus, or chronic respiratory disease), cardiac injury, elevated D-dimer and lactate dehydrogenase (LDH) concentrate, leukocytosis and lymphocytopenia were associated with adverse clinical outcomes of COVID-19 (2,6,10,13-18). However, the cumulative risk rendered by their combination on mortality has not been fully determined. In addition, the prognostic factors associated with COVID-19 are still uncertain. This study aimed to define the prognostic factors associated with mortality in hospitalized patients with COVID-19, and create a biomarker-based risk score for patients’ stratification and clinical decision-making. The clinical use was compared with the CURB-65 mortality risk score—a widely used severity score for community-acquired pneumonia (19). We present the following article in accordance with the TRIPOD reporting checklist (available at http://dx.doi.org/10.21037/atm-20-6205) (20).
Methods
Study design and participants
The derivation cohort was retrieved from Tongji Hospital of Huazhong University of Science and Technology (the largest teaching center in Hubei province), which is one of the designated hospitals for severely or critically ill COVID-19 cases. The study was conducted according to the principles of the Declaration of Helsinki (as revised in 2013). The study was approved by ethics board of Tongji Hospital and Hubei Xin Hua Hospital (NO.: TJ-IRB20200373) and informed consent was waived since the study was retrospectively designed and did not cause any harm to the patients. We retrospectively analyzed hospitalized adult patients (≥18 years old) from January 1, 2020, to April 10, 2020 (time of cut-off), who had been diagnosed with COVID-19 according to WHO interim guidance (21). A confirmed case of COVID-19 was defined as a positive result on reverse transcription-polymerase chain reaction (RT-PCR) of nasal and pharyngeal swab specimens. Only laboratory-confirmed patients were included in the final analysis.
An independent cohort of patients with the same clinical characteristics at Hubei Xinhua Hospital (another designated hospital for COVID-19) formed the external validation cohort, and it was also approved by the Ethics Commission of this institution.
Data collection
The patient list was filtered and derived from the His Medical System. We reviewed clinical electronic medical records, laboratory findings and radiological examinations for all laboratory confirmed COVID-19 cases. The admission data (demographic characteristics, major symptoms or signs) of these cases were documented. If the core records were missing or uncertain, requests for clarification were sent to involved health-care providers or their families. Age, sex, important chronic medical histories, key laboratory findings on admission, as well as living status up to the cut-off time were collected. Data were then entered into a computerized database for evaluation and cross-checking.
Statistical analysis
The endpoint for building this prediction model was death. Continuous variables were presented as median [interquartile range (IQR)], and were compared between survivors and non-survivors using Mann-Whitney U test or t-test as appropriate. Then, they were re-categorized into categorical variables using threshold values implicit in the existing severity assessment rules or clinical significance. Categorical variables were reported as whole numbers and proportions, and were compared by the χ2 test or Fisher exact test where appropriate.
Clinical and biomarker variables associated with death were assessed a-priori based on scientific knowledge, clinical importance, and risk factors identified in previously published articles (6,12,15,16,22-25). The significance of each variable in the derivation cohort was evaluated by univariate Cox regression analysis. A correlation matrix was used to evaluate the variables for collinearity, and plausible interaction terms were tested, including interactions between aspartate aminotransferase (AST)-alanine transaminase (ALT), creatinine-blood urea nitrogen, neutrophil-white blood cell and lymphocytes-white blood cell. Variables were excluded if they had high collinearity with global scores. CURB-65 was used as a comparison to our nomogram, and it was not involved in the model construction. Then, backward stepwise selection based on the Akaike information criterion (AIC) was applied to identify independent variables for developing the multivariable Cox proportional hazards regression model. Hazard ratios (HRs) and their 95% confidence intervals (CIs) were calculated (26). Selected variables were used to construct the nomogram to predict the probability of 30-day mortality on admission. For allocating points in the nomogram, the regression coefficients of each variable were applied to define the linear predictor (27).
The discriminating ability of the nomogram was evaluated using the Harrell’s concordance index (C-index) in the derivation and validation cohort, which estimates the probability of concordance between predicted and observed event rate in rank order (28). By using a bootstrapped sample (n=1,000) of the derivation and validation group, we plotted calibration curves to evaluate the accuracy of prediction of the nomogram.
The nomogram predicted probability of 30-day mortality for each patient were calculated according to the multivariable Cox model. Then, we plotted Kaplan-Meier curves of the tertiles of predictions in the derivation and validation cohort, and three prognostic categories were created (low, intermediate and high mortality risk group) based on the tertiles. Differences between the three groups were compared by the log-rank test. Clinical usefulness and net benefit were assessed with decision curve analysis (29).
Statistical analyses were conducted with RStudio version 1.1.419 (RStudio Inc.) and SPSS 20.0 software (SPSS Inc., Chicago, IL, USA). A two-sided P value less than 0.05 was considered statistically significant.
Results
Demographic and clinical characteristics
The derivation cohort consisted of 1,717 adult patients who had been confirmed as COVID-19 between January 1, 2020, and April 10, 2020. The external validation cohort consisted of 188 patients that had been confirmed as COVID-19 between January 13, 2020, and March 25, 2020. The median follow-up was 45 days (IQR, 35–98 days) for the derivation cohort, and 39 days (IQR, 31–87 days) for the validation cohort.
In the derivation cohort, the median age was 64 years (IQR, 53–71 years) and 50.9% (874 of 1,717) were male. The Kaplan-Meier estimated survival was 85.2% (95% CI, 83.0–87.4%) at 30 days. The in-hospital mortality was 11.7% (201 of 1,717), 67.2% (135 of 201) of the deceased patients were male and 68.7% (138 of 201) were older than 65 years. Of the 201 total deaths, 198 (98.5%) happened within 30 days. The median duration from admission to death was 10 days (IQR, 4.5–15 days). Baseline characteristics of patients who remained alive and who had died up to the time of analysis are compared in Table 1. The baseline clinical characteristics were similar between the derivation and validation cohort for most comparisons.
Table1. Demographics and baseline characteristics of enrolled patients on admission.
| Variable | Derivation cohort (n=1,717) | Validation cohort (n=188) | P† | |||||
|---|---|---|---|---|---|---|---|---|
| Survivors (n=1,516) | Non-survivors (n=201) | P* | Survivors (n=161) | Non-survivors (n=27) | P* | |||
| Major clinical characteristics | ||||||||
| Age, years | 63.0 (51.0–70.0) | 70.0 (63.0–78.0) | <0.001 | 65.0 (54.0–72.0) | 65.0 (61.0–76.0) | 0.952 | 0.340 | |
| Male sex | 739 (48.7) | 135 (67.2) | <0.001 | 78 (48.4) | 20 (74.1) | 0.014 | 0.750 | |
| Fever (temperature ≥37.3 °C) | 1,061 (70.0) | 146 (72.6) | 0.440 | 116 (72.0) | 20 (74.1) | 0.828 | 0.560 | |
| Cough | 698 (46.0) | 70 (34.8) | 0.003 | 76 (47.2) | 13 (48.1) | 0.928 | 0.494 | |
| Dyspnea | 350 (23.1) | 60 (29.9) | 0.035 | 35 (21.7) | 11 (40.7) | 0.034 | 0.857 | |
| Hypertension | 481 (31.7) | 91 (45.3) | <0.001 | 49 (30.4) | 9 (33.3) | 0.763 | 0.496 | |
| Diabetes mellitus | 150 (9.9) | 29 (14.4) | 0.048 | 17 (10.6) | 5 (18.5) | 0.234 | 0.588 | |
| COPD | 77 (5.1) | 27 (13.4) | <0.001 | 14 (8.7) | 3 (11.1) | 0.685 | 0.111 | |
| Cardiovascular disease | 114 (7.5) | 33 (16.4) | <0.001 | 13 (8.1) | 3 (11.1) | 0.601 | 0.981 | |
| Smoking history | 136 (9.0) | 25 (12.4) | 0.113 | 11 (6.8) | 2 (7.4) | 0.913 | 0.266 | |
| qSOFA score | 0 (0–0) | 1 (0–1) | <0.001 | 0 (0–1) | 1 (0–1) | 0.102 | <0.001 | |
| 0–1 | 1,504 (99.2) | 181 (90.0) | <0.001‡ | 97 (60.2) | 13 (48.1) | 0.004‡ | ||
| 2 | 10 (0.7) | 16 (8.0) | 61 (37.9) | 10 (37.0) | ||||
| 3 | 2 (0.1) | 4 (2.0) | 3 (1.9) | 4 (14.8) | ||||
| CURB-65 score | <0.001‡ | <0.001‡ | 0.198 | |||||
| 0–1 | 1,327 (87.5) | 93 (46.3) | 132 (82.0) | 16 (59.3) | ||||
| 2 | 169 (11.1) | 75 (37.3) | 26 (16.1) | 4 (14.8) | ||||
| 3–5 | 20 (1.3) | 33 (16.4) | 3 (1.9) | 7 (25.9) | ||||
| Key laboratory findings [normal range] | ||||||||
| White blood cell count, ×109/L [4–10] | 5.7 (4.5–7.4) | 9.1 (5.8–13.4) | <0.001 | 5.6 (4.4–7.6) | 9.2 (5.6–12.9) | 0.038 | 0.995 | |
| <4 | 255 (16.8) | 18 (9.0) | <0.001‡ | 31 (19.3) | 3 (11.1) | <0.001‡ | ||
| 4–10 | 1,126 (74.3) | 93 (46.3) | 116 (72.0) | 11 (40.7) | ||||
| >10 | 135 (8.9) | 90 (44.8) | 14 (8.7) | 13 (48.1) | ||||
| Lymphocyte count, ×109/L [1.1–3.2] | 1.09 (0.76–1.52) | 0.57 (0.42–0.85) | <0.001 | 1.01 (0.72–1.39) | 0.57 (0.42–0.75) | <0.001 | 0.039 | |
| ≥1.0 | 868 (57.3) | 35 (17.4) | <0.001‡ | 83 (51.6) | 2 (7.4) | <0.001‡ | ||
| ≥0.8 to <1.0 | 232 (15.3) | 23 (11.4) | 28 (17.4) | 2 (7.4) | ||||
| ≥0.5 to <0.8 | 298 (19.7) | 61 (30.3) | 36 (22.4) | 15 (55.6) | ||||
| <0.5 | 118 (7.8) | 82 (40.8) | 14 (8.7) | 8 (29.6) | ||||
| Neutrophil count, ×109/L [1.8–6.3] | 3.9 (2.7–5.5) | 7.9 (4.5–12.0) | <0.001 | 3.8 (2.7–5.8) | 8.0 (4.8–11.9) | 0.004 | 0.800 | |
| <1.8 | 95 (6.3) | 7 (3.5) | <0.001‡ | 11 (6.8) | 2 (7.4) | <0.001‡ | ||
| ≥1.8 to <6.3 | 1,157 (76.3) | 70 (34.8) | 118 (73.3) | 9 (33.3) | ||||
| ≥6.3 | 264 (17.4) | 124 (61.7) | 32 (19.9) | 16 (59.3) | ||||
| Platelet count, ×109/L [125–350] | 224.0 (169.0–296.8) | 159.0 (103.5–223.5) | <0.001 | 228.0 (169.5–293.0) | 149.0 (97.0–215.0) | 0.004 | 0.650 | |
| <125 | 124 (8.2) | 66 (32.8) | <0.001 | 13 (8.1) | 10 (37.0) | <0.001 | ||
| Haemoglobin, g/L [130–175] | 126.0 (115.0–137.0) | 128.0 (112.5–144.5) | 0.417 | 125.0 (114.0–137.0) | 126.0 (112.0–143.0) | 1.000 | 0.657 | |
| <130 | 549 (36.2) | 85 (42.3) | 0.094 | 59 (36.6) | 11 (40.7) | 0.684 | ||
| Procalcitonin, ng/mL [0.02–0.05] | 0.06 (0.04–0.10) | 0.30 (0.13–0.88) | <0.001 | 0.06 (0.04–0.13) | 0.18 (0.10–0.95) | <0.001 | 0.387 | |
| <0.05 | 466 (30.7) | 8 (4.0) | <0.001‡ | 49 (30.4) | 1 (3.7) | <0.001‡ | ||
| ≥0.05 to <0.5 | 978 (64.5) | 120 (59.7) | 103 (64.0) | 14 (51.9) | ||||
| ≥0.5 to <2 | 55 (3.6) | 48 (23.9) | 5 (3.1) | 9 (33.3) | ||||
| ≥2 | 17 (1.1) | 25 (12.4) | 4 (2.5) | 3 (11.1) | ||||
| hs-TnI, pg/mL [≤15.6] | 4.0 (0.0–9.1) | 33.0 (11.2–205.6) | <0.001 | 4.4 (2.0–12.4) | 13.9 (4.6–140.8) | 0.009 | 0.775 | |
| >15.6 | 149 (9.8) | 117 (58.2) | <0.001 | 24 (14.9) | 13 (48.1) | <0.001 | ||
| hs-CRP, mg/L [<1] | 20.2 (3.5–63.6) | 102.9 (58.7–162.4) | <0.001 | 22.9 (3.7–71.5) | 82.5 (39.7–139.7) | <0.001 | 0.246 | |
| >10 | 936 (61.7) | 195 (97.0) | <0.001 | 102 (63.4) | 25 (92.6) | 0.003 | ||
| LDH, U/L [135–225] | 264.0 (209.0–345.8) | 504.0 (369.5–673.0) | <0.001 | 280.0 (217.5–369.5) | 439.0 (353.0–695.0) | <0.001 | 0.102 | |
| >225 | 1,062 (70.1) | 193 (96.0) | <0.001 | 123 (76.4) | 26 (96.3) | 0.018 | ||
| Serum creatinine, μmol/L [59–104] | 68.0 (56.0–83.0) | 86.0 (66.5–112.5) | <0.001 | 66.0 (56.0–81.5) | 87.0 (66.0–141.0) | 0.011 | 0.197 | |
| >104 | 148 (9.8) | 67 (33.3) | <0.001 | 23 (14.3) | 9 (33.3) | 0.015 | ||
| Blood urea nitrogen, mmol/L [3.1–8.1] | 4.4 (3.3–5.7) | 8.3 (5.8–12.7) | <0.001 | 4.3 (3.4–5.8) | 8.1 (4.8–16.2) | 0.002 | 0.942 | |
| >8 | 111 (7.3) | 95 (47.3) | <0.001 | 11 (6.8) | 11 (40.7) | <0.001 | ||
| Total bilirubin, μmol/L [≤26] | 8.7 (6.4–12.0) | 12.8 (9.2–18.9) | <0.001 | 8.8 (6.0–12.2) | 12.4 (8.3–19.2) | 0.013 | 0.993 | |
| >26 | 38 (2.5) | 28 (13.9) | <0.001 | 4 (2.5) | 3 (11.1) | 0.028 | ||
| AST, U/L [≤40] | 25 (19–37) | 42.0 (29.5–63.5) | <0.001 | 27.0 (19.0–42.0) | 47.0 (27.0–63.0) | 0.074 | 0.166 | |
| >40 | 393 (25.9) | 119 (59.2) | <0.001 | 55 (34.2) | 17 (63.0) | 0.004 | ||
| Alanine aminotransferase, U/L [≤41] | 23 (14–38) | 26.0 (18.0–41.5) | 0.204 | 23.0 (14.0–41.0) | 29.0 (17.0–59.0) | 0.191 | 0.979 | |
| >41 | 366 (24.1) | 54 (26.9) | 0.399 | 43 (26.7) | 11 (40.7) | 0.136 | ||
| Albumin, g/L [35–52] | 35.1 (31.6–38.8) | 31.0 (27.8–33.6) | <0.001 | 34.9 (30.7–38.4) | 31.1 (28.0–35.1) | 0.015 | 0.595 | |
| <35 | 736 (48.5) | 170 (84.6) | <0.001 | 82 (50.9) | 20 (74.1) | 0.026 | ||
| D-dimer, µg/mL [≤0.5] | 0.74 (0.36–1.60) | 1.30 (0.00–2.95) | <0.001 | 0.70 (0.31–1.75) | 1.39 (0.00–3.38) | 0.191 | 0.661 | |
| ≤0.5 | 508 (33.5) | 7 (3.5) | <0.001‡ | 55 (34.2) | 2 (7.4) | <0.001‡ | ||
| >0.5 to ≤1 | 358 (23.6) | 21 (10.4) | 39 (24.2) | 0 (0) | ||||
| >1 | 650 (42.9) | 173 (86.1) | 67 (41.6) | 25 (92.6) | ||||
| Prothrombin time, s [11.5–14.5] | 13.9 (13.3–14.5) | 15.3 (14.3–16.6) | <0.001 | 13.9 (13.3–14.7) | 15.3 (14.2–17.1) | 0.001 | 0.306 | |
| >14.5 | 344 (22.7) | 140 (69.7) | <0.001 | 48 (29.8) | 18 (66.7) | <0.001 | ||
Data are median (IQR) or n (%). P values were calculated by Mann-Whitney U test, t-test, χ2 test, or Fisher’s exact test, as appropriate. *, represents difference between survivors and non-survivors; †, represents difference between derivation and validation cohort; ‡, χ2 test comparing all subcategories. COPD, chronic obstructive pulmonary disease; qSOFA, quick Sequential Organ Failure Assessment; hs-TnI, high-sensitivity troponin I; hs-CRP, high sensitivity C-reactive protein; LDH, lactate dehydrogenase; AST, aspartate aminotransferase.
Predictors of mortality and model construction
The results of Cox regression analysis are presented in Table 2. Backward selection procedure based on the AIC identified the following ten variables that had the strongest association with death: age, sex, chronic obstructive pulmonary disease (COPD), AST, high-sensitivity C-reactive protein (hs-CRP), high-sensitive troponin I (hs-TnI), white blood cell count, lymphocyte count, D-dimer, and procalcitonin (all Wald tests =405.2, P<0.001). A nomogram for predicting 30-day mortality on admission was developed based on the 10 independent prognostic factors (Figure 1). The model was given the acronym age, biomarkers, COPD, sex (ABCS)-mortality score. The scoring system is shown in Table 3.
Table 2. Cox proportional Hazards regression model showing the association of variables with mortality in the derivation cohort.
| Variable | Univariable | Multivariable | |||
|---|---|---|---|---|---|
| HR (95% CI) | P value | HR (95% CI) | P value | ||
| Factors selected | |||||
| Age, years | 1.05 (1.04–1.07) | <0.001 | 1.02 (1.01–1.36) | 0.001 | |
| Sex | |||||
| Female | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| Male | 2.07 (1.54–2.78) | <0.001 | 1.52 (1.11–2.08) | 0.009 | |
| COPD | |||||
| No | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| Yes | 2.71 (1.81–4.07) | <0.001 | 1.58 (1.04–2.41) | 0.034 | |
| AST, U/L | |||||
| ≤40 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| >40 | 3.01 (2.50–4.38) | <0.001 | 1.56 (1.16–2.10) | 0.003 | |
| hs-CRP, mg/L | |||||
| ≤10 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| >10 | 14.59 (6.47–32.89) | <0.001 | 2.86 (1.20–6.84) | 0.018 | |
| hs-TnI, pg/mL | |||||
| ≤15.6 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| >15.6 | 8.86 (6.69–11.73) | <0.001 | 2.88 (2.06–4.02) | <0.001 | |
| White blood cell count, ×109/L | |||||
| 4–10 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| <4 | 0.84 (0.51–1.39) | 0.499 | 1.14 (0.68–1.92) | 0.610 | |
| >10 | 5.96 (4.46–7.97) | <0.001 | 2.27 (1.65–3.13) | <0.001 | |
| Lymphocyte count, ×109/L | |||||
| ≥1.0 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| ≥0.8 to <1.0 | 2.07 (1.22–3.50) | 0.007 | 1.03 (0.60–1.76) | 0.926 | |
| ≥0.5 to <0.8 | 4.02 (2.65–6.10) | <0.001 | 1.29 (0.83–2.00) | 0.260 | |
| <0.5 | 10.63 (7.15–15.79) | <0.001 | 2.40 (1.56–3.68) | <0.001 | |
| D-dimer, µg/L | |||||
| ≤0.5 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| >0.5 to ≤1 | 3.44 (1.46–8.11) | 0.005 | 1.17 (0.48–2.82) | 0.735 | |
| >1 | 14.07 (6.61–29.97) | <0.001 | 2.19 (1.97–4.92) | 0.029 | |
| Procalcitonin, ng/mL | |||||
| <0.05 | 1.00 (ref.) | – | 1.00 (ref.) | – | |
| ≥0.05 to <0.5 | 6.44 (3.15–13.18) | <0.001 | 2.16 (1.02–4.58) | 0.044 | |
| ≥0.5 to <2 | 32.28 (15.27–68.23) | <0.001 | 3.06 (1.34–6.99) | 0.008 | |
| ≥2 | 51.38 (23.16–113.96) | <0.001 | 4..12 (1.72–9.87) | 0.001 | |
| Factors not selected | |||||
| Hypertension | |||||
| No | 1.00 (ref.) | – | – | – | |
| Yes | 1.70 (1.29–2.25) | <0.001 | – | – | |
| Diabetes mellitus | |||||
| No | 1.00 (ref.) | – | – | – | |
| Yes | 1.50 (1.01–2.22) | 0.043 | – | – | |
| Cardiovascular disease | |||||
| No | 1.00 (ref.) | – | – | – | |
| Yes | 3.33 (1.60–3.38) | <0.001 | – | – | |
| Smoking history | |||||
| No | 1.00 (ref.) | – | – | – | |
| Yes | 1.50 (0.98–2.28) | 0.059 | – | – | |
| qSOFA score | 0.79 (0.56–1.13) | 0.194 | – | – | |
| CURB-65 score | |||||
| 0–1 | 1.00 (ref.) | – | – | – | |
| 2 | 5.05 (3.72–6.85) | <0.001 | – | – | |
| 3–5 | 12.52 (8.41–18.64) | <0.001 | – | – | |
| Neutrophil count, ×109/L | |||||
| 1.8–6.3 | 1.00 (ref.) | – | – | – | |
| <1.8 | 1.21 (0.55–2.62) | 0.637 | – | – | |
| >6.3 | 5.94 (4.43–7.96) | <0.001 | – | – | |
| Platelet count, ×109/L | |||||
| ≥125 | 1.00 (ref.) | – | – | – | |
| <125 | 4.31 (3.21–5.78) | <0.001 | – | – | |
| Haemoglobin, g/L | |||||
| ≥130 | 1.00 (ref.) | – | – | – | |
| <130 | 1.27 (0.96–1.68) | 0.092 | – | – | |
| LDH, U/L | |||||
| ≤225 | 1.00 (ref.) | – | – | – | |
| >225 | 7.53 (3.71–15.28) | <0.001 | – | – | |
| Serum creatinine, μmol/L | |||||
| ≤104.0 | 1.00 (ref.) | – | – | – | |
| >104.0 | 3.92 (2.92–5.25) | <0.001 | – | – | |
| Total bilirubin, μmol/L | |||||
| ≤21.0 | 1.00 (ref.) | – | – | – | |
| >21.0 | 5.47 (3.67–8.16) | <0.001 | – | – | |
| Alanine aminotransferase, U/L | |||||
| ≤41.0 | 1.00 (ref.) | – | – | – | |
| >41.0 | 1.08 (0.79–1.47) | 0.650 | – | – | |
| Albumin, g/L | |||||
| ≥35.0 | 1.00 (ref.) | – | – | – | |
| <35.0 | 4.44 (3.03–6.52) | <0.001 | – | – | |
| Blood urea nitrogen, mmol/L | |||||
| ≤8.0 | 1.00 (ref.) | – | – | – | |
| >8.0 | 7.94 (6.01–10.47) | <0.001 | – | – | |
| Prothrombin time, s | |||||
| ≤14.5 | 1.00 (ref.) | – | – | – | |
| >14.5 | 6.03 (4.46–8.15) | <0.001 | – | – | |
HR, hazard ratio; COPD; chronic obstructive pulmonary disease; AST, aspartate aminotransferase; hs-CRP, high sensitivity C-reactive protein; hs-TnI, high-sensitivity troponin I; qSOFA, quick Sequential Organ Failure Assessment; LDH, lactate dehydrogenase.
Figure 1.
Nomogram predicting 30-day mortality for patients with COVID-19. COPD, chronic obstructive pulmonary disease; AST, aspartate aminotransferase; hs-CRP, high sensitivity C-reactive protein; hs-TnI, high-sensitivity troponin I.
Table 3. Point assignment and prognostic score.
| Variables | Points | Probability of 30-day mortality |
|---|---|---|
| Age (years) | ||
| 10 | 0 | |
| 20 | 11 | |
| 30 | 22 | |
| 40 | 33 | |
| 50 | 44 | |
| 60 | 56 | |
| 70 | 67 | |
| 80 | 78 | |
| 90 | 89 | |
| 100 | 100 | |
| hs-CRP, mg/L | ||
| ≤10 | 0 | |
| >10 | 50 | |
| White blood cell count, 109/L | ||
| 4–10 | 0 | |
| <4 | 7 | |
| ≥10 | 40 | |
| D-dimer, µg/L | ||
| ≤0.5 | 0 | |
| ≥0.5 to <1.0 | 7 | |
| ≥1 | 37 | |
| Sex | ||
| Female | 0 | |
| Male | 20 | |
| COPD | ||
| No | 0 | |
| Yes | 23 | |
| AST, U/L | ||
| ≤40 | 0 | |
| >40 | 22 | |
| hs-TnI, pg/mL | ||
| ≤15.6 | 0 | |
| >15.6 | 52 | |
| Lymphocyte count, 109/L | ||
| ≥1.0 | 0 | |
| ≥0.8 to <1.0 | 1 | |
| ≥0.5 to <0.8 | 12 | |
| <0.5 | 43 | |
| Procalcitonin, ng/mL | ||
| <0.05 | 0 | |
| ≥0.05 to <0.5 | 37 | |
| ≥0.5 to <2 | 54 | |
| ≥2 | 69 | |
| Total points | ||
| 216 | 0.1 | |
| 252 | 0.2 | |
| 275 | 0.3 | |
| 292 | 0.4 | |
| 307 | 0.5 | |
| 320 | 0.6 | |
| 333 | 0.7 | |
| 347 | 0.8 | |
| 365 | 0.9 |
hs-CRP, high sensitivity C-reactive protein; COPD, chronic obstructive pulmonary disease; AST, aspartate aminotransferase; hs-TnI, high-sensitivity troponin I.
Model performance
The ABCS-mortality score showed good accuracy in estimating the risk of 30-day mortality in the derivation cohort, with a bootstrap-corrected C-index of 0.888 (95% CI, 0.869–0.907) (Table 4). In addition, calibration plots graphically demonstrated good agreement between the risk estimation by the prediction model and the Kaplan-Meier estimated 30-day mortality rate (Figure 2A). In the validation cohort, the model was also well calibrated (Figure 2B), achieving a C-index of 0.838 (95% CI, 0.777–0.899) for the prediction of 30-day mortality risk (Table 4). The CURB-65 score achieved a C-index of 0.696 (95% CI, 0.660–0.731) and 0.619 (95% CI, 0.519–0.720) in the derivation and validation cohort, respectively (Table 4).
Table 4. C-indexes for predicting 30-day mortality for the ABCS-mortality score compared with the CURB-65 score in the derivation and validation cohort.
| Risk model | Derivation cohort | Validation cohort |
|---|---|---|
| Cohort (events/n) | 201/1,717 | 27/188 |
| ABCS-mortality | 0.888 (0.869–0.907) | 0.838 (0.777–0.899) |
| ABCS-mortality (LDH replace AST) | 0.886 (0.867–0.905) | 0.836 (0.776–0.896) |
| ABCS-mortality (excluding hs-TnI) | 0.878 (0.859–0.896) | 0.830 (0.763–0.898) |
| CURB-65 | 0.696 (0.660–0.731) | 0.619 (0.519–0.720) |
Data are C-indices (95% CI) for each score. CURB-65 score is confusion, urea >7 mmol/L, respiratory rate >30/min, blood pressure (SBP <90 mmHg or DBP ≤60 mmHg), age ≥65 years. ABCS, age, biomarkers, COPD, sex; AST, aspartate aminotransferase; LDH, lactate dehydrogenase; hs-TnI, high-sensitivity troponin I; COPD, chronic obstructive pulmonary disease.
Figure 2.
Calibration plots for estimating mortality rate at 30-day. Calibration plots are shown for the derivation (A) and external validating cohort (B). The 45° dotted line is the reference line that indicates where a perfect calibration would lie. Error bars represent the 95% confidence interval of the Kaplan-Meier calculation.
Clinical utility of the model
The cutoff value of the tertiles of the predictions was 2.0% and 9.0% based on the derivation cohort, and was applied to the validation cohort. Risk categories were created according to less than 2.0% (low mortality risk group), 2.0–9.0% (intermediate mortality risk group), and greater than 9.0% (high mortality risk group) risk for death within 30 days.
To further evaluate the discriminative ability of the model, the risk stratified Kaplan-Meier curves based on the risk categories were plotted for the derivation (Figure 3A) and validation cohort (Figure 3B). As shown, patients with the highest predicted 30-day mortality rate (>9%) had a substantially worse outcome compared with patients in the other two groups. Comparison of mortality frequencies within different risk categories for both the derivation (Figure 3C) and the validation cohort (Figure 3D) shows that the ABCS-mortality score had good discriminative ability in different subgroups of patients with COVID-19. The ABCS-mortality score displayed improved risk stratification within the range of the CURB-65 score risk classes (Figure 4), including patients with a low CURB-65 score of 0–1 point. By contrast, the CURB-65 score did not improve the risk stratification within the different ABCS-mortality risk groups (Figure S1).
Figure 3.
Risk stratified Kaplan-Meier curves and frequencies of death in different predefined risk groups based on tertiles of the predicted 30-day mortality. Cumulative mortality rate for the derivation (A) and validation cohort (B) in different mortality risk groups. Comparison of mortality frequencies within different mortality risk groups for the derivation (C) and validation cohort (D).
Figure 4.
Cumulative rates of mortality by ABCS-mortality risk classes in the CURB-65 score. Kaplan-Meier estimated cumulative event rate by the three ABCS-mortality risk classes (low, intermediate, and high) for the CURB-65 score (0-1 points, 2 points, and 3-5 points). ABCS, age, biomarkers, COPD, sex; COPD, chronic obstructive pulmonary disease.
Finally, decision curve analysis was conducted to compare different prediction models. As seen in Figure 5, the decision curve analysis reveals the clinical usefulness of each model based on a continuum of potential thresholds for mortality risk (x axis) and the net benefit of applying the model to stratify patients (y axis). In this analysis, the ABCS-mortality risk score provided a larger net benefit across the range of mortality risk compared with the CURB-65 score.
Figure 5.
Decision curve analysis for the ABCS-mortality score and CURB-65 score. ABCS, age, biomarkers, COPD, sex; COPD, chronic obstructive pulmonary disease.
Considering differences between healthcare systems, additional models were created in the same manner using alternative biomarkers, i.e., replacing AST with LDH, or excluding the hs-TnI (for it might not be routinely tested in other hospitals). The ABCS-mortality score including LDH (instead of AST) provided similar results with C-index of 0.886 (95% CI, 0.867–0.905) in the derivation and 0.836 (95% CI, 0.776–0.896) in the external validation cohort (Table 4). The nomogram and calibration curves for this alternative ABCS-mortality model are shown in the Figure S2 and Figure S3, and the scoring system is shown in Table S1.
When excluded hs-TnI, while the other variables remained in the model (Figure S4), the ABCS-mortality score was slightly poorer than the model with hs-TnI, but still consistently outperformed the CURB-65 score (Table 4). The calibration curves for this alternative model is shown in Figure S5, and the scoring system is shown in Table S2.
Discussion
COVID-19 is spreading rapidly worldwide, and about 81% cases were classified as mild (non-pneumonia and mild pneumonia). However, it can also be deadly, for 14% cases were severe and presented as dyspnea, lung infiltrates and hypoxemia, and 5% progressed rapidly with acute respiratory distress syndrome, septic shock, multiorgan failure, and died finally (2,8). Therefore, early identification and a proper therapeutic strategy for cases with high mortality risk is of crucial importance to save lives.
CURB-65 is a widely used tool for severity assessment of community acquired pneumonia (19), but has not been developed in the setting of acute viral infection. A previously reported tool, the MuLBSTA scoring system, has been developed to predict mortality risk among patients with viral pneumonia, and it provided accurate estimates of patient prognosis (22). However, as a new coronavirus pneumonia, the intrinsic pathogenesis of SARS-CoV-2 caused pneumonia may differ significantly from other viral pneumonias, and a severity assessment model focusing on this specific subgroup of patients is urgently needed. Recently, a prediction model for assessing progression risk in patients with COVID-19 was proposed (18). However, we don’t know whether the identified risk factors were also associated with short-term mortality risk of COVID-19. What’s more, the sample size was relatively small and a validation dataset was absent. In this context, we developed and validated an ABCS-mortality score for predicting 30-day mortality risk in adult patients with COVID-19 on admission, and we hope that this model could be helpful for stratifying patients into different prognostic groups, and facilitate clinical decision-making.
The strength of this study lies in the focusing of statistical analyses of association on a-priori chosen predictors based on the current clinical evidence (2,6,8,13,15,19,22), which could avoid spurious associations. Although ten risk factors were eventually included in the model, it is an easy-to-use visual graphic tool for clinical application. Apart from demographic characteristics like age, sex and clinical history of COPD, the other biomarkers are routinely tested in many hospitals and are easily obtained. As shown in Figure 1, the model has incorporated important biomarkers that have been confirmed to provide incremental information about the risk of mortality in patients with COVID-19: AST, a marker of liver function; hs-TnI, indicating cardiac injury; white blood cell count and lymphocyte count, reflecting the status of immune system; D-dimer, a marker of coagulation disorders (4,6,7,15,24). Additionally, biomarkers reflecting inflammatory activity and bacterial infection such as hs-CRP and procalcitonin were also reliable indicators of adverse outcomes for COVID-19 (13). This information can be applied to inform the prognosis of the patients, and to make individualized decisions regarding the treatment and follow-up.
Accurate risk stratification is important because the prognosis of the COVID-19 is heterogeneous. Since no specific treatment (such as targeted antiviral drugs) is available for COVID-19 currently, the mainstay of treatment has been supportive care. For non-severe patients, isolation and close follow-up might be sufficient to manage this disease (1,2). However, for potential critical patients, aggressive therapeutic strategies or intensive care are needed. Thus, stratifying patients into different prognostic groups on admission is crucial for rational medical resources allocation and management selection.
For clinical use of the ABCS-mortality score, patients were stratified into different mortality risk groups based on the tertiles of predicted 30-day mortality. This model showed good discriminative ability in distinct groups of patients who were at different risk of mortality. It also consistently predicted 30-day mortality with a higher accuracy than the CURB-65 risk score and provided better clinical usefulness throughout the range of mortality risk as assessed by the decision curve analysis. The ABCS-mortality score could enable patients to be categorized into suitable groups for three distinct management options: patients at low risk of mortality (<2.0%) might be suitable for isolation at temporary health clinics (such as Fang Cang Shelter in China) and close medical surveillance; patients at intermediate risk of mortality (2.0–9.0%) should be considered for supervised treatment at designated hospital; for patients at high risk of mortality on admission (>9.0% or even higher), initial care in an intensive care unit may be appropriate.
Limitations of this study should be acknowledged. First, the external validation was only performed in a small Chinese dataset, validation in larger, international datasets is needed to further improve the generalizability and robustness of this model. Second, owing to differences between healthcare systems in managing patients with COVID-19, this model will need to be adjusted to the local setting, even though two alternative models have been proposed in the current study. Third, the patients who were asymptomatic or had mild symptoms that were managed at home or isolation sites were not included, so our study cohort may represent a more severe subgroup of COVID-19 cases. Furthermore, during the onset of the pandemic, several drugs (hydroxychloroquine, azithromycin, lopinavir/ritonavir, dexamethasone, remdesivir etc.) had been proposed that might be useful to improve the prognosis of COVID-19. The use of any of these drugs might change the predictions of the proposed model and lessen its applicability. However, since none treatment standards (dosage and timing of administration) were available about these drugs, it was hard to evaluate the effect they had on mortality. More high-level evidence based on randomized controlled trials are needed to prove their effectiveness. Finally, objective parameters on lung function at disease presentation (such as oxygen saturation percentage, PaO2/FiO2, SaO2/FiO2, oxygen requirements) were not included in the model due to missing data in the His Medical System. In addition, some potential mortality predictors identified in previous literatures (such as hypertension, diabetes, and chronic cardiovascular diseases) were linked to mortality by univariate Cox regression analysis in the current study (P<0.05 for all), but were excluded during the backward stepwise selection procedure due to relatively low prediction power. Other variables like obesity and chronic kidney diseases were also not involved in the analysis due to incomplete information. Therefore, the choice of potential prognostic factors could not be entirely informed by a systematic review of previous literatures.
In conclusion, this study offers a simple prediction model based on the clinical features of age, sex, clinical history of COPD, as well as seven biomarkers as a practical tool of stratifying COVID-19 patients into distinct prognostic groups. The ABCS-mortality score performed well on the derivation and independent external validation cohort. This tool can be offered to clinicians to improve their ability to predict patients’ prognosis on admission, and strengthen the prognosis-based clinical decision making, which might be helpful for rational allocation of scarce medical resources and reducing mortality of COVID-19 patients.
Supplementary
The article’s supplementary files as
Acknowledgments
Funding: This study is supported by the youth medical talent support program of Hubei Province.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted according to the principles of the Declaration of Helsinki (as revised in 2013). The study was approved by Ethics Board of Tongji Hospital and Hubei Xin Hua Hospital (No.: TJ-IRB20200373) and informed consent was waived since the study was retrospectively designed and did not cause any harm to the patients.
Footnotes
Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at http://dx.doi.org/10.21037/atm-20-6205
Data Sharing Statement: Available at http://dx.doi.org/10.21037/atm-20-6205
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/atm-20-6205). The authors have no conflicts of interest to declare.
References
- 1.Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020;395:497-506. 10.1016/S0140-6736(20)30183-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Chen N, Zhou M, Dong X, et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study. Lancet 2020;395:507-13. 10.1016/S0140-6736(20)30211-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Zhu N, Zhang D, Wang W, et al. A Novel Coronavirus from Patients with Pneumonia in China, 2019. New Engl J Med 2020;382:727-33. 10.1056/NEJMoa2001017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Guan WJ, Ni ZY, Hu Y, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. New Engl J Med 2020;382:1708-20. 10.1056/NEJMoa2002032 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Available online: https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6
- 6.Yang X, Yu Y, Xu J, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study. Lancet Respir Med 2020;8:475-81. 10.1016/S2213-2600(20)30079-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Wang D, Hu B, Hu C, et al. Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus–Infected Pneumonia in Wuhan, China. JAMA 2020;323:1061-9. 10.1001/jama.2020.1585 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wu Z, McGoogan JM. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72 314 Cases From the Chinese Center for Disease Control and Prevention. JAMA 2020;323:1239-42. 10.1001/jama.2020.2648 [DOI] [PubMed] [Google Scholar]
- 9.Chen H, Guo J, Wang C, et al. Clinical characteristics and intrauterine vertical transmission potential of COVID-19 infection in nine pregnant women: a retrospective review of medical records. Lancet 2020;395:809-15. 10.1016/S0140-6736(20)30360-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Chan JF, Yuan S, Kok K, et al. A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-to-person transmission: a study of a family cluster. Lancet 2020;395:514-23. 10.1016/S0140-6736(20)30154-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wu JT, Leung K, Leung GM. Nowcasting and forecasting the potential domestic and international spread of the 2019-nCoV outbreak originating in Wuhan, China: a modelling study. Lancet 2020;395:689-97. 10.1016/S0140-6736(20)30260-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Xu XW, Wu XX, Jiang XG, et al. Clinical findings in a group of patients infected with the 2019 novel coronavirus (SARS-Cov-2) outside of Wuhan, China: retrospective case series. BMJ 2020;368:m606. 10.1136/bmj.m606 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chen T, Wu D, Chen H, et al. Clinical characteristics of 113 deceased patients with coronavirus disease 2019: retrospective study. BMJ 2020;368:m1091. 10.1136/bmj.m1091 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Zhang J, Zhou L, Yang Y, et al. Therapeutic and triage strategies for 2019 novel coronavirus disease in fever clinics. Lancet Respir Med 2020;8:e11-e12. 10.1016/S2213-2600(20)30071-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Zhou F, Yu T, Du R, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet 2020;395:1054-62. 10.1016/S0140-6736(20)30566-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Shi S, Qin M, Shen B, et al. Association of Cardiac Injury With Mortality in Hospitalized Patients With COVID-19 in Wuhan, China. JAMA Cardiol 2020;5:802-10. 10.1001/jamacardio.2020.0950 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Inciardi RM, Lupi L, Zaccone G, et al. Cardiac Involvement in a Patient With Coronavirus Disease 2019 (COVID-19). JAMA Cardiol 2020;5:819-24. 10.1001/jamacardio.2020.1096 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ji D, Zhang D, Xu J, et al. Prediction for Progression Risk in Patients with COVID-19 Pneumonia: the CALL Score. Clin Infect Dis 2020;71:1393-9. 10.1093/cid/ciaa414 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Lim WS, van der Eerden MM, Laing R, et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax 2003;58:377-82. 10.1136/thorax.58.5.377 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. 10.7326/M14-0698 [DOI] [PubMed] [Google Scholar]
- 21.World Health Organization. Clinical management of severe acute respiratory infection when novel coronavirus (2019-nCoV) infection is suspected: interim guidance. January 28, 2020 Available online: https://www.who.int/docs/default-source/coronaviruse/clinical-management-of-novel-cov.pdf
- 22.Guo L, Wei D, Zhang X, et al. Clinical Features Predicting Mortality Risk in Patients With Viral Pneumonia: The MuLBSTA Score. Front Microbiol 2019;10:2752. 10.3389/fmicb.2019.02752 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Uranga A, Quintana JM, Aguirre U, et al. Predicting 1-year mortality after hospitalization for community-acquired pneumonia. PLoS One 2018;13:e0192750. 10.1371/journal.pone.0192750 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Ruan Q, Yang K, Wang W, et al. Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China. Intensive Care Med 2020;46:1294-7. 10.1007/s00134-020-06028-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Wynants L, Van Calster B, Bonten MMJ, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ 2020;369:m1328. 10.1136/bmj.m1328 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Simon R. Confidence intervals for reporting results of clinical trials. Ann Intern Med 1986;105:429-35. 10.7326/0003-4819-105-3-429 [DOI] [PubMed] [Google Scholar]
- 27.Shariat SF, Karakiewicz PI, Suardi N, et al. Comparison of Nomograms With Other Methods for Predicting Outcomes in Prostate Cancer: A Critical Analysis of the Literature. Clin Cancer Res 2008;14:4400-7. 10.1158/1078-0432.CCR-07-4713 [DOI] [PubMed] [Google Scholar]
- 28.Harrell FE, Jr, Califf RM, Pryor DB, et al. Evaluating the yield of medical tests. JAMA 1982;247:2543-6. 10.1001/jama.1982.03320430047030 [DOI] [PubMed] [Google Scholar]
- 29.Vickers AJ, Cronin AM, Elkin EB, et al. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak 2008;8:53. 10.1186/1472-6947-8-53 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
The article’s supplementary files as