Abstract
Background
The prognosis of trauma patients is highly dependent on early medical diagnosis. By constructing a nomogram model, the risk of adverse outcomes can be displayed intuitively and individually, which has important clinical implications for medical diagnosis.
Objective
To develop and evaluate models for predicting patients with adverse outcomes of trauma that can be used in different data availability settings in China.
Methods
This was a retrospective prognostic study using data from 8 public tertiary hospitals in China from 2018. The data were randomly divided into a development set and a validation set. Simple, improved and extended models predicting adverse outcomes were developed, with adverse outcomes defined as in-hospital death or ICU transfer, and patient clinical characteristics, vital signs, diagnoses, and laboratory test values as predictors. The results of the models were presented in the form of nomograms, and performance was evaluated using area under the receiver operating characteristic curve (ROC-AUC), precision-recall (PR) curves (PR-AUC), Hosmer-Lemeshow goodness-of-fit test, calibration curve, and decision curve analysis (DCA).
Results
Our final dataset consisted of 18,629 patients (40.2% female, mean age of 52.3), 1,089 (5.85%) of whom resulted in adverse outcomes. In the external validation set, three models achieved ROC-AUC of 0.872, 0.881, and 0.903, and a PR-AUC of 0.339, 0.337, and 0.403, respectively. In terms of the calibration curves and DCA, the models also performed well.
Conclusions
This prognostic study found that three prediction models and nomograms including the patient clinical characteristics, vital signs, diagnoses, and laboratory test values can support clinicians in more accurately identifying patients who are at risk of adverse outcomes in different settings based on data availability.
Keywords: Trauma patients, adverse outcomes, risk prediction model, nomogram
Introduction
Trauma is a major global health threat, causing more deaths than HIV/AIDS, tuberculosis, malaria and maternal mortality combined [1, 2]. It is estimated that more than five million people worldwide die each year from traumatic injuries [3]. The situation is even worse in low- and middle-income countries. On the one hand, more than 90% of deaths occur in these countries [4, 5], on the other hand, the mortality rate from trauma is higher than in high-income countries. In China, in particular, the trauma mortality rate is twice that of most developed countries [6].
As important tools for assessing the condition of trauma patients, outcome prediction models for trauma patients play an important role in accurately assessing injuries and improving outcomes. It is important to select an appropriate scoring system and scoring indices for the accurate assessment of trauma patients [7]. From the last century to the present, a number of clinical predictive models have been developed to support clinicians in the assessment of trauma severity, thus reducing mortality or adverse outcome rates [8–17]. Despite the fact that trauma mortality is higher in developing countries than in developed countries, more of these prediction models have been developed in developed countries, such as the Laboratory Frailty Index (FI-lab) [18], the Croce’s and updated Croce’s model [19, 20] by Kobes et al. [21]. However, these models rely on more detailed data from electronic medical records (EMRs) [22, 23], which are usually difficult to collect and access in developing countries, creating barriers to the widespread adoption of these models. For example, Glasgow Coma Scale (GCS) is a very common and important predictor in these models, but GCS is either not specifically recorded in the information systems of many hospitals in China or is recorded in an unstructured way, making it difficult to use in these models.
Due to limited medical data, particularly the unavailability of some key predictors, we were unable to refine the existing model. Instead, using data from hospitals in China, this study focused more on developing prediction models that are suitable for different scenarios, i.e. dealing with patients who need to be assessed quickly when there is insufficient data, or could be assessed more accurately with sufficient data, and also aimed to shed light on assessing the severity of trauma patients in environments with limited access to data.
Methods
Data source and participants
This retrospective study used the military hospital public service database of the National Engineering Laboratory of Application Technology in Medical Big Data, which was established by the China Development and Reform Commission [24]. Patients were eligible for inclusion if they were 18 years or older, had a primary diagnosis of trauma according to the International Classification of Diseases-Tenth Revision (ICD-10), and were discharged from eight hospitals in Beijing, China, from January 1 to December 31, 2018. Patients with missing values for the predictors were excluded.
Outcome and predictor variables
Adverse outcome was defined as in-hospital death or ICU transfer, with in-hospital death defined as all-cause death during hospitalization and ICU transfer defined as any transfer (including injury-related).
A total of 46 candidate predictors of adverse outcomes were selected with reference to the UK National Early Warning Score (NEWS) and the Laboratory-based Acute Physiological Function Score 2nd Edition (LAPS2), and also on the basis that they were easily accessible. These predictors were categorized as follows: (1) patient clinical characteristics, including sex, age, and admission route; (2) patient vital signs, including temperature, pulse, respiratory rate, systolic blood pressure, and shock index (shock index was defined as the pulse divided by systolic blood pressure); (3) patient diagnoses, including Elixhauser Comorbidity Index [25], New Injury Severity Score (NISS), nature of injury, body region, and mechanism of injury; and (4) patient laboratory tests values, including hemoglobin, blood sodium, serum bilirubin, white blood cell count (WBCC), serum albumin, blood glucose, hematocrit, creatinine, urea nitrogen-creatinine ratio, and blood urea nitrogen. Patients’ vital signs and laboratory tests were the first measurements taken within 24 h of admission.
As we evaluated 46 predictor variables that will be included in the logistic regression model, the sample size for the derivation phase was a minimum of 460 events, according to the recommended sample size requirement for model validation [26, 27]. Our case ascertainment for the derivation and validation cohorts met the minimum sample size requirement.
Statistical analysis
The cohort of trauma patients was randomly divided 7:3 into a training set and an external validation set. Multivariate stepwise logistic regression was used to develop risk prediction models for adverse patient outcomes using the training set. The variance inflation factor (VIF) was calculated to check for multicollinearity among predictors.
The nomogram, a good representation of the prediction model, can convert complex regression results into visual graphics, making the results easier to read and more convenient for patient assessment [28]. Therefore, we used nomograms to visually illustrate the importance of variables in the logistic regression prediction model for adverse outcomes [29]. To further validate the stability of the prediction models, the 200-iteration cross-validation method was used to internally validate the models. The area under the receiver operating characteristic (ROC) curves (ROC-AUC) and the precision-recall (PR) curves (PR-AUC) were used to assess the discrimination of the models, the Hosmer-Lemeshow goodness-of-fit test and the calibration curve were used to assess the model calibration. Clinical decision curve analysis (DCA) was performed to evaluate the clinical utility of the models [30].
Because vital signs, injury characteristics, and routine laboratory tests were more common and easily accessible, these variables were included in the model in this study, while patients with missing values for these variables were excluded. As a result, respiratory rate, pulse rate, mechanism of injury, and blood sodium had many missing values, all of which were excluded. Subsequently, a small percentage of missing serum bilirubin, all less than or equal to 1%, were also excluded and did not affect our analysis.
Model derivation
In the multivariable analyses, we developed three models for predicting adverse outcomes in order of increasing complexity: (1) a simple model that included only demographics, vital signs, and New Injury Severity Score (NISS) as predictors; (2) an improved model that added the nature of injury, body region, and mechanism of injury as predictors based on the simple model; and (3) an extended in-hospital assessment model that added laboratory values as predictors based on the improved model.
A p value < 0.05 was considered statistically significant for the final predictors included in the models. All statistical analyses were performed using R version 4.3.1.
The study protocol was reviewed and approved by the Ethics Committee of Chinese People’s Liberation Army General Hospital, and informed consent was waived due to the retrospective nature of the study and the use of anonymized data (approval number: S2021-187-01). This study followed the reporting guidelines of the Transparent Reporting of a multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) [31].
Results
Characteristics of the patients
A total of 45,677 inpatient medical records were initially identified according to the inclusion criteria. Twenty-seven thousand forty-eight cases were excluded due to missing important predictor variables. The final study set consisted of 18,629 trauma inpatients, and the training and external validation sets consisted of 13,040 and 5,589 patients, respectively (Figure 1). The mean age of the study cohort was 52.3 years (SD, 20.9 years), 7,488 (40.2%) were female, and 1,089 (5.85%) had adverse outcomes. Patient characteristics are shown in Table 1 in detail.
Figure 1.
The flowchart of participants’ enrollment in the training set and external validation set.
Table 1.
Characteristics of the patients in the study cohort.
| Variable | Training set |
Validation set |
p Value* | |||
|---|---|---|---|---|---|---|
| Overall | Without adverse outcomes | Adverse outcomes | Without adverse outcomes | Adverse outcomes | ||
| (N = 18629) | (N = 12295) | (N = 745) | (N = 5245) | (N = 344) | ||
| Sex (%) | 0.794 | |||||
| Female | 7488 (40.2%) | 4910 (39.9%) | 340 (45.6%) | 2090 (39.8%) | 148 (43.0%) | |
| Male | 11141 (59.8%) | 7385 (60.1%) | 405 (54.4%) | 3155 (60.2%) | 196 (57.0%) | |
| Age (years) | 0.983 | |||||
| Mean (SD) | 52.3 (20.9) | 51.3 (20.6) | 67.9 (20.5) | 51.3 (20.6) | 67.0 (21.3) | |
| Admission route (%) | 0.458 | |||||
| Emergency admission | 8575 (46.0%) | 5902 (48.0%) | 124 (16.6%) | 2491 (47.5%) | 58 (16.9%) | |
| Outpatient admission | 10054 (54.0%) | 6393 (52.0%) | 621 (83.4%) | 2754 (52.5%) | 286 (83.1%) | |
| Condition at admission (%) | 0.481 | |||||
| Danger | 161 (0.9%) | 41 (0.3%) | 65 (8.7%) | 22 (0.4%) | 33 (9.6%) | |
| Urgent | 6920 (37.1%) | 4475 (36.4%) | 382 (51.3%) | 1889 (36.0%) | 174 (50.6%) | |
| General | 11548 (62.0%) | 7779 (63.3%) | 298 (40.0%) | 3334 (63.6%) | 137 (39.8%) | |
| Body temperature (%) | 0.492 | |||||
| 0–36.7 | 6850 (36.8%) | 4516 (36.7%) | 243 (32.6%) | 2000 (38.1%) | 91 (26.5%) | |
| 36.7–38 | 11205 (60.1%) | 7430 (60.4%) | 447 (60.0%) | 3109 (59.3%) | 219 (63.7%) | |
| ≥38 | 574 (3.1%) | 349 (2.8%) | 55 (7.4%) | 136 (2.6%) | 34 (9.9%) | |
| Pulse (%) | 0.408 | |||||
| 0–60 | 11555 (62.0%) | 7741 (63.0%) | 317 (42.6%) | 3338 (63.6%) | 159 (46.2%) | |
| 60–100 | 6176 (33.2%) | 4059 (33.0%) | 302 (40.5%) | 1686 (32.1%) | 129 (37.5%) | |
| ≥100 | 898 (4.8%) | 495 (4.0%) | 126 (16.9%) | 221 (4.2%) | 56 (16.3%) | |
| Respiration rate (%) | 0.116 | |||||
| 0–16 | 240 (1.3%) | 116 (0.9%) | 61 (8.2%) | 35 (0.7%) | 28 (8.1%) | |
| 16–19 | 8088 (43.4%) | 5367 (43.7%) | 240 (32.2%) | 2387 (45.5%) | 94 (27.3%) | |
| ≥19 | 10301 (55.3%) | 6812 (55.4%) | 444 (59.6%) | 2823 (53.8%) | 222 (64.5%) | |
| Systolic blood pressure (%) | 0.334 | |||||
| 0–100 | 424 (2.3%) | 258 (2.1%) | 43 (5.8%) | 105 (2.0%) | 18 (5.2%) | |
| 100–120 | 3492 (18.7%) | 2362 (19.2%) | 122 (16.4%) | 949 (18.1%) | 59 (17.2%) | |
| 120–140 | 9622 (51.7%) | 6456 (52.5%) | 268 (36.0%) | 2772 (52.9%) | 126 (36.6%) | |
| ≥140 | 5091 (27.3%) | 3219 (26.2%) | 312 (41.9%) | 1419 (27.1%) | 141 (41.0%) | |
| Shock Index (%) | 0.003 | |||||
| 0–0.65 | 12640 (67.9%) | 8301 (67.5%) | 455 (61.1%) | 3665 (69.9%) | 219 (63.7%) | |
| 0.65–0.85 | 5324 (28.6%) | 3623 (29.5%) | 201 (27.0%) | 1419 (27.1%) | 81 (23.5%) | |
| ≥0.85 | 665 (3.6%) | 371 (3.0%) | 89 (11.9%) | 161 (3.1%) | 44 (12.8%) | |
| Elixhauser Comorbidity Index (%) | 0.044 | |||||
| 0 | 17497 (93.9%) | 11609 (94.4%) | 608 (81.6%) | 5006 (95.4%) | 274 (79.7%) | |
| 1 | 1132 (6.1%) | 686 (5.6%) | 137 (18.4%) | 239 (4.6%) | 70 (20.3%) | |
| NISS (%) | 0.634 | |||||
| 1–4 | 2625 (14.1%) | 1840 (15.0%) | 24 (3.2%) | 750 (14.3%) | 11 (3.2%) | |
| 4–9 | 9746 (52.3%) | 6736 (54.8%) | 88 (11.8%) | 2875 (54.8%) | 47 (13.7%) | |
| 9–16 | 4247 (22.8%) | 2570 (20.9%) | 397 (53.3%) | 1091 (20.8%) | 189 (54.9%) | |
| 16–25 | 676 (3.6%) | 394 (3.2%) | 70 (9.4%) | 185 (3.5%) | 27 (7.8%) | |
| > =25 | 1335 (7.2%) | 755 (6.1%) | 166 (22.3%) | 344 (6.6%) | 70 (20.3%) | |
| Body region (%) | 0.691 | |||||
| Chest torso | 771 (4.1%) | 482 (3.9%) | 58 (7.8%) | 198 (3.8%) | 33 (9.6%) | |
| Extremities | 11985 (64.3%) | 8032 (65.3%) | 374 (50.2%) | 3402 (64.9%) | 177 (51.5%) | |
| Head & neck all | 3194 (17.1%) | 1961 (15.9%) | 284 (38.1%) | 829 (15.8%) | 120 (34.9%) | |
| Spine and upper back | 2679 (14.4%) | 1820 (14.8%) | 29 (3.9%) | 816 (15.6%) | 14 (4.1%) | |
| Nature of injury (%) | 0.523 | |||||
| Fracture | 10594 (56.9%) | 7005 (57.0%) | 424 (56.9%) | 2962 (56.5%) | 203 (59.0%) | |
| Internal organ injuries | 2360 (12.7%) | 1371 (11.2%) | 267 (35.8%) | 607 (11.6%) | 115 (33.4%) | |
| Open wound | 611 (3.3%) | 435 (3.5%) | 7 (0.9%) | 160 (3.1%) | 9 (2.6%) | |
| Others | 5064 (27.2%) | 3484 (28.3%) | 47 (6.3%) | 1516 (28.9%) | 17 (4.9%) | |
| Mechanism of injury (%) | 0.02 | |||||
| Fall | 6545 (35.1%) | 4144 (33.7%) | 358 (48.1%) | 1888 (36.0%) | 155 (45.1%) | |
| Transport | 2324 (12.5%) | 1507 (12.3%) | 152 (20.4%) | 594 (11.3%) | 71 (20.6%) | |
| Unspecified | 9760 (52.4%) | 6644 (54.0%) | 235 (31.5%) | 2763 (52.7%) | 118 (34.3%) | |
| Blood sodium (%) | 0.734 | |||||
| 0–135 | 1132 (6.1%) | 691 (5.6%) | 112 (15.0%) | 276 (5.3%) | 53 (15.4%) | |
| 135–146 | 16916 (90.8%) | 11269 (91.7%) | 558 (74.9%) | 4829 (92.1%) | 260 (75.6%) | |
| ≥146 | 581 (3.1%) | 335 (2.7%) | 75 (10.1%) | 140 (2.7%) | 31 (9.0%) | |
| Serum bilirubin (%) | 0.186 | |||||
| 0–2 | 18186 (97.6%) | 12033 (97.9%) | 710 (95.3%) | 5123 (97.7%) | 320 (93.0%) | |
| ≥2 | 443 (2.4%) | 262 (2.1%) | 35 (4.7%) | 122 (2.3%) | 24 (7.0%) | |
| WBCC (%) | 0.155 | |||||
| 0–5 | 2066 (11.1%) | 1420 (11.5%) | 49 (6.6%) | 586 (11.2%) | 11 (3.2%) | |
| 5–13 | 15237 (81.8%) | 10108 (82.2%) | 512 (68.7%) | 4368 (83.3%) | 249 (72.4%) | |
| > =13 | 1326 (7.1%) | 767 (6.2%) | 184 (24.7%) | 291 (5.5%) | 84 (24.4%) | |
| Serum albumin (%) | 0.976 | |||||
| 0–2.5 | 229 (1.2%) | 86 (0.7%) | 75 (10.1%) | 31 (0.6%) | 37 (10.8%) | |
| ≥2.5 | 18400 (98.8%) | 12209 (99.3%) | 670 (89.9%) | 5214 (99.4%) | 307 (89.2%) | |
| Blood glucose (%) | 0.652 | |||||
| 0–200 | 17771 (95.4%) | 11833 (96.2%) | 600 (80.5%) | 5044 (96.2%) | 294 (85.5%) | |
| ≥200 | 858 (4.6%) | 462 (3.8%) | 145 (19.5%) | 201 (3.8%) | 50 (14.5%) | |
| Hematocrit (%) | 0.515 | |||||
| 0–0.2 | 3154 (16.9%) | 2036 (16.6%) | 176 (23.6%) | 860 (16.4%) | 82 (23.8%) | |
| 0.2–0.4 | 7530 (40.4%) | 4795 (39.0%) | 469 (63.0%) | 2036 (38.8%) | 230 (66.9%) | |
| 0.4–0.5 | 7612 (40.9%) | 5248 (42.7%) | 95 (12.8%) | 2238 (42.7%) | 31 (9.0%) | |
| ≥0.5 | 333 (1.8%) | 216 (1.8%) | 5 (0.7%) | 111 (2.1%) | 1 (0.3%) | |
| Creatinine (%) | 0.626 | |||||
| 0–1 | 15701 (84.3%) | 10431 (84.8%) | 543 (72.9%) | 4465 (85.1%) | 262 (76.2%) | |
| 1–2 | 2731 (14.7%) | 1771 (14.4%) | 160 (21.5%) | 736 (14.0%) | 64 (18.6%) | |
| ≥2 | 197 (1.1%) | 93 (0.8%) | 42 (5.6%) | 44 (0.8%) | 18 (5.2%) | |
| Urea nitrogen-creatinine ratio (%) | 0.727 | |||||
| 0–25 | 14621 (78.5%) | 9742 (79.2%) | 483 (64.8%) | 4187 (79.8%) | 209 (60.8%) | |
| ≥25 | 4008 (21.5%) | 2553 (20.8%) | 262 (35.2%) | 1058 (20.2%) | 135 (39.2%) | |
| Hemoglobin (%) | 0.393 | |||||
| 0-M130F150 | 5467 (29.3%) | 3277 (26.7%) | 525 (70.5%) | 1405 (26.8%) | 260 (75.6%) | |
| ≥M130F150 | 13162 (70.7%) | 9018 (73.3%) | 220 (29.5%) | 3840 (73.2%) | 84 (24.4%) | |
| Blood urea nitrogen (%) | 0.328 | |||||
| 0–18 | 13806 (74.1%) | 9317 (75.8%) | 384 (51.5%) | 3941 (75.1%) | 164 (47.7%) | |
| 18–20 | 1626 (8.7%) | 1032 (8.4%) | 75 (10.1%) | 493 (9.4%) | 26 (7.6%) | |
| 20–40 | 2942 (15.8%) | 1818 (14.8%) | 234 (31.4%) | 763 (14.5%) | 127 (36.9%) | |
| ≥40 | 255 (1.4%) | 128 (1.0%) | 52 (7.0%) | 48 (0.9%) | 27 (7.8%) | |
Notes: ‘-’ was defined as (a,b): the interval range contains a without b; *: the comparison of the improved model and the extended model.
Derivation of the models
In the stepwise logistic regression process for all three models, body temperature, shock index, blood urea nitrogen and urea nitrogen-creatinine ratio were removed, and sex was also removed in the extended model. The detailed regression results of the models were presented in Table 2. None of the VIF values for the predictor variables in these three models were greater than 5 (eTables 1–3 in Supplement), indicating that multicollinearity would not be a problem in the regression models.
Table 2.
Logistic regression for factors associated with adverse outcomes.
| Variable | Simple model |
Improved model |
Extended model |
|||
|---|---|---|---|---|---|---|
| OR_with_CI | p Value | OR_with_CI | p Value | OR_with_CI | p Value | |
| (Intercept) | 0.068(0.034 ∼ 0.135) | <0.001 | 0.028(0.013 ∼ 0.062) | <0.001 | 0.116(0.044 ∼ 0.304) | <0.001 |
| Sex | ||||||
| Female | 1.0[Reference] | – | 1.0[Reference] | – | – | – |
| Male | 1.285(1.074 ∼ 1.538) | 0.006 | 1.169(0.974 ∼ 1.404) | 0.094 | – | – |
| Age | 1.037(1.031 ∼ 1.042) | <0.001 | 1.038(1.033 ∼ 1.044) | <0.001 | 1.031(1.026 ∼ 1.038) | <0.001 |
| Admission route | ||||||
| Emergency admission | 0.396(0.307 ∼ 0.509) | <0.001 | 0.373(0.287 ∼ 0.482) | <0.001 | 0.41(0.313 ∼ 0.535) | <0.001 |
| Outpatient admission | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| Condition at admission | ||||||
| Danger | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| Urgent | 0.082(0.05 ∼ 0.135) | <0.001 | 0.097(0.058 ∼ 0.158) | <0.001 | 0.106(0.063 ∼ 0.178) | <0.001 |
| General | 0.085(0.051 ∼ 0.141) | <0.001 | 0.111(0.066 ∼ 0.185) | <0.001 | 0.123(0.071 ∼ 0.21) | <0.001 |
| Pulse | ||||||
| 0–60 | 0.758(0.634 ∼ 0.907) | 0.002 | 0.796(0.664 ∼ 0.953) | 0.013 | 0.871(0.722 ∼ 1.051) | 0.149 |
| 60–100 | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| ≥100 | 2.501(1.888 ∼ 3.3) | <0.001 | 2.373(1.788 ∼ 3.134) | <0.001 | 1.827(1.356 ∼ 2.448) | <0.001 |
| Respiration rate | ||||||
| 0–16 | 3.032(1.976 ∼ 4.601) | <0.001 | 2.622(1.705 ∼ 3.991) | <0.001 | 1.991(1.266 ∼ 3.098) | 0.003 |
| 16–19 | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| ≥19 | 0.864(0.721 ∼ 1.037) | 0.114 | 0.817(0.68 ∼ 0.983) | 0.032 | 0.806(0.665 ∼ 0.976) | 0.027 |
| Systolic blood pressure | ||||||
| 0–100 | 2.627(1.652 ∼ 4.109) | <0.001 | 2.644(1.651 ∼ 4.166) | <0.001 | 2.024(1.206 ∼ 3.328) | 0.006 |
| 100–120 | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| 120–140 | 0.7(0.548 ∼ 0.9) | 0.005 | 0.698(0.545 ∼ 0.898) | 0.005 | 0.76(0.587 ∼ 0.988) | 0.038 |
| ≥140 | 0.93(0.723 ∼ 1.201) | 0.574 | 0.931(0.723 ∼ 1.206) | 0.586 | 1.056(0.812 ∼ 1.381) | 0.686 |
| Elixhauser Comorbidity Index | ||||||
| 0 | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| 1 | 2.132(1.695 ∼ 2.668) | <0.001 | 2.052(1.631 ∼ 2.57) | <0.001 | 1.805(1.41 ∼ 2.299) | <0.001 |
| NISS | ||||||
| 1–4 | 1.0[Reference] | – | 1.0[Reference] | – | 1.0[Reference] | – |
| 4–9 | 0.681(0.435 ∼ 1.107) | 0.106 | 0.6(0.361 ∼ 1.026) | 0.054 | 0.564(0.332 ∼ 0.985) | 0.039 |
| 9–16 | 3.929(2.583 ∼ 6.247) | <0.001 | 3.361(2.066 ∼ 5.656) | <0.001 | 2.874(1.723 ∼ 4.948) | <0.001 |
| 16–25 | 5.728(3.525 ∼ 9.593) | <0.001 | 2.824(1.636 ∼ 4.991) | <0.001 | 2.343(1.325 ∼ 4.238) | 0.004 |
| > =25 | 6.979(4.478 ∼ 11.309) | <0.001 | 3.199(1.942 ∼ 5.435) | <0.001 | 2.094(1.228 ∼ 3.667) | 0.008 |
| Body region | ||||||
| Chest Torso | – | – | 3.471(2.369 ∼ 5.029) | <0.001 | 3.11(2.07 ∼ 4.614) | <0.001 |
| Extremities | – | – | 1.0[Reference] | – | 1.0[Reference] | – |
| Head & Neck all | – | – | 2.735(1.885 ∼ 3.946) | <0.001 | 2.806(1.899 ∼ 4.127) | <0.001 |
| Spine and upper back | – | – | 0.608(0.393 ∼ 0.912) | 0.02 | 0.752(0.482 ∼ 1.137) | 0.191 |
| Nature of injury | ||||||
| Fracture | – | – | 1.687(1.142 ∼ 2.532) | 0.01 | 1.448(0.954 ∼ 2.233) | 0.088 |
| Internal organ injuries | – | – | 1.657(1.107 ∼ 2.523) | 0.016 | 1.161(0.755 ∼ 1.814) | 0.504 |
| Open wound | – | – | 0.408(0.158 ∼ 0.914) | 0.043 | 0.359(0.137 ∼ 0.822) | 0.024 |
| Others | – | – | 1.0[Reference] | – | 1.0[Reference] | – |
| Mechanism of injury | ||||||
| Fall | – | – | – | – | 1.246(1.017 ∼ 1.528) | 0.034 |
| Transport | – | – | – | – | 1.079(0.815 ∼ 1.422) | 0.593 |
| Unspecified | – | – | – | – | 1.0[Reference] | – |
| Blood sodium | ||||||
| 0–135 | – | – | – | – | 1.15(0.894 ∼ 1.47) | 0.27 |
| 135–146 | – | – | – | – | 1.0[Reference] | – |
| ≥146 | – | – | – | – | 2.644(1.796 ∼ 3.847) | <0.001 |
| Serum bilirubin | ||||||
| 0–2 | – | – | – | – | 1.0[Reference] | – |
| ≥2 | – | – | – | – | 1.525(0.98 ∼ 2.308) | 0.053 |
| WBCC | ||||||
| 0–5 | – | – | – | – | 1.0[Reference] | – |
| 5–13 | – | – | – | – | 1.141(0.823 ∼ 1.612) | 0.442 |
| > =13 | – | – | – | – | 2.687(1.836 ∼ 3.986) | <0.001 |
| Serum albumin | ||||||
| 0–2.5 | – | – | – | – | 1.0[Reference] | – |
| ≥2.5 | – | – | – | – | 0.339(0.225 ∼ 0.511) | <0.001 |
| Blood glucose | ||||||
| 0–200 | – | – | – | – | 1.0[Reference] | – |
| ≥200 | 2.268(1.759 ∼ 2.91) | <0.001 | ||||
| Hematocrit | ||||||
| 0–0.2 | – | – | – | – | 1.0[Reference] | – |
| 0.2–0.4 | – | – | – | – | 0.766(0.603 ∼ 0.975) | 0.029 |
| 0.4–0.5 | – | – | – | – | 0.653(0.469 ∼ 0.909) | 0.012 |
| ≥0.5 | – | – | – | – | 0.482(0.158 ∼ 1.185) | 0.148 |
| Creatinine | ||||||
| 0–1 | – | – | – | – | 1.0[Reference] | – |
| 1–2 | – | – | – | – | 1.286(1.024 ∼ 1.608) | 0.029 |
| ≥2 | – | – | – | – | 1.875(1.153 ∼ 3.007) | 0.01 |
| Hemoglobin | ||||||
| 0-M130F150 | – | – | – | – | 1.929(1.55 ∼ 2.412) | <0.001 |
| ≥M130F150 | – | – | – | – | 1.0[Reference] | – |
Nomogram of adverse outcomes
A nomogram, which calculated the risk of adverse outcomes for trauma patients based on predictors, was developed for the simple model (eFigure 1 in Supplement), the improved model (eFigure 2 in Supplement), and the extended model (Figure 2).
Figure 2.
Nomogram of the extended model.
Validation of the models
In the training set, the ROC-AUCs for the three models were 0.882, 0.891, and 0.911, respectively. Meanwhile, the models discriminated and calibrated well in both internal and external validation. In the internal validation, a three-fold cross-validation method with 200 iterations was used, and the ROC-AUCs were 0.879 (95% CI: 0.878, 0.880), 0.887 (95% CI: 0.886, 0.888), 0.905 (95% CI: 0.904, 0.906), respectively. In the external validation set, the AUCs of the ROC curve were 0.872, 0.881, and 0.903, and the AUCs of the PR curve were 0.339, 0.337, and 0.403, respectively. The ROC and PR curves showed that the goodness-of-fit of the models improved as the number of the predictors increased (Figure 3).
Figure 3.
ROC curves and PR curves for the external validation sets of the 3 models.
Additional sensitivity analyses for dropping patients with severe traumatic brain injury (42.1%) yielded the ROC-AUC of 0.888 for the simple model, 0.902 for the improved model, and 0.919 for the extended model. The detailed regression results of the models were presented in the supplement (eTables 4–6, and eFigure 3).
The calibration curves for the three models were close to the actual predicted value (Figure 4). The models slightly underestimated the risk of trauma patients when the actual incidence rate of adverse outcomes was less than 30%, while they progressively overestimated the risk of adverse outcomes as the actual incidence rate of adverse outcomes became greater than 30%.
Figure 4.
Calibration curves for assessing the goodness of fit of the 3 prediction models.
The decision curve showed that using these nomograms to predict the probability of an adverse outcome would result in a greater net benefit than in an all-or-none patient intervention scenario. When the threshold probability was less than 60%, it indicated a high potential for clinical application. The net benefit of the extended model was significantly higher than the other two models, suggesting that the extended model should be the first choice (Figure 5).
Figure 5.
Decision curves associated with three models.
Discussion
Given the limitations of realistic conditions for obtaining predictors of adverse outcomes in trauma patients, this multicenter retrospective prognostic study performed internal-external validation on three prediction models for adverse outcomes in patients with trauma, which were suitable for both rich and limited data settings. We found that the extended model (including patient clinical characteristics, vital signs, diagnoses, and laboratory values) had better discrimination, accuracy, and greater net benefit than the simple model (including patient clinical characteristics and vital signs) or the improved model (laboratory values added based on the simple model). The calibration of the extended model was also better than the other two models when the incidence of adverse outcomes was greater than 30%. However, the calibration of the extended model was somewhat poor when the incidence rate of adverse outcomes was less than 30%. The three models developed in our study also had better discrimination than most other logistic models [32]. Additionally, they performed even better than machine learning models [32, 33] in the literature.
In terms of the calibration curves, the models overestimated the risk of adverse outcomes as the observed rate of adverse outcomes was greater than 30%. However, the actual rate in our total sample, training set and validation set was 5.85%, 5.71%, and 6.15%, respectively, so the models could still be considered to perform well.
Our study is clinically important because trauma patients need continuous assessment and care from pre-hospital to in-hospital, and these three models developed in this work can serve as necessary tools for different time points in the whole medical care process. We believe that these data can be obtained within 2 h upon arrival to the ED [34]. The easy-to-use nomograms developed in the study were novel and allowed the calculation of adverse outcomes in individual trauma patients in clinical practice, with great potential for use in a variety of settings. The simple, improved, or extended nomogram can be selected for use by medical staff on the basis of available patient information and will help to rapidly assess the severity of injury in trauma patients, improving diagnostic efficiency with consequent benefits for medical resource allocation. The nomograms can also assist clinicians in initiating discussions with injured patients or their families to support advance care planning.
Compared to the simple model in which NISS was controlled for, the performance of the model that further included nature of injury, body region and injury mechanism improved and the prediction was more accurate, suggesting that injury type, body region and injury mechanism were important predictors in the hospital setting in addition to NISS.
To our knowledge, existing prediction models for the outcome of patients with injury have generally included vital signs, laboratory test values alongside demographics, and body region, mechanisms of injury and NISS or ISS have also been considered [32, 35, 36], but the nature of injury has rarely been taken into account. This study, however, included nature of injury in the model and resulted in better performance, making a tentative contribution in this area. This has two implications. Firstly, nature of injury should be introduced as an important factor in the process of clinical decision making in emergency department or hospital settings, which will provide more information. Secondly, this factor should also be given more attention in research on injury outcome prediction or analysis of influence factors.
There are several limitations in this study. First, trauma diagnoses and treatment variables were determined based on data from Chinese trauma patients, which is more applicable to China. Our data were obtained from the EMRs, which provide a large amount of medical information on trauma patients. Initially, the data were collected from 8 tertiary hospitals in Beijing. At the same time, only patients admitted to hospitals were included, but in reality, a very large number of trauma patients were treated in outpatient departments. Therefore, the patients may not be a good representation of all trauma patients in the whole country. Second, the analysis and modelling were appropriately compromised by the poor quality of the data. The data used in this study were not from a specialized trauma research registry such as the National Trauma Data Bank [37], and some key variables commonly used in clinical practice were missing. For example, the terrible triad (hypocoagulation, acidosis, hypothermia) is associated with early death, but laboratory values for hypocoagulation and acidosis were not available in our dataset, making it impossible to control for them in the extended model. Another example is about mechanism of injury. A significant number of patients were classified as missing for mechanism of injury due to blank entries in the EMRs. Also, the categories were not detailed enough to further elaborate on the mechanism of injury, including specific categories such as fall from height, pedestrian, motorcycle, etc., and data quality led to the removal of over 50% of the sample from the original data.
However, it is worth noting that, on the one hand, the data used in this study were not from a professional dataset, which made it difficult to predict adverse outcomes in trauma patients in a limited data set, but on the other hand, it highlighted the importance and value of our study.
Conclusions
This prognostic study has found that three prediction models and nomograms, including the patient clinical characteristics, vital signs, diagnoses, and laboratory tests values, can help clinicians more accurately identify patients at risk of adverse outcomes (in-hospital death or transfer to ICU) in various settings based on data availability. Further research should focus on external validation, feasibility, and their use for initiating advance care planning discussions with patients and relatives, which could enhance decision-making and medical treatment for trauma patients.
Supplementary Material
Funding Statement
We acknowledge grants from the National Natural Science Foundation of China (72274211).
Authors contributions
QZ wrote the manuscript. LL and CF designed the study. JCL collected the data, and QZ and WL performed the data search and analysis. XFY prepared figures and tables. LL, QZ, XC and SMS interpreted the data and revised the manuscript accordingly. All authors have read and approved the final manuscript.
Disclosure statement
The authors declare no conflict of interest.
Data availability statement
The data are available upon a reasonable request from the corresponding authors.
References
- 1.Gerdin M, Roy N, Khajanchi M, et al. Validation of a novel prediction model for early mortality in adult trauma patients in three public university hospitals in urban India. BMC Emerg Med. 2016;16(1):15. doi: 10.1186/s12873-016-0079-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Mortality GBD, GBD 2013 Mortality and Causes of Death Collaborators . Causes of death C. Global, regional, and national age-sex specific all-cause and cause-specific mortality for 240 causes of death, 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet. 2015;385(9963):117–171. doi: 10.1016/S0140-6736(14)61682-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.GBDCoD C, GBD 2016 Causes of Death Collaborators . Global, regional, and national age-sex specific mortality for 264 causes of death, 1980-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet. 2017;390(10100):1151–1210. doi: 10.1016/S0140-6736(17)32152-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mathers C, Fat DM, Boerma JT.. World Health Organization. The global burden of disease: 2004 update. Geneva, Switzerland: World Health Organization; 2008. [Google Scholar]
- 5.Chandran A, Hyder AA, Peek-Asa C.. The global burden of unintentional injuries and an agenda for progress. Epidemiol Rev. 2010;32(1):110–120. doi: 10.1093/epirev/mxq009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Liu T, Bai XJ.. Trauma care system in China. Chin J Traumatol. 2018;21(2):80–83. doi: 10.1016/j.cjtee.2017.06.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Champion HR, Sacco WJ, Hunt TK.. Trauma severity scoring to predict mortality. World J Surg. 1983;7(1):4–11. doi: 10.1007/BF01655906. [DOI] [PubMed] [Google Scholar]
- 8.Rehn M, Perel P, Blackhall K, et al. Prognostic models for the early care of trauma patients: a systematic review. Scand J Trauma Resusc Emerg Med. 2011;19(1):17. doi: 10.1186/1757-7241-19-17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Champion HR, Sacco WJ, Copes WS, et al. A revision of the trauma score. J Trauma. 1989;29(5):623–629. doi: 10.1097/00005373-198905000-00017. [DOI] [PubMed] [Google Scholar]
- 10.Hannan EL, Farrell LS, Gorthy SF, et al. Predictors of mortality in adult patients with blunt injuries in New York State: a comparison of the Trauma and Injury Severity Score (TRISS) and the International Classification of Disease, Ninth Revision-based Injury Severity Score (ICISS). J Trauma. 1999;47(1):8–14. doi: 10.1097/00005373-199907000-00003. [DOI] [PubMed] [Google Scholar]
- 11.Gormican SP. CRAMS scale: field triage of trauma victims. Ann Emerg Med. 1982;11(3):132–135. doi: 10.1016/s0196-0644(82)80237-0. [DOI] [PubMed] [Google Scholar]
- 12.Koehler JJ, Baer LJ, Malafa SA, et al. Prehospital Index: a scoring system for field triage of trauma victims. Ann Emerg Med. 1986;15(2):178–182. doi: 10.1016/s0196-0644(86)80016-6. [DOI] [PubMed] [Google Scholar]
- 13.Husum H, Gilbert M, Wisborg T, et al. Respiratory rate as a prehospital triage tool in rural trauma. J Trauma. 2003;55(3):466–470. doi: 10.1097/01.Ta.0000044634.98189.De. [DOI] [PubMed] [Google Scholar]
- 14.Perel P, Arango M, Clayton T, et al. Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ. 2008;336(7641):425–429. doi: 10.1136/bmj.39461.643438.25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Perel P, Prieto-Merino D, Shakur H, et al. Predicting early death in patients with traumatic bleeding: development and validation of prognostic model. BMJ. 2012;345(aug15 1):e5166–e5166. doi: 10.1136/bmj.e5166. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Farzan N, Foroghi Ghomi SY, Mohammadi AR.. A retrospective study on evaluating GAP, MGAP, RTS and ISS trauma scoring system for the prediction of mortality among multiple trauma patients. Ann Med Surg (Lond). 2022;76:103536. doi: 10.1016/j.amsu.2022.103536. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Larkin EJ, Jones MK, Young SD, et al. Interest of the MGAP score on in-hospital trauma patients: comparison with TRISS, ISS and NISS scores. Injury. 2022;53(9):3059–3064. doi: 10.1016/j.injury.2022.05.024. [DOI] [PubMed] [Google Scholar]
- 18.Cheung A, Haas B, Ringer TJ, et al. Canadian study of health and aging clinical frailty scale: does it predict adverse outcomes among geriatric trauma patients? J Am Coll Surg. 2017;225(5):658–665 e3. doi: 10.1016/j.jamcollsurg.2017.08.008. [DOI] [PubMed] [Google Scholar]
- 19.Croce MA, Fabian TC, Waddle-Smith L, et al. Identification of early predictors for post-traumatic pneumonia. Am Surg. 2001;67(2):105–110. doi: 10.1177/000313480106700201. [DOI] [PubMed] [Google Scholar]
- 20.Croce MA, Tolley EA, Fabian TC.. A formula for prediction of posttraumatic pneumonia based on early anatomic and physiologic parameters. J Trauma. 2003;54(4):724–730. doi: 10.1097/01.Ta.0000054643.54218.C5. [DOI] [PubMed] [Google Scholar]
- 21.Kobes T, Terpstra AM, IJpma FFA, et al. A recalibrated prediction model can identify level-1 trauma patients at risk of nosocomial pneumonia. Arch Orthop Trauma Surg. 2023;143(8):4933–4941. doi: 10.1007/s00402-023-04766-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Zhuang Y, Tu H, Feng Q, et al. Development and validation of a nomogram for adverse outcomes of geriatric trauma patients based on frailty syndrome. Int J Gen Med. 2022;15:5499–5512. doi: 10.2147/IJGM.S365635. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.de Munter L, Polinder S, Lansink KW, et al. Mortality prediction models in the general trauma population: a systematic review. Injury. 2017;48(2):221–229. doi: 10.1016/j.injury.2016.12.009. [DOI] [PubMed] [Google Scholar]
- 24.Jia H, Wang S, Liu J, et al. A novel risk score for in-hospital perioperative mortality of five major surgeries. Int J Qual Health Care. 2021;33(2):1–8. doi: 10.1093/intqhc/mzab080. [DOI] [PubMed] [Google Scholar]
- 25.Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8–27. doi: 10.1097/00005650-199801000-00004. [DOI] [PubMed] [Google Scholar]
- 26.Pavlou M, Ambler G, Seaman SR, et al. How to develop a more accurate risk prediction model when there are few events. BMJ. 2015;351:h3868. doi: 10.1136/bmj.h3868.h3868. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Collins GS, Ogundimu EO, Altman DG.. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–226. doi: 10.1002/sim.6787. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Pietrantonio F, Lonardi S, Corti F, et al. Nomogram to predict the outcomes of patients with microsatellite instability-high metastatic colorectal cancer receiving immune checkpoint inhibitors. J Immunother Cancer. 2021;9(8):e003370. doi: 10.1136/jitc-2021-003370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Ing EB, Ing R.. The use of a nomogram to visually interpret a logistic regression prediction model for giant cell arteritis. Neuroophthalmology. 2018;42(5):284–286. doi: 10.1080/01658107.2018.1425728. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Parekh N, Ali K, Davies JG, et al. Medication-related harm in older adults following hospital discharge: development and validation of a prediction tool. BMJ Qual Saf. 2020;29(2):142–153. doi: 10.1136/bmjqs-2019-009587. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350(jan07 4):g7594–g7594. doi: 10.1136/bmj.g7594. [DOI] [PubMed] [Google Scholar]
- 32.Rau C-S, Wu S-C, Chuang J-F, et al. Machine learning models of survival prediction in trauma patients. J Clin Med. 2019;8(6):799. doi: 10.3390/jcm8060799. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Cardosi JD, Shen H, Groner JI, et al. Machine learning for outcome predictions of patients with trauma during emergency department care. BMJ Health Care Inform. 2021;28(1):e100407. doi: 10.1136/bmjhci-2021-100407. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Lee K-C, Lin T-C, Chiang H-F, et al. Predicting outcomes after trauma: prognostic model development based on admission features through machine learning. Medicine (Baltimore). 2021;100(49):e27753. doi: 10.1097/md.0000000000027753. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Clark DE, Hannan EL, Wu C.. Predicting risk-adjusted mortality for trauma patients: logistic versus multilevel logistic models. J Am Coll Surg. 2010;211(2):224–231. doi: 10.1016/j.jamcollsurg.2010.03.033. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Snoek A, Dekker M, Lagrand T, et al. A clinical decision model identifies patients at risk for delayed diagnosed injuries after high-energy trauma. Eur J Emerg Med. 2013;20(3):167–172. doi: 10.1097/MEJ.0b013e328353d926. [DOI] [PubMed] [Google Scholar]
- 37.Hashmi ZG, Kaji AH, Nathens AB.. Practical guide to surgical data sets: National Trauma Data Bank (NTDB). JAMA Surg. 2018;153(9):852–853. doi: 10.1001/jamasurg.2018.0483. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data are available upon a reasonable request from the corresponding authors.