Accuracy and external validation of the modified rapid emergency medicine score in road traffic injuries in a Bangkok level I trauma center

Abstract

Background

Trauma is a significant public health problem. Therefore, many injury scores have been created to predict mortality and triage patients. This study aims to validate the modified Rapid Emergency Medicine Score (mREMS) for in-hospital mortality prediction in road traffic injuries and compare the mREMS with the revised trauma score (RTS) and the mechanisms, Glasgow Coma Scale (GCS), age, and arterial pressure (MGAP) score.

Methods

Data were retrospectively collected from the Vajira Hospital (1,033 cases). The mREMS was calculated from six predictors: age, systolic blood pressure, heart rate, respiratory rate, pulse oxygen saturation, and GCS. The receiver operating characteristic curve was plotted, and the area under the curve (AUC) was calculated. The AUC and 95% confidence interval (CI) of the mREMS were compared with the AUCs of other scores. Model calibration was assessed using the Hosmer–Lemeshow goodness-of-fit test.

Results

The mREMS was significantly better than the RTS at predicting death in road traffic injury patients [mREMS: AUCs, 0.909 (95% CI, 0.866–0.951); RTS: AUCs, 0.859 (95% CI, 0.791–0.927] (p = 0.023). However, the difference between the AUCs of the mREMS and MGAP score was not statistically significant (p = 0.150). The mREMS’ calibration performance was also satisfactory in this dataset based on the Hosmer–Lemeshow goodness-of-fit test (p = 0.277).

Conclusion

In the road traffic injury population, the mREMS is an excellent predictor of in-hospital mortality. These results can be applied to improve triage. However, this score should be further validated in other trauma centers before nationwide implementation.

Modified rapid emergency medicine score; Road traffic injury; External validation.

1. Introduction

Injury is a leading global health problem. Approximately 5.8 million people are killed annually by injuries; nearly one-quarter of these deaths come from road traffic crashes [1]. The mortality rate of injured patients depends on injury severity, patient characteristics, trauma referral system, and care process. An effective triage, if not all, is a substantial factor in improving patients’ outcomes. Therefore, many trauma scores have been developed to predict mortality and morbidity and triage patients.

Trauma scores can be classified into anatomical [e.g., Injury Severity Score (ISS) [2]], physiological [e.g., Revised Trauma Score (RTS) [3]; Mechanisms, Glasgow Coma Scale (GCS), Age, arterial Pressure (MGAP) score [4]; and modified Rapid Emergency Medicine Score (mREMS) [5]], and combined [e.g., Trauma and Injury Severity Score (TRISS) [6]; and Kampala Trauma Score (KTS) [7]] scoring systems. The ISS is a well-known and widely used trauma score; however, the ISS requires a definite diagnosis of injuries in each body region for the score calculation. Although ISS yields excellent survival prediction [8], this score cannot always be used in the early phase of trauma care. Therefore, a more flexible trauma score, based on physiological data, is needed. Among various physiological scoring systems, RTS is widely used. RTS demonstrated excellent mortality prediction in many studies outside the country of origin [9, 10, 11]. However, many physiological scoring systems have been developed to replace RTS, including the mREMS [5].

The Rapid Emergency Medicine Score (REMS) was first developed for in-hospital mortality prediction in non-trauma patients. Scores are given for each physiological variable, including age, systolic blood pressure (SBP), heart rate (HR), respiratory rate (RR), pulse oxygen saturation (SpO₂), and GCS. The sum of these scores is used for outcome prediction [12]. The REMS also demonstrated excellent mortality prediction in the trauma population, but re-weighting of some variables was required [13]. Henceforth, a modified version of REMS (i.e., mREMS) was developed. The mREMS provides better in-hospital mortality prediction than did the original version and other physiological scores [5]; however, the mREMS has not been externally validated.

The emergency severity index (ESI) is the only triage tool used in Thai practice. Even though ISS, RTS, and TRISS are well-known prognostic scores in Thailand, none of these has been used in the triage process. Integrating trauma scores could help the triage process performed with more insight. However, prediction scores usually perform less well outside derivation setting. As a result, using prediction scores in a new population requires external validation [14]. Only TRISS has been validated in Thailand practice [15, 16]. Due to mREMS′ better mortality prediction performance than other scores [5], it was interesting and should be selected for validation in Thai patients, especially road traffic injury victims, a substantial burden to Thailand's healthcare system. This study has 2 objectives: (1) to validate and revise mREMS before using it in our clinical practice (2) to compare mREMS′ performance with other physiological scores.

2. Methods

2.1. Study setting

This study was conducted in Vajira Hospital, where is a 774-bed University Hospital. This hospital is one of the level I referral trauma centers in Bangkok. More than 500 trauma cases were treated per year, in which approximately 200–300 cases were admitted.

2.2. Data collection

The mREMS was validated in Vajira's road traffic injury population. Patients were identified based on the ICD-10 codes, V00–V89. Using these codes, all road transport injuries were included. The medical records of trauma patients, injured from January 1, 2015, to December 31, 2018, were retrospectively reviewed. Only hospitalized cases with complete data for mREMS calculation were included in this study (no patient with incomplete data of mREMS predictors was included). Pregnancy, age less than 15 years, and referred cases for postoperative care or rehabilitation were the exclusion criteria. Data on age, sex, and the mechanism of injury were collected. All variables required for the mREMS, RTS, and MGAP score calculations, obtained at emergency department arrival, were also retrieved. The outcome of interest was in-hospital mortality within 30 days after admission. This study was approved by Vajira's institutional review board (COA 047/2563) before data collection.

2.3. Sample size estimation

According to Vajira's data, death would be observed in 3% of trauma hospitalization. The area under the receiver operating characteristic (ROC) curve (AUC) of the mREMS from the Vajira cohort was hypothesized to be 0.900. The null AUC value is 0.500; however, it was set to be 0.750, which was consensus among researchers that this value should be the lowest acceptable AUC for the mREMS to be implemented. Type I error and power were set to be 0.05 and 0.8, respectively. Using the sample size calculation proposed by Hanley and McNeil [17], 800 patients (24 deaths and 776 survivors) were needed. The sample size estimation was performed using the web-based calculator (SciStat.com).

2.4. Statistical analysis

Categorical variables, including sex, the mechanism of injury, and the severity of head injury, were reported using numbers and percentages and compared between patient groups using chi-square tests. The mean and standard deviation (SD) or median and interquartile range (IQR) were used to report continuous outcomes (i.e., age, physiological variables, and trauma scores), depending on the normality of data distribution. Independent t-tests were used for normally distributed data; otherwise, Mann–Whitney U tests were used.

Age and physiological components, including SBP, HR, RR, SpO₂, and GCS, were scored using the mREMS scoring scheme (see Table S1 in Supplementary material). Then, the mREMS was stratified following the original mREMS study, and death rates in each stratum were reported. RTS and MGAP scores were also calculated using the corresponding formulas.

Logistic regression was performed for death outcomes for each trauma score. The ROC curves for each trauma score were plotted, assigning false-positive rate (or 1 – specificity) on the x-axis and sensitivity on the y-axis. From ROC curve analysis, appropriate cutoffs were suggested by the original article and selected considering the likelihood ratio. Sensitivity, specificity, positive and negative predictive values (PPVs and NPVs), and positive and negative likelihood ratios (LR+ and LR–) were estimated for each cutoff accordingly. The AUC of the mREMS was compared with the AUCs of the RTS and MGAP scores using DeLong et al.’s approach [18].

The mREMS model was assessed for goodness-of-fit using Hosmer–Lemeshow chi-square. The observed-to-expected (O/E) value ratio and the plotting of the observed versus expected value were also used for calibration assessment. The O/E ratio closing to 1 and the fitted line closing to the reference line in the O/E plot indicate excellent model calibration. To further improve model performance, recalibration was performed by adjusting the model intercept (i.e., baseline risk) and multiplying the mREMS score with the overall correcting factor derived from the logistic regression of death outcome with the mREMS [19, 20].

The model's performance was reported along with its 95% confidence interval (95% CI). If not otherwise stated, a p-value of less than 0.05 would be considered statistical significance. Stata version 16 (StataCorp, Texas, USA) was used for statistical analysis. This study was reported in line with the STARD guideline [21].

3. Results

This analysis included 1,033 road traffic injured patients; 43 (4.2%) of these patients died. The median age was 35 (IQR; 21, 49) years, and 78.7% of the patients were male. Ages and sex did not significantly differ between patients who survived and patients who died. The most common mechanisms of injury were bicycle and motorcycle crashes, which accounted for 84.7% of the patient cohort, whereas pedestrian injury and motor vehicle crashes accounted for 12.9% and 2.4% of the patients, respectively. A higher proportion of patients who died were pedestrian injuries and motor vehicle crashes compared with the proportion of patients who survived (see Table 1).

Table 1.

Baseline characteristics, physiological variables, and trauma scores.

	Survived (N = 990)	Died (N = 43)	p-value
Age, years (Median, IQR)	35 (21, 49)	36 (25, 49)	0.306
Sex
Male, N (%)	781 (78.9)	32 (74.4)	0.483
Mechanism of injury, N (%)
Pedestrian	123 (12.4)	10 (23.3)	0.012
Bicycle/Motorcycle	845 (85.4)	30 (69.8)
Motor vehicle	22 (2.2)	3 (7.0)
SBP, mmHg (Mean ± SD)	136 ± 26	119 ± 59	0.061
HR, beats/min (Mean ± SD)	89 ± 16	79 ± 38	0.077
RR, breaths/min (Mean ± SD)	20 ± 3	20 ± 10	0.960
SpO2, % (Median, IQR)	99 (98, 100)	97 (88, 100)	<0.001
Head injury severity, N (%)
Mild (GCS 14–15)	902 (91.1)	12 (27.9)	<0.001
Moderate-Severe (GCS 3–13)	88 (8.9)	31 (72.1)
mREMS
Overall score (Median, IQR)	1 (0, 2)	6 (5, 10)	<0.001
Age score (Median, IQR)	0 (0, 1)	0 (0, 1)	0.446
SBP score (Median, IQR)	0 (0, 1)	0 (0, 1)	<0.001
HR score (Median, IQR)	0 (0, 0)	2 (0, 2)	<0.001
RR score (Median, IQR)	0 (0, 0)	0 (0, 1)	<0.001
SpO2 score (Median, IQR)	0 (0, 0)	0 (0, 1)	<0.001
GCS score (Median, IQR)	0 (0, 0)	5 (0, 6)	<0.001
RTS (Median, IQR)	7.8408 (7.8408, 7.8408)	5.9672 (4.0936, 7.5500)	<0.001
MGAP (Median, IQR)	29 (27, 29)	19 (17, 23)	<0.001

GCS Glasgow coma scale; HR heart rate; IQR interquartile range; MGAP mechanism of injury, Glasgow coma scale, age, and systolic blood pressure; min minute; mmHg millimeter mercury; mREMS modified rapid emergency medicine score; RR respiratory rate; RTS revised trauma score; SBP systolic blood pressure; SD standard deviation; SpO₂ pulse oxygen saturation.

Most of the physiological variables, except SpO₂ and head injury severity, did not significantly differ between survivors and patients who died. However, SBP, HR, RR, SpO₂, and GCS scores, assigned by the mREMS scoring scheme, substantially differed between the two groups. The median of the mREMS was higher in patients who died than in patient who survived [6 (IQR, 5–10) versus 1 (IQR, 0–2), respectively]. In addition, the medians (IQR) of both RTS and MGAP scores were significantly lower in patients who died than in patients who survived (see Table 1). The number of patients who fell into each score category for the mREMS components is reported in Table S2 in Supplementary material. The majority of patients had low scores in each mREMS component. The score distribution among the patients who died is shown in Table S3 in Supplementary material.

When the mREMS was applied to Vajira's road traffic injury dataset, the model demonstrated outstanding discrimination properties, with an AUC of 0.909 (95% CI, 0.866–0.951). The AUCs of the RTS and MGAP scores were 0.859 (95% CI, 0.791–0.927) and 0.878 (95% CI, 0.810–0.946), respectively. The AUC of the mREMS was significantly higher than that of the RTS, but not higher than that of the MGAP score (p = 0.023 and 0.150, respectively). The ROC curves for all trauma scores are shown in Figure 1.

Figure 1.

Receiver operating characteristic curve of trauma scores.

When patients in our cohort were stratified as that of the original mREMS study, higher death rates were associated with higher mREMS. Nevertheless, death in each stratum was much higher in our cohort than that reported in the original mREMS cohort (see Table 2). Only seven patients had mREMS higher than 13, and all seven of these patients died. Sensitivity, specificity, PPVs, NPVs, LR+, and LR– varied depending on the score cutoff, as demonstrated in Table 3.

Table 2.

Mortality rates across modified Rapid Emergency Medicine Score (mREMS) strata compared with the original study.

mREMS	Total N	Deaths N (%)	% Death (Original study)
0–2	761	6 (0.8)	0.03
3–5	198	11 (5.6)	0.08
6–8	48	12 (25.0)	0.3
9–13	19	7 (36.8)	2.9
14–17	1	1 (100)	11.1
18–21	1	1 (100)	54.4
22–26	5	5 (100)	91.4

Table 3.

Predictive performance across modified Rapid Emergency Medicine Score cutoffs.

Cut off	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)	LR+ (95% CI)	LR– (95% CI)
≥3	86.1 (72.1, 94.7)	76.3 (73.5, 78.9)	13.6 (11.8, 15.6)	99.2 (98.4, 99.6)	3.6 (3.1, 4.3)	0.2 (0.1, 0.4)
≥6	60.5 (44.4, 75.0)	95.2 (93.6, 96.4)	35.1 (27.3, 43.9)	98.2 (97.5, 98.8)	12.5 (8.6, 18.0)	0.4 (0.3, 0.6)
≥9	32.6 (19.1, 48.5)	98.8 (97.9, 99.4)	53.8 (36.5, 70.3)	97.1 (96.5, 97.6)	26.9 (13.2, 54.5)	0.7 (0.6, 0.8)
≥15	16.3 (6.8, 30.7)	100.0 (99.6, 100.0)	100.0 (N/A)	96.5 (96.0, 96.9)	N/A	0.8 (0.7, 1.0)

CI confidence interval, LR likelihood ratio, N/A not available, NPV negative predictive value, PPV positive predictive value.

The mREMS model goodness-of-fit is indicated by a Hosmer–Lemeshow chi-square of 3.86, corresponding with a p-value of 0.277. Furthermore, the O/E ratio of 1.015 (95% CI, 0.886–1.144) and the O/E plot (Figure 2) also confirmed an excellent model calibration in this dataset. Model recalibration by adding new intercept and multiplying the mREMS with the overall correcting factor failed to improve the performance of the mREMS.

Figure 2.

Calibration plot of the modified Rapid Emergency Medicine Score.

4. Discussion

REMS [12] has been proposed since 2004 and was modified in 2017 for use in trauma patients [5]. Since then, this score has never been externally validated. This study is the first one validating the mREMS outside derivative population using data from Thai road traffic injury patients. When the mREMS was applied, a little lower AUC was observed in our cohort than that originally derived (0.909 versus 0.967, respectively) [5]. Due to the difference in population characteristics between the original mREMS and Vajira cohort, this lower AUC was predictable. Even though most of the predictor variables themselves did not differ between dead patients and survivors in our dataset, transformed variables regarding the mREMS scheme did. This result is satisfying when looking from the user's perspective.

Among various trauma scores, anatomical scores cannot be promptly used [22]. ISS can only be calculated in a relatively late phase of trauma care because its accuracy depends on a definite diagnosis of organ injury, which requires complex investigations. TRISS and KTS have similar disadvantages, which require anatomical information. For this reason, scores solely constructed on easily measured physiological variables might be more appropriate when timeliness matters in the early stages of trauma care. Thus, physiological scores are more effective in triage, especially in developing countries where the complete investigation may not be achieved timely [23]. In this study, we intended to compare the mREMS with scores from the same class (i.e., physiological scores). However, the mREMS was compared with ISS in its original study, and the result indicated significantly better discrimination performance from the mREMS [5].

Many physiological predictors were repeatedly used to construct trauma scores. For instance, GCS and SBP are included in the mREMS, RTS, and MGAP scores, whereas RR were components of both mREMS and RTS [3, 4, 5]. Therefore, a considerable gap in model performance was not expected among physiological trauma scores. However, the mREMS was significantly better in mortality prediction compared with the RTS in the present study. The mREMS also predicted in-hospital death better than the MGAP score, although the AUC difference did not reach statistical significance. Our findings were similar to the findings of the original mREMS study [5]. Different predictors’ assigned weight, especially GCS, could explain performance variation among prediction scores. MGAP score performed better than RTS in this and the original mREMS studies, which did not contradict results from non-mREMS articles [24, 25].

The emergency trauma score (EMTRAS), incorporating age, GCS, base excess, and prothrombin time, is another physiological score that provided an excellent mortality prediction and has been compared with REMS [26, 27] but was not investigated in this study. Base excess indicates the degree of inadequate tissue perfusion, whereas prothrombin time represents massive blood loss and organ system failure. Integrating laboratory data could increase prediction performance [27], but it takes time. Given the retrospective data collection, these laboratory predictors were available for only a limited number of victims (only critically injured patients would have these laboratory data). Prospective data collection is required for studying EMTRAS in our setting. While acknowledged, a comparison between mREMS and EMTRAS was not conducted in this study.

The mREMS demonstrated excellent model performance in Vajira's road traffic injury cohort. Nevertheless, substantially higher mortality than in the original mREMS study [5] was observed in this study. When patients were stratified into the mREMS strata as that of the original study, markedly high death rates were observed in every stratum. This finding may be due to a higher baseline risk of death in Thai road traffic injuries. In the original mREMS study, scores were stratified into seven strata corresponding with incremental death risk in each stratum. All patients whose scores ≥14 had 100% mortality in our study. Thus, we decided to stratify mREMS into five levels: very low, low, moderate, high, and very high. Corresponded risks of death were 0.8%, 5.6%, 25.0%, 36.8%, and 100%, respectively (see Table 2). Severe head injuries, represented by low GCS, were observed in 63% of deaths, whereas only 14% of victims who died experienced severe hypotension (see Table S3 in Supplementary material). Thus, head injuries might be a target group to improve the care process and implement preventive policies.

The main weakness of available trauma prediction scores is that they do not consider transfer time and pre-hospital resuscitation. Pre-hospital logistics and management significantly affect mortality. For instance, short transfer time and high-quality pre-hospital resuscitation are associated with better outcomes. However, physiological parameters used for score computation should already reveal the quality of these pre-hospital factors. A patient's co-morbidity is also crucial. Again, it is not considered in any trauma prediction score. Since the information about a patient's co-morbidity might not be available on arrival, incorporating this factor might reduce score utility in early prognosis prediction. Because pre-hospital management and co-morbidity status are not included in any physiological score, these data were not collected.

The present study has several limitations. First, few patients had mREMS higher than 13, and all died. The 100% mortality rates in this patient group may be inaccurate. Second, model performance could be further improved by model revision (i.e., adjusting coefficients of each variable plus adding new predictors); however, this procedure was not performed because of the unavailability of the mREMS model's equation. Third, this study was based on a complete case analysis which could result in selection bias. Fourth, sensitivity analysis stratifying patients regarding transport time, which could relate to mortality, was not performed due to a lack of data. Fifth, the results from this single-center study might not apply to other trauma centers. The mREMS still needs further studies in Thailand. Finally, no significant information was added from the results of the RTS and MGAP score performance, given that RTS and MGAP score were already validated in many studies [9, 10, 11, 28].

However, this study has several strengths. To the best of our knowledge, this study is the first external validation of the mREMS. Because external validation is required before adopting a prediction model in a new population, this study, which indicates excellent mREMS performance, could be a substantial reference. The selected cohort was road traffic injury patients, a major health problem in Thailand. Thus, this score could be confidently used in this group of patients. Furthermore, the mREMS is the first physiological score officially validated in this country. The impact of the mREMS implementation should be evaluated in further study.

In conclusion, this study is the first external validation of the mREMS, which confirmed the excellent score performance in predicting in-hospital mortality of trauma patients. The mREMS achieved excellent model discrimination (between patients who died and patients who survived) and calibration in the study cohort. In road traffic injury, the mREMS significantly provided a better mortality prediction when compared with the RTS.

Declaration

Author contribution statement

Naralin Phunghassaporn: Performed the experiments; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Pakkapol Sukhvibul: Contributed reagents, materials, analysis tools or data; Wrote the paper.

Suphakarn Techapongsatorn: Conceived and designed the experiments; Wrote the paper.

Amarit Tansawet: Conceived and designed the experiments; Analysed and interpreted the data; Wrote the paper.

Funding statement

The authors gratefully acknowledge the Navamindradhiraj University Research Fund for providing English proofreading and editing support (Grant number: N/A).

Data availability statement

Data will be made available on request.

Declaration of interest's statement

The authors declare no conflict of interest.

Additional information

Supplementary content related to this article has been published online at https://doi.org/10.1016/j.heliyon.2022.e12225.

Appendix A. Supplementary data

The following are the supplementary data related to this article: