Injury Severity Score (ISS) vs. ICD-derived Injury Severity Score (ICISS) in a patient population treated in a designated Hong Kong trauma centre

Abstract

Trauma and Injury Severity Score (TRISS) has been the benchmark of mortality risk in trauma centers for over 30 years. TRISS utilizes the Injury Severity Score (ISS) as an index of anatomical injury. This study investigated the efficacy of a new type of index of anatomical injury called the ICD-derived Injury Severity Score (ICISS) compared to the ISS using a logistic regression analysis and a global chi-square test of the areas under the Receiver Operator Characteristic (ROC) curves. We found that the empirically derived ICISS performed as well as the consensus derived ISS with no statistical differences between their respective area under the ROC curves.

INTRODUCTION

Background

For a trauma center intending to perform an effective review of their service, as well as for the scientific study of trauma, it is important to have an accurate benchmark of mortality risk. This benchmark serves as a predictor of mortality or “expected” outcome for any patient presenting with certain injuries. The expected result can then be compared to the “actual” outcomes in order to provide quality assurance of care provision. For many years, this benchmark has been the Trauma and Injury Severity Score (TRISS)(1–10). TRISS utilizes the patient’s age, type of injury, Revised Trauma Score (RTS), and the Injury Severity Score to estimate the probability of survival. It takes into account the patient’s physiological injury, physiological response and anatomic injury. The Injury Severity Score (ISS), first developed by Baker et al., supplies the anatomic index for TRISS, and has been a standard tool for three decades (1).

Lately, there have been new ideas about anatomic trauma scoring that have brought the ISS under a more critical light. One main disadvantage of the ISS is its innate attachment to the Abbreviated Injury Scale (AIS) for severity estimates, as the AIS is a consensus rather than an empirically derived scale (11). Also, the ISS uses data from the top three different anatomic regions with the most severe injuries, neglecting to account for other important injuries within a single region. In many scenarios, one region may have several severe injuries, only one of which will be accounted for, along with two less significant injuries in two other anatomic regions. In addition, because the different regions aren’t weighted, a severe foot injury can have the same impact on the score as a moderate head injury. Lastly, the ISS combines injury with therapy in its calculation. Apoorly managed minor head injury allowed to progress to coma may result in the same score as a quickly and effectively managed severe head injury. However, despite these important drawbacks, the ISS has remained a robust standard of anatomic trauma scoring during these past thirty years. Past challengers to ISS such as the Anatomic Index (AI) introduced by Champion et al. (2), and the Revised Estimate Survival Probability (RESP) index introduced by Levy et al. have failed to replace the ISS. They were not shown to improve enough upon the ISS as a predictor of survival (6,12).

Recently, a new system has come to the fore. In the middle of the 1990s, Osler introduced the ICD derived Injury Severity Score (ICISS), a survival score based on the ICD-9 classification of trauma injuries (13). It was also created in an attempt to address the main limitations of ISS. However, it also has the added feature of convenience, a clear advantage, as most trauma centers already collect and classify patients based on their ICD-9 injuries. Compared to ISS, it is easy to compute. To calculate the ISS score, there must be a trained individual who can correctly apply the AIS/ISS ratio. The ICISS however, is a simple likelihood value. It is based on the assumption that a patient’s probability of survival can be predicted based on the survival rates of prior patients with similar injuries as classified by the ICD-9. The ICISS value is the product of survival risk ratios (SRRs) from each injury sustained. These SRRs are established based on trauma data from large patient databases, the original of which was the North Carolina State Discharge Database (13) with data from over 300 000 patients. Using these risk ratios, later studies did in fact show that the ICISS was superior to the ISS alone as a predictor of survival (13–18).

In 2006, a Canadian database was created from the National Trauma Registry (NTR) of Canada, consisting of over one million cases – the largest yet in the world (19). This benchmark database attempted to address an issue of severity overestimation by the ICISS by creating inclusive and exclusive SRRs for each ICD-9 classed trauma injury. Inclusive SRRs are derived from the proportion of patients who survived the injury in question irrespective of other associated injuries present. To determine exclusive SRRs, the patient population with multiple injuries was excluded, resulting in ratios which are thought to be more representative of the “true” risk of the injury. Unsurprisingly, the exclusive SRRs produce ICISS values which are much more optimistic in terms of survival, compared to the inclusive SRR determined ICISS values.

PRESENT STUDY

The objective of the current study is to investigate the predictive value of ICISS compared to ISS. It makes use of the Canadian benchmark to compare inclusive and exclusive ICISS values to ISS values of 1167 trauma patients who presented to Queens Mary Hospital in Hong Kong over a period of 8 years.

At this time, the TRISS coefficients are over fifteen years old. They are derived from studies in the U.S. and may not always be a good predictor of survival in other parts of the world. Therefore, a secondary objective of this study is to examine the effectiveness of the ICISS using Canadian determined SRRs applied to an external database, in a different population.

MATERIALS AND METHODS

Setting

Queen Mary Hospital (QMH) in Hong Kong is one of the five designated trauma centers in Hong Kong as well as the main teaching hospital for the University of Hong Kong Li Ka Shing Faculty of Medicine. It has a well kept registry of trauma patients from 1998 onwards, complete with diagnoses, ISS, ICD-9 codes, and TRISS values calculated by a trained trauma nurse coordinator, along with survival outcomes. QMH has a 24 hour service trauma team with an audit panel, trauma director, and staffed by ATLS trained trauma surgeons. These services render QMH equivalent to a U.S. level 1 trauma centre, with case volume being the only parameter precluding its qualification as such.

SRRs

Survival risk ratios were obtained from tables as published by Bergeron et al. in the Journal of Trauma (19).

ICISS Scores

Inclusive and exclusive ICISS scores were calculated for trauma patients treated at QMH between the years of 1998 and 2005, numbering 1298 cases. Because the SRRs derived from the NTR of Canada do not include penetrating injuries, only blunt trauma cases were included in this study. Instances with incomplete diagnoses or injuries with unclassified SRRs were also excluded, leaving 1167 cases for comparison. An example of Inclusive ICISS score calculation is provided below.

Example trauma case with four diagnoses (inclusive SRRs taken from table 1):

Table 1.

Means, standard deviations and ranges of ISS, inclusive and exclusive ICISS

	Survivors			Mortalities			Overall
	Mean	SD	Range	Mean	SD	Range	Mean	SD	Range
ISS	13.75	11.82	65	34.32	15.05	75	16.56	14.18	75
Inclusive ICISS	0.87	0.13	0.64	0.67	0.15	0.73	0.84	0.15	0.75
Exclusive ICISS	0.91	0.11	0.49	0.75	0.13	0.76	0.89	0.12	0.76

Diagnosis 1: Epidural hematoma (ICD code 852), SRR = 0.8211

Diagnosis 2: Cerebral contusion (ICD code 851), SRR = 0.8816

Diagnosis 3: Fracture of skull base (ICD code 801), SRR = 0.9233

Diagnosis 4: Traumatic pneumothorax (ICD code 860), SRR = 0.9248

ICISS = 0.8211 × 0.8816 × 0.9233 × 0.9248 = 0.6181

Logistic Regression Analysis

Multivariate logistic regression analysis was performed with SPSS v.13.0 (SPSS Inc., USA). Exclusive and inclusive ICISS values were compared separately. In a multivariate logistic regression analysis with age, RTS, mechanism of injury and ISS, both inclusive and exclusive ICISS values were determined to be independent predictors of mortality, as were mechanism of injury, RTS and ISS.

Receiver Operator Characteristic Curves

The predictive value of ISS and ICISS was determined by calculation of the respective receiver operator characteristic (ROC) curves, a graphic representation of the sensitivity divided by 1-specificity of a diagnostic test. A perfect test has a sensitivity of 1 and a 1-specificity of 0, denoting detection of all true positives and no false positives. Graphically, this is represented by a point at the top left corner of the graph and an area under the curve (AUC) equal to 1. Pure chance is represented by the diagonal, and an AUC of 0.5. The higher the AUC of a ROC curve produced by a test, the more effective the test is at discriminating between true positives and false positives. After the ROC curves were found, the AUC between the ISS and ICISS values were compared in a global chi-square test for statistical significance.

Statistical Methods

Basic statistics including mean, range and standard deviation were calculated with Microsoft Excel.

The ROC curves along with AUC for ISS, inclusive ICISS and exclusive ICISS were calculated with SPSS v.13.0.

Statistical analyses for AUC comparison was done with Stata v.9.2 (StataCorp, USA), as SPSS does not feature the function. Stata performs a global chi-square test comparing the AUCs, with a significance level of 0.05.

RESULTS

Basic Statistical Measures

There were a total of 1168 cases suitable for ICISS calculation. The mean age was 45.9 years, ranging from 3 months to 99 years old. 28% of cases were females. The case volume ranged from 115 to 352 cases per year. Overall mortality was 13.78%. (Table 1)

Logistic Regression Analysis

In a multivariate logistic regression analysis with ISS, age, RTS and mechanism of injury, inclusive and exclusive ICISS values were found to be independent predictors of mortality. Odds ratios were 38.086 (95% CI [5.835,248.584]) and 46.954 (95% CI [5.736, 384.356]), p<0.001. ISS was also an independent predictor with an odds ratio of 1.057 (95% CI [1.036,1.078]) and p<0.001. Hosmer and Lemeshow goodness of fit tests were insignificant at p=0.255 for analysis with inclusive ICISS and p=0.141 with exclusive ICISS, indicating a reasonably good fit of the logistic regression model with the data (Table 2).

Table 2.

Multivariate logistic regression analysis with inclusive and exclusive ICISS

	df	Sig. Value	Exp(B)*	Lower 95% CI for Exp(B)*	Upper 95% CI for Exp(B)*
Inclusive ICISS	1	0.000	38.086	5.835	248.584
ISS	1	0.000	1.057	1.036	1.078
Age	1	0.082	1.002	1.000	1.005
RTS	1	0.494	0.165	0.001	28.834
Mechanism of Injury	7	0.033
Exclusive ICISS	1	0.000	46.954	5.736	384.356
ISS	1	0.000	1.062	1.042	1.082
Age	1	0.089	1.002	1.000	1.005
RTS	1	0.000	2.095	1.831	2.397
Mechanism of Injury	7	0.063

^*Exp(B) is an exponentiation of B coefficient, which gives the odds ratio.

ROC Curves and AUC

The AUC values for ISS, inclusive ICISS and exclusive ICISS were 0.868, 0.851 and 0.838 respectively, with standard errors of 0.015, 0.014 and 0.015 (Table 3).

Table 3.

Global chi-square analysis of AUC between ISS, inclusive and exclusive ICISS

	ROC			Asymptomatic Normal
	Observed	Area	Std. Error	95% CI
ISS	1166	0.8677	0.0148	0.83873	0.89676
Inclusive ICISS	1166	0.8510	0.0138	0.82400	0.87795
Exclusive ICISS	1166	0.8379	0.0148	0.80893	0.86697

Area(ISS) = Area(Inclusive ICISS) = Area(Exclusive ICISS)

Chi2 Value = 5.22 Prob>Chi2 (p-value) = 0.0734

Comparison of AUC

The global chi-square test with 2 degrees of freedom between the three ROC areas was insignificant. (p = 0.0734). In this study, there was no difference between the ICISS and the ISS systems – they are equally predictive of survival. (Table 3)

DISCUSSION

From the results, ICISS appears to perform as well as ISS given the same dataset. It was not however, able to outperform its predecessor in this study. Despite the somewhat optimistic results, there is an issue which limits this interpretation. At a sample size of only 1167, this is a relatively low powered study. This of course brings the question of whether the non-significance can be due to type II error. In spite of this fact, the current findings may yet have something to offer. It certainly raises interesting questions and provides for speculation of potential solutions.

Other minor limitations may have affected the results. Firstly, the SRRs developed by Bergeron et al (19). do not take into account penetrating injuries. The second issue involves the organization of injury types used by Bergeron et al. The SRRs derived from the NTR are divided into subtypes such that similar injuries within a range of ICD-9 classifications share the same SRR value. This may take away from the specificity of the SRRs and effectively under or over-estimate certain important injuries in those groups (i.e., head injuries).

While there is a recognized risk of interpreting these results too liberally, we can nonetheless safely reflect on the performance of ICISS as long as we qualify these conjectures. Our study suggests that ICISS is a good predictor of survival. For this dataset, its performance was on par with ISS. This conclusion is in keeping with the current literature available. There are several studies showing showing that ICISS consistently performs better than ISS in predicting mortality (13,18,11,20,21). There have also been studies demonstrating that ICISS predicts duration of stay and use of hospital resources more accurately than ISS. (18,22,23). So far, ICISS does not appear to have any glaring deficits.

An interesting focus that this study brings up is the external validity of trauma indices. The literature on this is sparse, but based on results of this study, the SRRs derived by Bergeron et al. appear to function quite adequately in Hong Kong despite being designed for a Canadian population. In future investigations, it would be interesting to compare ISS and ICISS in terms of external validity in different settings. The general opinion to date favours derivation of local SRR databases, theoretically boosting the performance of ICISS with local (similar) populations. But the use of a universal scale has certain advantages as well. For example, it allows for the evaluation and comparison of services around the world, and the determination of a standard of care for all injury types regardless of locality. It enables a service to improve aspects of their care by learning from teams with more experience in the area. Disaster protocols are a good example of a service that could conceivably benefit from a universal scale. ISS has proven useful in this respect for many years, it is one of the few advantages it still holds over the ICISS.

The debate on adopting the new ICISS system doesn’t seem likely to end soon and rightly so. There should be a sizeable amount of evidence available before giving up an anatomical index that has proven its worth time and again, yet ICISS appears to be adequately providing it. The evidence of ICISS’ advantages is increasing and there is more on the way. In addition to outperforming ISS, ICISS is easier to use. Calculation of ISS can be rather difficult, requiring an individual who is extensively trained in the AIS lexicon. In comparison ICISS is a simple product, the computation of which offers less opportunity for error. Yet ICISS seemed to perform as well as its predecessor. And because many large hospitals already keep extensive records of ICD-9 diagnoses in trauma cases, there is by now a very large collection of data ready and waiting for conversion into ICISS scores, for analysis, and for SRR dataset development. Several hospitals are already beginning to keep track of both ICISS and ISS for their records, owing mainly to ICISS’ simplicity.

CONCLUSION

The push towards an empirical anatomical index in this age of evidence-based medicine has seen more than its share of newcomers. So far, the ISS has proven very robust in the face of these challenges. Yet there is no denying that ICISS is gaining ground quickly. With performance that is at least on par with, and usually above, the standard, its more sophisticated empirically-derived pedigree, and its ease of operation, it appears that the long awaited successor to the ISS may have finally arrived.

Figure 1.

ROC curves of ISS, inclusive ICISS and exclusive ICISS