Predicting 14-Day Mortality after Severe Traumatic Brain Injury: Application of the IMPACT Models in the Brain Trauma Foundation TBI-trac^® New York State Database

Abstract

Prognostic models for outcome prediction in patients with traumatic brain injury (TBI) are important instruments in both clinical practice and research. To remain current a continuous process of model validation is necessary. We aimed to investigate the performance of the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) prognostic models in predicting mortality in a contemporary New York State TBI registry developed and maintained by the Brain Trauma Foundation. The Brain Trauma Foundation (BTF) TBI-trac^® database contains data on 3125 patients who sustained severe TBI (Glasgow Coma Scale [GCS] score ≤8) in New York State between 2000 and 2009. The outcome measure was 14-day mortality. To predict 14-day mortality with admission data, we adapted the IMPACT Core and Extended models. Performance of the models was assessed by determining calibration (agreement between observed and predicted outcomes), and discrimination (separation of those patients who die from those who survive). Calibration was explored graphically with calibration plots. Discrimination was expressed by the area under the receiver operating characteristic (ROC) curve (AUC). A total of 2513 out of 3125 patients in the BTF database met the inclusion criteria. The 14-day mortality rate was 23%. The models showed excellent calibration. Mean predicted probabilities were 20% for the Core model and 24% for the Extended model. Both models showed good discrimination with AUCs of 0.79 (Core) and 0.83 (Extended). We conclude that the IMPACT models validly predict 14-day mortality in the BTF database, confirming generalizability of these models for outcome prediction in TBI patients.

Introduction

Traumatic brain injury (TBI) is a heterogeneous disease in terms of injury mechanism, pathology, severity, and prognosis (Ghajar et al., 2000; Maas et al., 2008). Outcome prediction is relevant for both clinical practice and research (Lingsma et al., 2010). In clinical practice, evidence-based prediction of individual patient outcome is important for realistic counseling of patients and relatives. In research, predictions may be used to compare actual outcome to predicted outcome for benchmarking quality of care, for classification, to identify best practices in comparative effectiveness research, or as instruments to adjust for heterogeneity in prognostic risk in clinical trials.

In 2000, the Brain Trauma Foundation (BTF) initiated a quality improvement program in New York State to track the treatment of severe TBI patients (Glasgow Coma Scale [GCS] score<9), which expanded to involve 22 of the 46 state-designated level 1, 2, and 3 trauma centers. The program was funded through the New York State Department of Health Bureau of Emergency Medical Services, and has a database (TBI-trac^®) containing prospective data on more than 3000 patients. The project was designed to assess and implement adoption of the evidence-based Guidelines for the Management of Severe TBI (Brain Trauma Foundation, 1995,2000,2007), but the data also offer opportunities for further quality-of-care research, outcomes research, and comparative effectiveness research. Each of these requires valid predictions of outcome for individual patients.

Outcome predictions require a good prognostic model. Methodological requirements to develop a valid prognostic model include sufficiently large patient samples, and internal and external validation (Steyerberg, 2009). In TBI, many prognostic models have been published, but only few fulfill these requirements (Perel et al., 2006; Mushkudiani et al., 2008), resulting in limited generalizability beyond the development sample.

The aim of this study was to establish the validity of an existing model to predict 14-day mortality in the TBI-trac database. The model we used was the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) model, which was developed to predict 6-month mortality and unfavorable outcome, and already showed good performance in an external validation set (Steyerberg et al., 2008). For the present study, the IMPACT model was modified to predict 14-day mortality and applied to the TBI-trac database.

Methods

Study population

The BTF TBI-trac database contains prospectively collected data of patients who sustained a severe TBI (GCS score≤8). Patients were included between 2000 and 2009 in 22 trauma centers (20 Level 1 and 2 Level 2 centers) enrolled in a New York State quality improvement program. The database includes information on mechanism of injury, demographics (age, sex, and race), injury severity (GCS score and pupillary reactivity), and brain CT characteristics. Data were also collected from the patient's stay in the intensive care unit on physiologic variables, the presence of secondary insults (hypotension, hypoxia, and cerebral hypoperfusion), and any therapies to reduce elevated intracranial pressure (ICP). Outcome is assessed at 2 weeks after injury. For our analyses, we used data recorded after resuscitation on the first day of admission to the hospital. Patients were considered hypoxic when the arterial oxygen pressure on day 1 was<60 mm Hg, or when the lowest oxygen saturation before admission to the hospital was<90%. If the patient was sedated during GCS assessment, the variable GCS was recoded as “untestable.” We excluded patients who died in the emergency department, those who had a GCS motor score of 1 or 2 with bilateral fixed and dilated pupils on admission, those with a GCS score≥9 on day 1, those with a GCS motor score of 6 on day 1, age<14 years, penetrating TBI, and missing 14-day outcome.

Models

The IMPACT models were developed in the IMPACT database, which included prospectively collected data of moderate and severe TBI patients from eight randomized controlled trials and three observational series (total n=8509). Details on the development and validation of the IMPACT prognostic models have been described previously (Steyerberg et al., 2008). The IMPACT Core Model includes the predictors age, GCS motor score, and pupillary reactivity. The Extended Model added variables on secondary insults (hypoxia and hypotension), and CT scan characteristics (Marshall CT classification, traumatic subarachnoid hemorrhage [TSAH], and epidural hematoma). Both models were developed for prediction of mortality and unfavorable outcome at 6 months, according to the dichotomized Glasgow Outcome Scale (GOS; dead or vegetative state, and severe disability).

The IMPACT predictors present in the TBI-trac data were age, GCS motor score, pupillary reactivity, hypoxia, hypotension, and TSAH. The Marshall CT classification and epidural hematoma were not recorded in the TBI-trac data, and were replaced by the status of basal cisterns and the presence of midline shift as CT parameters. We therefore refitted the IMPACT models in the IMPACT database to obtain new coefficients and intercepts for prediction of 14-day mortality:

Calibration and discrimination

The external validity of the models was assessed by studying calibration and discrimination. Calibration refers to the agreement between observed and predicted outcomes. The extent of over- or underestimation relative to the observed and predicted rate was explored graphically using validation plots. We assessed calibration-in-the-large by fitting a logistic regression model with the model predictions as an offset variable. The intercept indicates whether predictions are systematically too low or too high, and should ideally be zero. The calibration slope reflects the average effects of the predictors in the model, and was estimated in a logistic regression model with the logit of the model predictions as the only predictor. For a perfect model, the slope is equal to 1.

The area under the receiver operating characteristic Curve (AUC) and 95% confidence interval (CI) was used to assess the ability of the model to discriminate between death and survival. The 95% CI of the AUC was calculated by a bootstrap resampling method.

Statistical analyses

Missing values in the BTF dataset were statistically imputed using single imputation with the AregImpute function in R statistical software (R Foundation, Vienna, Austria). In the dataset with eight independent variables and one outcome variable per patient, 832 of the 20,104 data points (4%) were missing and imputed. The highest percentage of missings was in the variables hypoxia and status of basal cisterns. All the analyses were done in both the IMPACT and BTF TBI-trac databases. Patient baseline characteristics were described as standard summary statistics: median (range) for continuous variables, and frequency (percentage) for categorical variables. Univariate and multivariable logistic regressions were performed in both the imputed data and the complete cases, and associations were expressed as odds ratios and 95% CIs. Since the means and standard deviations as well as the prognostic effects and the results of the validation were very similar, we report only odds ratios and 95% CI from the imputed data. The calibration plots were created with an adapted version of the val.prob function from the Design library in the R package.

Results

Study population

The BTF TBI-trac dataset contained data on 3125 severe TBI patients. Patients were excluded if they had a GCS score≥9 on day 1 (152 patients), or a GCS motor score of 6 on day 1 (23 patients), since they did not meet the definition of severe TBI. Patients younger than 14 years of age (215 patients), and those with penetrating or gunshot TBI (132 patients) were excluded, since these were exclusion criteria of the IMPACT study. Finally, patients who had a GCS motor score of 1 or 2 and bilateral fixed and dilated pupils on admission or died in the emergency department (43 patients), and those without daily records (7 patients) or 14-day outcome assessment (40 patients) were excluded. After exclusion criteria were applied, a total of 2513 patients were eligible for the validation.

In both the IMPACT database and the TBI-trac database (over a 10-year period), 23% of the patients had died 14 days after injury. The patients in the TBI-trac dataset were older (median 35 versus 30 years), more often had an untestable GCS motor score (15% versus 5%), and less often had two unreactive pupils (9% versus 25%). Only 6.7% of the patients in the TBI-trac database had hypoxia, while this was 20% in the IMPACT data. The percentage of patients with hypotension and the various CT characteristics were similar (Table 1).

Table 1.

Baseline Characteristics of Patients in the IMPACT Database and in the BTF TBI-trac Database

Characteristics	Measure or category	IMPACT database n=8428	BTF database n=2513	p Value*
Age, years	Median (25th–75th percentile)	30 (21–45)	35 (22–52)	<0.0001
Motor score of GCS	Total available	8428 (100%)	2513 (100%)
	None (1)	1381 (16%)	709 (28%)
	Extension (2)	1028 (12%)	147 (6%)	<0.0001
	Abnormal flexion (3)	1078 (13%)	191 (8%)
	Normal flexion (4)	1926 (23%)	537 (21%)
	Localizes/obeys (5/6)	2567 (30%)	540 (21%)
	Untestable/missing (9)	448 (5%)	389 (15%)
Pupillary reactivity	Total available	7051 (84%)	2492 (99%)
	Both pupils reactive	4442 (63%)	1789 (72%)
	One pupil reactive	876 (12%)	479 (19%)	<0.0001
	No pupil reactive	1733 (25%)	224 (9%)
		n=6918	n=2513
Hypoxia	Total available	5375 (78%)	2255 (90%)	<0.0001
	Yes or suspected	1087 (20%)	151 (7%)
Hypotension	Total available	6342 (92%)	2513 (100%)	0.036
	Yes or suspected	1152 (19%)	505 (20%)
Basal cisterns	Total available	3833 (55%)	2266 (90%)	0.794
	Partially compressed or absent	1653 (53%)	985 (43%)
Midline shift	Total available	4676 (68%)	2365 (94%)	<0.0001
	Yes	1866 (40%)	797 (34%)
Traumatic subarachnoid hemorrhage	Total available	5813 (84%)	2355 (94%)	0.451
	Yes	2652 (46%)	1096 (47%)

^*p Values were calculated by either chi-square test and Fisher's exact test wherever applicable.

IMPACT, International Mission on Prognosis and Analysis of Clinical Trials in TBI; BTF, Brain Trauma Foundation; GCS, Glasgow Coma Scale; TBI, traumatic brain injury.

Prognostic effects

Overall, the prognostic effects were very similar in the IMPACT and the TBI-trac data (Table 2). The largest difference was in the odds ratios (OR) for no reactive pupils (univariate OR=6.5, 95% CI 5.8,7.4 in IMPACT, and OR=18.4, 95% CI 13.2,25.6 in TBI-trac). In the Core Model from both datasets, older age, poor GCS motor score, and unreactive pupils were highly predictive for 14-day mortality. In addition, hypoxia, hypotension, compressed or absent basal cisterns, midline shift, and TSAH were significant predictors of 14-day mortality from the Extended Model in both datasets.

Table 2.

Associations between Predictors and 14-Day Mortality in the IMPACT Database (n=8428) and the Btf TBI-trac Database (n=2513)

		IMPACT			BTF TBI-trac
Characteristics	Measure or category	Univariate	Core model n=8428	Extended model n=6918	Univariate	Core model n=2513	Extended model n=2513
Age, years	OR 25th–75th percentile	1.8 (1.6–1.9)	1.9 (1.8–2.1)	1.7 (1.5–1.9)	1.8 (1.6–2.1)	2.1 (1.8–2.4)	1.9 (1.6–2.3)
Motor score of GCS	None (1)	5.2 (4.4–6.1)	3.1 (2.7–3.8)	2.7 (2.2–3.3)	6.6 (4.7–9.2)	4.1 (2.9–5.9)	3.5 (2.4–5.1)
	Extension (2)	5.2 (4.4–6.3)	3.0 (2.5–3.6)	2.3 (1.8–2.8)	5.7 (3.7–9.0)	4.2 (2.6–6.8)	3.8 (2.3–6.4)
	Abnormal flexion (3)	2.7 (2.2–3.2)	1.9 (1.6–2.3)	1.7 (1.4–2.1)	3.3 (2.1–5.2)	2.5 (1.5–4.0)	2.3 (1.4–3.7)
	Normal flexion (4)	1.5 (1.3–1.8)	1.3 (1.0–1.5)	1.2 (1.0–1.4)	1.7 (1.1–2.4)	1.5 (1.0–2.2)	1.5 (1.0–2.3)
	Localizes/obeys (5/6)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)
	Untestable/missing (9)	3.1 (2.4–3.9)	2.0 (1.6–2.6)	1.5 (1.2–2.0)	2.7 (1.8–3.9)	2.3 (1.5–3.5)	2.3 (1.5–3.5)
Pupillary reactivity	Both pupils reactive (1)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)	1.0 (Ref)
	One pupil reactive (2)	2.8 (2.4–3.2)	2.2 (1.9–2.6)	1.9 (1.6–2.2)	2.6 (2.1–3.3)	2.3 (1.8–2.9)	1.7 (1.3–2.2)
	No pupil reactive (3)	6.5 (5.8–7.4)	4.8 (4.2–5.5)	4.4 (3.8–5.2)	18.4 (13.2–25.6)	14.2 (10.0–20.1)	8.5 (5.9–12.2)
Hypoxia	Yes or suspected	2.1 (1.8–2.4)		1.3 (1.1–1.6)	1.8 (1.3–2.5)		1.7 (1.1–2.5)
Hypotension	Yes or suspected	2.9 (2.6–3.4)		2.1 (1.8–2.5)	3.1 (2.5–3.8)		2.1 (1.6–2.7)
Basal cisterns	Partially compressed or absent	3.0 (2.6–3.4)		1.9 (1.6–2.2)	5.2 (4.2–6.4)		2.8 (2.1–3.7)
Midline shift	Yes	2.2 (2.0–2.5)		1.3 (1.1–1.5)	3.1 (2.5–3.8)		1.3 (1.0–1.7)
Traumatic subarachnoid hemorrhage	Yes	2.5 (2.2–2.7)		2.1 (1.9–2.4)	2.5 (2.0–3.0)		1.3 (1.0–1.7)

IMPACT, International Mission on Prognosis and Analysis of Clinical Trials in TBI; BTF, Brain Trauma Foundation; GCS, Glasgow Coma Scale; TBI, traumatic brain injury; OR, odds ratio.

Model performance

The AUCs of the refitted models in the IMPACT data were 0.763 for the Core Model and 0.806 for the Extended Model (Table 3). In the validation sample the models discriminated even better between patients who had died within 14 days and those who were alive. AUCs were 0.787 (95% CI 0.765,0.809) for the Core Model, and 0.827 (95% CI 0.805,0.845) for the Extended Model. Also the calibration of the models was good (Fig. 1). The Core Model predicted 20% 14-day mortality, and the Extended Model 24%, while the observed 14-day mortality was 23%. The calibration slopes were 1.239 for the Core Model and 1.121 for the Extended Model, indicating that the prognostic effects were stronger in the validation sample.

Table 3.

Discrimination (Area Under the Curve [95% Confidence Interval]) in the Development (IMPACT; n=8428 [Core] and n=6918 [Extended]) and Validation (BTF TBI-trac; n=2513) Data

	Core model	Extended model
Development	0.763 (0.750,0.775)	0.806 (0.794,0.817)
External validation	0.787 (0.765,0.809)	0.827 (0.805,0.845)

Core model: logodds (14-day mortality)=–3.342+0.027 ^* (age)+1.154 ^* (GCS motor score=1)+1.089 ^* (GCS motor=2)+0.642 ^* (GCS motor score=3)+0.225 ^* (GCS motor score=4)+0.707 ^* (GCS motor score=9)+0.780 ^* (pupils=2)+1.567 ^* (pupils=3)

Extended model: logodds (14-day mortality)=−3.957+0.022 ^* (age)+0.990 ^* (GCS motor score=1)+0.861 ^* (GCS motor score=2)+0.522 ^* (GCS motor score=3)+0.166 ^* (GCS motor score=4)+0.424 ^* (GCS motor score=9)+0.625 ^* (pupils=2)+1.488 ^* (pupils=3)+0.282 ^* (hypoxia)+0.752 ^* (hypotension)+0.618 ^* (obliterated cisterns)+0.226 ^* (midline shift)+0.749 ^* (TSAH)

IMPACT, International Mission on Prognosis and Analysis of Clinical Trials in TBI; BTF, Brain Trauma Foundation; GCS, Glasgow Coma Scale; TBI, traumatic brain injury; OR, odds ratio; TSAH, traumatic subarachnoid hemorrhage.

FIG. 1.

Calibration plots of the (A) Core and Extended (B) IMPACT models in BTF TBI-trac data (IMPACT, International Mission on Prognosis and Analysis of Clinical Trials in TBI; BTF, Brain Trauma Foundation; TBI, traumatic brain injury; ROC, receiver operating characteristic).

The models were also tested in the non-imputed dataset, which showed similar values for the performance measures. The Core Model (n=2492) had an AUC of 0.787 and calibration slope of 1.24. For the Extended Model (n=1976) the AUC was 0.833 and the calibration slope 1.17.

Discussion

The relevance of prognostic modeling in TBI is being increasingly recognized (Lingsma et al., 2010). This has resulted in the development of various prognostic models. Many of these, however, have shown methodological shortcomings, in particular overfitting due to incorrect approaches to predictor selection and lack of internal and external validation (Mushkudiani et al., 2008; Perel et al., 2006). The IMPACT models were developed on large data sets with state-of-the-art methodology, including external validation (Steyerberg et al., 2008). The development phase thus meets quality standards for model development (Steyerberg, 2009). The need for validation, however, does not end with the development phase. In general, prognostic models should be submitted to a continuing process of validation, and where appropriate, updating in order to remain current. In this study, we validated the IMPACT prognostic models in the contemporary data set of the BTF TBI-trac New York State TBI registry. By this validation the models showed high discrimination and good calibration. These results confirm generalizability, and further demonstrate the validity of their use for current data.

In this validation study the IMPACT models were refitted, as CT parameters were scored differently in the BTF TBI-trac database, and the outcome in this observational study was 14-day mortality rather than 6-month outcome. In this case, refitting was essential due to the different coding of CT parameters. In general, however, the concept of refitting a prognostic model is a highly relevant issue, in particular for the application of prognostic models in the design and analysis of clinical trials. The IMPACT studies have shown that the statistical power of clinical trials can be increased by up to 50% with the combined use of covariate adjustment and ordinal analysis (Maas et al., 2010a; McHugh et al., 2010). Both covariate adjustment and the use of the sliding dichotomy as an approach to ordinal analysis require robust prognostic models. Debate exists as to whether these models should be completely pre-specified, or whether the weighting of the variables included in the model may be refitted to the data set under study. In the latter case a better discrimination and performance of the model on the new data set may be anticipated. We suggest that such refitting is admissible and perhaps even preferable. A minimum requirement, however, is the pre-specification of the variables included in the model. We note that this approach to refitting is different from what we did in this study. In the current study the existing IMPACT models were refitted within the development data set rather than on the new data set under study. As a sensitivity analysis we additionally tested the existing IMPACT Core Model and the BTF TBI-trac database. Discrimination was very similar: the AUC for the existing Core Model was 0.773, compared to 0.786 for the refitted model. Calibration was less optimal: the intercept was −0.605 for the existing Core Model, compared to 0.037 for the refitted variant. The calibration slopes were again quite similar: 1.059 for the existing model versus 1.132 for the refitted model.

We found greater discriminative ability of the IMPACT models in the validation data set than previously described in the development data set. This can be explained by the greater heterogeneity of the validation data set (observational series without restrictive enrolment criteria), and illustrates that the AUC as a measure for discrimination is also influenced by the case-mix of the validation set (Vergouwe et al., 2010).

The calibration plots (Fig. 1) for the IMPACT Core Model show a small but systematic underestimation of observed mortality. It is tempting to attribute this to improvements in treatment over time. However, differences in coding or distribution of variables between the data sets may offer an alternative explanation. We noted a larger number of patients in the TBI-trac dataset with an absent motor score (most likely including “false absent” due to sedation), and a lower number with unreactive pupils. It would seem possible that this lack of information is compensated for in the Extended Model by adding details on CT characteristics, as the fit of the extended IMPACT model is excellent. This excellent fit is remarkable, as the data sets underpinning the development of the prognostic models included series collected between 1984 and 1997 (Marmarou et al., 2007). The excellent fit emphasizes the broad generalizability of the IMPACT models both across time and settings.

Several limitations of our study should be acknowledged. First, the BTF TBI-trac database did not contain all variables in a similar coding format as that used in the IMPACT prognostic model. In particular, CT parameters were scored differently. We therefore had to modify and refit the IMPACT extended models to include these parameters; we consider it, however, unlikely that this will have influenced performance. The discrepancy in coding essential variables emphasizes the necessity for standardization of data collection in TBI studies, as recommended by the Common Data Elements initiative (Duhaime et al., 2010; Maas et al., 2010b,2011; Manley et al., 2010; Wilde et al., 2010).

Second, 14-day mortality was the outcome measure recorded in the BTF TBI-trac database, while the IMPACT models were developed to assess 6-month mortality and unfavorable outcome. We therefore refitted the IMPACT models for 14-day mortality in the development population. As a sensitivity analysis we also applied the original IMPACT Core Model to the BTF TBI-trac dataset and found no major differences in performance. This agreement is possibly explained by the fact that the highest proportion of mortality occurs in the first weeks after injury. In the CRASH trial investigating the effect of corticosteroid therapy after TBI, for example, 85% of deaths occurred in the first 2 weeks (CRASH Trial Collaborators, 2004,2005). The difference in outcome measures between the IMPACT and BTF TBI-trac datasets may, however, be considered more an asset than a limitation, as the good performance of the models on external validation versus 14-day mortality demonstrates the broad generalizability of these models across different settings.

Several other studies have also validated the IMPACT models. Panczykowski and associates reported good performance of the IMPACT models (AUC 0.76–0.83) at validation on an unselected series of 587 patients with severe TBI admitted to a single Level 1 trauma center (Panczykowski et al., 2011). Roozenbeek and associates (in press) externally validated the IMPACT models on 5 new datasets (2 observational studies and 3 clinical trials), also finding good performance with regard to both discrimination and calibration. Poorer performance was, however, found in validating the models on one of the most recent Phase III clinical trials in TBI (dexanabinol study; Maas et al., 2006). In combination with the results of these other validation studies, the current study has now shown good performance and broad generalizability for the IMPACT models across broad and widely different settings.

Conclusions

The current validation study of the contemporary BTF TBI-trac database from New York State demonstrates that the IMPACT prognostic models for TBI accurately predict 14-day mortality. In combination with the results of other validation studies, the current study has proven the generalizability of the IMPACT prognostic models across different settings and outcome measures.

Acknowledgments

Development of the IMPACT models was supported by National Institutes of Health grant 046291. The BTF TBI-trac data collection was funded through the New York State Department of Health Bureau of Emergency Medical Services and by the Brain Trauma Foundation.

Author Disclosure Statement

No competing financial interests exist.