Abstract
Traumatic brain injury (TBI) is common and debilitating. Randomized trials of interventions for TBI ideally assess effectiveness by using long-term functional neurological outcomes, but such outcomes are difficult to obtain and costly. If there is little change between functional status at hospital discharge versus 6 months, then shorter-term outcomes may be adequate for use in future clinical trials. Using data from a previously published multi-center, randomized, placebo-controlled TBI clinical trial, we evaluated patterns of missing outcome data, changes in functional status between hospital discharge and 6 months, and three prognostic models to predict long-term functional outcome from covariates available at hospital discharge (functional measures, demographics, and injury characteristics). The Resuscitation Outcomes Consortium Hypertonic Saline trial enrolled 1282 TBI patients, obtaining the primary outcome of 6-month Glasgow Outcome Score Extended (GOSE) for 85% of patients, but missing the primary outcome for the remaining 15%. Patients with missing outcomes had less-severe injuries, higher neurological function at discharge (GOSE), and shorter hospital stays than patients whose GOSE was obtained. Of 1066 (83%) patients whose GOSE was obtained both at hospital discharge and at 6-months, 71% of patients had the same dichotomized functional status (severe disability/death vs. moderate/no disability) after 6 months as at discharge, 28% had an improved functional status, and 1% had worsened. Performance was excellent (C-statistic between 0.88 and 0.91) for all three prognostic models and calibration adequate for two models (p values, 0.22 and 0.85). Our results suggest that multiple imputation of the standard 6-month GOSE may be reasonable in TBI research when the primary outcome cannot be obtained through other means.
Key words: : clinical trial design, functional outcomes, prognostic models, traumatic brain injury
Introduction
Traumatic brain injury (TBI) is a significant cause of death and disability in the United States. Each year, approximately 1.4 million U.S. residents are treated in emergency departments (EDs) for TBI, whereas, on average, 52,000 die.1 Of those who survive, the Centers for Disease Control and Prevention estimate that between 80,000 and 90,000 will suffer permanent disability, with annual U.S. direct and indirect costs of $60 billion.2,3 Clinical trials in this area seek to illuminate the effect of risk factors and treatments for this condition.
Different outcome measures have been used to evaluate the effect of TBI on functional outcome. The Glasgow Outcome Scale (GOS) and the more detailed Glasgow Outcome Scale Extended (GOSE), often evaluated 6 months after severe TBI (sTBI), are acknowledged to be the gold standard for assessment of functional neurological outcome for clinical trials.3–5 However, there are significant challenges in obtaining complete 6-month outcome data in the TBI population, including loss to follow-up and the need for caregivers to assist with data completion. Further, because of the nature of sTBI, patients are often initially unable to provide consent for themselves. Efforts to obtain long-term functional outcomes and minimize missing data among patients with TBI are time-consuming and expensive. Clinical trials studying early TBI interventions under exception from informed consent regulations are presumed to have even greater difficulty in obtaining long-term outcomes as a result of factors, such as patient refusal for continued participation and a broader patient population, that may be harder to track over time. However, published literature quantifying these factors in exception from informed consent studies is sparse. Several large trials operating partially under exception from informed consent have skirted these issues by excluding patients deemed unlikely to be available for follow-up.6,7 Other trials operating partially or fully under exception from informed consent that have not made these restrictions have reported rates of missing long-term functional outcomes up to 14%.8–12 In some cases, similar rates have been noted in clinical trials of TBI conducted with consent given by patients or family members before randomization.13 Among patients enrolled in the Resuscitation Outcomes Consortium (ROC) Hypertonic Saline (HS) TBI trial conducted under exception from informed consent, which was powered to detect a 7.5% absolute reduction in the incidence of poor outcomes (GOSE ≤4),14 the primary outcome of 6-month GOSE score was missing in 15% of enrolled patients.15
In this study, we evaluated patterns of missing functional outcome (6-month GOSE) data in a TBI study conducted under exception from informed consent, assessed whether a shorter follow-up period adequately represents 6-month functional outcome, and developed a series of prognostic models for long-term neurological outcomes based on variables available at the time of hospital discharge. The resulting models are analyzed for model performance and calibrated on a validation data set. We hypothesized that patient characteristics available at hospital discharge would be highly predictive of 6-month functional outcome and would support the use of such models in future clinical trials among patients when long-term outcome is missing.
Methods
Study design and setting
We performed a secondary analysis of data from a multi-center, double-blind, randomized, placebo-controlled trial conducted by ROC and administered under exception from informed consent.16 The study involved 13 regional clinical centers, 75 emergency medical service (EMS) agencies, and 53 hospitals in the United States and Canada. The trial, which was conducted between May 2006 and May 2009, tested the effect of out-of-hospital hypertonic saline with or without dextran on 6-month functional outcome (GOSE) after sTBI. Two cohorts of trauma patients, 1 with hypovolemic shock and 1 with TBI, were enrolled in the trial; this article considers only the TBI cohort. Further details of the trial have been described previously.15
Patients
Patients ≥15 years of age with blunt trauma and an out-of-hospital Glasgow Coma Scale (GCS) score ≤8 were randomized to receive hypertonic saline/dextran, hypertonic saline, or normal saline in the out-of-hospital setting. The study was terminated after exceeding prespecified futility boundaries, with 1282 enrolled patients with blunt trauma, a prehospital GCS score of ≤8, and without hypovolemic shock.
Outcome
The primary outcome measure was 6-month GOSE, which we considered to be the gold standard for functional neurological outcome in this study. When possible, GOSE was collected directly from the patient by a structured phone interview. If the patient was unable to participate, a family member or caregiver was allowed to provide information. Mortality information was used to supplement the GOSE outcome and was ascertained by hospital records, phone follow-up, and public records. For this analysis, we dichotomized functional outcome into two categories: 1) severe disability or death (GOSE ≤4) and 2) moderate or no disability (GOSE ≥5).
Predictors
For prognostic models predicting 6-month GOSE, we considered nine standard patient variables that are easily acquired either at enrollment or at hospital discharge (Table 1). Predictors included age, sex, Injury Severity Score (ISS), maximum head Abbreviated Injury Scale (AIS) score, discharge GOSE score, discharge Disability Rating Scale (DRS) score, length of hospital stay, number of days alive out of intensive care unit (ICU) through day 28 (ICU-free days), and discharge disposition (death, home, inpatient rehabilitation, skilled nursing facility, left against medical advice, or other). Similar to the 6-month GOSE outcome, the discharge GOSE score could be calculated from responses from the patient alone, caregiver alone, patient and caregiver together, or from hospital records. Length of hospital stay was defined as the time from presentation to the ED until death or discharge from hospital or ED. All covariates were modeled as continuous, except for sex and discharge disposition, which were modeled as categorical variables.
Table 1.
Covariates Selected for Inclusion in Prognostic Models
| Covariate | Model 1 | Model 2 | Model 3 |
|---|---|---|---|
| Age, years | X | ||
| Discharge disposition | |||
| Discharge Disability Rating Scale (range, 0–30) | X | ||
| Discharge Glasgow Outcome Scale Extended (range, 1–8) | X | X | X |
| Length of hospital stay, days | X | X | |
| Injury Severity Score (range, 0–75) | |||
| Maximum head Abbreviated Injury Severity score (range, 0–6) | |||
| Number of days alive out of ICU through day 28 | X | ||
| Sex | |||
| C-statistic (95% bootstrap confidence interval) | 0.882 (0.854, 0.907) | 0.910 (0.883, 0.934) | 0.912 (0.886, 0.937) |
| le Cessie–van Houwelingen p value | 0.22 | 0.85 | 0.03 |
ICU, intensive care unit.
Statistical analysis
We used descriptive statistics to investigate potential differences in relevant covariates between patients with and without missing 6-month GOSE among patients alive at the time of hospital discharge. We also investigated the reasons for missing data and described the number of attempts at contacting patients with missing and nonmissing outcomes. Finally, to evaluate whether functional status at hospital discharge adequately represents 6-month functional outcome, we examined the concordance between patients' discharge GOSE and 6-month GOSE.
Model development
Using patients with complete covariate and outcome information, we investigated the ability of three logistic regression models to predict 6-month severe disability or death (GOSE score ≤4). Model 1 included discharge GOSE as the only covariate, whereas model 2 included discharge GOSE and the length of hospital stay, both thought to be clinically important predictors of long-term functional outcome. Model 3 included multiple covariates selected via exhaustive search using Akaike information criterion (AIC) values from the nine predictors listed in Table 1. We randomly split the sample 60%/40% into a training data set (used for model development) and a validation data set (used to assess model performance). Individuals without a 6-month GOSE outcome were excluded from the primary analysis, but were included in a sensitivity analysis to evaluate the effect of patients with missing values on model development (described in detail below).
Because the DRS (an alternative measure of functional outcome) has been suggested to be more sensitive to subtle changes in functional status than the GOSE,17 we also investigated discharge DRS as a better predictor of 6-month outcome than discharge GOSE in a sensitivity analysis. To assess this possibility, we replaced discharge GOSE with discharge DRS in models 1 and 2. It was not necessary to do a similar analysis for model 3, which already included the potential to incorporate either discharge DRS and/or discharge GOSE as predictors.
Model performance and validation
We evaluated the performance characteristics of each prognostic model, including sensitivity and specificity, on a randomly selected validation data set consisting of 40% of the TBI sample. Receiver operating characteristic curves were plotted for the proposed models and the C-statistic calculated to quantify discrimination among models. For purposes of internal validation, we also investigated the calibration of each model, using the le Cessie–van Houwelingen goodness-of-fit test.18 A calibration plot for each model provided graphical support by plotting a Lowess smoother of observed versus predicted probabilities.19
In a sensitivity analysis, we reanalyzed the data, including patients in the validation data set who had complete predictors but missing outcomes. We assumed that, if their outcomes had been ascertained, these patients would have had the same misclassification rate as patients with the same discharge GOSE who had complete outcomes. We assigned the stratum-specific error rate, defined as the proportion of patients with complete outcomes in that discharge GOSE stratum who were misclassified, to patients with missing outcomes and reevaluated model performance. In a similar analysis, we analyzed discharge disposition strata. Patients missing 6-month GOSE were excluded from the primary analysis; including them by their expected misclassification rate provided a potentially improved representation of model performance (e.g., if strata that were harder to classify were more likely to be missing).
The GOSE interview focuses on the functional effect of the injury on disability. However, patients may not have full insight into their disability, particularly with respect to psychological changes; conversely, “worrywart” caregivers may paint an exaggerated picture of the extent of disability.20 To quantify the effect of interview respondent, we evaluated the prognostic models from the primary analysis when an indicator that the patient was the respondent was added as a predictor to each model. Finally, for comparison of the models in the primary analysis with existing prognostic models,21–24 we evaluated a CRASH-based model that included as predictors age, prehospital GCS, first ED pupillary reactivity, and Marshall score as well as a “known predictors” model that included age, prehospital GCS, first ED pupillary reactivity, lowest ED hemoglobin, Marshall score, maximum head AIS, and ISS as predictors.
All analyses were done using R for Windows (version 2.15.2).25
Results
The ROC TBI trial15 enrolled 1282 patients who met out-of-hospital criteria for TBI (GCS ≤8) without evidence of shock and formed the primary sample for analysis. Of the 1282 TBI patients, 321 (25.0%) died in hospital, 948 (73.9%) were discharged alive, and 13 (1.0%) had an unknown discharge status. There were 893 (69.7%) patients with a head AIS ≥3, and 346 (27.0%) died within 6 months of injury. Of the 1205 patients with known GOSE at hospital discharge (including in-hospital death), 985 (81.7%) had discharge GOSE ≤4. Of the 1087 patients whose functional outcome (including death) was assessable after 6 months, 628 (57.8%) had GOSE ≤4. Secondary short-term outcomes of GOSE and DRS on discharge were available in 94% and 93.8% (respectively) of all patients.
There were particular patterns of missing data in this sample. Of patients known to be discharged alive from the hospital (N=948), 64 (6.8%) were missing discharge GOSE and 184 (19.4%) were missing the primary functional outcome of 6-month GOSE. Among these 948 patients, characteristics of patients missing 6-month GOSE differed from those with observed 6-month GOSE (Table 2). Patients with missing outcomes were less likely to have had a critical head injury as measured by head AIS ≥3 and more likely to be discharged to home. Further, patients with missing outcomes had less-severe injuries as measured by ISS, higher neurological function at discharge (GOSE and DRS), more days alive out of hospital and out of ICU through day 28, and shorter hospital stays than those whose outcome was not missing.
Table 2.
Patient and Injury Characteristics Based on the Primary Outcome Measure of 6-Month GOSE
| Patients discharged alive (N=948) | ||||
|---|---|---|---|---|
| Covariate | All patients (N=1282) Mean (SD) | 6-month GOSE observed (N=764) Mean (SD) | 6-month GOSE missing (N=184) Mean (SD) | p value |
| Age, years | 39 (18.5) | 36.2 (16.6) | 34.2 (13.8) | 0.091 |
| Days alive out of hospital (through day 28) | 10.2 (11.1) | 12.3 (10.5) | 19.6 (9.8) | <0.001 |
| Days alive out of intensive care unit (through day 28) | 15.5 (11.5) | 19.8 (8.5) | 24.3 (6.8) | <0.001 |
| Length of hospital stay (days) | 16.9 (25.7) | 23.1 (28.7) | 11.3 (19) | <0.001 |
| Injury severity scale | 26.1 (15.8) | 25.1 (14.4) | 15.5 (14.2) | <0.001 |
| Discharge GOSE | 3.3 (2.1) | 3.9 (1.7) | 5 (2.3) | <0.001 |
| Discharge Disability Rating Score | 13.3 (11.7) | 7.5 (6.8) | 5.9 (7.6) | 0.022 |
| N (%) | N (%) | N (%) | p value | |
|---|---|---|---|---|
| Sex | ||||
| Male | 976 (76.1) | 582 (76.2) | 153 (83.2) | 0.049 |
| Unknown/NA | 2 (0.2) | 0 (0) | 0 (0) | |
| Injury severity: head category | ||||
| No injury | 237 (18.5) | 137 (17.9) | 67 (36.4) | |
| Minor | 6 (0.5) | 3 (0.4) | 2 (1.1) | |
| Moderate | 114 (8.9) | 86 (11.3) | 24 (13) | |
| Serious | 161 (12.6) | 120 (15.7) | 29 (15.8) | |
| Severe | 275 (21.5) | 199 (26) | 30 (16.3) | |
| Critical | 452 (35.3) | 207 (27.1) | 31 (16.8) | |
| Unsurvivable | 5 (0.4) | 0 (0) | 0 (0) | |
| Unknown/NA | 32 (2.5) | 12 (1.6) | 1 (0.5) | <0.001 |
| Marshall score, first head CT32 | ||||
| Diffuse Injury I (no visible pathology) | 375 (29.3) | 260 (34) | 99 (53.8) | |
| Diffuse Injury II | 434 (33.9) | 329 (43.1) | 60 (32.6) | |
| Diffuse Injury III (swelling) | 151 (11.8) | 77 (10.1) | 9 (4.9) | |
| Diffuse Injury IV (shift) | 51 (4) | 19 (2.5) | 4 (2.2) | |
| Evacuated mass lesion V | 207 (16.1) | 70 (9.2) | 8 (4.3) | |
| Nonevacuated mass lesion VI | 16 (1.2) | 5 (0.7) | 0 (0) | |
| Unknown/NA | 48 (3.7) | 4 (0.5) | 4 (2.2) | <0.001 |
| Discharge disposition | ||||
| Death | 317 (24.7) | 0 (0) | 0 (0) | |
| Home | 512 (39.9) | 386 (50.5) | 126 (68.5) | |
| Inpatient rehabilitation | 296 (23.1) | 258 (33.8) | 38 (20.7) | |
| Skilled nursing facility | 105 (8.2) | 96 (12.6) | 9 (4.9) | |
| Left against medical advice | 13 (1) | 5 (0.7) | 8 (4.3) | |
| Other | 39 (3) | 19 (2.5) | 3 (1.6) | <0.001 |
| Discharge GOSE interview respondent | ||||
| Patient only | 251 (19.6) | 209 (27.4) | 42 (22.8) | |
| Caregiver only | 212 (16.5) | 185 (24.2) | 27 (14.7) | |
| Patient and caregiver | 246 (19.2) | 231 (30.2) | 15 (8.2) | |
| Chart | 179 (14) | 122 (16) | 57 (31) | |
| Patient deceased | 321 (25) | 0 (0) | 0 (0) | |
| Unknown/NA | 73 (5.7) | 17 (2.2) | 43 (23.4) | <0.001 |
| 6-month GOSE interview respondent | ||||
| Patient only | 399 (31.1) | — | — | |
| Caregiver only | 287 (22.4) | — | — | |
| Patient and caregiver | 55 (4.3) | — | — | |
| Chart | 0 (0) | — | — | |
| Patient deceased | 346 (27) | — | — | |
| Unknown/NA | 195 (15.2) | — | — | NA |
For continuous covariates, p values are from a two-sample t-test assuming unequal variances. For categorical covariates, p values are from Fisher's exact test.
GOSE, Glasgow Outcome Scale Extended; NA, not available; CT, computed tomography; SD, standard deviation.
We also investigated the causes of missing outcomes in the sample (Fig. 1). Of the 1282 original patients, 321 (25.0%) were known to have died in the hospital/ED within 180 days and another 11 (0.9%) were ascertained to be dead subsequent to hospital discharge at the 1-month follow-up; these patients were not considered to have missing outcomes. Nine hundred and fifty (74.1%) patients were eligible to be contacted at the 6-month follow-up, with 195 (15.2%) missing 6-month GOSE. Of these, we ascertained that in 63 (32.3%) cases the patient/family/legally authorized representative actively refused consent, in 73 (37.4%) cases consent could not otherwise be obtained, and in 59 (30.3%) cases consent was obtained, but follow-up contact was unsuccessful. Among the 59 patients who were lost to follow-up, the median number of attempts was 5 (interquartile range, 2–7); in 6 cases, the number of attempts was not documented.
FIG. 1.
Flow chart of missing outcomes in Resuscitation Outcomes Consortium Hypertonic Saline trial, Traumatic Brain Injured (TBI) patients (N=1282). Patients known to be dead are considered not to have missing outcomes, by definition. Percentages are of total patients. GOSE, Glasgow Outcome Score Extended.
A priori, we anticipated discharge GOSE to be an important predictor of 6-month GOSE. Figure 2 shows the relationship between a patient's discharge GOSE and 6-month GOSE, with each line representing the GOSE trajectory of 1 patient. Of the 1066 patients that had both scores available, 540 (50.7%) had the same 6-month GOSE score as at discharge, whereas 176 (16.5%) had 6-month scores that differed by 1 point from discharge GOSE. Seven hundred and fifty-nine (71.2%) had the same dichotomized GOSE scores (severe disability/death [GOSE ≤4] vs. moderate/no disability [GOSE ≥5]) at 6 months as at discharge, 300 (28.1%) had a functional status that improved, and 7 (0.7%) had a worse functional status at 6 months than at hospital discharge.
FIG. 2.
Correspondence between discharge and 6-month Glasgow Outcome Scale Extended (GOSE) scores (N=1066). Each line shows 1 patient's trajectory from discharge GOSE score to 6-month GOSE score, with darker lines indicating more common trajectories. Analysis restricted to1066 patients who had both scores available; patients whose 6-month GOSE score was collected more than 300 days postinjury have been truncated to 300 days.
We considered three models for predicting a patient's 6-month severe disability, as measured by GOSE ≤4. Model 1 included the patient's discharge GOSE, whereas model 2 added length of hospital stay to this covariate. Model 3 included age, discharge GOSE, discharge DRS, length of hospital stay, and number of days out of the ICU through day 28 as predictors identified by exhaustive search using AIC. All three models had good discriminative ability when applied to the validation subset of 424 patients with complete predictors and outcomes. Model 1 had a C-statistic of 0.882 (95% bootstrap confidence interval [CI], 0.854, 0.907). Model 3 had the highest discrimination with a C-statistic of 0.912 (CI, 0.886, 0.937), whereas model 2 was similar with a C-statistic of 0.910 (CI, 0.883, 0.934). Models 2 and 3 were significantly different from model 1 in terms of discriminative ability; there was no difference between models 2 and 3 in this regard. Figure 3 shows the corresponding receiver operating characteristic curve for each of the models, and prediction equations for each model are given in Table 3.
FIG. 3.
Receiver operating characteristic curves for primary analysis, using validation data set (N=424). Model 1: discharge Glasgow Outcome Score Extended; model 2: model 1 plus length of hospital stay; model 3: model 2 plus age, discharge Disability Rating Scale, and number of intensive care unit–free days.
Table 3.
Prognostic Model Equations for Predicting Unfavorable 6-Month Functional Outcome
| Prognostic model | Equation for predicting P (6-month GOSE ≤4) |
|---|---|
| Model 1 |  |
| Model 2 |  |
| Model 3 |  |
Coefficients are estimated using both training and validation sample.
GOSEDc, Glasgow Outcome Scale Extended score at discharge (0–8); Hosptime, length of hospital stay (days); DRSDc=Disability Rating Scale score at discharge (0–30); OoICU, number of days alive out of intensive care unit through day 28.
Calibration of each model was assessed both graphically and analytically. The le Cessie–van Houwelingen test was carried out to determine the goodness of fit of the proposed models in the validation data set. Models 1, 2, and 3 had p values of 0.22, 0.85, and 0.03, respectively, indicating adequate agreement for models 1 and 2 and a lack of fit in the validation sample for Model 3. Figure 4 gives a graphical depiction of each model's calibration using the validation data set. As a sensitivity analysis, we explored the possibility that discharge DRS, instead of discharge GOSE, would be most predictive of 6-month functional outcome. A model using discharge DRS as the sole predictor of 6-month GOSE had a C-statistic of 0.895 (95% bootstrap CI, 0.865, 0.922), whereas adding a patient's length of hospital stay increased the C-statistic to 0.905 (0.877, 0.930).
FIG. 4.
Calibration plot using validation data set (N=424), for each prognostic model. Model 1: discharge GOSE; model 2: model 1 plus length of hospital stay; model 3: model 2 plus age, discharge Disability Rating Scale, and number of intensive care unit–free days. C-H, le Cessie–van Houwelingen; GOSE, Glasgow Outcome Score Extended.
In practice, an investigator may prefer to summarize a model's performance by the number of patients misclassified at a given threshold, depending on the relative cost of making a particular kind of error. As an illustration, for model 2, setting the false-positive rate at 5% (specificity of 95%) corresponded with a sensitivity of 71.4%; using this rule in the validation set, 73 of 255 patients with 6-month GOSE ≤4 (poor functional outcome) and 8 of 169 with 6-month GOSE ≥5 (good functional outcome) would be misclassified, for an overall misclassification rate of 81/424=19%. Likewise, setting the false-negative rate at 5% (sensitivity of 95%) corresponded with a specificity of 46.8% for model 2. Using this rule in the validation set, 13 of 255 patients with 6-month GOSE ≤4 and 90 of 169 with 6-month GOSE ≥5 would be misclassified, or an overall misclassification rate of 103/424=24%.
We performed a sensitivity analysis in which patients in the validation data set with missing outcomes, but available predictors, were reweighted by the error rate noted in that stratum of discharge GOSE. After reweighting, the C-statistic for model 1 was 0.883, for model 2 was 0.909, and for model 3 was 0.913. A similar analysis that reweighted based on categories of discharge disposition for model 3 saw similarly small differences (not shown). A sensitivity analysis that accounted for the effect of GOSE respondent found no difference in discriminatory ability from the corresponding models in the primary analysis. A CRASH22-based prognostic model that included age, prehospital GCS, first ED pupillary reactivity, and Marshall score had a C-statistic of 0.805 in the validation sample; an expanded model that included all available known predictors (age, prehospital GCS, first ED pupillary reactivity, lowest ED hemoglobin, Marshall score, head AIS, and ISS) had a C-statistic of 0.813 in the validation sample.
Discussion
In this study, we investigated several aspects of obtaining functional outcome measures among patients with serious TBI enrolled in an out-of-hospital interventional trauma trial conducted under exception from informed consent. The characteristics of TBI patients with and without missing functional outcome (GOSE) at 6 months differed, with those missing 6-month GOSE generally being less severely injury (less-severe brain injuries, better functional status at discharge, and shorter length of hospital stay). Comparing functional status at hospital discharge versus 6 months postinjury suggests that changes in mental function (mostly improvements) continue after the initial hospitalization. Our findings also suggest that patient characteristics, injury severity, and function at hospital discharge can be used to accurately predict 6-month functional status for patients with missing primary outcome data. Whereas previous studies21–24 have investigated this topic, we selectively considered predictors that would be routinely collected in a large-scale clinical trial. We also explored the reasons for missing outcomes and found that a majority were secondary to refusal or inability to obtain consent and a smaller portion being truly lost to follow-up.
Patients with TBI enrolled in a trauma trial with and without measured GOSE obtained at 6 months postinjury appear to be inherently different. This finding is important for several reasons. Assuming that all patients missing the 6-month functional outcome have either “good” neurological function or “poor” neurological function is likely to introduce bias to study findings. Simply excluding patients missing the primary outcome measure can also introduce bias by skewing the analyzed sample to patients with more-severe brain injuries and apparently worse functional outcomes. Thus, the mechanism of missingness for functional outcome was not “missing completely at random” (MCAR), where missing values have no association with observed or unobserved values. Though simplistic methods for handling missing data appear flawed, our results demonstrate that patient information available at or before hospital discharge adequately predicts 6-month GOSE. These findings suggest that the mechanism of missingness is more likely “missing at random” (MAR), whereby observed values can be used to accurately impute missing outcomes. For illustration, using only observed primary outcomes, we would estimate 58% (628 of 1087) of patients to have an unfavorable 6-month outcome (GOSE ≤4); when we supplement observed data by using model 2 to predict 6-month outcome for those missing this measurement, we estimate only 54% of patients to have an unfavorable outcome. This difference, though small, could have great impact on trials powered to detect modest differences. Having either MCAR or MAR as the mechanism of missingness is a key assumption in using such methods as multiple imputation to handle missing values,26–29 thereby preserving study power and minimizing bias.
Although securing functional status for 100% of patients is ideal, this is generally unrealistic, even under the best of circumstances. Using standard clinical information available during a patient's hospital stay, we evaluated three prognostic models for a patient's 6-month severe disability, which were all highly predictive (C-statistic >0.88) of 6-month GOSE. The two most parsimonious models were adequately calibrated, whereas the lack of calibration observed in the most complicated model could possibly have been a result of overfitting. Balancing both model performance and fit, model 2 (which included discharge GOSE and length of hospital stay) appeared to be the best of the three prognostic models evaluated and may help guide future efforts to handle missing long-term outcome data in TBI research. Using discharge DRS in place of discharge GOSE as a predictor did not strongly affect the discriminatory performance of the resulting models, whereas using a combination of nonfunctional predictors had inferior discrimination, compared with the models from the primary analysis.
Although prognostic models involving discharge GOSE were shown to be highly predictive of long-term functional outcome, our findings suggest against using discharge GOSE in place of 6-month GOSE as the primary clinical outcome. In our sample, just over half of patients had the same GOSE at discharge as at 6 months. When GOSE was categorized into standard good/bad neurological outcomes, over one quarter of patients changed GOSE categories (from bad to good) between hospital discharge and 6 months postinjury. Whereas very few patients had worse functional outcome in this time period, using discharge GOSE in place of 6-month GOSE appears not to be adequate by itself.
These results illustrate the importance of effective methods for handling long-term follow-up data for patients with TBI and the challenges inherent in emergency care research tracking long-term functional outcomes. We recommend the multiple imputation strategy demonstrated here as a way to minimize bias potentially introduced by missing data. The prognostic models we developed had excellent performance in our data set, but we recommend thoughtful calibration of these models for new applications. We demonstrate that even among high-functioning research sites participating in a large research network, there will be missing outcome data for a portion of patients. Injury, demographic, and outcome measures are typically easier to obtain during the hospital stay, because once patients leave the hospital, they may be more reluctant to continue participation in the trial or more difficult to contact, even when consent is obtained. This reluctance to participate or loss to follow-up appears more likely among patients with less-severe injuries and less disability, thus skewing the sample and likely distorting study findings if such patients are simply excluded. An investigation into the causes of missing data in this trial found that many patients had refused to consent to continued participation in the trial or were so difficult to contact that consent or the outcome of interest could not be obtained. In TBI trials not conducted under exception from informed consent, it is possible that these patients would have refused participation from the start, thereby reducing the proportion of patients missing the primary outcome. Our findings should provide insight and guidance to investigators pursuing future interventional TBI trials and evaluating long-term functional status as an outcome.
Limitations
For the prognostic models, we conducted a complete-case analysis, which excluded patients with missing information for 6-month GOSE score or predictors. If missingness in 6-month GOSE were related to factors that were not included in the prognostic models we developed, then this approach could have introduced bias. We addressed this potential limitation through a sensitivity analysis including patients with missing 6-month GOSE in the models, with qualitatively similar results. By including discharge GOSE in all models considered here, our prognostic models were considered to be valid, provided one agrees with the premise that discharge GOSE adequately captures patient health, and missingness does not depend on other factors; we believed both assumptions were reasonable. The GOSE asks respondents to compare current and preinjury functionality; at hospital discharge, patients' sense of their own social abilities and work abilities may not reflect their true functionality in everyday life. This is an acknowledged limitation of the GOSE and thus of our analysis, too.20 Also, although the dichotomization of 6-month GOSE into favorable/unfavorable outcomes is common in the TBI literature, a dichotomized outcome cannot capture the full trajectory of recovery after TBI. Smaller changes in GOSE score from hospital discharge to 6 months may still have large effects on patients' daily lives; our analysis does not attempt to describe these smaller changes. Finally, although our prognostic models had excellent discrimination in this data set under the assumptions described above, the prediction equations used here would not be valid for a population that had a different relationship between discharge and 6-month GOSE. We believe the multiple imputation strategy itself to be exportable, but researchers are advised to develop imputation models appropriate to the situation at hand.
Conclusions
In the context of an out-of-hospital trauma trial conducted under exception from informed consent, we demonstrate that TBI patients with and without measured 6-month GOSE are inherently different. Prognostic models built from information obtained during hospitalization appear highly predictive of 6-month GOSE, and the multiple imputation strategy may be reasonable for 6-month functional status when it cannot otherwise be obtained. Using such models (e.g., through multiple imputation) appears less biased than alternative methods for handling missing outcomes. A shorter duration of follow-up appears inadequate in representing long-term functional outcome after TBI, underestimating the improvements that patients may see after hospital discharge. Whereas minimizing missing data wherever possible is critically important, especially in trials designed to detect modest treatment effects,30,31 our findings provide guidance for clinical trial design and handling missing outcome data in future TBI studies.
Contributor Information
Collaborators: the Resuscitation Outcomes Consortium Investigators
Acknowledgments
The authors acknowledge the hard work and dedication of all the EMS and fire agencies that participated in the ROC hypertonic saline/dextran (HSD) study. Research in the prehospital setting would not be possible without the tireless efforts of their paramedics and firefighters. Special thanks go to the pre- and in-hospital data guardians and research coordinators at each of the participating sites for their diligence and patience in abstracting the additional data required to contribute to this substudy of ROC HSD.13
The ROC is supported by a series of cooperative agreements to 10 regional clinical centers and one data coordinating center (5U01 HL077863-University of Washington Data Coordinating Center, HL077865-University of Iowa, HL077866-Medical College of Wisconsin, HL077867-University of Washington, HL077871-University of Pittsburgh, HL077872-St. Michael's Hospital, HL077873-Oregon Health & Science University, HL077881-University of Alabama at Birmingham, HL077885-Ottawa Hospital Research Institute, HL077887-University of Texas SW Medical Center Dallas, and HL077908-University of California San Diego) from the National Heart, Lung and Blood Institute (NHLBI) in partnership with the National Institute of Neurological Disorders and Stroke, U.S. Army Medical Research and Material Command, The Canadian Institutes of Health Research (CIHR), the Institute of Circulatory and Respiratory Health, Defense Research and Development Canada, the Heart, Stroke Foundation of Canada, and the American Heart Association. Laurie J. Morrison (Toronto Regional Coordinating Center [RCC]), Eileen M. Bulger (Data Coordinating Center, University of Washington, Seattle RCC), Karen J. Brasel (Milwaukee RCC), Kellie Sheehan (Data Coordinating Center, University of Washington), Joseph P. Minei (Dallas RCC), Jeffrey D. Kerby (Alabama RCC), and Samuel A. Tisherman (Pittsburgh RCC) all receive ROC funding. Sandro Rizoli receives ROC funding for the PROPPR study, plus honoraria from CSL Behring and KCI and, until recently, a CIHR new investigator award in partnership with NovoNordisk. Leila R. Zelnick receives salary support from the National Institutes of Health (NIH; grant no.: T32 CA09168). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NHLBI or the NIH.
Author Disclosure Statement
No competing financial interests exist.
References
- 1.Faul M., Xu L., Wald M.M., and Coronado V.G. (2010). Traumatic brain injury in the United States: emergency department visits, hospitalizations, and deaths 2002–2006. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control: Atlanta, GA [Google Scholar]
- 2.Langlois J., Rutland-Brown W., and Thomas K.E. (2006). Traumatic brain injury in the United States: emergency department visits, hospitalizations, and deaths. Centers for Disease Control and Prevention, National Center for Injury Protection: Atlanta, GA [Google Scholar]
- 3.Maas A.I.R., Roozenbeek B., and Manley G.T. (2010). Clinical trials in traumatic brain injury: past experience and current developments. Neurotherapeutics 7, 115–126 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Clifton G.L., Hayes R.L., Levin H.S., Michel M.E., and Choi S.C. (1992). Outcome measures for clinical trials involving traumatically brain-injured patients: report of a conference. Neurosurgery 31, 975–978 [DOI] [PubMed] [Google Scholar]
- 5.Teasdale G.M., Pettigrew L.E.L., Wilson J.T.L., Murray G., and Jennett B. (1998). Analyzing outcome of treatment of severe head injury: a review and update on advancing the use of the Glasgow Outcome Scale. J. Neurotrauma 15, 587–597 [DOI] [PubMed] [Google Scholar]
- 6.Maas A.I.R., Murray G., Henney H., Kassem N., Legrand V., Mangelus M., Muizelaar J.P., Stocchetti N., Knoller N., and the Pharmos TBI Investigators. (2006). Efficacy and safety of dexanabinol in severe traumatic brain injury: results of a phase III randomised, placebo-controlled, clinical trial. Lancet Neurol. 5, 38–45 [DOI] [PubMed] [Google Scholar]
- 7.European Study Group of Nimodipine in Severe Head Injury. (1994). A multicenter trial of the efficacy of nimodipine on outcome after severe head injury. J. Neurosurg. 80, 797–804 [DOI] [PubMed] [Google Scholar]
- 8.Clifton G.L., Miller E.R., Choi S.C., Levin H.S., McCauley S., Smith K.R., Muizelaar J.P., Wagner F.C., Marion D.W., Luerssen T.G., Chesnut R.M., and Schwartz M. (2001). Lack of effect of induction of hypothermia after acute brain injury. N. Engl. J. Med. 344, 556–563 [DOI] [PubMed] [Google Scholar]
- 9.Edwards P., Arango M., Balica L., Cottingham R., El-Sayed H., Farrell B., Fernandes J., Gogichaisvili T., Golden N, Hartzenberg B., Husain M., Ulloa M.I., Jerbi Z., Khamis H., Komolafe E., Laloe V., Lomas G., Ludwig S., Mazairac G., Munoz Sanchez M., Nasi L., Olldashi F., Plunkett P., Roberts I., Sandercock P., Shakur H., Soler C., Stocker R., Svoboda P., Trenkler S., Venkataramana N.K., Wasserberg J., Yates D., Yutthakasemsunt S., and the MRC CRASH Trial Collaborators. (2005). Final results of MRC CRASH, a randomized placebo-controlled trial of intravenous corticosteroid in adults with head injury—outcomes at 6 months. Lancet 365, 1957–1959 [DOI] [PubMed] [Google Scholar]
- 10.Marshall L.F., Maas A.I.R., Marshall S.B., Bricolo A., Fearnside M., Iannotti F., Klauber M., Lagarrigue J., Lobato R., Persson L., Pickard J.D., Piek J., Servadei F., Wellis G.N., Morris G.F., Means E.D., and Musch B. (1998). A multicenter trial on the efficacy of using tirilazad mesylate in cases of head injury. J. Neurosurg. 89, 519–525 [DOI] [PubMed] [Google Scholar]
- 11.Temkin N.R., Anderson G.D., Winn H.R., Ellenbogen R.G., Britz G.W., Schuster J., Lucas T., Newell D.W., Mansfield P.N., Machamer J.E., Barber J., and Dikmen S.S. (2007). Magnesium sulfate for neuroprotection after traumatic brain injury: a randomized controlled trial. Lancet Neurol. 6, 29–38 [DOI] [PubMed] [Google Scholar]
- 12.Young B., Runge J.W., Waxman K.S., Harrington T., Wilberger J., Muizelaar J.P., Boddy A., and Kupiec J.W. (1996). Effects of pegorgotein on neurologic outcome of patients with severe head injury: a multicenter, randomized controlled trial. JAMA 276, 538–543 [PubMed] [Google Scholar]
- 13.Marmarou A., Nichols J., Burgess J., Newell D., Troha J., Burnham D., Pitts L., and the American Brain Injury Consortium Study Group. (1999). Effects of the bradykinin antagonist Bradycor™ (Deltibant, CP-1027) in severe traumatic brain injury: results of a multi-center, randomized, placebo-controlled trial. J. Neurotrauma 16, 431–444 [DOI] [PubMed] [Google Scholar]
- 14.Brasel K.J., Bulger E., Cook A.J., Morrison L.J., Newgard C.D., Tisherman S.A., Kerby J.D., Coimbra R., Hata J.S., Hoyt D.B., and the Resuscitation Outcomes Consortium. (2008). Hypertonic resuscitation: design and implementation of a prehospital intervention trial. J. Am. Coll. Surg. 206, 220–232 [DOI] [PubMed] [Google Scholar]
- 15.Bulger E.M., May S., Brasel K.J., Schreiber M., Kerby J.D., Tisherman S.A., Newgard C., Slutsky A., Coimbra R., Emerson S., Minei J.P., Bardarson B., Kudenchuk P., Baker A., Christenson J., Idris A., Davis D., Fabian T.C., Aufderheide T.P., Callaway C., Williams C., Banek J., Vaillancourt C., van Heest R., Sopko G., Hata J.S., Hoyt D.B., and the Resuscitation Outcomes Consortium Investigators. (2010). Out-of-hospital hypertonic resuscitation following severe traumatic brain injury: a randomized controlled trial. JAMA 304, 1455–1464 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Davis D.P., Garberson L.A., Andrusiek D.L., Hostler D., Daya M., Pirrallo R., Craig A., Stephens S., Larsen J., Drum A.F., Fowler R., and the Resuscitation Outcomes Consortium Investigators. (2007). A descriptive analysis of emergency medical service systems participating in the Resuscitation Outcomes Consortium (ROC) network. Prehosp. Emerg. Care 11, 369–382 [DOI] [PubMed] [Google Scholar]
- 17.Struchen M.A., Hannay H.J., Contant C.F., and Robertson C.S. (2001). The relation between acute physiological variables and outcome on the Glasgow Outcome Scale and Disability Rating Scale following severe traumatic brain injury. J. Neurotrauma 18, 115–125 [DOI] [PubMed] [Google Scholar]
- 18.Le Cessie S., and van Houwelingen J.C., (1991). A goodness-of-fit test for binary regression models, based on smoothing methods. Biometrics 47, 1267–1282 [Google Scholar]
- 19.Cleveland W.S. (1979). Robust locally weighted regression and smoothing scatterplots. J. Am. Stat. Assoc. 74, 829–936 [Google Scholar]
- 20.Wilson J.T.L., Pettigrew L.E.L., and Teasdale G.M. (1998). Structured interviews for the Glasgow Outcome Scale and the Extended Glasgow Outcome Scale: guidelines for their use. J. Neurotrauma 15, 573–580 [DOI] [PubMed] [Google Scholar]
- 21.Hukkelhoven C.W.P.M., Steyerberg E.W., Habbema J.D.F., Farace E., Marmarou A., Murray G.D., Marshall L.F., and Maas A.I.R. (2005). Predicting outcome after traumatic brain injury: development and validation of a prognostic score based on admission characteristics. J. Neurotrauma 22, 1025–1039 [DOI] [PubMed] [Google Scholar]
- 22.MRC CRASH Trial Collaborators, Perel P., Arango M., Clayton T., Edwards P., Komolafe E., Pocock S., Roberts I., Shakur H., Steyerberg E., and Yutthakasemsunt S. (2008). Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 336, 425–429 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Steyerberg E.W., Mushkudiani N., Perel P., Butcher I., Lu J., McHugh G.S., Murray G.D., Marmarou A., Roberts I., Habbema J.D.F., and Maas A.I.R. (2008). Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med. 5, e165. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Yuan F., Ding J., Chen H., Guo Y., Wang G., Gao W.W., Chen S.W., and Tian H.L. (2012). Predicting outcomes after traumatic brain injury: the development and validation of prognostic models based on admission characteristics. J. Trauma Acute Care Surg. 73, 137–145 [DOI] [PubMed] [Google Scholar]
- 25.R Development Core Team (2012). A language and environment for statistical computing. http://www.R-project.org
- 26.Crawford S.L., Tennstedt S.L., and McKinlay J.B. (1995). A comparison of analytic methods for non-random missingness of outcome data. J. Clin. Epidemiol. 48, 209–219 [DOI] [PubMed] [Google Scholar]
- 27.Greenland S., and Finkle W.D. (1995). A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am. J. Epidemiol. 142, 1255–1264 [DOI] [PubMed] [Google Scholar]
- 28.Newgard C.D., and Haukoos J.S. (2007). Advanced statistics: missing data in clinical research—part 2: multiple imputation. Acad. Emerg. Med. 14, 669–678 [DOI] [PubMed] [Google Scholar]
- 29.Rubin D.B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons, Inc.: New York [Google Scholar]
- 30.Fleming T.R. (2011). Addressing missing data in clinical trials. Ann. Intern. Med. 154, 113–117 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Little R.J., D'Agostino R., Cohen M.L., Dickersin K., Emerson S.S., Farrar J.T., Frangakis C., Hogan J.W., Molenberghs G., Murphy S.A., Neaton J.D., Rotnitzky A., Scharfstein D., Shih W.J., Siegel J.P., and Stern H. (2012). The prevention and treatment of missing data in clinical trials. N. Engl. J. Med. 367, 1355–1360 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Maas A.I.R., Hukkelhoven C.W.P.M., Marshall L.F., and Steyerberg E.W. (2005). Prediction of outcome in traumatic brain injury with computed tomographic characteristics: a comparison between the computed tomographic classification and combinations of computed tomographic predictors. Neurosurgery 57, 1173–1182 [DOI] [PubMed] [Google Scholar]