The Influence of Enrollment Criteria on Recruitment and Outcome Distribution in Traumatic Brain Injury Studies: Results from the Impact Study

Abstract

Substantial heterogeneity exists among patients who suffer from traumatic brain injury (TBI). Strict enrollment criteria may diminish heterogeneity in randomized controlled trials (RCTs), but will also decrease recruitment and may affect the outcome distribution. The aim of this study was to investigate the influences of commonly used enrollment criteria for RCTs in TBI on potential recruitment and on outcome distribution. We used individual patient data from the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) database, including six therapeutic phase III RCTs (n = 5816) and three surveys (n = 2217) in TBI. The primary outcome was the Glasgow Outcome Scale (GOS) at 6 months after injury, which we dichotomized as favorable/unfavorable. We investigated the influences of commonly used enrollment criteria on recruitment and outcome distribution: time window between injury and admission to study hospital ≤ 8 h; age at injury ≤ 65 years; ≥ 1 reactive pupil; motor score > 1; Glasgow Coma Scale ≤ 8. Application of all enrollment criteria resulted in a large reduction of recruitment in both the surveys (up to 65%) and the RCTs (up to 41%). Among the remaining patients, fewer had an unfavorable outcome in both the surveys (original, 60%; remaining, 44%) and the RCTs (original, 43%; remaining, 38%). Applying these enrollment criteria to patients from the surveys resulted in an outcome distribution that approximated the outcome observed in the RCTs. The use of strict enrollment criteria leads to substantial reductions in the recruitment of RCTs in TBI. The outcome in TBI studies depends strongly on the enrollment criteria.

Introduction

Traumatic brain injury (TBI) represents a serious health problem worldwide, posing high costs to society. Much research is undertaken to improve outcome after TBI. Comparing outcome between different series of TBI patients and demonstrating efficacy of new approaches in randomized clinical trials (RCTs) is notoriously difficult due to the inherent heterogeneity of TBI populations (Maas et al., 1999). Heterogeneity in terms of type and severity of injury, as well as in terms of prognostic risk, may cause imbalances between treatment groups and dilute treatment effects.

Imbalances in baseline characteristics between treatment groups have been reported in some RCTs (Marshall et al., 1998). Such imbalances may be prevented by block randomization (Bullock, 2002), stratified randomization (Bullock, 2002; Choi and Bullock, 2001; Maas et al., 1999), and minimization. We may achieve similar numbers of patients in each arm of the trial by random allocation in blocks (“block randomization”). Stratified randomization may be used to ensure that equal numbers will be allocated to each arm with a characteristic thought to affect prognosis. Furthermore, minimization may be used to ensure balance between trial arms for several patient factors. The first patient is truly randomly allocated; for each subsequent patient, the treatment allocation is identified, which minimizes the imbalance between groups at that time. Approaches for dealing with the heterogeneity in the analysis phase (e.g., covariate adjustment) have also been employed in TBI trials (Hernández et al., 2006).

Heterogeneity amongst the population under study may also be reduced by using strict enrollment criteria. Enrollment criteria aim to select patients that are most likely to benefit from the studied treatment, both from a mechanistic and prognostic perspective (Maas et al., 1999; Machado et al., 1999; Narayan et al., 2002). Three main goals are aimed for: (1) exclusion of patients with a very good or very bad prognosis, because one may expect that these patients will do well without any treatment or will do poorly no matter what treatment is given; (2) inclusion of patients who are likely to benefit from the treatment, because of the pathophysiologic background of the injury and the studied treatment; (3) reaching a distribution of approximately 50% of patients with a favorable outcome and 50% of patients with an unfavorable outcome in the study population, to increase statistical power for the detection of a true treatment effect.

Machado et al. (1999) have demonstrated that targeting a clinical trial to patients with an intermediate prognostic risk would permit a potential reduction in sample size of approximately 30%. The disadvantage of this approach, however, is that it may substantially reduce recruitment.

The objective of this study is to investigate the influences of common enrollment criteria on the potential recruitment and on the outcome distribution of RCTs in TBI.

Methods

Study population

We used individual patient data from the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) database. This database contains data of patients with moderate (Glasgow Coma Scale [GCS], 9–12) and severe (GCS ≤ 8) TBI from eight RCTs and three surveys conducted between 1984 and 1996. IMPACT links researchers in Belgium, the Netherlands, the United Kingdom, and the United States in a project addressing methodological problems in design and analysis of RCTs in TBI. Details of the different studies and data management of the IMPACT database have been described previously (Marmarou et al., 2007). The primary outcome measurement was the Glasgow Outcome Scale (GOS) at 6 months after injury. For our analyses, we dichotomized the GOS into unfavorable (GOS 1–3) versus favorable outcome (GOS 4–5). We selected patients over 14 years of age at the time of injury. The data from two RCTs (HIT-I and SKB) were not used in our analysis, because the numbers of patients in these studies were too small for the purpose of our study. This resulted in a cohort of 8033 adults: 2217 patients from surveys and 5816 patients from RCTs.

Selection by enrollment criteria

We investigated the influences of five commonly used enrollment criteria for TBI studies: time window between injury and admission to study hospital ≤ 8 h; age at injury ≤ 65 years; ≥1 reactive pupil; motor score > 1; and GCS ≤ 8. We further explored the influence of other cut-off values for time window (≤4, ≤6, and ≤12 h) and for age (≤55, ≤60, and ≤70 years). The enrollment criteria were applied separately and simultaneously. The influence on the recruitment was studied by comparing the original sample sizes to the new sample sizes with the investigated enrollment criteria applied. The change in outcome distribution was studied by comparing unfavorable outcome fractions at 6 months after the injury.

Statistical analysis

Descriptive statistics included medians and percentages. The software used was SPSS, version 11.0.1 (SPSS Inc., Chicago, IL). Figure 1 was produced with R software version 2.6.0 (The R Foundation, 2007).

FIG. 1.

The influence of the application of the investigated enrollment criteria (time window ≤ 8 h; age at injury ≤ 65 years; ≥1 reactive pupil; motor score > 1; GCS ≤ 8) on recruitment and outcome distribution in three surveys and six randomized controlled trials (RCTs) of the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) database. For each study, the selected (gray bars) and the excluded patients (black bars) are depicted. The width of the bars indicates the numbers of selected and excluded patients.

Results

The baseline characteristics of the original study population are summarized in Table 1. These reflect the effects of the different initial enrollment criteria per study.

Table 1.

Distribution of Patient Characteristics Across Three Surveys and Six RCTs from the IMPACT Study

			Pupillary reactivity	Motor score
	Time window (hours), median (interquartile range)	Age (years), median (interquartile range)	(1) both pupils pos (2) one pupil pos (3) both pupils neg	(1,2) none/extension (3) abnormal flexion (4) normal flexion (5,6) localizes/obeys (9) untestable/missing	GCS, median (interquartile range)
				243 (40%)
			300 (50%)	74 (12%)
TCDB (n = 604)	1.2 (0.8–3.1)	26 (21–40)	55 (9%)	122 (20%)	6 (4–8)
			249 (41%)	134 (22%)
				31 (5%)
				198 (25%)
			429 (54%)	37 (5%)
UK4 (n = 791)	3.5 (2.0–8.9)	36 (22–55)	115 (15%)	141 (18%)	6 (4–8)
			247 (31%)	221 (28%)
				194 (25%)
				230 (28%)
			527 (64%)	55 (7%)
EBIC (n = 822)	3.7 (1.5–8.0)	38 (24–59)	87 (11%)	113 (14%)	7 (4–11)
			208 (25%)	281 (34%)
				143 (17%)
				671 (30%)
			1256 (57%)	166 (8%)
Total surveys (n = 2217)	2.8 (1.2–6.1)	32 (22–53)	257 (12%)	376 (17%)	7 (4–9)
			704 (32%)	636 (29%)
				368 (16%)
				280 (34%)
			583 (71%)	92 (11%)
HIT II (n = 819)	2.0 (1.0–4.0)	33 (22–49)	101 (12%)	181 (22%)	8 (6–10)
			135 (16%)	207 (25%)
				59 (7%)
				152 (15%)
			709 (68%)	132 (13%)
TIUS (n = 1041)	.7 (0.5–1.2)	30 (23–41)	122 (12%)	300 (29%)	7 (6–10)
			210 (20%)	457 (44%)
				0 (0%)
				141 (13%)
			813 (73%)	237 (21%)
TINT (n = 1118)	1.2 (0.8–2.1)	30 (21–45)	170 (15%)	327 (29%)	6 (5–7)
			135 (12%)	413 (37%)
				0 (0%)
				655 (43%)
			779 (52%)	165 (11%)
PEGSOD (n = 1510)	1.1 (0.6–2.2)	27 (20–38)	160 (11%)	334 (22%)	5 (3–7)
			571 (38%)	356 (24%)
				0 (0%)
				55 (13%)
			316 (78%)	91 (22%)
SLIN (n = 409)	No data	28 (21–43)	79 (19%)	127 (31%)	6 (5–7)
			14 (3%)	136 (33%)
				0 (0%)
				264 (29%)
			612 (67%)	143 (16%)
SAPHIR (n = 919)	2.3 (1.0–4.2)	32 (20–38)	307 (33%)	223 (24%)	6 (5–8)
			0 (0%)	286 (31%)
				3 (0%)
				1547 (27%)
			3812 (66%)	860 (15%)
Total RCTs (n = 5816)	1.2 (0.7–2.6)	30 (21–43)	939 (16%)	1492 (26%)	7 (5–10)
			1065 (18%)	1855 (32%)
				62 (1%)

Effects of enrollment criteria on recruitment

Effects of enrollment criteria on recruitment were more pronounced in the surveys than in the RCTs, since these had less stringent initial enrollment criteria (Table 1). When each of the studied criteria was applied separately in the surveys, “at least one reactive pupil” led to the largest reduction in recruitment (32%), followed by “time window < 8 h” (20%), “motor score > 1” (18%), “GCS ≤ 8” (15%), and “age at injury ≤ 65” (12%; Table 2). In the RCTs, the reductions were less (Table 2).

Table 2.

Numbers of Excluded Patients Across Studies by Application of Enrollment Criteria

	Original study population		Time window ≤8 h		Age ≤ 65 years		Unreactive pupils		Motor score > 1		GCS ≤ 8		All criteria applied		Total remaining N if criteria applied
	N	% Unfav of N	No. of patients excluded from orig. N	% Unfav of excluded patients	No. of patients excluded from orig. N	% Unfav of excluded patients	No. of patients excluded from orig. N	% Unfav of excluded patients	No. of patients excluded from orig. N	% Unfav of excluded patients	No. of patients excluded from orig. N	% Unfav of excluded patients	No. of patients excluded from orig. N	% Unfav of excluded patients	% Unfav of excluded patients	N	% Unfav of remaining N
TCDBa	604	65	38	74	41	95	249	88	136	91	40	45	351	58	81	253	44
UK4a	791	66	210	70	90	89	247	90	113	89	80	54	508	64	76	283	47
EBICa	822	51	198	53	135	79	208	81	150	71	207	29	583	71	55	239	42
Total surveys	2217	60	446	63	266	85	704	87	399	82	327	37	1442	65	67	775	44
HIT IIb	819	40	62	42	35	71	135	70	210	47	42	21	391	48	48	428	43
TIUSb	1041	38	0	—	7	100	210	70	9	67	226	16	436	42	43	605	34
TINTb	1118	41	0	—	13	69	135	68	5	60	161	17	310	28	42	808	41
PEGSODb	1510	51	1	0	32	84	571	69	475	63	68	12	904	60	59	606	40
SLINb	409	43	0	—	1	100	14	93	0	—	0	—	15	4	93	394	41
SAPHIRb	919	41	32	56	34	82	0	—	141	50	138	25	327	36	43	592	40
Total RCTs	5816	43	95	46	122	80	1065	70	841	57	635	18	2383	41	50	3433	38

^aSurvey.

^bRandomized controlled trial.

As expected, using shorter time windows led to a greater reduction of recruitment in both the surveys and the RCTs (Table 3). In the surveys, the recruitment reduction decreased gradually with the use of higher maximum ages: 22% (≤55 years); 17% (≤60 years); 12% (≤65 years); and 9% (≤70 years; Table 4). The RCTs showed less reduction: 10% (≤55 years); 5% (≤60 years); 2% (≤65 years); and 0% (≤70 years; Table 4).

Table 3.

Numbers of Excluded Patients by Selection on Time Window

		Original study population		Time window ≤ 4 h		Time window ≤ 6 h		The window ≤ 8 h		Time window ≤ 12 h
	Original enrollment criteria	N	% Unfav of N	No. of patients excluded	% Unfav of patients excluded	No. of patients excluded	% Unfav of patients excluded	No. of patients excluded	% Unfav of patients excluded	No. of patients excluded	% Unfav of patients excluded
TCDBa	≤48 h	604	65	106	64	57	70	38	74	22	77
UK4a	≤72 h	791	66	359	70	252	69	210	70	154	71
EBICa	≤24 h	822	51	372	49	252	52	198	53	152	55
Total surveys		2217	60	837	60	561	62	446	63	328	64
HIT IIb	≤12 h no longer obeying command or ≤24 h after injury	819	40	248	40	121	41	62	42	28	43
TIUSb	≤4 h	1041	38	—	—	—	—	—	—	—	—
TINTb	≤4 h	1118	41	—	—	—	—	—	—	—	—
PEGSODb	≤8 h	1510	51	107	49	16	38	—	—	—	—
SLINb	≤8 h and within 4 h of admission	409	43	0	—	0	—	—	—	—	—
SAPHIRb	≤12 h	919	41	265	43	103	43	32	56	—	—
Total RCTs		5816	43	620	43	240	42	94	47	28	43

^aSurvey.

^bRandomized controlled trial.

Table 4.

Numbers of Excluded Patients by Selection on Age at Injury

		Original study population		Age ≤ 55		Age ≤ 60		Age ≤ 65		Age ≤ 70
	Original enrollment criteria	N	% Unfav of N	No. of patients excluded	% Unfav of patients excluded	No. of patients excluded	% Unfav of patients excluded	No. of patients excluded	% Unfav of patients excluded	No. of patients excluded	% Unfav of patients excluded
TCDBa	—	604	65	76	95	59	95	41	95	32	97
UK4a	—	791	66	192	84	143	85	90	89	63	89
EBICa	> 16 yrs	822	51	226	74	172	77	135	79	101	80
Total surveys		2217	60	494	81	374	83	266	85	196	86
HIT IIb	16–70 yrs	819	40	121	64	70	64	35	71	—	—
TIUSb	14–65 yrs	1041	38	63	60	31	61	7	100	—	—
TINTb	14–65 yrs	1118	41	121	60	57	67	13	69	—	—
PEGSODb	15–70 yrs	1510	51	103	80	60	83	32	84	—	—
SLINb	16–65 yrs	409	43	29	72	16	81	1	100	—	—
SAPHIRb	15–70 yrs	919	41	145	61	85	67	34	82	—	—
Total RCTs		5816	43	582	65	319	69	122	80	—	—

^aSurvey.

^bRandomized controlled trial.

Application of all five studied enrollment criteria simultaneously led to a mean recruitment reduction of 65% in the surveys (Table 2, Fig. 1). In the RCTs, the mean reduction of the recruitment was 41%.

Effects of enrollment criteria on outcome distribution

The application of more stringent enrollment criteria had profound effects on the outcome distribution (Table 2). Effects were greatest for age, pupils, motor score, and GCS. In contrast to all other selection criteria, the GCS criterion mainly excluded patients with a favorable outcome. Patients excluded for a high age, bilaterally unresponsive pupils or an absent motor score had more unfavorable outcomes. Only modest effects of different time windows were seen on the outcome distribution (Table 3).

Combining all enrollment criteria had a pronounced effect in the outcome distribution, which was most evident in the surveys, reflecting the greater initial heterogeneity. In the surveys the percentage of unfavorable outcome prior to applying the enrollment criteria was 60%, and decreased to 44% after applying the enrollment criteria, approximating the outcome distribution observed in the RCTs.

Effects of the application of all enrollment criteria on the recruitment and ouctome distribution are illustrated in Figure 1. For example, the first gray bar of Figure 1 shows that application of all investigated enrollment criteria on the population of the Traumatic Coma Data Bank (TCDB) survey, resulted in selection of 253 patients, from whom 43% had an unfavorable outcome. The first black bar depicts 351 excluded patients from the TCDB population with an unfavorable proportion of 81%.

Discussion

Our evaluations show substantial effects of enrollment criteria both on potential recruitment and on outcome. The five studied enrollment criteria led to a joint reduction of recruitment of 65% in the surveys and 41% in the RCTs, and substantially narrowed the differences in outcome between surveys and RCTs that were initially present. Clearly, effects in individual studies varied with the stringency of original criteria.

As a limitation of our study, it must be noticed that we only focused on the influences of five enrollment criteria applied separately and simultaneously. We did not investigate any interactions between the different enrollment criteria. Structural abnormalities as visualized by computed tomography (CT) examination are currently commonly used as selection criterion for enrollment in RCTs, but we did not include these in our analyses, as the available information was not detailed enough in all IMPACT studies.

In TBI, Phase III RCTs traditionally aim to target the population that is most likely to benefit from the therapy under investigation, both from a mechanistic and prognostic perspective. Mechanistic targeting is based upon our—often very poor—potential for identifying patients in whom specific pathophysiologic mechanisms are active. In prognostic targeting, enrollment criteria aim to obtain a homogeneous population with respect to expected outcome, excluding those with either a very poor or very good prognosis. For this reason, most trials have excluded patients over the age of 65, and patients with an absent motor response or unreactive pupils. Indeed, our results show a very high rate of unfavorable outcome in patients excluded for these reasons, indicating that this approach is appropriate.

Nevertheless, interpretation should be with caution: TBI is being seen more frequently in elderly patients and, with the ageing population, the average age of TBI patients is increasing (Maas et al., 2007). Elderly patients with TBI therefore represent an important subpopulation. The poor outcome in elderly patients observed in our series is primarily caused by severe TBI (GCS ≤ 8), and no inference may be made as to patients with moderate TBI.

With the current frequent use of early sedation, intubation and neuromuscular blockade, accurate assessment of the motor scale may be confounded, leading to erroneously recording the motor score as being absent. Likewise, small unreactive pupils may be caused by effects of deep sedation. Consequently, the use of these criteria in isolation may be hazardous. Taken in combination, however, reliability will be increased and for this reason we prefer to classify by prognosis. A limited number of admission characteristics can be used to establish the baseline prognostic risk in individual patients (Hukkelhoven et al., 2005, 2006).

Machado et al. (1999) have shown that targeting a trial towards patients with an intermediate prognosis may reduce the required sample size by 30%. These conclusions were reached from analyses performed on the European Brain Injury Consortium (EBIC) survey. Our results, including also analysis of the EBIC survey, however, show that the reduction in recruitment due to more stringent enrollment criteria may be even higher, thus obviating the potential benefit. This touches on the debate whether focused, targeted Phase III trials, or large mega trials (with inherent heterogeneity) should be preferred in the field of TBI. To date, only one mega-trial has been reported in TBI (Edwards et al., 2005).

We found that time window is a crucial factor in determining recruitment, but that little effect exists on the outcome distribution. Recruitment reductions by time window selection varied from 38% (≤4 h) to 15% (≤12 h) in the surveys. In the RCTs, there was a variation of 11% (≤4 h) to <1% (≤12 h) in reduction of recruitment.

Determination of the appropriate time window for a study should preferably be based on knowledge of the pathophysiologic mechanism targeted, and the time at which this mechanism may be active. Extrapolating results from experimental studies to the clinical situation may, however, be difficult. In practice, time windows are frequently determined based on “best guess” and the clinical perception as to the time frame within which a reasonable number of subjects may be recruited. Our studies show a substantial decline in recruitment when time windows are shortened from 6 to 4 hours, but less so on comparing a 6 hour time window to 8 hours. However, these effects of time window on recruitment were based on the available data in the IMPACT database about the time of injury and the time of admission to study hospital. In practice, the actual time windows to study drug administration are longer, due to organizational and logistical reasons such as informed consent procedures (Kompanje et al., 2005). Therefore, our observations are likely to be an underestimation of the actual decrease in recruitment by time window selection.

We also probably underestimate the actual recruitment reduction by application of selection criteria because we investigated the effect of selection criteria on pre-selected study populations. In the SAPHIR trial, for example, we have learned from the screening logs that only 19% of the number of patients initially screened was enrolled in the trial; 81% was excluded because of not meeting enrollment criteria, inability to obtain informed consent, or logistic reasons (e.g., unavailability of study personnel or medication) (Slieker et al., 2008).

The profound effect of enrollment criteria on outcome is perhaps not surprising, but has hitherto been insufficiently recognized. Enrollment criteria may be deliberately applied (as in RCTs), but may also inadvertently result from selective hospital admission policies in different institutions, or be caused by effects of local trauma organization, outside the control of investigators. A detailed understanding of such factors is, in our opinion, a prerequisite for comparing outcome results between series, originating from different settings or times.

It is remarkable that the application of common enrollment criteria to surveys resulted in an approximation of outcome distribution observed in RCTs.

Conclusion

Strict selection with common enrollment criteria leads to substantial recruitment reductions in TBI studies. The outcome distribution strongly depends on the enrollment criteria.

Acknowledgments

Grant support was provided by NIH NS-42691.

Author Disclosure Statement

No competing financial interests exist.