Randomization methods in emergency setting trials: a descriptive review

Abstract

Background

Quasi‐randomization might expedite recruitment into trials in emergency care settings but may also introduce selection bias.

Methods

We searched the Cochrane Library and other databases for systematic reviews of interventions in emergency medicine or urgent care settings. We assessed selection bias (baseline imbalances) in prognostic indicators between treatment groups in trials using true randomization versus trials using quasi‐randomization.

Results

Seven reviews contained 16 trials that used true randomization and 11 that used quasi‐randomization. Baseline group imbalance was identified in four trials using true randomization (25%) and in two quasi‐randomized trials (18%). Of the four truly randomized trials with imbalance, three concealed treatment allocation adequately. Clinical heterogeneity and poor reporting limited the assessment of trial recruitment outcomes.

Conclusions

We did not find strong or consistent evidence that quasi‐randomization is associated with selection bias more often than true randomization. High risk of bias judgements for quasi‐randomized emergency studies should therefore not be assumed in systematic reviews. Clinical heterogeneity across trials within reviews, coupled with limited availability of relevant trial accrual data, meant it was not possible to adequately explore the possibility that true randomization might result in slower trial recruitment rates, or the recruitment of less representative populations. © 2015 The Authors. Research Synthesis Methods published by John Wiley & Sons, Ltd.

1. Background

Recruitment to emergency medicine clinical trials may be complicated by the short time frames available for obtaining consent and for identifying, enrolling, randomizing and treating eligible participants (Cofield et al., 2010). Possible drawbacks of using methodologically sound randomization processes in emergency settings (such as telephone or web‐based systems, or systems using sequentially numbered sealed, opaque envelopes) might be a delay in treatment, and complexity of trial administration (Zhao et al., 2010). Recruitment difficulties can arise where treatment delays are clinically unacceptable. Trial investigators may therefore sometimes need to consider adopting more pragmatic approaches to recruitment that involve balancing methodological rigour with expediency in enrolment and randomization.

One approach that has been used with the aim of reducing delay in enrolment is a ‘quasi‐random’ allocation of treatment. This involves the use of a pre‐defined participant or setting characteristic, such as date of birth, to determine which treatment a participant receives. The major concern when using quasi‐randomization is that trial investigators have prior knowledge of the treatment that an individual is due to receive. This lack of allocation concealment increases the risk of selection bias during the trial recruitment phase. Selection bias would be expected to adversely impact trial validity if it introduced an imbalance between trial treatment groups in an important prognostic indicator. Important baseline imbalances can also arise by chance, especially when sample sizes are small. Regardless of cause, such imbalances can make it difficult to ascribe any outcome effects to trial interventions alone.

Systematic reviews that appraise and synthesize evidence from clinical trials deal with quasi‐randomization in different ways. Review authors may decide a priori to exclude quasi‐randomized trials, or they may opt to include them with pre‐specified plans for subgroup analyses, usually based on adequacy of allocation concealment. When quasi‐randomized trials are included in systematic reviews, their findings may be undervalued, because they are almost always automatically judged to be at high risk of selection bias, even though evidence of actual selection bias is either not sought or may not be apparent (Corbett et al., 2014).

This study had two main objectives, both relating to clinical trials performed in an emergency or urgent care setting. Firstly, we aimed to obtain an estimate of the prevalence of important baseline imbalances that may have been a consequence of selection bias, in trials that used quasi‐randomization versus those that used true randomization. Secondly, we wished to examine whether there is any evidence to suggest that any possible benefits of using true randomization might be offset in other areas of trial recruitment, such as slower recruitment rates, or the recruitment of less representative populations.

2. Methods

In December 2013, we used two approaches to identify relevant systematic reviews. First, we searched the reviews included in an overview of reviews project (ongoing at the Centre for Reviews and Dissemination (CRD)), which is evaluating reviews of delivery room interventions. Second, we identified systematic reviews classified as ‘emergency medicine’ in the Cochrane Library. Eligible reviews had to include at least one clinical trial that clearly reported using true randomization and at least one quasi‐randomized trial that clearly reported how interventions were allocated to patients. We defined quasi‐randomization as allocation methods that use easily accessible information such as patient hospital number, date of birth and date of admission, or by using alternate allocation. For the purposes of this study, we defined true randomization as sequence generation using a method that has a random component, regardless of the level of allocation concealment; we did this because in systematic reviews, eligibility of trial design is often based on whether random sequence generation methods were used. We then distinguished between the true randomized trials that used adequate allocation concealment methods, from those using inadequate allocation concealment methods. Where closely related reviews were identified, across which there was overlap of trials, we selected the review with the largest number of quasi‐randomized trials.

Eligible trials had to include participants with acute injury or illness, requiring immediate intervention as quickly as was clinically practicable. In the event of any uncertainty regarding how quickly the intervention was given, a decision on eligibility for our study was made based on the type of consent obtained; studies using time‐saving strategies such as waived, deferred or implied consent were included, and studies requiring (pre‐randomization) patient consent were excluded. Trials of prophylactic interventions or surgery/post‐surgery interventions were excluded. Cross‐over trials, cluster‐randomized trials and trials that were reported only as conference abstracts were also excluded. One particular investigator has performed many trials in emergency settings, most of which have since been retracted; all studies by this author were deemed ineligible for this study (Oransky, 2013).

We included all the quasi‐randomized trials in eligible systematic reviews. In reviews where the total number of studies was ≤10, we included all eligible trials providing the ratio of randomized to quasi‐randomized studies was not greater than 2:1. Where higher than a 2:1 ratio existed and the total number of studies in the review was ≤10, we achieved a 2:1 ratio by selecting the most recently published truly randomized trials.

For reviews with more than 10 studies in total, we selected an equal number of true and quasi‐randomized trials, again by prioritizing those published most recently. For example, for a review with five quasi‐randomized controlled trials (RCTs) and 25 trials with true randomization, we would select all five quasi‐RCTs. We would then select the five most recently published eligible trials that used true randomization.

Evidence of possible selection bias was sought by assessing baseline imbalances in important prognostic indicators across treatment groups within individual trials (Corbett et al., 2014). Two authors (W. M. and S. O.) provided advice on the important prognostic indicators for neonatal trials (including references for relevant studies); for the remaining trials, information from published studies was identified. These approaches were also used to define what constituted an important baseline difference between trial treatment groups. When necessary, we made arbitrary but conservative judgements on cut‐offs.

For each trial, we extracted the following data: methods of sequence generation and allocation concealment (including any reasons given for using quasi‐random methods), trial eligibility criteria and corresponding details of the populations enrolled, the target number of patients to recruit and the number actually recruited, the number of eligible patients who were not enrolled (with reasons), the number of ineligible patients enrolled, data to calculate an estimate of the rate of recruitment (per centre), type of consent obtained and the country/countries where the trials were performed. We made risk of bias judgements on methods of allocation concealment using information from both the published trial reports and the systematic reviews. One author extracted data that were independently checked by a second author.

3. Results

We identified seven eligible systematic reviews, including 27 eligible clinical trials: 11 used quasi‐random methods, and 16 used true randomization. Of the seven included reviews, three were of fluid resuscitation for critically ill patients (Kwan et al., 2003, Bunn et al., 2004, Perel and Roberts, 2011), two were of neonatal interventions (one investigating respiratory oxygen levels (Saugstad et al., 2008) and one the effect of intubation (Halliday and Sweet, 2001)), one was of intubation for adults or children (Lecky et al., 2008) and one was of hypothermia following cardiopulmonary resuscitation (Arrich et al., 2009). Six of the 11 quasi‐randomized trials did not report a rationale for using quasi‐randomization (Caldwell and Bowser, 1979, Linder et al., 1988, Ramji et al., 1993, Evans et al., 1996, Gausche et al., 2000, Rabitsch et al., 2003). Three trials stated that quasi‐randomization was used to avoid detrimental delay in care (Bickell et al., 1994, Ramji et al., 2003, Bajaj et al., 2005); a fourth trial also stated this reason adding a desire to avoid a reduction in the recruitment of the most depressed infants (possibly leading to a non‐representative sample; Saugstad et al., 1998). One trial viewed quasi‐randomization as being the only feasible method for immediate use by large numbers of ambulance officers and emergency department physicians (Bernard et al., 2002).

Details of individual trials (with a full trial reference list) are reported in Supporting information S1. The important prognostic indicators identified for each systematic review are listed in Supporting information S2, which also details the associated magnitudes of group difference used to decide whether a trial had an important baseline imbalance.

The methods and results of randomization in the included clinical trials are listed in Table 1; details on the methods used for sequence generation and allocation concealment in each trial are available in Supporting information S1. Important imbalance between groups within a trial was identified in two of the 11 quasi‐randomized trials (18%) and four of the 16 trials using true randomization (25%); these trials are presented in bold in Table 1. In the four trials that used true randomization that had imbalance, three described appropriate methods to conceal treatment allocation, and one used an inappropriate method.

Table 1.

Trial randomization methods and baseline similarity of groups.

Trial	Sequence generation methoda	Risk of bias from allocation concealment methodsa	Important imbalance identified?b	Number randomized (number of groups)
Review: Resuscitation of newborns with room air or pure oxygen (Saugstad et al., 2008)
Bajaj et al., 2005	Quasi‐random	High	No	204 (2)
Ramji et al., 1993	Quasi‐random	High	No	84 (2)
Ramji et al., 2003	Quasi‐random	High	No	433 (2)
Saugstad et al., 1998	Quasi‐random	High	No	703 (2)
Toma, 2006a	Truly random	Unclear	No	54 (2)
Toma, 2006b	Truly random	Unclear	No	44 (2)
Toma, 2007	Truly random	Unclear	No	56 (2)
Vento, 2001	Truly random	Unclear	No	527 (2)
Vento, 2003	Truly random	Unclear	No	151 (2)
Vento, 2005	Truly random	Unclear	No	53 (2)
Review: Endotracheal intubation in meconium‐stained newborns (Halliday and Sweet, 2001)
Linder et al., 1988	Quasi‐random	High	No	572 (2)
Wiswell et al., 2000	Truly random	High	Unclear	2094 (2)
Review: Colloids versus crystalloids for fluid resuscitation in critically ill patients (Perel and Roberts, 2011)
Evans et al., 1996	Quasi‐random	High	No	25 (2)
Bulger et al., 2011	Truly random	Low	No	895 (3)
James et al., 2011	Truly random	Low	Yes	115 (4)
Review: Timing and volume of fluid administration for patients with bleeding (Kwan et al., 2003)
Bickell et al., 1994	Quasi‐random	High	No	598 (2)
Dutton, 2002	Truly random	Low	No	110 (2)
Turner et al., 2000	Truly random	High	No	1309c (2)
Review: Hypertonic versus near isotonic crystalloids for fluid resuscitation in critically ill patients (Bunn et al., 2004)
Caldwell and Bowser, 1979	Quasi‐random	High	Unclear	37 (2)
Cooper et al., 2004	Truly random	Low	No	229 (2)
Vassar et al., 1993	Truly random	Low	Yes	233 (4)d
Review: Hypothermia for neuroprotection after cardiopulmonary resuscitation (Arrich et al., 2009)
Bernard et al., 2002	Quasi‐random	High	Yes	84 (2)
HACA, 2002	Truly random	Low	No	275 (2)
Laurent, 2005	Truly random	Low	Yes	61 (3)
Review: Intubation for acutely ill and injured patients (Lecky et al., 2008)
Gausche et al., 2000	Quasi‐random	High	No	830 (2)
Rabitsch et al., 2003	Quasi‐random	High	Yes	172 (2)
Goldenberg, 1986	Truly random	High	Yes	175 (2)

^aSee Supporting information S1 for details.

^bSee Supporting information S1 and S2 for details.

^cParamedics were randomized, with 1309 patients subsequently recruited.

^dBaseline data only presented for 194 patients (as 39 were ineligible).

An assessment of how representative the trial populations were could only be made for the review of resuscitation approaches in newborns (Saugstad et al., 2008). Eligibility criteria with respect to weight and age varied between trials: all the quasi‐randomized studies had no age criteria and very broad weight criteria (all using ≥1000 g); all the truly randomized trials recruited only term infants (except for one trial that recruited infants >34 weeks), with no specific weight criteria.

Table 2 summarizes the trial accrual and recruitment data. In two reviews, quasi‐randomization was associated with faster accrual, with recruitment rates being double (Lecky et al., 2008) and triple (Halliday and Sweet, 2001) those achieved in equivalent trials using true randomization. In two reviews, there was little or no indication of differences in accrual rates (Saugstad et al., 2008, Arrich et al., 2009) although for one of these reviews it was not possible to estimate monthly accrual rates in half the trials (Saugstad et al., 2008). Clinical and methodological heterogeneity across trials precluded any meaningful comparisons in the remaining reviews (Kwan et al., 2003, Bunn et al., 2004, Perel and Roberts, 2011). Data on how many eligible patients were not recruited and on how many ineligible patients were recruited were often not reported.

Table 2.

Trial accrual and recruitment data.

Trial	Sequence generation method	Target sample size	Patients randomized	RA ratea	No. of eligible patients not randomized	No. of ineligible patients randomized
Review: Resuscitation of newborns with room air or pure oxygen (Saugstad et al., 2008)
Bajaj et al., 2005	Quasi‐random	146	204	14	0	0
Ramji et al., 1993	Quasi‐random	72	84	–c	0	0
Ramji et al., 2003	Quasi‐random	300	433	4	0	2
Saugstad et al., 1998	Quasi‐random	648	703	3	107	90
Toma, 2006a	Truly random	NR	54	–c	NR	NR
Toma, 2006b	Truly random	NR	44	15	NR	NR
Toma, 2007	Truly random	NR	56	–c	NR	NR
Vento, 2001	Truly random	NR	527	–c	NR	NR
Vento, 2003	Truly random	NR	151	–c	NR	24
Vento, 2005	Truly random	NR	53	1	Unclear, although 3 ‘improperly randomized’	0
Review: Endotracheal intubation in meconium‐stained newborns (Halliday and Sweet, 2001)
Linder et al., 1988	Quasi‐random	NR	572	18	0	0
Wiswell et al., 2000	Truly random	2058	2094	6	NR	Unclear
Review: Colloids versus crystalloids for fluid resuscitation in critically ill patients (Perel and Roberts, 2011)
Evans et al., 1996	Quasi‐random	NR	25	25	NR	0
Bulger et al., 2011	Truly random	3726	895	0.3	NR	23
James et al., 2011	Truly random	140	115	3	0	5
Review: Timing and volume of fluid administration for patients with bleeding (Kwan et al., 2003)
Bickell et al., 1994	Quasi‐random	~600	598	16	0	471
Dutton, 2002	Truly random	NR	110	6	NR	NR
Turner et al., 2000	Truly random	NRb	1309	5	0	0
Review: Hypertonic versus near isotonic crystalloids for fluid resuscitation in critically ill patients (Bunn et al., 2004)
Caldwell and Bowser, 1979	Quasi‐random	NR	37	1	NR	NR
Cooper et al., 2004	Truly random	220	229	0.5	0	0
Vassar et al., 1993	Truly random	600	233	2	0	39
Review: Hypothermia for neuroprotection after cardiopulmonary resuscitation (Arrich et al., 2009)
Bernard et al., 2002	Quasi‐random	62	84	0.6	0	0
HACA, 2002	Truly random	NR	275	0.6	30	0
Laurent, 2005	Truly random	90	61	1	0	0
Review: Intubation for acutely ill and injured patients (Lecky et al., 2008)
Gausche et al., 2000	Quasi‐random	800	830	–c	1	None
Rabitsch et al., 2003	Quasi‐random	NR	172	14	NR	NR
Goldenberg, 1986	Truly random	NR	175	7	0	~10

RA, randomization; NR, not reported.

^aEstimated monthly rate, per centre.

^bNot reported for patients but 420 for paramedics.

^cUnable to calculate an estimate.

4. Discussion

This descriptive review of emergency care setting clinical trials did not find any evidence that quasi‐randomization results in selection bias more often than true randomization; these results suggest that high risk of bias judgements for quasi‐randomized studies should therefore not be assumed in systematic reviews of interventions delivered in emergency or urgent care settings.

Important imbalance between groups within a trial was identified in two of the 11 quasi‐randomized trials (18%) and four of the 16 trials using true randomization (25%). These results suggest that when baseline imbalance does occur, it may be a consequence of chance effects, which become evident (and problematic) because of the small trial populations. Three trials had important baseline imbalances despite using both true randomization and adequate allocation concealment methods. All three had small group sizes – having 50 or fewer participants per arm. Chance imbalances may be more prevalent in small emergency setting trials because of difficulties in implementing methods to reduce the possibility of imbalances. The use of stratified or minimization randomization methods is likely to be impractical in most emergency settings, although feasible in some (Zhao et al., 2010).

Possible reasons for the low incidence of selection bias in emergency setting trials might include the following: lack of (pre‐intervention) time for trial investigators/staff to judge prognosis; investigators being less inclined to allow their biases to influence the care of such acutely ill patients; the possibility of regulatory authority audit (and having to justify inappropriate exclusions); the team nature of intervention delivery, precluding opportunities for bias; and the fact that interventions may be administered by staff with limited involvement in trial design (e.g. paramedics) who might be less likely to have strong enough opinions to result in biased selection. Emergency setting trials are also quite likely to assess mortality or other objectively assessed outcomes; trials with inadequate or unclear allocation concealment show no evidence of bias for all‐cause mortality, and little evidence of bias for objective outcomes (Wood et al., 2008).

Clinical heterogeneity across trials within reviews, coupled with a shortage of quasi‐randomized trials, meant it was only possible to examine one review to evaluate whether population variability differed between the different randomization approaches. Furthermore, accrual and recruitment data were often unavailable. The degree of recruitment of ineligible patients was not well documented in several trials, although in those not reporting any actual data it was nevertheless evident from the methods used that some trials must have randomized many ineligible patients. Practices such as sealed envelopes being assigned to the records of expectant mothers on admission (before eligibility can be known), and the discarding of randomization assignments when infants were not eligible, were evident in neonatal trials (Wiswell et al., 2000, Vento et al., 2003).

Our study has some limitations, the main one being that it was quite small and was exploratory in nature. In terms of assessing biases within trials, we investigated only the impact of randomization methods on selection bias and did not attempt to evaluate other biases that might result from the randomization methods. It was our intention, when planning the study, that we would also try to compare outcome results data across the different methods of randomization. However, it became apparent during piloting that there was too much variation in the outcomes reported to make this a worthwhile exercise. Furthermore, we were aware that other possible biases (e.g. lack of blinding of the treating clinician) will have sometimes differed between the types of randomization method, which may also have had an impact on effect estimates. Our focus was therefore on selection bias and on how this may be assessed in systematic reviews. Although our lists of prognostic indicators and cut‐offs were thorough and quite conservative (i.e. small differences were flagged as being potentially important), they were nevertheless devised pragmatically, with the main aim being to compare the two methods of randomization. Our study relates only to imbalances in known prognostic indicators; the possibility of selection bias resulting in imbalances in unknown prognostic factors remains for quasi‐randomized trials, but not for truly randomized trials (Urbach, 1993, Worrall, 2002).

Our results provide supportive evidence for the idea of systematic reviewers utilizing data on important baseline covariates when judging risk of selection bias in clinical trials, rather than using randomization method details alone; the results also highlight the value of assessing for chance imbalances (Corbett et al., 2014). Selection bias was not evident in eight of the 11 quasi‐randomized trials included in our study; all eight would normally have been judged as being at high risk of bias. The results from our study also help to inform consideration and discussion about why quasi‐randomized trials are excluded from systematic reviews (Herbison, 2012). It is unclear why trials that use true randomization, but inadequate allocation concealment, are frequently deemed to be more suitable for inclusion than quasi‐randomized trials. One further issue arose to help inform future systematic reviews of emergency setting interventions: considering the difficulties that may be encountered when recruiting participants into emergency setting trials, we suggest that an assessment of the external validity and applicability of trial results is essential. Such assessments may be complex, which may partly explain why they are often neglected in systematic reviews (Dekkers et al., 2010, Burchett et al., 2011).

In one of the truly randomized trials in our study, a delay in administering treatment was avoided by opening envelopes before eligibility could be confirmed (Wiswell et al., 2000); another trial saved time by randomizing paramedics, rather than patients (Turner et al., 2000). However, the use of these methods meant that allocation was not properly concealed and eligible patients could potentially have then been wrongly excluded (because eligibility assessments would have been performed with foreknowledge of the allocated treatment). Nevertheless, in some trials, methodologically sound randomization was used without causing delays in treatment. This was evident in the fluid resuscitation reviews; in many of the trials, the randomization sequence was applied (in code) physically to the interventions (the bags of fluid; Mattox et al., 1991, Bulger et al., 2011, James et al., 2011, Cooper et al., 2004, Vassar et al., 1993). This appears to be a time‐saving and resource‐efficient method that would obviate the need for quasi‐random methods (assuming good trial administration, with the supply of code‐labelled bags not running out at any point). However, of the trials in the remaining reviews in our study, such methods were not an option, because the interventions could not be delivered in discrete packs.

Considering the reporting limitations seen in many of the trials in our study, further methodological research might best be focussed only on evaluating baseline imbalance outcomes in populations that are relatively simple to define prognostically, such as preterm neonates or trauma patients. Future studies might also identify how frequently chance imbalances arise in neonatal or trauma trials using methodologically sound randomization methods, regardless of level of emergency status. Assessment of whether minimization or stratified randomization techniques have been used or whether statistically adjusted results (to allow for the effect of confounders) have been calculated would also be informative. In terms of clinical research, an example area where quasi‐randomization might be considered to help simplify and facilitate trial recruitment is the effect of timing on umbilical cord clamping in preterm infants; a Cochrane review has concluded that there were insufficient data for all the review's primary outcomes, despite an evidence base of 15 randomized trials, which were mostly small studies (Rabe et al., 2012). Although our results relate to emergency setting clinical trials, the use of randomized trials has expanded to areas of study beyond clinical medicine; our results may be of interest to any investigators who are studying interventions that are given in time‐limited settings.

5. Conclusion

This descriptive review of emergency care setting clinical trials did not find any evidence that quasi‐randomization results in selection bias more often than true randomization; these results suggest that high risk of bias judgements for quasi‐randomized studies should therefore not be assumed in systematic reviews of interventions delivered in emergency or urgent care settings.

Our results also suggest that the likelihood of chance imbalances affecting trial results may also be an important issue to consider, for both trial investigators and systematic reviewers. Clinical heterogeneity across trials within reviews, coupled with limited availability of relevant trial accrual data, meant it was not possible to adequately explore the possibility that true randomization might result in slower trial recruitment rates, or the recruitment of less representative populations.

Ethical approval

None required.

Conflict of interests

The authors declare that they have no competing interests.

Authors' contributions

M. C. conceived of the study, developed its design and coordination, identified studies, extracted and analysed the data, drafted the manuscript and coordinated the authors' comments. T. M. B. helped to identify relevant studies, extracted data and helped to revise the manuscript. W. M. participated in the design of the study, provided clinical advice, contributed to the interpretation of data and helped to revise the manuscript. S. O. provided clinical advice, contributed to the interpretation of data and helped to revise the manuscript.

Supporting information

Supporting information S1: Individual trial data. For all the trials included in this study this file provides the methods, recruitment and population data. A reference list of all the included studies is also provided.

Supporting information S2: Important prognostic indicators and magnitudes of group difference. This file provides the review‐specific details of important prognostic indicators, along with the associated magnitudes of group differences. The sources and references which we used are also provided.

Corbett, M. S. , Moe‐Byrne, T. , Oddie, S. , and McGuire, W. (2016) Randomization methods in emergency setting trials: a descriptive review. Res. Syn. Meth., 7: 46–54. doi: 10.1002/jrsm.1163.