Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals

Abstract

Objective To examine the prevalence of a risk of bias associated with the design and conduct of cluster randomised controlled trials among a sample of recently published studies.

Design Retrospective review of cluster randomised trials published in the BMJ, Lancet, and New England Journal of Medicine from January 1997 to October 2002.

Main outcome measures Prevalence of secure randomisation of clusters, identification of participants before randomisation (to avoid foreknowledge of allocation), differential recruitment between treatment arms, differential application of inclusion and exclusion criteria, and differential attrition.

Results Of the 36 trials identified, 24 were published in the BMJ,11 in the Lancet, and a single trial in the New England Journal of Medicine. At the cluster level, 15 (42%) trials provided evidence for secure allocation and 25 (69%) used stratified allocation. Few trials showed evidence of imbalance at the cluster level. However, some evidence of susceptibility to risk of bias at the individual level existed in 14 (39%) studies.

Conclusions Some recently published cluster randomised trials may not have taken adequate precautions to guard against threats to the internal validity of their design.

Introduction

In most clinical trials participants are randomised as individuals to different treatments. Sometimes individual allocation is not possible or desirable, and groups of individuals are randomised instead: this is known as cluster or group randomisation. Many reasons for using cluster allocation exist. For example, evaluation of clinical guidelines or medical education on patient outcomes almost always requires that healthcare professionals are the “unit” of allocation.

Although randomised trials are the most robust evaluative method, poorly conducted studies are susceptible to different forms of selection bias that can make their results unsound. Methodological reviews of individually randomised trials have shown that rigorously conducted trials produce different effect estimates from poorly conducted studies.¹,² Less attention has been paid, however, to cluster trials. Cluster trials are generally more difficult to design and execute than individually randomised studies, and some design features of a cluster trial may make it especially vulnerable to a range of threats that can introduce selection bias.

In cluster trials potential bias in the execution of the trial can occur at two levels, the first of which is the cluster level. Randomisation of clusters needs to be undertaken carefully and preferably independently. Otherwise, biased allocation may occur (certain clusters being allocated to a particular arm on the basis of reasons that might affect outcome). It is theoretically possible for allocation of clusters to be subverted, as has happened in individually randomised trials.³ Similarly, once clusters have been allocated it is important, as with individually randomised trials, to try to retain the cluster in its allocated group and avoid the cluster dropping out, to avoid the risk of attrition bias.

The second level at which bias can occur in cluster trials is after the clusters have been allocated and when individual participants are recruited into the study. Sometimes identification and recruitment of participants and assessment of outcome in a cluster trial are relatively straightforward with little scope for bias. For example, in an evaluation of the effect of offering routine influenza vaccination to healthcare workers on patient mortality, hospitals were randomised to offer routine vaccination to staff or not.^w1 Any differences between the groups were then observed by using mortality data. Two important methodological aspects to this trial, and other similar cluster trials, limit the risk of bias. These are complete identification and inclusion of participants, partly owing to the fact that consent was not needed for either treatment or collection of data. Because all the participants were identified and included at the point of randomisation, except for chance imbalances the two groups should be similar at baseline (assuming that the allocation procedure was fair), which avoids the threat of selection bias.

In some cluster trials identification and inclusion of participants and assessment of outcome are less straightforward. Often participants have to be recruited prospectively after randomisation. For example, in a trial of the effectiveness of a training package for general practitioners, patients had to be identified prospectively after the general practitioner had been randomised.^w2 The prospective inclusion of participants can potentially lead to selection bias through the recruitment of different types of participant by the researcher or clinician. If the person prospectively recruiting participants has “foreknowledge” of the allocation group then, as shown in individually randomised trials, bias can result.³ In addition to this source of selection bias, another can be introduced by the participant if consent is needed after randomisation.

Selection bias can be introduced if consent is withheld for either treatment or data collection. This is a well known disadvantage of acquiring consent after randomisation in individually randomised trials (known as Zelen's method⁴), because some refusal of treatment or data collection will usually occur.⁵ This is less of a problem in non-Zelen designs, as participants are told in advance about the treatment options and if they decline to be exposed to one of the options they are not randomised (although some may decline in the period between allocation and receipt of treatment).

Several ways of avoiding the biases outlined above exist. One is to try to identify trial participants before randomisation and obtain consent for treatment, data collection, or both before allocation. Use of prior identification and prior consent avoids potential biases occurring through foreknowledge of the allocation schedule, by the researcher and patient. If this is not possible, identification and recruitment of participants should ideally be undertaken by someone blinded to the group allocation.

Another problem that can lead to bias, in both individual and cluster randomised trials, is the differential application of inclusion and exclusion criteria. Differential exclusion between groups in an individually randomised trial of breast cancer screening, identified in a systematic review, has led to questions about its rigour.⁶ Again this problem can be reduced if the person applying the criteria is blinded to the group allocation.

In this paper we review some recently published cluster trials to determine the extent of their risk of bias. We also describe the steps that some authors took to reduce these risks.

Methods

Searching and data extraction

We hand searched the BMJ, Lancet, and New England Journal of Medicine for all cluster randomised trials published from January 1997 to October 2002. We based our choice of journals on anecdotal experience that the BMJ regularly publishes cluster trials, as does the Lancet, and a wish to include a non-British general medical journal. We limited our search to five years merely so that we had a sample of fairly recent trials. We did not have a predetermined sample size.

Definition of outcomes

Selection bias can be introduced into a trial in several different ways. In this paper we sought evidence for the risk of bias from several sources.

Secure cluster allocation—This is where evidence exists that cluster randomisation was securely undertaken.

Cluster attrition—This occurs when clusters are lost to follow up after randomisation.

Cluster imbalance—This is where evidence exists of imbalance in important variables at the cluster level.

Differential individual recruitment or consent—This is when different proportions of participants are recruited to the different arms of the trial. If recruitment rates differ between groups this may lead to the risk of bias.

Differential individual exclusion or inclusion—This can occur when eligibility criteria are applied differentially after randomisation, which can introduce bias.

Two of us (SP, JW) hand searched the journals and independently extracted data. The three authors met to discuss all the papers and any disagreements. If we observed differences in proportions between the randomised groups in recruitment, consent, and exclusion or inclusion rates we used χ² to test for significance.

Results

We identified 42 potentially eligible trials. We excluded six studies: one was a 14 year follow up of an earlier trial,⁷ another measured the intervention and outcome on only one level,⁸ another had a switchback design,⁹ two guideline studies did not provide any data on individual participants,¹⁰,¹¹ and the sixth trial had a mixture of cluster and individual allocation.¹² Of the 36 trials included,^w1-w36 24 were published in the BMJ, 11 in the Lancet, and one in the New England Journal of Medicine. In table 1 we describe the basic characteristics of the trials. In table 2 we examine whether the trials identified participants before random allocation and any evidence of bias occurring in the trials.

Table 1.

Characteristics of included cluster trials

Study	No of clusters	No of participants	Description	Clustering accounted for in sample size estimation?
Aveyard 1999w3	52	9 301	Expert system for smoking prevention and cessation in schools	Yes
Bennewith 2002w4	98	2 141	Prevention of repeat episodes of deliberate self harm	Yes
Carman 2000w1	20	1 437	Influenza vaccination of healthcare workers	Yes
Chapman 2000w5	8	346	Educational intervention to prevent dog bites	No
Field 2001w6	9	494	Two methods of data collection	Yes
Flottorp 2002w7	142	12 369	To improve general practice management of sore throat and urinary tract infections	Yes
Gavgani 2002w8	18	4 498	Insecticide impregnated dog collars on incidence of zoonotic visceral leishmaniasis	No
Graham 2002w9	24	3 794	Teenagers' knowledge of emergency contraception	Yes
Haider 2000w10	40	726	Community based peer counsellors on breast feeding	Not clear
Jolly 1999w11	67	686	Programme to coordinate and support follow up care in general practice	Not clear
Jordhoy 2000w12	6	707	Palliative care intervention	Not clear
Kannus 2000w13	22	1 725	Hip protectors	No
Kendrick 1999w14	36	2 152	Prevent unintentional injuries in children	Yes
Kidane 2000w15	37	70 506	Maternal education for early treatment of paediatric malaria	Yes
King 2002w2	116	410	Behavioural therapy to treat patients with depression	Yes
Kinmonth 1998w16	43	360	Patient centred care for diabetes in general practice	Yes
Kroeger 2002w17	14	2 913	Insecticide impregnated curtains to control transmission of cutaneous leishmaniasis	No
MacArthur 2002w18	37	3 580	Community postnatal care	Yes
McCartney 1997w19	28	182 200	General practitioner feedback to increase aspirin use	No
Moher 2001w20	21	2 142	Secondary prevention of coronary heart disease	Yes
Montgomery 2000w21	27	810	Interventions for management of hypertension	Yes
Morrison 2001w22	221	689	Infertility guidelines for general practitioners	Yes
Morrow 1999w23	39	130	Home based counselling to promote breast feeding	Yes
O'Cathain 2002w24	13	10 327	Leaflets to promote informed choice in maternity care	Yes
Olivarius 2001w25	311	1 470	Structured personal care of type 2 diabetes mellitus	No
Premaratne 1999w26	40	48 800	Effectiveness of an asthma resource centre	No
Sagliocca 1999w27	146	404	Hepatitis A vaccine	No
Sahota 2001w28	10	636	School intervention to reduce risk factors for obesity	Yes
Shah 2001w29	6	325	Peer led programme for asthma education in adolescents	No
Smeeth 2001w30	106	42 278	Methods to administer a screening questionnaire	No
Steptoe 1999w31	20	883	Behavioural counselling in general practice	Yes
Thompson 2000w32	59	4 192	Detection and outcome of depression in primary care	Yes
Van Eijk 2001w33	21	46 078	Academic detailing to reduce antidepressant use	No
Wawer 1999w34	10	44 107	Prevention of sexually transmitted disease	No
West 1999w35	270	44 646	Supplementation with vitamin A or β carotene on mortality related to pregnancy	No
Wight 2000w36	25	8 430	Teacher delivered sex education	Yes

Table 2.

Potential sources of bias

Study	Did cluster allocation seem secure?	Cluster allocation stratified?	Evidence of Cluster imbalance?	How many Clusters lost after randomisation?	Patients identified before randomisation?	Could selection have been biased?	Evidence of risk of bias?
Aveyard 1999w3	Yes	Yes	No	1	Yes	No	No
Bennewith 2002w4	Yes	Yes	No	1	No	No	No
Carman 2000w1	Yes	Yes	Yes	0	Yes	No	No/Yes*
Chapman 2000w5	Unclear	Unclear	Yes	0	No	No	No
Field 2001w6	Yes	Yes	No	0	No	Yes	No
Flottorp 2002w7	Yes	No	No	22	Yes	No	No
Gavgani 2002w8	Unclear	Yes	No	0	Yes	No	Attrition
Graham 2002w9	Yes	Yes	Unclear	0	Yes	Yes	Consent
Haider 2000w10	Unclear	No	Unclear	0	No	Yes	No
Jolly 1999w11	Yes	Yes	Unclear	0	No	Yes	Recruitment
Jordhoy 2000w12	Unclear	Yes	Unclear	0	No	Yes	No
Kannus 2000w13	Unclear	No	Unclear	0	Yes	Yes	Consent
Kendrick 1999w14	Yes	Yes	Unclear	0	Unclear	Unclear	No
Kidane 2000w15	Unclear	Yes	Unclear	0	Yes	No	No
King 2002w2	Unclear	No	No	32	No	No	No
Kinmonth 1998w16	Yes	Yes	No	2	No	Yes	Recruitment
Kroeger 2002w17	Yes	Yes	No	1	Yes	No	No
MacArthur 2002w18	Yes	Yes	Unclear	1	No	Yes	Attrition
McCartney 1997w19	Unclear	Unclear	No	0	Yes	No	No
Moher 2001w20	Yes	Yes	No	0	Yes	Yes	Exclusion
Montgomery 2000w21	Yes	Yes	Unclear	0	No	No	No
Morrison 2001w22	Unclear	Yes	No	7	No	Yes	No
Morrow 1999w23	Yes	Yes	Unclear	8	No	Yes	No
O'Cathain 2002w24	No	Yes	Unclear	0	No	Yes	No
Olivarius 2001w25	Unclear	Yes	No	10	No	Yes	Exclusion
Premaratne 1999w26	Unclear	Yes	No	0	No	No	No
Sagliocca 1999w27	Unclear	No	Unclear	0	No	No	Attrition
Sahota 2001w28	No	Yes	Unclear	0	Yes	No	No
Shah 2001w29	Unclear	No	Yes	0	No	No	Inclusion
Smeeth 2001w30	Unclear	Unclear	Unclear	0	Yes	No	No
Steptoe 1999w31	Unclear	Yes	No	0	No	Yes	Recruitment
Thompson 2000w32	Yes	Yes	Unclear	4	No	Yes	No
Van Eijk 2001w33	Unclear	Unclear	No	0	Yes	No	No
Wawer 1999w34	Unclear	Yes	No	0	Yes	No	Consent
West 1999w35	Unclear	Yes	No	0	No	Unclear	No
Wight 2002w36	Unclear	Unclear	No	0	Unclear	No	Attrition

^*Not for main outcome, possibly for secondary outcome.

Secure cluster allocation—Fifteen trials seemed to use a secure method of allocating clusters; the remainder did not clearly describe who undertook the allocation (table 2) or how this was done. Most trials used some form of stratified random allocation to reduce the possibility of “chance bias.”

Cluster attrition—In 10 trials a loss of clusters occurred between randomisation and follow up. Most of the trials lost only a small proportion of their clusters, but one study lost more than half (56%) of all the randomised clusters.^w2

Differential consent or recruitment—We found some evidence for differential consent or recruitment in seven of the 23 trials that had not undertaken prior identification of participants (table 2). Three trials recruited more participants from one group than the other,^{w11 w16 w31} and the other four studies differentially obtained consent from more participants in one arm than the other.^{w9 w13 w29 w34} One trial, although it seemed to identify all the participants before allocation for the main mortality outcome, seemed to have introduced the risk of selection bias into the measurement of its secondary outcome.^w1

Differential application of inclusion or exclusion criteria—We found two trials that seemed to have applied inclusion or exclusion criteria differentially between groups after randomisation.^{w20 w25} Moher et al, in a study promoting methods of secondary prevention of coronary disease, excluded significantly more participants owing to misdiagnosis in the intervention groups than in the control group.^w20 Similarly, Olivarius et al excluded twice as many participants because of illness in the intervention group than in the control arm.^w25

Differential attrition—Evidence of differential attrition between the randomised groups existed in four trials.^{w8 w18 w27 w36}

Table 3 summarises the potential sources of bias risk in 14 trials in which we observed differences between the groups that indicate a risk of selection bias. Authors of six studies alerted the reader to the potential risk of bias in their study.

Table 3.

Evidence of risk of bias

Study	Potential source of bias	Acknowledgment of risk by authors and steps taken
Carmen 2000w1	48% v 33% of patients having influenza vaccination, P<0.01; 69% (258/375) v 78% (269/344) accepted virological screening for secondary outcome assessment, P=0.004	Yes, for cluster imbalance, used adjusted odds ratios
Gavgani 2002w8	7.6% (143/1870) v 11.4% (229/2006) attrition in control and intervention groups, P<0.001	No
Graham 2002w9	12.2% (216/1768) of control group refused consent v 17% (344/2026) of intervention group, P<0.001	No
Jolly 1999w11	Control group recruited 15.5% more than intervention group; practice population not given, so impossible to see if significantly different	Noted recruitment “imbalance” in discussion
Kannus 2000w13	31.4% (204/650) of intervention group refused consent v 8.7% (94/1075) of control group, P<0.001	Acknowledged potential for selection bias in discussion
Kinmonth 1998w16	0.06% (142/225 015) v 0.047% (108/230 560) of practice populations were recruited for intervention and control groups, P=0.02	No
MacArthur 2002w18	4.2% (46/1087) v 2.6% (25/977) of intervention and control groups withdrew or moved away, P=0.04	Commented on loss to follow up in discussion
Moher 2001w20	0.32% (2/623) v 2.6% (20/772) and 1.3% (10/747) misdiagnoses for control and two intervention groups, P=0.002	No
Olivarius 2001w25	8.7% (67/774) v 4.0% (28/696) excluded owing to illness in intervention group and control group, P<0.01	Noted in results more post-randomisation exclusions
Sagliocca 1999w27	3.9% (7/178) v 0% (0/173) lost to follow up in control and intervention groups, P=0.02	No
Shah 2001w29	22.3% (148/662) v 17.3% (124/717) included in control and intervention groups, P=0.02	No
Steptoe 1999w31	Control practices recruited 567 v 316 participants given similar sized populations (P value not calculable)	Differential recruitment rate mentioned in discussion, but not as a potential source of bias
Wawer 1999w34	17.5% (4002/22 915) v 14.7% (3125/21 192) refused consent or treatment in intervention and control groups, P<0.001	No
Wight 2002w36	30.6% (1070/3493) v 27.5% (1069/3892) attrition in numbers reporting intercourse in intervention and control groups, P=0.003	No

Discussion

Cluster trials can be difficult to do; nevertheless, they are needed to evaluate some interventions. Although a large literature exists about sources of potential bias that can occur in individually randomised trials, less evidence is available about the special problems encountered in cluster trials.

Evidence of bias risk at cluster level

Some authors did not clearly describe the allocation process of the clusters, which is important as this can be subverted; other trialists were clear in stating that an independent person undertook the allocation. In most trials some form of stratification was used to reduce the element of chance bias, although this was not always successful. Some trials lost complete clusters after randomisation. However, with the exception of one trial,^w2 the proportion of clusters lost was relatively low and therefore would be unlikely to introduce bias.

Evidence of bias risk at individual level

One of the major risks for introduction of bias is when prospective recruitment is needed. This difficulty can be overcome and the risk of bias reduced, as two examples serve to illustrate. Bennewith et al reduced the possibility of recruitment bias by blinding the clinician identifying participants until after the patient was assessed as being eligible or not.^w4 Similarly, King et al reduced the same threat by asking a trained receptionist to recruit patients.^w2 Because that trial evaluated a training package to help general practitioners to manage depressed patients, the training would probably have reduced the diagnostic threshold of the general practitioners. Thus, had the doctors recruited participants themselves, this would have increased the risk that they could have recruited either more or less seriously depressed participants than the control doctors. Use of receptionists reduced this risk.

As well as differences between groups in recruitment and consent, we found that differential post-randomisation exclusion or inclusion was a problem in some studies. Inclusion of the “wrong” participant is likely to be a problem in some cluster trials. Two ways exist to deal with wrong inclusions and avoid bias. Firstly, all participants could be retained within the trial after allocation whether or not they fitted the inclusion criteria and even if they could not or did not receive the allocated treatment (that is, intention to treat analysis).¹³ This could lead, however, to some dilution of treatment effect. As an alternative, decisions on exclusion could be made by a person blind to the group allocation.

A new CONSORT statement?

Elbourne and Campbell have recently argued for amending the CONSORT statement to allow for the special methodological circumstances of cluster trials.¹⁴ We would echo this call. We found it very difficult in several of the trials to ascertain whether a risk of bias was likely or not. We would wish the following additions to be made. Firstly, a clear statement as to whether the population was identified before or after the allocation decision had been made. Secondly, was the person who recruited the participants blind to group allocation? Thirdly, what was the size of the population within the clusters? For example, Steptoe et al^w31 did not state the size of the general practice populations in their trial arms and Kinmonth et al^w15 gave only means. For the first of these studies we could only assume that the recruitment was significantly different, and for the second study we had to make an estimate. This missing information also meant that for some studies we could not be completely sure if recruitment bias had taken place. For example, in Kendrick et al no suggestion of any recruitment bias was apparent; however, we could not be absolutely sure as the authors did not present the practice population sizes.^w14

Conclusion

Cluster trials are vulnerable to the risk of bias. Careful planning and execution of such trials can avoid these biases.

What is already known on this topic

Reviews of individually randomised trials show that results can differ according to quality of methods

Foreknowledge of allocation and failure to use intention to treat analysis can lead to bias

What this study adds

Cluster randomised trials are susceptible to forms of selection bias

Careful planning and execution of such trials can avoid these biases