Abstract
Objectives To compare how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and to estimate how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.
Design Cohort study of 102 sets of trial protocols and corresponding publications.
Setting Protocols of randomised trials approved by the scientific and ethical committees for Copenhagen and Frederiksberg, 1994 and 1995.
Main outcome measures Frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols, and the proportion of protocols where methods were reported to be adequate but descriptions were unclear in the trial publications.
Results 96 of the 102 trials had unclear allocation concealment according to the trial publication. According to the protocols, 15 of these 96 trials had adequate allocation concealment (16%, 95% confidence interval 9% to 24%), 80 had unclear concealment (83%, 74% to 90%), and one had inadequate concealment. When retrospectively defined loose criteria for concealment were applied, 83 of the 102 trial publications had unclear concealment. According to their protocol, 33 of these 83 trials had adequate allocation concealment (40%, 29% to 51%), 49 had unclear concealment (59%, 48% to 70%), and one had inadequate concealment.
Conclusions Most randomised clinical trials have unclear allocation concealment on the basis of the trial publication alone. Most of these trials also have unclear allocation concealment according to their protocol.
Introduction
Selection bias occurs in randomised clinical trials if patients with a better prognosis are preferentially allocated to one of the treatment arms. The results of the trial will then to some degree reflect this difference in prognosis rather than just a difference in the effects of the compared treatments.
The purpose of randomisation is to avoid selection bias, as patients with known and unknown differences in prognosis will tend to be equally distributed between the treatment groups. To ensure true randomisation, however, the random allocation sequence should not only be sequentially and irreversibly administered but should also be concealed to the individuals in charge of enrolment and treatment allocation. Otherwise, knowledge of the upcoming allocation will permit selective assignment of patients by manipulation of either the sequence of treatments to be allocated or the sequence of patients to be enrolled.1
Surveys have shown that 44% to 93% of publications of randomised controlled trials lack a clear description of allocation concealment.2 Empirical studies have shown that publications of trials in which allocation concealment is unclear or inadequate are associated with, on average, a 20-30% exaggeration of the treatment effect (measured as a ratio of odds ratios) compared with trials of the same interventions with adequate concealment.3-9 Generation of a truly random sequence is an interrelated issue for which there is suggestive empirical evidence of an associated inflation of the treatment effect if the trial publication does not document adequate procedures.4-6 Both issues are of major concern because the effect of many treatments is less than these average biases. We compared how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and we estimated how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.
Methods
Our cohort consisted of all published randomised trials (apart from trials in dentistry) whose protocols were approved by the scientific and ethical committees for Copenhagen and Frederiksberg in 1994 and 1995. We identified trial publications by contacting the principal investigators and by searching PubMed, Embase, and the Cochrane central register of controlled trials (final search in May 2003; median publication year 1999, range 1995-2003).10 In total, 102 protocols were published in 122 trial reports. When there was more than one publication of a trial, we examined all publications for information on allocation concealment.
Outcomes and data extraction
Our outcome measures were frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols; the proportion of protocols where methods were reported to be adequate when the trial publications gave unclear descriptions; the type and frequency of methods used for allocation concealment; and the prevalence of other trial characteristics that might undermine concealment (for example, stating the block size in the protocol).
Two authors extracted data from the first half of the published reports and the second half of the protocols while another pair of authors extracted data from the rest. Disagreements were resolved within each pair, thus avoiding assessment of both trial publication and protocol for the same trial.
Assessment of adequacy of allocation concealment
We considered the following methods for allocation concealment as adequate:3-6,11-13 central randomisation; numbered coded vehicles; opaque, sealed, and sequentially numbered envelopes; and other methods containing convincing means of concealment. Inadequate methods concerned open or predictable sequences of allocation (for example, alternation), date of birth, case record number or similar, and open tables of random numbers. We categorised studies as unclear that did not fall into one of these categories or that provided no information.
To ensure consistency and transparency and to capture how strict application compared with loose application of our criteria might influence our results, we operationalised our interpretations of authors' descriptions of allocation concealment (see table on bmj.com). The strict criteria are those recommended for Cochrane reviews,11 except for an elaboration on central randomisation, as specified in the table on bmj.com.12,13 The loose criteria, which we defined retrospectively, comprised the most liberal criteria used in any of the previous empirical studies of bias associated with unclear or inadequate allocation concealment.3-9 For instance, in a study by Schulz et al4 envelopes had to be opaque, sealed, and sequentially numbered to qualify as adequate concealment, whereas in a study by Kjaergard et al6 use of sealed envelopes without further details qualified as adequate (see bmj.com for examples of how the criteria were applied on our sample).
Assessment of adequacy of sequence generation
Adequate methods of sequence generation included computer generated random numbers, tables of random numbers, or drawing lots or envelopes. Inadequate methods could be related to prognosis such as date of birth or year of admission. Unclear methods were methods not falling into one of these two categories or where the methods were not described.
Statistical analysis
We calculated 95% confidence intervals using the exact binomial method in Stata version 8.
Results
Allocation concealment
Using the strict criteria, 96 of the 102 trials (94%, 95% confidence interval 88% to 98%) had unclear allocation concealment according to their publications. According to their protocols, 15 of these 96 trials (16%, 9% to 24%) had adequate allocation concealment and one had inadequate concealment, whereas most (80 of 96; 83%, 74% to 90%) had unclear concealment (table 1).
Table 1.
Adequacy of allocation concealment as described in pairs of protocols and corresponding trial publications in 102 trials according to strict and loose criteria. Values are numbers
|
Description of allocation concealment in protocols
|
|||||
|---|---|---|---|---|---|
| Adequate | Unclear | Inadequate | Total | ||
| Strict criteria | |||||
| Description of allocation concealment in trial publications
|
Adequate
|
4
|
1
|
0
|
5
|
| Unclear
|
15
|
80
|
1
|
96
|
|
| Inadequate
|
0
|
0
|
1
|
1
|
|
| Total
|
19
|
81
|
2
|
102
|
|
| Loose criteria | |||||
| Description of allocation concealment in trial publications
|
Adequate
|
12
|
6
|
0
|
18
|
| Unclear
|
33
|
49
|
1
|
83
|
|
| Inadequate
|
0
|
0
|
1
|
1
|
|
| Total | 45 | 55 | 2 | 102 | |
Using the loose criteria, 83 of the 102 trials had unclear allocation concealment (81%, 72% to 88%). According to the protocols, 33 of these 83 publications (40%, 29% to 51%) had adequate allocation concealment, one had inadequate concealment, and 49 (59%; 48% to 70%) had unclear concealment (see table 1).
According to the strict criteria, 20 of the 102 studies (five publications and 19 protocols; see table 1) described adequate allocation concealment. When the loose criteria were applied, however, 51 studies (18 publications and 45 protocols) described adequate concealment.
Sequence generation
Eighty one of the 102 trial publications gave no information on how the allocation sequence was generated; 16 of these 81 trials (20%; 12% to 30%) described adequate sequence generation in the protocol. No protocols or trial publications reported inadequate methods of sequence generation.
Methods used for allocation concealment
Table 2 lists the methods used to achieve allocation concealment. Numbered coded vehicles was the most frequently applied method according to the protocols (26 of 102) but had the lowest rate of appearance in the trial publications (three of 26). None of the 17 trials using central randomisation fulfilled the strict criteria, as none described concealment of the randomisation sequence from the central staff, only four described irreversibility of the treatment assignment, and none described that prognostic data irrelevant to stratification must not be revealed to the central office (in three trials such data were positively requested). In 39 of the 102 trials neither the protocols nor the publications provided any information on attempts to conceal the allocation. In four trials, the protocol and the publication gave conflicting information on which method was used.
Table 2.
Methods for allocation concealment in pairs of protocols and corresponding trial publications. Values are numbers
|
Method of concealment in protocols
|
|||||||
|---|---|---|---|---|---|---|---|
| Method of concealment in trial publications | Centralised | Envelopes | Numbered coded vehicles | Other | Uncertain | No information available | Total |
| Centralised | 3 | 1 | 2 | 6 | |||
| Envelopes | 7 | 1 | 1 | 9 | |||
| Numbered coded vehicles | 3 | 1 | 4 | ||||
| Other | 1 | 1 | 1 | 3 | |||
| Uncertain | 1 | 3 | 1 | 5 | |||
| No information available | 11 | 2 | 17 | 6 | 39 | 75 | |
| Total | 14 | 11 | 26 | 7 | 44 | 102 | |
Trial characteristics that might weaken an otherwise adequate allocation concealment regimen
Block randomisation
In 14 trials, block randomisation could partly have compromised allocation concealment because the block size was explicitly stated in the protocol. This is problematic since a known block size enables qualified guesswork to predict upcoming allocations towards the end of the block. This can weaken allocation concealment even in multicentre studies if they are stratified per centre and in double blind studies if the blinding becomes compromised—for example, because of adverse effects.
Tasks that should not be carried out by the same party
The preparation of envelopes for concealment was described in the passive tense in nine of the 13 studies using the envelope method for allocation concealment (see table 2). Thus it is unknown whether the same person prepared the envelopes, enrolled the patients, and administered the envelopes, particularly as seven of the nine studies were single centre studies. An example of lack of separation of functions for central randomisation was when the same party had information on the prognosis of the next patient to be enrolled and was involved in concealing the sequence and in administering it (see bmj.com).
Code envelopes
In 42 of the 55 double blind studies, a security system for emergency code breaking was described in the protocol but mentioned in only one publication. Overall, 90% (38 of 42) of these protocols specified that envelopes or a similar system would be present at the clinical location. Deciphering the contents of such envelopes, for instance by holding them against strong light, might have revealed the allocation for the next patient; yet only one of the 38 protocols (3%) described the envelopes as opaque. Although such code envelopes are a theoretical threat to the allocation concealment, it is unknown whether their presence on the clinical location is associated with exaggerated effect estimates. Consequently, our criteria for assessment of allocation concealment by the means of envelopes did not include assessment of code envelopes.
Discussion
Most trial publications provided unclear information on allocation concealment. When we applied strict criteria the corresponding protocols clarified that 16% had adequate concealment compared with 40% when we applied loose criteria. Thus, regardless of the criteria applied, most of the protocols also provided unclear information or gave rise to additional concern that the allocation concealment might have been compromised (for example, by disclosing the block size). A similar pattern of insufficient reporting was found for sequence generation. The lack of clarity in the protocols is consistent with, but does not prove, the notion that unclear reporting of allocation concealment in trial publications often reflects inadequate safeguards against selection bias.4
Our results make it reasonable to assume that the empirical surveys, which show a 20-30% exaggeration of the treatment effect for trial publications with unclear or inadequate allocation concealment, included some trials with allocation concealment that was adequately carried out but insufficiently reported.3-6 This implies that if inadequate concealment with ensuing selection bias is to explain the observed exaggeration in the previous studies,3-6 then an even larger exaggeration would be expected for those trials where neither the publication nor the protocol indicated adequate concealment.
Strengths and limitations of study
The strength of our study is that it is the first account of how allocation concealment is described in a representative cohort of trial protocols and subsequent publications of trials. The detailed data extraction allowed for sensitivity analysis of the strictness of the applied criteria and for finding additional elements that could compromise allocation concealment.
One limitation is that even in the cases where the protocols provided explicit descriptions of allocation concealment, the assumption that the trials were conducted according to the protocol, might not always be true.10 However, only four of 102 trials gave conflicting information when the publications were compared with their protocols. Another limitation is that it is still unresolved as to what extent the exaggeration associated with unclear allocation concealment in trial publications can be explained by inadequate concealment and ensuing selection bias, as opposed to unclear concealment being a marker of other sources of bias.4
Relation of our findings to those of other studies
Our strict criteria might have been too stringent, and four related studies used criteria with a stringency somewhere between our strict and loose criteria.14-17
Three studies indicated that trial publications with unclear allocation concealment reflect poor reporting of adequate methods, rather than poor methods.14-16
In a retrospective questionnaire survey of investigators by Hill et al, 78% of 32 trials with unclear allocation concealment in trial publications were adequately concealed according to the primary investigators.14 The finding, however, centred on a small sample, on the reliability and memory of the investigators, and on assumptions of what the 20% of non-responders would have replied.
Devereaux et al found that 54 of 56 trials with unclear allocation concealment in the trial publication were adequately concealed according to a pre-announced telephone interview of the investigators.15 These trials were published in journals with higher impact factors than ours and might be of higher methodological quality. Or maybe some of the protocols in our cohort failed to adequately detail all the procedures adopted to protect against bias. Devereaux et al argue that since investigators were willing to report lack of blinding of some parties, they would probably answer reliably on lack of allocation concealment. However, although lack of blinding may be a question of feasibility, lack of allocation concealment is inexcusable and hence potentially less likely to be admitted. The reliability of surveyed investigators has previously been reported on in two surveys where 86% (42 of 49) and 80% (28 or 35) of investigators denied the existence of unreported outcomes, although there was evidence to the contrary in their study protocols.10,18
Another survey was done on trials carried out within the framework of the Radiation Therapy Oncology Group, where alltrial protocols undergo a rigorous six step peer review process.16 Although all studies had adequate allocation concealment (central randomisation) only 42% reported adequate concealment in the trial publication. However, as the authors pointed out, their result has limited generalisability since few trial protocols undergo such rigorous peer review and, as documented in our broad cohort, central randomisation is not the most commonly used method across medical specialties.
What is already known on this topic
In most trial publications, allocation concealment is unclear or inadequate
Unclear or inadequate concealment in publications is associated with an exaggeration of the treatment effect by 20-30%, on average
What this study adds
Most often allocation concealment is unclear in trial protocols
Gatekeepers who sanction protocols should require that adequate methods of allocation concealment are described and used
Protocols should be publicly accessible to enhance critical appraisal of trials
Finally, Liberati et al17 reported results similar to ours; among 47 trials with unclear allocation concealment in the publications, 11 (23%) used adequate randomisation methods (defined as central randomisation) according to a subsequent telephone interview of all but one investigator. The discrepancy with the findings of Hill et al and Devereaux et al might reflect the difference in response rate, criteria for adequate concealment, recentness of the included trials, or the strategies for contacting and phrasing the questions to the investigators.
Implications for clinicians and policy makers
It is prudent to assume that a notable fraction of the overestimation of the treatment effect associated with unclear allocation concealment is caused by selection bias. This fraction can be reduced through several mechanisms. Journals should endorse and enforce the consolidated standards of reporting trials statement (www.consort-statement.org), which recommends explicit description of the allocation procedures in publications of trials, and the gatekeepers who sanction protocols for funding and approval should demand that adequate methods are described in protocols and implemented in trials. Furthermore, our study adds to the argument that protocols should be made publicly available,10,19,20 because public access would increase the reliability of critical appraisal of the fraction of trials where the protocol does describe methods for allocation concealment. Such access would most likely require international legislation and implementation by drug regulatory authorities for trials on pharmaceutical interventions, and research ethics committees for trials on non-pharmaceutical interventions. Both necessitates appropriate investment because these institutions are already pressured to review too much, too quickly.21,22
Supplementary Material
Further details of methods, and examples, are on bmj.com
We thank the scientific and ethical committees for Copenhagen and Frederiksberg, who provided access to the trial protocols and offered administrative support; the researchers who responded to the questionnaires; and J Hedegaard for secretarial support.
Contributors: All authors conceived and designed the study, analysed and interpreted the data, critically revised the manuscript and approved the final version of the manuscript. JP, AH, EF, and PCG acquired the data. PGC provided administrative support for the study. JP drafted the manuscript. She is guarantor for the paper and accepts full responsibility for the conduct of the study, had access to the data, and controlled the decision to publish.
Funding: JP, AH, EF, and PCG are funded by Copenhagen Hospital Corporation. AWC was supported by the Rhodes Trust. DGA is supported by Cancer Research UK. The funding organisations had no role in any aspect of the study, the manuscript, or the decision to publish.
Competing interests: None declared.
Ethical approval: Not required.
References
- 1.Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet 2002;359: 614-8. [DOI] [PubMed] [Google Scholar]
- 2.Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: 663-94. [DOI] [PubMed] [Google Scholar]
- 3.Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323: 42-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273: 408-12. [DOI] [PubMed] [Google Scholar]
- 5.Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352: 609-13. [DOI] [PubMed] [Google Scholar]
- 6.Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135: 982-9. [DOI] [PubMed] [Google Scholar]
- 7.Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002;287: 2973-82. [DOI] [PubMed] [Google Scholar]
- 8.Jüni P, Egger M. Allocation concealment in clinical trials. JAMA 2002;288: 2407-8. [DOI] [PubMed] [Google Scholar]
- 9.Schulz KF, Altman DG, Moher D. Allocation concealment in clinical trials. JAMA 2002;288: 2406-7. [PubMed] [Google Scholar]
- 10.Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291: 2457-65. [DOI] [PubMed] [Google Scholar]
- 11.Alderson P, Green S, Higgins JPT, ed. Assessment of study quality. Cochrane reviewers' handbook 4.2.2 [Updated Mar 2004]; sect 6. In: Cochrane Library, Issue 2. Chichester: Wiley, 2004.
- 12.Meinert C. Clinical trials. Design, conduct and analysis. Oxford: Oxford University Press, 1986.
- 13.Berger VW, Bears JD. When can a clinical trial be called `randomized'? Vaccine 2003;21: 468-72. [DOI] [PubMed] [Google Scholar]
- 14.Hill CL, LaValley MP, Felson DT. Discrepancy between published report and actual conduct of randomized clinical trials. J Clin Epidemiol 2002;55: 783-6. [DOI] [PubMed] [Google Scholar]
- 15.Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol 2004;57: 1232-6. [DOI] [PubMed] [Google Scholar]
- 16.Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004;328: 22-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986;4: 942-51. [DOI] [PubMed] [Google Scholar]
- 18.Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171: 735-40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Jüni P, Rutjes AW, Dieppe PA. Are selective COX 2 inhibitors superior to traditional non steroidal anti-inflammatory drugs? BMJ 2002;324: 1287-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet 1999;353: 490-3. [DOI] [PubMed] [Google Scholar]
- 21.Ghersi D, Campbell EG, Pentz R, Cox Macpherson C. The future of institutional review boards. Lancet Oncol 2004;5: 325-9. [DOI] [PubMed] [Google Scholar]
- 22.Abraham J. The pharmaceutical industry as a political player. Lancet 2002;360: 1498-502. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.