Abstract
OBJECTIVE--To determine whether inappropriate subgroup analysis together with chance could change the conclusion of a systematic review of several randomised trials of an ineffective treatment. DESIGN--44 randomised controlled trials of DICE therapy for stroke were performed (simulated by rolling different coloured dice; two trials per investigator). Each roll of the dice yielded the outcome (death or survival) for that "patient." Publication bias was also simulated. The results were combined in a systematic review. SETTING--Edinburgh. MAIN OUTCOME MEASURE--Mortality. RESULTS--The "hypothesis generating" trial suggested that DICE therapy provided complete protection against death from acute stroke. However, analysis of all the trials suggested a reduction of only 11% (SD 11) in the odds of death. A predefined subgroup analysis by colour of dice suggested that red dice therapy increased the odds by 9% (22). If the analysis excluded red dice trials and those of poor methodological quality the odds decreased by 22% (13, 2P = 0.09). Analysis of "published" trials showed a decrease of 23% (13, 2P = 0.07) while analysis of only those in which the trialist had become familiar with the intervention showed a decrease of 39% (17, 2P = 0.02). CONCLUSION--The early benefits of DICE therapy were not confirmed by subsequent trials. A plausible (but inappropriate) subset analysis of the effects of treatment led to the qualitatively different conclusion that DICE therapy reduced mortality, whereas in truth it was ineffective. Chance influences the outcome of clinical trials and systematic reviews of trials much more than many investigators realise, and its effects may lead to incorrect conclusions about the benefits of treatment.
Full text
PDF
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Easterbrook P. J., Berlin J. A., Gopalan R., Matthews D. R. Publication bias in clinical research. Lancet. 1991 Apr 13;337(8746):867–872. doi: 10.1016/0140-6736(91)90201-y. [DOI] [PubMed] [Google Scholar]
- Hine L. K., Laird N. M., Hewitt P., Chalmers T. C. Meta-analysis of empirical long-term antiarrhythmic therapy after myocardial infarction. JAMA. 1989 Dec 1;262(21):3037–3040. [PubMed] [Google Scholar]
- Imperiale T. F., Petrulis A. S. A meta-analysis of low-dose aspirin for the prevention of pregnancy-induced hypertensive disease. JAMA. 1991 Jul 10;266(2):260–264. [PubMed] [Google Scholar]
- Koudstaal P. J., Stibbe J., Vermeulen M. Fatal ischaemic brain oedema after early thrombolysis with tissue plasminogen activator in acute stroke. BMJ. 1988 Dec 17;297(6663):1571–1574. doi: 10.1136/bmj.297.6663.1571. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Peto R., Collins R., Gray R. Large-scale randomized evidence: large, simple trials and overviews of trials. Ann N Y Acad Sci. 1993 Dec 31;703:314–340. doi: 10.1111/j.1749-6632.1993.tb26369.x. [DOI] [PubMed] [Google Scholar]
- Peto R. Why do we need systematic overviews of randomized trials? Stat Med. 1987 Apr-May;6(3):233–244. doi: 10.1002/sim.4780060306. [DOI] [PubMed] [Google Scholar]
- Simes R. J. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986 Oct;4(10):1529–1541. doi: 10.1200/JCO.1986.4.10.1529. [DOI] [PubMed] [Google Scholar]
- Yusuf S., Collins R., Peto R., Furberg C., Stampfer M. J., Goldhaber S. Z., Hennekens C. H. Intravenous and intracoronary fibrinolytic therapy in acute myocardial infarction: overview of results on mortality, reinfarction and side-effects from 33 randomized controlled trials. Eur Heart J. 1985 Jul;6(7):556–585. doi: 10.1093/oxfordjournals.eurheartj.a061905. [DOI] [PubMed] [Google Scholar]
- Yusuf S., Collins R., Peto R. Why do we need some large, simple randomized trials? Stat Med. 1984 Oct-Dec;3(4):409–422. doi: 10.1002/sim.4780030421. [DOI] [PubMed] [Google Scholar]