A Britton, M McKee, N Black, K McPherson, C Sanderson, C Bain
Randomised controlled trials are often held up as the “gold standard” of medical research, and it is commonly believed that the size of a treatment effect is exaggerated in non-randomised studies. In these days of evidence based medicine, however, where is the empirical evidence that this is so? A widespread criticism of randomised controlled trials is that they are based on highly selected individuals. Are there systematic differences between patients included and excluded in such trials, and do these influence the measured treatment effect?
These are just a few of the questions that Britton and colleagues have attempted to address in their review. The other questions they consider are to what extent it is possible to adjust for baseline differences between study groups, and how important is patients’ preferences in terms of outcome. The answers to these questions inevitably depend on the quality of the available evidence. The authors found 18 papers comparing randomised controlled trials and prospective non-randomised studies with the same intervention and similar settings and found no consistent pattern in the effect size. They give a variety of reasons both for a larger effect and for a smaller effect in randomised controlled trials. For example, higher quality interventions in randomised controlled trials may produce a larger effect, and the fact that non-randomised studies may contain a disproportionate number of patients with a greater capacity to benefit (in one arm) may produce a smaller effect. Four separate chapters consider case studies in surgical interventions, drug interventions, organisational interventions, and preventive interventions.
The answers to the other questions posed by the authors are more vague. As one might expect, there are differences in patients included in treatment trials and those included in prevention trials: patients in the latter are more affluent, better educated, and more likely to adopt a healthy lifestyle, although evidence is limited because few studies report details of the non-participants. Britton and colleagues conclude that adjusting for differences in baseline prognostic factors often changed the estimated treatment effect, but not “significantly” and not consistently. There were only four studies that directly examined the effects of patients’ preferences on treatment estimates, and there was some suggestion that preferences could account for some of the differences between randomised controlled trials and non randomised studies.
Britton and colleagues conclude with some recommendations for more representative, pragmatic trials, and for better reporting of characteristics of eligible patients who did not participate. Overall this is a useful review, and if the authors did not answer all the questions they set themselves at the outset, they have at least demonstrated that the reason for this apparent failure is that the evidence is not available yet and map out useful areas for future research.
Figure.
Health Technology Assessment
1998;2(13):pp 214
Rating: ★★★