US researchers have claimed that most randomised trials of new treatments published in leading medical journals are reported in a potentially misleading way, with statistics designed to make the results more positive than if other statistical tests were used.
Researchers from the University of California at Davis, near Sacramento, reviewed 359 randomised clinical trials of new treatments published between 1989 and 1998 in five major medical journals: the Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine.
They found that most of the trials report results based on relative risk reduction—the percentage difference in end points between the active treatment and the placebo or comparison treatment. The study showed that only 18 of the papers reviewed considered absolute risk reduction—the actual difference between the treatment and placebo results. Only eight of the 359 trials reported the number needed to treat—the number of patients needed to be treated with a drug to prevent one adverse outcome, for example a myocardial infarction (JAMA 5June 2002).
One member of the research group, Dr Jim Nuovo, said in a statement on the study: “Our review of the literature showed that, in the majority of cases, only the most favourable statistic—the relative risk reduction—was used when reporting the results of these studies.”
He warned that this could potentially be misleading: “The public expects advertisers to use the most flattering statistics to bolster claims of effectiveness in promoting products. However, most of us expect that medical journals will provide complete reporting of all important aspects of research on a new treatment. Without more comparative data, readers may be basing their decisions to use a new treatment on incomplete information.”
Dr Nuovo said that medical journals should require all authors to follow the recommendations of the CONSORT statement, which include use of absolute risk reduction and number needed to treat. These recommendations were introduced in 1996 by a group of journals, including those in the study, but did not apply to the reporting of trials before this date, which covered much of the period included in Dr Nuovo's study.
He said: “No single number can entirely explain the results of a trial, so it would be in the best interest of physicians and their patients to require complete reporting of the potential benefits and the potential risks of any new treatment. Doing so should increase the chance of the appropriate implementation of research on new treatments.”
Dr Ike Iheanacho, deputy editor of the Drug and Therapeutics Bulletin, agreed: “It is not unusual to see new drugs presented in terms of relative risk reduction rather than absolute risk reduction. Relative risk reduction is often much more impressive and seems more tangible.”
He added: “If you are assessing the effects of any drug, you want as much information as possible. At the very least, you need absolute risk reduction. Relative risk reduction can appear quite large when the absolute risk reduction is actually very small.” For example, if a disease kills two in every million people, a drug that reduces the death rate to one in a million would give a relative risk reduction of 50% which appears to be a major benefit. However, the absolute risk reduction would only be one in a million.