Comparator bias: why comparisons must address genuine uncertainties

Controlled trials are for reducing uncertainties about the relative merits of different treatments

Researchers may believe – and patients and physicians may hope – that a particular treatment (perhaps because it is new) is better than other available treatments; but it may often turn out to be worse.^1,2 When the British Medical Research Council's controlled trial of streptomycin for pulmonary tuberculosis was conceived in 1946,³ none of the therapies used to treat the disease had been shown in controlled clinical trials to be useful; indeed, one controlled trial had shown gold salt therapy to do more harm than good.⁴ Although streptomycin was known to be useful in forms of tuberculosis which had previously always been fatal, there was uncertainty about how useful the new drug would be in pulmonary tuberculosis, from which patients often recovered after treatment with bed rest alone. Patients in the MRC trial were accordingly randomized either to bed rest alone, or to bed rest and streptomycin.

The same reasoning is applicable when controlled trials are designed today. After considering systematic reviews of the relevant existing evidence, patients and their doctors must be substantially uncertain about which among the treatment options – including no active treatment – is preferable. This implies ensuring that no patient who agrees to participate in the trial will knowingly be disadvantaged, whichever one of the comparison treatments the patient is assigned to receive.

Clinical trials are done to reduce uncertainties, and they should only be done if clinicians and their patients are uncertain which of the existing alternatives is preferable.^5,6,7,8 This requirement is sometimes referred to as ‘the uncertainty principle’⁹ or ‘equipoise’.^10,11

If one or more of the treatments selected for the comparison in a trial is known to be worse than others, not only will some participants in the trial be denied effective treatment, but this ‘comparator bias’ will result in unfair tests of treatments. Even if other sources of bias have been well controlled in such studies, their results will mislead patients and their doctors. Unfortunately, comparator bias is sometimes deliberately introduced for just this purpose, usually with a view to showing that new treatments are preferable to existing alternatives.^12,13

Inappropriate use of inactive comparators

Comparator bias is introduced when treatments known to be beneficial are withheld from patients participating in controlled trials. The reason that bed rest alone was an acceptable treatment for half the patients in the MRC trial of streptomycin for pulmonary tuberculosis was that there was no known effective treatment for the condition. When systematic reviews of existing evidence show that existing treatments are more helpful than doing nothing, or than using placebos, comparator bias will result if patients are denied these effective treatments, thus giving the active treatments in the trial an unfair advantage.

Although users of clinical research evidence are usually interested in the relative merits and disadvantages of active alternative treatments,^14,15 available comparative efficacy data were used in only half of 100 applications for marketing licences for new molecular entities approved by the US Food and Drug Administration between 2000 and 2010.¹⁶

Estellat and Ravaud¹⁷ have shown that placebos were used as comparators in four out of five (81/102) trials of biologic disease-modifying drugs for rheumatoid arthritis done during the past decade. In 54 (86%) of 63 trials involving patients with a high level of active disease, placebos (or treatments known to have been ineffective) were used, with the result that potentially helpful treatments were being withheld from 9,224 out of 13,095 patients randomized to the control arms.

Even though the efficacy of erythropoietin in preventing anemia in cancer patients had been convincingly demonstrated, some researchers continued to assign participants in clinicial trials to placebos instead of testing the drug's effects on other outcomes.¹⁸ Uncertainty about the effect of the drug on survival continues more than 20 years after drug was approved for clinical use in 1989.

Using inappropriate ‘active’ comparators

Predictable results favouring new treatments can be obtained when inappropriate ‘active’ comparators are used. For example,¹⁹ Psaty et al. noted that three out of four large industry-sponsored trials evaluating newer antihypertensive drugs used the beta-blocker atenolol as the comparator, even though this drug had been shown to be inferior to a low-dose thiazide diuretic.

Comparator bias can also result when a treatment is compared with an inappropriately low dose of a comparator intervention. This occurred in comparisons of newer non-steroidal anti-inflammatory agents used for arthritis with older drugs in the same class.²⁰ Inappropriately low doses can also result when treatments are given by an inappropriate route, for example, by comparing intravenous administration of one drug with oral administration of another that is poorly absorbed from the gastro-intestinal tract.²¹

The net usefulness of treatments often requires trade-offs between wanted and unwanted effects. Treatments may be preferable even if their beneficial effects are no better than alternatives if they have fewer adverse effects. Some of the newer drugs for treating schizophrenia, for example, may be preferable to established drugs for this reason. However, this apparent advantage may be because the newer agents have been compared with inappropriately high doses of the older comparator drugs.²² Safer reported eight trials sponsored by three different drug companies which compared newer second-generation neuroleptic agents to a fixed high dose (20 mg/day) of haloperidol. Predictably, patients using the new agents had fewer extrapyramidal side effects.²³

Rheumatological research provides a further example of the use of inappropriate active comparators. For example, in the MEDAL (Multinational Etoricoxib and Diclofenac Arthritis Long-term) trial, when 24,913 patients with osteoarthritis and 9,787 patients with rheumatoid arthritis were randomly assigned to receive COX-2 inhibitor etoricoxib or COX 1 inhibitor diclofenac,²⁴ no difference was detected in the frequency of bleeding or adverse cardiovascular events. However,²⁵ Psaty and Weiss have noted that the results were predictable because diclofenac is known to have a toxicity profile similar to that of the COX-2 inhibitor celecoxib. They suggested that naproxen would have been an appropriate comparator because it was known to be associated with a lower risk of cardiovascular events: a meta-analysis of 121 placebo-controlled trials of COX-2 inhibitors yielded a relative risk (of vascular events of 0.92 (95% CI = 0.81–1.05) for COX-2s when diclofenac was used as the comparator compared with 1.57 (95% CI = 1.21–2.03) when naproxen was used as the comparator.²⁶

How can comparator bias be reduced?

Reducing some forms of bias is straightforward: allocation bias, for example, is controlled by strict random allocation of patients to treatment comparison groups. Comparator bias cannot be dealt with so straightforwardly. In fact, a precise mathematical solution of the choice of appropriate comparator is theoretically not possible.²⁷

However, comparator bias would be less of a problem if the choice of comparison groups in controlled trials became informed routinely and transparently by systematic reviews of relevant existing evidence. A 2005 survey of authors of clinical trials which had recently been added to systematic reviews revealed that less than half were even aware of the relevant reviews when they designed their new studies;²⁸ and a 2011 analysis of clinical trials reported over four decades showed that, regardless of the number of relevant previous trials, fewer than a quarter and a median of only two trials had been cited in trial reports.²⁹ When, in the light of the existing evidence and other considerations, patients and doctors are uncertain which among treatment options is preferable, the preconditions for avoiding comparator bias exist.³⁰

The choice of comparators in clinical trials inevitably involves judgements and values that go beyond scientific considerations. It is not surprising, therefore, that researchers, sponsors, patients and government regulators may have different views on the selection of comparators.^31,32,33 Some authors believe that as long as the drugs are listed on the national pharmacotherapy reference books comparison against such treatment may be justified even if it is not supported by evidence-based clinical guidelines published in the literature.³⁴ Avoidance of inappropriate use of inactive and active comparators would seem most likely to result from greater involvement of the patients and clinicians for whom research should be producing relevant knowledge,³⁵ with those who prioritise, fund and design clinical research, and the entities that approve the marketing of new interventions.

DECLARATIONS

Competing interests

None

Funding

Not relevant

Ethical approval

Not relevant

Guarantor

HM

Contributorship

Both authors contributed to the conception and drafting of the manuscript.

Acknowledgment

We thank Drs. Adams, Power, Paul, Alexander, Price and Singh for providing some of the examples of comparator bias cited in this article.