Hard-Wired Bias: How Even Double-Blind Randomized Controlled Trials Can Be Skewed from the Start

Well designed, adequately powered randomized controlled trials (RCTs) are rightfully considered the highest form of evidence on which to base treatment and diagnostic decisions, minimizing potential biases, particularly confounding, that plague alternate, lesser forms of evidence^1–3. At the same time, simply being an RCT is not sufficient to ensure that conclusions are free from bias. In recent years, sponsors and trialists have incorporated subtle design choices into RCTs that have skewed the final trial results. Here, we describe the phenomenon of “hard wired” bias—bias that is introduced at the outset of randomized trials⁴. We hope that recognition of hard-wired bias serves as a reminder that there remains room for improvement in the analysis and conduct of clinical trials.

Bias in the interpretation of clinical data may occur at many junctures, as depicted in, but not limited to, the Figure. Selective reporting of outcomes, and publication bias occur at the final step in the presentation of data. Bias in the interpretation and analysis of data, including deviations from preplanned statistical analysis occur penultimately. Bias can also occur prior to the collection of data—implicit in the design of clinical trials. Unlike bias in the analysis or reporting of data, bias in trial design cannot be overcome by statistical methods or reanalysis⁵; instead it can only be noted as a limitation of the study. What are some of the ways in which bias in clinical trials can become hard-wired?

Figure 1.

The origins of bias in clinical trials

Crossover

Crossover from placebo or control to the investigational agent is typically done in pharmacokinetic studies, or for interventions aimed at assessing a subjective endpoint (permitting intra-user comparisons). However, in modern clinical trials, crossover is increasingly used in studies testing the basic efficacy of a novel compound. For instance, many randomized controlled trials of cancer therapeutics allow patients assigned to placebo to crossover to treatment upon progression of their disease⁶. Such crossover affects interpretations regarding the drug’s effect on survival. For instance, if a cancer drug delays progression, but does not improve survival, crossover is often cited as the reason for these findings. The drug would have improved survival had it not been for crossover. However, despite this common interpretation, there are other valid interpretations in this setting. For instance, a drug may slow progression, but increase off-target deaths, such that it has no net benefit⁷. In this case, crossover can mask the harms of the medication, and provide a misleading inference regarding benefits.

Sipuleucil T (Provenge, Dendreon) is a cancer vaccine approved by the US Food and Drug Administration (FDA) for the treatment of metastatic prostate cancer. In the seminal trial leading to approval, the drug managed to improve overall survival without any evidence that it slowed disease progression⁸. In the trial, many patients in the control arm received the frozen vaccine when their cancer progressed, and fewer patients received docetaxel—a drug with a proven survival benefit—or received it after a delay. This design led to the suggestion that Sipuleucel-T demonstrated efficacy not by improving outcomes, but rather because crossover harmed the control group—by delaying alternate, effective therapy⁹. When it comes to crossover, better conclusions cannot be obtained simply through access to the data—all interpretations of the data have to make some assumptions about whether crossover is benefitting or harming the control group. Consider RECORD-1, a randomized trial comparing everolimus to placebo in patients with metastatic renal cell cancer for whom prior therapy had failed. RECORD-1 demonstrated an improvement in progression free survival, but no change in overall survival¹⁰. This deficiency was attributed to crossover¹⁰, and the manufacturer provided modeling experiments arguing what the survival would have been were it not for crossover¹¹. However, this exercise relies on assumptions, which may not be true. Because we cannot know the true effect of everolimus on survival, it was rejected by the UK’s National Institute for Clinical Excellence (NICE).¹²

For a different set of questions however, a lack of crossover can bias results. When clinical trials seek to establish the basic efficacy of a novel compound, the presence of crossover can distort inferences, as we have shown. However, when clinical trials test the proper sequencing of agents already known to confer benefit, the absence of crossover can be equally problematic. Consider some contemporary examples. In a randomized trial from Spain that showed lenalidomide and dexamethasone improved overall survival among patients with smoldering myeloma over current standard of care of observation¹³, patients in the control arm were not routinely given access to lenalidomide when they developed overt multiple myeloma. Thus we cannot be sure that the survival advantage of early treatment would still exist if control patients had fair access to this drug, as they would have had in the US. The failure to proscribe lenalidomide upon progression is also a limitation of randomized trials of maintenance therapy^14,15, which (should) ask the question of whether continuous administration of a drug is better than receiving the agent upon having progressive disease.

In a final example, the proteasome inhibitor bortezomib, which is FDA approved for relapsed mantle cell lymphoma, was tested in the first line setting of this disease. At the time of the trial, the drug was widely used in the relapsed setting. Thus, any trial seeking to advance the drug into the first line should show that early use of the drug improves survival beyond its second line use. However, the randomized trial testing this question was globally conducted, and, as such, patients in the control arm had poor access to the drug in the second line setting, with only 19% receiving it¹⁶. Thus, we cannot be sure that the survival advantage seen in the trial would exist had it been conducted in nations where bortezomib is a mainstay of second line treatment.

Selection Bias

Bias in choice of study subjects is a frequent concern in clinical trials. Selection bias can be caused by the inclusion and exclusion criteria to a study, which prevent the generalizability of results to a broader patient population. Although such concerns are increasingly appreciated, there remain pertinent even today. For example, among cancer drugs approved by US Food and Drug Administration between 1995 and 2002, demographics of patients were strikingly different from cancer patients in the United States¹⁷. While the proportions of patients aged ≥ 65, ≥ 70, and ≥ 75 years was 60%, 46%, and 31%, respectively among cancer patients in the United States, these age groups comprised only 36%, 20%, and 9% of patients in registration trials (P < .001). Fehrenbacher and colleagues extended these findings, and showed that inclusion criteria of contemporary randomized trials in non-small cell lung cancer would exclude the majority of patients treated at Kaiser Permanente, a large insurer with a representative patient population¹⁸.

Although these examples are illustrative, this problem is relatively tractable, as care can be taken to prevent extrapolation to untested populations. However, in other cases, selection bias cannot be accounted for.

Consider the open-label “run in period.” The Heart Protection Study, a randomized trial of 20,536 high cardiovascular risk individuals, tested whether simvastatin at 40mg daily could improve outcomes compared to placebo¹⁹. The trial found that the medication decreased major vascular events by 25%, and the authors go further, arguing that, without non-compliance, the improvement would have been 33%¹⁹. Notably, this trial utilized a 4-week placebo run in period followed by a 4–6 week simvastatin run in period prior to randomization. During this time, a patient’s primary doctor could remove a patient from randomization, and any patient could elect not to be randomized for “any reason²⁰.” All together, 11,609 patients who were eligible for the study and began the run-in period dropped out prior to randomization²⁰. Thus, over a third of the patients who began the study were not randomized, and no set of specified inclusion criteria can define the set of patients who remained. Others have noted that the use of run-in periods can limit the applicability of study findings, and can inflate estimates of benefits²¹. This occurs in part because run-in periods of the active drug tests a different clinical question—whether discontinuation of a therapy is harmful—rather than whether initiating a therapy is beneficial.

Open-label run in periods were also problematic in the PARADIGM-HF trial²². This study randomized 8442 patients predominantly with NY Heart II and III heart failure to a combination of valsartan (an angiotensin receptor blocker (ARB)) and neprylisin (the investigational agent) or enalapril (the control arm). Yet, prior to randomization, over 10,500 patients entered into a run in period. During this time, patients were sequentially treated with a median of 15 days of enalapril followed by 29 days of the combination medication. Nearly 20% of participants who began the study dropped out of the study during this time. Thus, the run in period created both an indefinable study population, viz. patients who met inclusion criteria and did not drop out after 15 days of enalapril followed by 29 days of the combination medication, as well as posed a different question, whether switching back to enalapril was better or worse than continuing the combination among such patients.

The Unlevel Playing Field

Another way modern trials have hard-wired bias is by promoting an unequal comparison. For instance, a head to head trial of the tyrosine kinase inhibitors axitinib and sorafenib in metastatic kidney cancer appears to be fair; however a closer examination reveals problems²³. Specifically, while the starting dose of both drugs is appropriate, the dose reductions for toxicity favor the axitinib arm. For similar side effects, sorafenib has steeper dose reductions, and for patients doing well on full dose axitinib, the dose could even be increased. Collectively this meant that axitinib was pushed to a higher dose, and penalized less than sorafenib²⁴. It should be no surprise which medication was declared the winner of that study.

Another type of unequal playing field occurs when two cancer drugs are tested head to head, but as a matter of fact more patients had already taken, and had the opportunity to have their cancer acquire resistance to, one of those drugs. In the ENDEAVOR randomized trial, patients with multiple myeloma who had had their disease progress on treatment were randomized to bortezomib or carfilzomib in combination with dexamethasone²⁵. The results showed a progression free survival benefit for carfilzomib, but as a matter of fact, given the date these drugs were approved, the majority of patients had the opportunity to be previously treated with bortezomib, while very few patients had the chance to previously receive carfilzomib. As a general rule in cancer, two drugs can be on average comparable, but each is less effective among patients already treated with that medication—a bias that the ENDEAVOR trial exploits.

Control Arms

The control for a clinical trial is ideally selected depending on the clinical question posed. Controls should reflect the best therapy currently being used in the target population, and for studies evaluating subjective endpoints, the control should be as close as possible to the investigational arm. If this condition is not met, a trial essentially uses a straw man comparator. Many times, the use of a sham control has unmasked bias in studies supporting the use of a medical procedure. For instance, vertebroplasty, epidural steroid injection, and arthroscopic meniscectomy all required a sham-controlled trial to demonstrate that the treatments had no benefit²⁶.

Censoring

Informative censoring²⁷ in clinical trials can distort our perception of the benefits of a treatment. All survival analysis is based on the premise that censoring is uninformative—the patients censored are no different than those who are followed. However, increasingly this assumption may be questioned. In many cancer treatment trials, censoring often occurs as patients withdraw from toxicity or intolerability. These patients are likely to be different from those who tolerate therapy well. In a recent study, Campigotto and Weller provide two examples in which patients who are censored are likely to have better or worse survival than those who remain on study²⁸. The authors then provide a range of estimates for the outcome had these patients not been excluded, based upon simulation. But, if patients come off treatment, and are no longer followed, we cannot reconstruct their outcomes without making assumptions. We are left with a hard wired bias.

Conclusion

Randomized control trials remain the best way to draw sound conclusions regarding the efficacy and impact of drugs, devices, screening and diagnostic tests, but unfortunately randomization does not ensure a fair trial. In this respect, randomization can be viewed as necessary but not sufficient for sound scientific decision making. The purpose of our analysis is not to disparage the growth of RCTs—which we believe is inevitable and unquestionably valuable—but to highlight persistent challenges. Many types of bias can be remedied by access to individual patient level data, while other types of bias, so called “hard-wired” bias, cannot be corrected for after the fact.

Medical trials involve the participation of human subjects, who donate their time and energy to further the altruistic pursuit of improved medical care. As such, we have a moral obligation to ensure that research is capable of most honestly answering an important clinical question. The elements of trial design that we discussed—crossover, drug run-in periods, the use of inadequate controls, early censoring, selection bias, and duration of follow up—are decisions made at the outset of a clinical trial, and cannot be later corrected. We must work to remove hard-wired bias from clinical trials, and only time will tell if our existing system (in which trials are predominantly funded and conducted by industry) can meet this challenge.