Commentary on Tinè et al

Tinè et al. present an interesting trial-level dataset from a successful drug development program for hepatitis C therapeutics, starting with interferon trials in the late 1980s and including interferon-combination therapy trials up through 2012.¹ The authors compare the rate of sustained viral response across decades of studies and demonstrate that the study-arm label of “experimental” vs “control” is a statistically significant factor associated with response rate. When comparing response rates across experiments, the same drug tends to have better response rates when labelled as “experimental” relative to when labelled as “control.” On the surface, this result is puzzling, as a label should not impact efficacy. The authors suggest that this bias may be caused by poor trial design and/or conduct. This conclusion seems unwarranted. While this is a meta-regression of study arms from randomized controlled trials, the analysis is observational and results must be evaluated with the same skepticism as any observational study. Meta-analyses typically summarize the treatment effect (e.g., differences in response rates between arms) across studies, rather than comparing point estimates from individual arms across studies. The paper by Tinè et al. involves the latter type of analysis. This is problematic because the value of randomization, which balances study-specific factors (e.g., a higher-risk patient cohort) across study arms, disappears with arm-specific comparisons. The resulting analysis is potentially confounded by many things, including differences in patient populations and patient-care practices that may evolve over time in complex ways.

Many factors changed over the course of this drug development program. For example, one can conjecture that the type of patient who would enroll in a study will depend on whether there are other therapies available. If there are approved therapies, perhaps experimental therapies are reserved for the sicker patients. Alternatively, if the existing therapy has toxicities and a new treatment has a more favorable safety profile, maybe healthier patients are enrolled, with the expectation that they can be salvaged by the more toxic treatment upon treatment failure. Many potential scenarios exist and preferences may even vary across clinicians and time. With respect to hepatitis C treatments, it is clear that many variables may have changed over time, including duration of treatment, timing of response evaluation, supportive care that may facilitate sustained therapy, assays for detecting residual virus, viral subtyping (e.g, type 1, type 2, etc), host genotyping (i.e., IL28B polymorphisms in the host), although this list is not exhaustive. These may all impact outcome. Controlling for them represents an enormous modelling challenge, but would be important to conclude whether the differences across time are due to the effect of label (“experimental” vs “control”) or due to changes across time. That said, the data collected by Tinè et al. give some clues about an important effect of time. An analysis of the IFN3 (“peg-interferon plus ribavirin”) studies alone reveals that arm (“control” vs “experimental”) is not significant, yet time is a significant predictor, with response rates decreasing over time (odds ratio: 0.93 per year, p=0.005). Further, time remains a significant predictor (odds ratio: 0.91 per year, p=0.05) when restricting the analysis to arms labelled as “experimental.” (See Figure 1). This suggests the presence of unmeasured covariates that changed over time are at play here.

Figure 1.

Peg-interferon plus ribavirin studies

Another potential bias in the analysis approach is worth mentioning, although it is difficult to fully evaluate the extent to which it is present in these data. A thought experiment is useful to demonstrate this point. Imagine a large pool of candidate drugs that are evaluated with many randomized controlled trials. From that pool, the best drug is selected to become the standard of care. This drug then becomes the control arm in a future study of the next generation of drugs. Because of this selection pressure, estimates of efficacy from the selected drug might be biased high. Another way to think of this is that the data represents a biased sample: only experimental drugs with successful results are eligible for inclusion. Unsuccessful drugs, by definition, do not become “control” drugs in later experiments. This would be exaggerated further with interim efficacy monitoring, although that does not appear to have been an issue in the data presented here.

Although the above arguments do not demonstrate a lack of bias in randomized controlled trials, they highlight problems with this approach to evaluating whether such bias exists. Randomized controlled trials remain the gold standard for evaluating new therapies. My comments reflect skepticism about analyses not protected by randomization, rather than concern about the conduct of randomized controlled trials.