Clinical trials in crisis: four simple methodologic fixes

Abstract

There is growing consensus that the US clinical trials system is broken, with trial costs and complexity increasing exponentially, and many trials failing to accrue. Yet concerns about the expense and failure rate of randomized trials are only the tip of the iceberg; perhaps what should worry us most is the number of trials that are not even considered because of projected costs and poor accrual. Several initiatives, including the Clinical Trials Transformation Initiative and the “Sensible Guidelines Group” seek to push back against current trends in clinical trials, arguing that all aspects of trials - including design, approval, conduct, monitoring, analysis and dissemination - should be based on evidence rather than contemporary norms. Proposed here are four methodologic fixes for current clinical trials. The first two aim to simplify trials, reducing costs and increasing patient acceptability by dramatically reducing eligibility criteria - often to the single criterion that the consenting physician is uncertain which of the two randomized arms is optimal - and by clinical integration, investment in data infrastructure to bring routinely collected data up to research grade to be used as endpoints in trials. The second two methodologic fixes aim to shed barriers to accrual, either by cluster randomization of clinicians (in the case of modifications to existing treatment) or by early consent, where patients are offered the chance of being randomly selected to be offered a novel intervention if disease progresses at a subsequent point. Such solutions may be partial, or result in a new set of problems of their own. Yet the current crisis in clinical trials mandates innovative approaches: randomized trials have resulted in enormous benefits for patients and we need to ensure that they continue to do so.

Introduction

There is growing consensus that the US clinical trials system is broken. It has been routine for researchers to wonder whether clinical trials might “go the way of the Oldsmobile”¹ (a once popular car that is now obsolete) and even for newspaper editorial writers to express concerns about “faltering clinical trials”. ² Consider, for example, that over 70% of Phase III cancer trials did not achieve accrual goals and that nearly half failed to accrue even one-third of the total number of patients required. ³ Alternatively, consider that costs to accrue a single patient to a clinical trial can approach $50,000 ⁴ and that opening a typical trial can involve 370 different steps and a median time from proposal to activation of over 18 months. ⁵ Moreover, concerns about the expense and failure rate of randomized trials are only the tip of the iceberg; perhaps what should worry us most is the number of trials that are not even considered because of anticipated cost and poor accrual.

Much has been written about the role that regulation plays in increasing clinical trial costs and complexity, including editorials in this journal.⁶ Authors have pointed to the need for multiple approvals from different ethics oversight boards and other regulatory agencies as particularly problematic,^6–8 arguing that repeat approvals are unlikely to improve trials, especially as recommendations from different oversight entities can be contradictory. Excessive trial monitoring is also a major area of concern, with calls to move away from standard reporting of all adverse events in all trials on a patient-by-patient basis and towards a more risk-based approach that focuses on randomized comparisons in the context of current knowledge about the experimental interventions.^{6, 9}

But blame can be also be shared with trialists: overly complicated protocols, with multiple primary endpoints, batteries of additional tests and seemingly endless eligibility criteria¹⁰ also serve to inflate costs and depress accrual.⁸ Empirical research comparing trial protocols has clearly demonstrated that trials have become more complex over time, with trial specific procedures, eligibility criteria and work burden increasing by about 10% per year.¹¹ DeMets and Califf have ascribed these recent developments to changes in clinical trial leadership: instead of academics leading trials with a public health orientation, industry “now drives … design, conduct and analysis” with the primary motive being sales and compliance with regulatory requirements.¹²

Several initiatives seek to push back against current trends in clinical trials, aiming to simplify trials and regulatory procedures. The Clinical Trials Transformation Initiative CITTI (http://www.ctti-clinicaltrials.org/) aims to “identify and promote practices that will increase the quality and efficiency of clinical trials”. CTTI suggests that all aspects of trials - including design, approval, conduct, monitoring, analysis and dissemination - should be based on evidence. This is in contrast to contemporary procedures that have generally evolved as a result of haphazard processes, often without a compelling scientific or ethical basis. CTTI’s goals are aligned with those of the “Sensible Guidelines” group (https://www.ctsu.ox.ac.uk/research/sensible-guidelines) that has held several conferences, with associated papers,^{7, 13–17} on clinical trial conduct and design. A typical example of a Sensible Guidelines group initiative is endpoint adjudication. Instead of seeing adjudication as an essential methodology, such that trials without adjudication are seen as flawed, the Sensible Guidelines group has attempted to determine the circumstances under which adjudication would improve trial validity.¹⁶ The NIH Collaboratory (https://www.nihcollaboratory.org) aims to “rethink clinical trials” and is involved in a number of demonstration projects of large, simple trials with a public health orientation. Perhaps the most ambitious recent effort is PCORnet (www.PCORnet.org), a national clinical trials infrastructure to facilitate pragmatic clinical trials using randomization in the course of routine care, and utilizing data from EMRs. No trials have yet been conducted using this network, but several are planned.

Here, I will build on this prior work by briefly outlining four simple methodologic approaches to address common problems of contemporary randomized trials. The four approaches are divided into two stages. First, trials can be simplified, and costs dramatically reduced, by reducing eligibility criteria and by integrating data collection into routine clinical practice. Once this is achieved the second stage is to remove barriers to accrual.

Stage 1: Simplify trials to reduce costs and improve tolerability

• 1Reducing eligibility criteria. Lengthy and complex eligibility criteria can often be replaced with just one: if the doctor has important uncertainty what to do, then the patient is eligible for the randomized trial.¹⁸ The uncertainty principle was pioneered by the ISIS trialists¹⁹ in a series of studies looking at interventions such as beta-blockers after suspected myocardial infarction. Even if the uncertainty principle is not formally invoked, eligibility criteria can often be dramatically simplified.²⁰ As a simple example, a trial comparing two statins will typically involve a set of criteria related to contraindications, such as elevated creatine kinase.²¹ But checking creatine kinase is often considered to be standard care before statin therapy, at least in older patients. In other words, a doctor would not consider a patient for a statin if creatine kinase was high and so the question of which statin was better would not even arise. Naturally, there will be trials where trial eligibility should not be reduced to just the uncertainty criterion. Nonetheless, investigators should still limit criteria to a small number, far less than the 50, 70 or even 100 criteria common in contemporary clinical trials.
• 2Clinical integration. Consider that a typical randomized trial requires creation of a data infrastructure entirely separate from that of clinical practice. This is problematic because, in general, whatever a researcher needs to know about a patient, a doctor should want to know. If a patient has a recurrence of cancer, for example, this is important data for a trialist studying adjuvant therapy, but the patient’s doctor will also need to be informed for consideration of appropriate salvage therapy. We have previously proposed²² investing in data infrastructure for clinical practice to bring routinely collected data up to research grade. Doctors support such investment because it helps them to provide better care. Once this infrastructure is in place, data collected routinely in clinical practice can be used as endpoints in clinical trials. As one well-known example, the long-term follow-up for the seminal West of Scotland Coronary Prevention Study comparing pravastatin with placebo was based entirely on routine records held by the National Health Service, including death, hospital discharges and cancer.²³ There is empirical evidence that such an approach, which avoids expensive and time-consuming event adjudication, is valid for cardiovascular outcomes.²⁴ Endpoints taken from routine clinical practice can also include patient-reported outcomes: our own group has completed a clinical trial comparing the effects of a surgical modification on urinary continence after radical prostatectomy²⁵ in which there was zero protocol specified data collection. As patient-reported outcomes were recorded as a routine part of clinical care, all trial endpoints were downloaded from the electronic medical record at the end of the trial. Experiences such of these have led to calls for “Embedding … research into practice”²⁶ and for “trials … to be conducted as a routine component of clinical care”.⁸ The use of a clinically-integrated approach does limit endpoints to those routinely collected and, because blinding is sometimes not possible, there is a possibility of bias in outcomes assessment. It is also possible that some trials require endpoints that should not be collected as a routine part of clinical practice. Clinically-integrated trials may also be associated with greater loss to follow-up, as there are no special procedures to ensure compliance with data recording, although the major power advantages of clinical integration should offset problems with drop-out.

Stage 2: Shed barriers to accrual

If a trial has minimal eligibility criteria and no additional data gathering, two additional procedures become possible.

• 3Cluster randomization. For studies comparing treatments that that are already widely used in practice – what is generally classed as “comparative effectiveness research” – cluster randomization offers the promise of dramatically improved accrual, simply because it is easier to randomize doctors than patients. Instead of a doctor having to remember to consent patients, and fit what can sometimes be a lengthy consent discussion into a busy clinic, cluster randomization might involve a much abbreviated consent process (see below for further comments on informed consent). For trials where the effects of treatment may depend on the clinician – such as surgery – cluster trials can involve crossover. For example, a surgeon could treat patients using one technique for three months and then crossover to the other for patients presenting during the subsequent three months, with the order determined at random. A further advantage of crossover for cluster randomized trials is that it offsets the power disadvantages associated with having the cluster as the unit of analysis, such that the number of patients in a cluster crossover trial is typically similar to that of a traditional trial where individual patients are randomized.²⁷ Cluster rather than individual randomization is also preferable for clinicians, because they can practice consistently over a period of time, instead of switching back and forth with each new patient.

  Cluster randomization raises the specter of selection bias: doctors know a patient’s allocation and could theoretically influence any consent or schedule patients for treatment during periods associated with particular interventions. This could be carefully monitored by tracking surgeon’s accrual rates over time, then comparing patient characteristics between crossover periods. Bias associated with unblinding can be limited by use of objective endpoints or studying therapies where patients are not likely to have strong preferences; e.g. between two similar analgesics.

• 4Early consent. Cluster randomization, with abbreviated consent, is not appropriate for novel treatments, where patients need to be fully informed that risks and benefits are uncertain. In many cases, the consent process will be less fraught for the patient and logistically easier for the trialist if it takes place early in the course of treatment, well before the patient needs to consider the intervention that is subject to randomization. At the time of initial treatment, patients can be told that, if additional treatment is indicated subsequently, then they may be randomly selected to be offered an experimental therapy. If they are selected, the risks and benefits would be fully explained and they could choose whether to accept or decline. Figure 1 shows the outline of an example trial. The initial consent would only be to release data to trialists and to have the chance of being selected for the novel treatment. Patients who consent would be registered and randomized to standard care or novel treatment. At the time of treatment, patients in the experimental arm would be given further information and the option whether to proceed or to choose standard care. Analysis would be by intent to treat. This methodology is similar to the cohort multiple randomized controlled design of Relton et al. ²⁸

  Early consent trials would be compromised if a high proportion of patients subsequently refused the experimental treatment, something that would bias the results towards the null.

Figure 1.

Early consent design.

Illustrative examples

To illustrate how these approaches might be used in practice, we will start with the INFORM-P trial, a comparison between using the oldest available red blood cells for in-patient transfusion (the current standard practice) versus the freshest available blood. Eligibility criteria were minimal – a few specific classes of patients were excluded – randomization was built into the order entry system and the primary endpoint was in-hospital mortality, which could be downloaded from the hospital database.^{29, 30}

A second example is the Veterans Affairs (VA) “point-of-care” randomized trial comparing two different management approaches to hyperglycemia, sliding scale versus weight-based insulin. There are only two eligibility criteria: the patient needs insulin but is not in ICU. There are no study specific diagnostic procedures or “follow-up events”. Study outcomes, which include length of stay, infections and kidney injury, and other trial data, such as patient demographics, baseline glucose and glomerular filtration rate, are “collected directly from the computerized patient record system”.³¹

INFORM-P and the VA hyperglycemia study incorporate the first two methodologic approaches associated with trial simplicity, namely reduced eligibility criteria and clinical integration. A trial that also includes cluster randomization is currently underway at our own institution. The trial is of factorial design, comparing modifications to radical prostatectomy: two different approaches to the lymph node dissection and two approaches to suturing of the urethra. All four approaches are widely used by urologists. The endpoints of this trial are cancer recurrence and complications for the lymph node comparison, both of which are routinely recorded in the clinical record, and patient-reported urinary continence for the suturing comparison, an endpoint that collected via an electronic questionnaire for all patients irrespective of participation in the trial.³² As outcomes may be surgeon dependent, the cluster randomization involves crossover. All patients undergoing surgery are eligible and surgeons are told always to offer what in their judgment is the best clinical care. For example, consider a surgeon operating during a time period when limited resection has been randomly assigned; the surgeon would be free to dissect more widely if the patient is at particularly high risk or if there is evidence of widespread nodal involvement on imaging or during surgery. Surgeons are asked to record which approach they used and, if they did not follow randomization, to state whether this decision was made before or during surgery.²⁵ There are plans to extend the study to include comparisons related to the timing of catheter removal and use of antibiotics.

As an example of the use of early consent to evaluate a new therapy, rather than cluster randomization for modifications of existing treatments, we will consider focal therapy for prostate cancer, a novel competitor to the established approaches of radiotherapy or surgery. In an early consent design, patients would be approached at the time of prostate biopsy, before diagnosis, and asked whether, were they to be diagnosed with an aggressive cancer in need of treatment, they would be willing to be randomly selected to be offered a new treatment. They would be informed that they could decide at that time whether to try the new treatment or opt for radiotherapy or surgery. They would also be informed that agreeing to take part would not require any additional visits, tests, procedures or questionnaires, because all data for the trial are routinely collected from patients irrespective of trial participation. Data on patient-reported and oncologic outcomes would be downloaded from hospital databases at the end of the trial and analyzed by intent-to-treat.³³

Consent

Clinical-integration and cluster randomization raise several issues concerning informed consent. For instance, in a clinically-integrated trial, consent does not need to be sought for trial-specific procedures (such as scans or questionnaires) because these are absent; similarly, consent for the intervention is not relevant in a cluster randomized trial because, patients receive the same treatment regardless of whether or not they consent. The general theme of recent initiatives such as CTTI and the Sensible Guidelines groups is that trial procedures need to be rationally applied on a trial-by-trial basis, rather than being universally imposed, irrespective of the trial design and study context. As such, the possibility has been raised as to whether a waiver of consent might be appropriate in some trials. This is particularly for trials where the randomized comparisons are routine in everyday clinical practice, such that patients might receive one or the other on a quasi-random basis when presenting for care depending on the provider they happen to see; consider, for example, a trial of two antibiotics for infection.³⁴ One line of argument has been to question the traditional sharp distinction between research and clinical care, with the former requiring more stringent ethical oversight and associated procedures, such as informed consent and safety monitoring, and the latter requiring little if any oversight. If this distinction falls, then there is room for a more practical approach that focuses on risk and burden irrespective of whether a medical activity is labeled research, clinical practice or quality assurance.³⁵

Yet the SUPPORT trial controversy³⁶ shows that such arguments are far from being universally accepted. In SUPPORT, premature infants were randomized to different levels of oxygen, both within clinical practice guidelines of the time. Critics of SUPPORT argue that randomization between different clinical approaches is “not the key issue” as “those giving consent to the trial should have been informed of all uncertainties”.³⁷ Such an argument of course ignores the question of whether parents of infants not on the trial were informed of all uncertainties during routine care. Nonetheless, it is clear the nature and content of consent will be a critical issue in considering solutions to the current crisis in clinical trials.

Pragmatic versus explanatory trials

The methodologic fixes proposed in this paper are clearly more applicable to trials that are towards the pragmatic rather than the explanatory end of the spectrum.³⁸ If a trial aims to test a very specific hypothesis about the causal effect of an intervention on an outcome, then it may not be appropriate at all to reduce eligibility criteria, or to restrict outcomes to those routinely collected in clinical practice. But implementation of the methodologies described above need not substantively modify the study question. As an example, the study question of a trial comparing two similar chemotherapy regimens is no more or less pragmatic in a trial that includes eligibility criteria related to chemotherapy contraindications than in a trial that avoids such eligibility criteria on the grounds that the choice of chemotherapy regimen is irrelevant for a contraindicated patient. The study question is similarly unaffected if the trial obtains patient-reported outcomes using a web-based system implemented in clinic for all patients rather than by paper questionnaire especially for trial patients. Randomization at the oncologist level rather than the patient level also does not require a change in the study question.

Concluding remarks

The approaches outlined in this paper developed from an extensive literature on clinical trial design. Trial simplification was described by Yusuf, Collins and Peto as long ago as 1984;³⁹ clinical-integration has been subject to considerable discussion since the advent of the electronic medical record;⁴⁰ cluster randomization has, of course, a long history in research; early consent is an adaptation of the “Zelen” design,⁴¹ and has been described previously.²⁸ I have tried to show how these ideas are complementary, and together provide a way to tackle important problems in contemporary clinical trials. There is the practical point that even if a trial based on the methodologic approaches described in this paper were regarded as imperfect, an imperfect trial that can be completed is preferable to a perfect trial that will never be done. An “imperfect” randomized trial is still qualitatively less biased than most observational alternatives.

The current crisis in clinical trials surely mandates that we explore some innovative solutions. Those solutions may be partial, or result in a new set of problems of their own. But that has to be a better option than trying the same thing over again, hoping for a different result, or perhaps just giving up altogether. Randomized trials have resulted in enormous benefits for patients: we need to ensure that they continue to do so.