The Prompted Optional Randomization Trial: A New Design for Comparative Effectiveness Research

Abstract

Randomized controlled trials are the gold standard for medical evidence because randomization provides the best-known protection against confounding of results.

Randomization has practical and ethical problems that limit the number of trials that can be conducted, however. A different method for collecting clinical data retains the statistically useful properties of randomization without incurring its practical and ethical challenges.

A computerized prompt introduces a random element into clinical decision-making that can be instantly overridden if it conflicts with optimal patient care. This creates a weak form of randomization that still eliminates the effect of all confounders, can be carried out without disturbing routine clinical care, and arguably will not require research-grade informed consent.

Clinicians who care for patients with common chronic diseases such as diabetes, hypertension, and dyslipidemia often must choose among multiple drugs with similar indications. For example, although solid clinical trial data establish that a diabetic who is inadequately treated by metformin will have better glycemic control with the addition of either a sulfonylurea or a dipeptidyl peptidase-4 inhibitor, data on which combination is more effective are lacking.¹ Without randomized controlled trials that make the direct comparison, clinicians have to rely on anecdote, expert opinion, and observational cohort studies to choose between these two drug classes. Because these choices differ significantly in their potential cost and may lead to different clinical outcomes, policymakers and researchers increasingly call for comparative effectiveness research designed to inform this kind of decision.²

Anecdote and expert opinion rank at the bottom of the hierarchy of evidence.³ Cohort studies rank higher, and they are easy to conduct because they simply observe the outcomes associated with exposure to a treatment and do not intrude into clinical decision-making. But this simplicity comes with the cost of increased vulnerability to bias and confounding, particularly confounding by indication.^4,5 Confounding by indication occurs when exposure to treatment is associated with both measured and unmeasured qualities of the patient, such as the underlying severity of disease. If patients with more severe disease tend to receive a particular treatment, that treatment may falsely appear to cause worse outcomes. Development of better databases and methodologies, such as propensity scores and instrumental variables, has attempted to address this problem, but the randomized controlled trial remains by far the most trusted technique for eliminating confounding and bias.^3,4

Randomized controlled trials are not available to answer many clinical questions because of the effort and expense involved and the ethical problems created by randomization. Randomization intrudes directly into a clinician’s prescribing decisions. It means exchanging a personalized treatment decision for one made by chance. Under most circumstances, patients should give informed consent for research before being randomized. Even with informed consent, ethicists debate when random assignment can be ethical. One influential standard, clinical equipoise, holds that randomization can be ethical as long as no consensus exists in the clinical community that one treatment is better than the other. This is in contrast to the more stringent standard, personal equipoise, which calls for each individual investigator to have no treatment preference at all.⁶

These practical and ethical concerns may make some useful randomized controlled trials infeasible. Suppose clinicians want to know whether drug A or drug B is a more effective treatment for diabetes. In a network of outpatient clinics, the choice between A and B might be made hundreds of times a year, enough to support a well-powered clinical trial. However, approaching hundreds of patients to determine eligibility, recruit them, and obtain informed consent would be expensive and disruptive to clinical care. Furthermore, physicians might decline to participate because they want to make an individualized choice for many of their patients.

If the question is sufficiently important clinically or the financial stakes are high enough, these barriers will be overcome. However, for many questions of modest but real importance, the need to obtain informed consent and ensure equipoise may be enough to prevent anybody from conducting a randomized controlled trial. This fact has preoccupied clinical researchers for many years.⁷ The current sense that comparative effectiveness research is a crucial tool for policymakers to contain health care costs and clinicians to make evidence-based decisions only makes the issue more acute.² Ethical standards cannot be compromised, but any way to make high-quality clinical research easier while still fully protecting patients would be valuable.

PROMPTED OPTIONAL RANDOMIZATION TRIAL

The prompted optional randomization trial (PORT) is a comparative effectiveness clinical research design that allows clinical practices sharing a network of electronic medical records to design and efficiently conduct comparative effectiveness research that gathers data that may approach the quality of a traditional randomized controlled trial. PORT is an effort to develop a form of randomization that retains randomization’s statistically useful properties without incurring the practical and ethical challenges of conducting a typical randomized controlled trial.

The key technologies are a computerized order entry system and the routine capture of clinical outcomes. Computerized order entry systems are already used to prompt a physician to reconsider certain orders, for example, orders for drugs not on an institution’s formulary. The same software can be used to introduce an element of random chance into therapeutic decision-making. Whenever physicians order drug A or B, the computer makes its own random choice between the drugs. The computer then prompts physicians to consider changing their prescription, but only when a physician’s order and the computer’s random choice are discordant. Otherwise, physicians’ orders, which are identical to the randomly generated orders, stand.

For example, if a physician orders drug A, 50% of the time the computer will also choose A. No prompt will be displayed, and the physician prescribes A. If the computer chooses B instead, it displays a prompt to consider prescribing drug B instead of drug A. A physician who prefers drug A for the particular patient dismisses the prompt with a single click and prescribes A. However, a physician in personal equipoise (having no preference between treatments) endorses the change with a single click and prescribes B (Figure 1).

FIGURE 1—

Flow of events and effect of random prompting in a prompted optional randomization trial.

Because the prompt is easily dismissed, and physicians and patients sometimes have reasons to prefer one drug over the other, the association between the group to which patients are randomly assigned and the actual prescription is less than 100%. This design increases the probability that a patient will receive the randomly assigned treatment. The ability to measure a causal association between treatment and clinical outcome is attenuated, but it is not eliminated. If drug A controls glucose better than drug B, and physicians are at least occasionally willing to alter their prescriptions when the computer prompts them, an unconfounded association between the direction of the prompt and the clinical outcome will develop. In a large enough study, this association could reveal a causal difference between glucose control on A versus B.

STATISTICAL ADVANTAGES

PORTs draw upon the validity of a well-studied concept in clinical epidemiology—instrumental variable analysis.^5,8 A sufficiently strong association between the randomized prompt and the actual treatment can be analyzed in a way that eliminates the effect of confounders, both measured and unmeasured. The method requires a variable with three properties: it must be associated reasonably strongly with treatment assignment, it must not be directly associated with the outcome of interest, and it must not be associated with any potential confounding variables. In normal observational data, instrumental variables are rare and rarely meet these criteria perfectly. Random assignment in a traditional randomized controlled trial is an artificially created perfect instrumental variable because it is highly correlated with the actual treatment received but has no direct association with the outcome or any association with confounding variables.

PORT creates an instrumental variable similar to traditional random assignment except that the association between the group to which the patient is randomly assigned and treatment is weaker. The most significant price of a weakened association in this context is a loss of statistical precision: if the correlation is only 50%, the standard error of the estimate of treatment effect doubles. Such losses of precision can be overcome by increased sample size, however. The ability to eliminate the effect of unmeasured confounders addresses confounding by indication, the crucial weakness of observational research, and delivers many of the advantages of a standard randomized controlled trial.⁸

Another limitation of instrumental variable techniques is that the estimates of treatment effect they produce apply only to marginal patients, that is, patients whose treatment assignment is actually altered by the prompt. Some patients may be destined to get drug A or drug B regardless of prompt direction, for example, if some providers always use drug B rather than drug A for patients with more severe diabetes. In a traditional observational analysis, such an association between disease severity and treatment assignment causes confounding. In an instrumental variable design, confounding does not occur, because such patients will be evenly distributed between the randomly assigned groups. However, they will not contribute to any observed difference in outcomes between groups, and any difference that is measured will not be generalizable to them. Results from PORTs should be interpreted with this limited generalizability in mind.⁹

ETHICAL ADVANTAGES

Because they make random assignment easy to override, PORTs are much less ethically problematic than a traditional randomized controlled trial. This research design addresses the issue of equipoise decisively, because if a physician or patient prefers one treatment, it is trivially easy to override the prompt. The strong assumption of personal equipoise is justified in this research design, because the prompt only affects treatment if the physician and the patient truly do not care which drug the patient receives.

In addition, PORTs arguably make research informed consent unnecessary. Informed consent is a critical element of clinical research because it protects patient autonomy and allows patients to ascertain that the research accords with their own values and preferences. However, informed consent is a means to achieve these goals, not an end in itself.¹⁰ Others have argued that the requirement of informed consent to a particular research study could be fulfilled by a general consent for treatment if the study would not significantly diminish participants' autonomy or seriously increase their risk.⁷ In this design, physicians can freely exercise their judgment to give the patient the best available treatment, preventing any increase in risk or reduction in benefit. Patients can also freely exercise any treatment preference, protecting their autonomy. In short, clinical informed consent ensures that clinicians respect patients’ goals and values.

This, together with the finding that the research could not be practicably carried out without a waiver of informed consent and that the waiver would not violate participant rights, would fulfill the Common Rule’s core criteria for a waiver of informed consent to research.¹¹ However, clinicians who are in equipoise over drug A versus B for a particular patient and change their recommendation in response to a prompt would likely want to disclose to the patient that, instead of prescribing the drug they initially considered, they are prescribing another drug they believe is equally effective, which was suggested by a prompt from their ordering database. This would be driven by the need for informed consent to treatment rather than informed consent to research.

The ethical issues this research design raises are complex, and PORTs may be classifiable in multiple coherent ways—for example, as research versus not and as requiring research informed consent versus not. The most effective way to understand these ethical issues may be to study them by implementing a PORT with a full waiver of informed consent in consultation with patients and their advocates and under the close supervision of an institutional review board.

CONCLUSIONS

This proposal has limitations. The weaker the association between prompt and treatment, the less precise the estimate of treatment effect.⁸ In addition, if physicians override the prompt more often for particular types of patient, the results will not be as applicable to that group, limiting the generalizability of results. Finally, blinding is not possible. This is a significant weakness, but its importance depends on the details of the clinical question being asked. Objective outcomes, such as biomarker assays that are checked routinely in practice, are relatively unsusceptible to problems with bias related to lack of blinding.⁵

Despite these limitations, a PORT design may be much more efficient and ethically acceptable than a standard randomized clinical trial. It permits clinicians who may not share clinical equipoise on drug A versus drug B to override the prompt because of their clinical judgment. In addition, data describing the situations in which the prompt is overridden could inform studies of clinical decision-making.

Conducting a PORT could become as simple as obtaining institutional review board approval, notifying physicians of the study, asking the systems administrator to program the prompt, and later reviewing electronic medical records for the outcome of interest. The cost of such a study will be low. Although imperfect, data generated by this study design offer a new option for comparative effectiveness research at a time when such evidence is increasingly critical and resources to conduct studies are scarce.