Comparative Effectiveness Trials: Generic Misassumptions Underlying the SUPPORT Controversy

As emphasized by the Institute of Medicine,¹ the effectiveness of more than half of therapies used in clinical practice is unclear. High priority should be given to promoting comparative effectiveness (CE) trials, trials of alternative therapies performed to facilitate better informed and more evidence-based decisions by clinicians, patients, third-party payers, and policy makers.^2,3 In such trials, the effectiveness of different treatment options is assessed under usual clinical circumstances across a broad range of patients treated clinically with these therapies. CE trials provide essential information to improve outcomes in everyday clinical practice that cannot be obtained from trials performed to assess the efficacy of a new experimental therapy. Such trials are conducted in ideal or restricted circumstances; enroll a limited number of carefully selected, and often uncomplicated, patients; and thus generally yield an optimistic view of efficacy. In comparing alternative therapies already used in clinical practice, CE trials have no “experimental” arm and no “control” arm, and the potential risks of one treatment are the potential benefits of the other. Yet CE trials fall under the same regulatory requirements for trials of experimental interventions never previously administered to patients.

The appropriate regulation of CE trials was a central issue in a public meeting in August 2013 held by the Office for Human Research Protection (OHRP) in response to heated controversy regarding the SUPPORT (Surfactant, Positive Pressure, and Oxgenation Randomized Trial) published in 2010 by the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.⁴ This 20-center CE trial included 1316 infants 24 to 27 weeks’ gestation randomized to oxygen saturation goals of 85% to 89% or 91% to 95%, the upper and lower ends of the goal range (85%–95%) suggested by the American Academy of Pediatrics. Caregivers were blinded to the saturation goal by offset oximeters. The lower goal reduced severe retinopathy of prematurity (ROP) as hypothesized but unexpectedly increased deaths, resulting in no significant difference between saturation groups in the composite primary outcome of death or severe ROP. Neurodevelopmental impairment rates were unaffected.⁵

In March 2013, the OHRP posted on its Web site its determination that SUPPORT violated regulatory requirements, that the consent form should have specified “substantial risks,” and that the low saturation goal “could increase risk of … death.” Public Citizen, a long-standing public advocacy group, asserted that the increased risk of death with the low saturation goal was foreseeable, randomization to the 2 saturation goals was highly unethical, and the parents were deliberately misled by not specifying an increased risk of death for the low saturation group on the consent form. Public Citizen demanded the Department of Health and Human Services issue a public apology and stop other Network trials (actions that have not occurred). A class action suit was then filed charging the principal investigator at the University of Alabama, the members of its institutional review board, and other defendants with negligence, lack of informed consent, breach of fiduciary duty, and wrongful death. Although the plaintiffs have not been successful to date, litigation continues.⁶

Many nonclinical ethicists have sided with OHRP.⁷ Those who have defended SUPPORT include clinical ethicists, clinical investigators, clinicians, the editor of the New England Journal of Medicine, and the director of the National Institutes of Health.^8,9,10 In response, OHRP placed the compliance enforcement action on hold until after further consideration and its issuance of further guidance. In our view as participants in the August 2013 meeting and as SUPPORT investigators, the criticisms of SUPPORT rest on multiple misassumptions about CE trials. If unchallenged, these misassumptions could seriously undermine proper trials in any area of medicine important to advancing evidence-based care and outcomes. The misassumptions that we consider to be most important are addressed here.

Misassumption 1

Participation in randomized trials increases the risk of adverse outcomes.

To the contrary, systematic reviews indicate that the overall risk of adverse outcomes in randomized trials are not increased and may be reduced among participants compared with similar nonparticipants¹¹ because of increased attention to optimizing patient evaluation and supportive care. Likewise, the risk-adjusted mortality in SUPPORT was lower among participants than among eligible nonenrolled infants, although the difference did not reach statistical significance (relative risk = 0.88; 95% confidence interval = 0.73–1.06; P = .16).¹²

Misassumption 2

Treatment risks can be accurately assessed by persons without a clear understanding of the clinical and research issues.

The assertion that the increased mortality with an 85% to 89% saturation goal was clearly foreseeable is based on studies performed >50 years ago when saturation monitoring and mechanical ventilation were unavailable and no more than 40% to 50% oxygen was administered even to deeply cyanotic infants. These studies are irrelevant to current care and to SUPPORT in which saturation was monitored continuously, and mechanical ventilation and 100% oxygen were provided if needed to meet the saturation goal. Mortality can be increased even in term infants by a brief exposure to too much oxygen.¹³ The best evidence in contemporary cohort studies and randomized trials conducted before SUPPORT suggested that saturation goals of 85% to 89% and perhaps as low as 70% would not increase and might decrease mortality among small preterm infants while reducing ROP below that with a 91% to 95% goal.¹⁴

Misassumption 3

Major treatment hazards viewed as highly plausible after trial completion should have been reasonably foreseeable before trial inception.

By definition, the foreseeability of hazards must be assessed using information known beforehand. In any legitimate randomized trial, the information required for truly informed treatment decisions is unknown. Surprises are sometimes unavoidable and unforeseeable. Even with 3 interim assessments of the accruing data, an independent Data Safety Monitoring Committee using standard stopping rules recommended that the trial be continued.

Misassumption 4

A composite primary outcome that includes death indicates that the investigators expected an effect on mortality.

This misassumption underlies the assertion that the investigators deliberately misled the parents. Yet the primary outcome in trials of high-risk patients is often a composite outcome that includes deaths when no effect on mortality is expected. Such patients may die before they can develop the outcome the intervention is hypothesized to prevent (eg, severe ROP in SUPPORT). Death is then a competing outcome. Failure to account for differences in mortality would violate the intention-to-treat principle¹⁵ and can seriously bias the primary analysis. Had the primary outcome in SUPPORT been limited to severe ROP without including deaths, the primary outcome would have indicated incorrectly that the low saturation goal was superior.

Misassumption 5

The consent form should list virtually any plausible hazard.

As emphasized by OHRP, only the incremental risks of the research need be considered under the Common Rule.¹⁶ Moreover, listing any treatment hazard that might seem plausible despite evidence to the contrary would likely be misleading and distract from important known hazards.

The criteria for reasonably foreseeable hazards are undefined. We would suggest criteria such as the following:

1. biologically plausible hazards that have not been well evaluated in clinical studies (as assessed using criteria like the Grading of Recommendations Assessment, Development and Evaluation or US Preventive Task Force criteria), and

2. hazards that are at least marginally associated with the treatment (P ≤.10–.15) in a systematic review of relevant clinical trials or in the absence of such a review, ≥1 trials or well-performed cohort studies.

As the evidence was judged by the SUPPORT investigators and the IRBs in their 20 centers, mortality was not a reasonably foreseeable risk of the 85% to 89% goal. SUPPORT critics contend that the threshold for foreseeable risk should be low enough to have required listing death as a risk for this goal. If this logic is followed through to its conclusion, death would have been listed for the higher saturation goal as well. Although listing death as a hazard for both treatment arms might have satisfied critics of SUPPORT, we doubt that the parents would have been better informed or their wants and needs better met.

Moving Forward

Even when treatment risks are well known, the most desirable disclosure and consent process requires study of such issues as patient/surrogate wants and needs in emergent and routine circumstances, the effects of differing approaches to risk disclosure (including negative placebo [nocebo] effects¹⁷), and factors that augment the validity of informed consent.

The central problem for consent in SUPPORT was the very reason for the trial: uncertain risks and benefits of different saturation goals after saturation monitoring was introduced in the 1980s. Ironically, the long delay in conducting this and other important trials in all areas of medicine is due partly to systematically different requirements in patient care and CE trials for administering the same unproven, possibly harmful treatment. As Fost has emphasized, it is not plausible to presume that a patient would want a therapy never properly tested for safety or efficacy with no prior review and no monitoring but would object to the same treatment being given with all the safeguards of controlled trials.¹⁸

In moving forward, the regulatory requirements for CE trials would be better based on the foreseeable risks than on simplistic and outmoded distinctions between practice and research.¹⁹ As Faden, Beauchamp, and Kass recently noted, “in a mature learning health care system … some randomized comparative effectiveness studies may justifiably proceed with a streamlined consent process and others may not require… consent at all.”²⁰ Although regulatory refinements are being developed to promote a learning health care system, IRBs can exercise the discretion allowed under current regulations to facilitate important CE trials involving minimal or reduced risk.²¹

Glossary

CE

comparative effectiveness

OHRP

Office for Human Research Protection

ROP

retinopathy of prematurity

SUPPORT

Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments