Reflections on FDA Draft Guidance on Bayes Methods in Trials

Introduction

The United States Food and Drug Administration (FDA) provides invaluable leadership in defining and enforcing evidentiary standards to protect research integrity. Their rigorous and impartial approach is critically important to ensuring reliable and robust evidence for informing clinical and public health practice and policy.

The FDA recently released draft guidance regarding using Bayesian methods in clinical trials.¹ We commend the FDA for its’ efforts to educate the research community on the fundamentals of Bayesian methods that have demonstrated value in settings such as diagnostic medicine, prediction, early phase drug and device trials, and rare diseases. The use of Bayesian methods in late-phase or confirmatory clinical trials has generally been limited to supplementary analyses, given recognition that Bayesian methods could be implemented in a manner that would compromise evidentiary and integrity standards and the reliability of results through: (i) concession of the benefits of randomization as Bayesian methods combine external (prior) information with high-quality randomized data; (ii) loss of objectivity by incorporating sponsor or investigator-specific priors; and (iii) reduced robustness through reliance on strong and sometimes unverifiable assumptions, (see Table 1)

Table 1:

Protecting Scientific Integrity and Evidentiary Standards in the Wake of the FDA Draft Guidance on Bayes Methods in Trials: Key Points

Clinical trials should be designed and conducted in a manner to ensure the replicability, integrity, and reliability of evidence for disease treatment and prevention.

Bayesian methods have been useful in settings such as diagnostic medicine, prediction, early phase drug and device trials, and rare diseases.

Use of Bayesian methods in late-phase or confirmatory clinical trials has generally been limited to supplementary analyses, given recognition that their implementation can compromise evidentiary and integrity standards and the reliability of results through: (i) concession of the benefits of randomization through their inclusion of external (prior) information; (ii) loss of objectivity by incorporating sponsor or investigator-specific priors; and (iii) reduced robustness via reliance on strong and sometimes unverifiable assumptions.

Transparency of Bayesian analyses is enhanced by quantifying the component influences contributing to the posterior that is the post-trial state of belief regarding treatment effects. These components include the prior or pre-trial state of belief and the observed trial data. Analyses of trial data without imposing informative priors should be standard supplementary analyses.

FDA guidance that provides substantive insights about the proper role and implementation of Bayesian methods is critically important, especially for confirmatory trials.

The fundamental structure of Bayesian analysis provides a method for combining preexisting information—the prior probability distribution—with observed data, to yield a posterior probability distribution that represents the updated state of the prior information in light of the observed data. Statistical conclusions or estimates can then be based on the posterior distribution. Key risks of Bayesian analysis, as indicated in Table 1, include those relating to the origin, quality, and potential bias associated with the prior.

In this Perspective, we provide insights into how the use of Bayesian methods in clinical trials could compromise research replicability, integrity, and reliability. We also discuss concerns in framing the recent FDA draft guidance on Bayes methods as “modernizing statistical methods” and emphasize the importance of having FDA guidance that provides substantive insights about the proper role and implementation of Bayesian methods.

The Primacy of Randomized Evidence

Randomized controlled trials (RCTs) provide the pinnacle of evidence evaluating medical interventions and resolving medical controversies. Randomization avoids systematic imbalance of confounding factors, known or unknown, and thereby provides safeguards against the limitations of our knowledge and measurement tools. Eligibility criteria ensure RCTs study a well-defined population, clarifying the generalizability of findings. These methodological attributes provide the foundational theory for valid statistical inference, producing invaluable benefits such as unbiased treatment effect estimates, correct confidence interval coverage probabilities, and error-rate control during hypothesis testing.

Yet, to preserve the benefits of randomization, its integrity must be protected through sound design, conduct, analyses, and interpretation². Without proper safeguards, RCTs are vulnerable to the same biases as observational studies. Some methods may compromise the integrity of randomized evidence whether the primary analysis is Bayesian or frequentist, e.g., use of post-randomization covariates, per protocol or on-treatment analyses, response-adaptive randomization³, or incorporation of external, historical, or non-randomized data. The integrity of randomization could be further compromised by incorporating prior beliefs, as occurs in the implementation of Bayesian methods.

Altering evidentiary standards is sometimes necessary. In some rare diseases, control groups may be non-randomized given challenges of enrolling trial participants. Alternatively, to enable randomization, the false-positive error rate may be set higher than the traditional two-sided 0.05 level in a deliberate and transparent manner. In these settings, assumptions and resulting uncertainties are acknowledged when describing conclusions.

Bayesian approaches, especially with the use of informative prior distributions, may introduce biases into treatment effect estimates and influence error rates in RCTs^4,5. While simulation studies may be used to evaluate bias and error rate control, they cannot cover the full range of possibilities. When using Bayesian methods, it remains important to assess frequentist-operating characteristics to ensure that evidentiary standards are being maintained.

Objectivity and the Prior

A prior or pre-study assumption about the distribution of treatment effects must be defined when implementing Bayesian analyses. Objectivity in constructing the prior is crucial for distinguishing scientific from wishful thinking and avoiding conflation of evidence-based belief with desired effects. Many RCTs have overturned expert opinion and conclusions derived from observational data, underscoring the need for caution when selecting a prior distribution. Scientific objectivity and avoidance of bias may be maximized by using uninformative priors, often yielding results equivalent to frequentist analyses⁵.

RCTs require pre-specification of primary endpoints and analysis methods to address multiplicity. Without pre-specification, there is increased risk of mistaking chance for true signal, leading to an increase in false-positive conclusions and random-high biased estimation. Similarly, when using evidence external to the RCT to develop a prior, objectivity is important when selecting that evidence. One must recognize the powerful influence of ‘cognitive dissonance’; if the investigator’s goal is to ‘establish’ rather than to ‘determine whether’ an intervention is safe and effective, then biased selection of evidence upon which the prior is based is likely. These concerns are enhanced if selection is post-hoc, i.e., when the favorability level of external evidence regarding intervention effects is known. Priors must be constructed based on relevant, high quality, objectively selected external data, implementing a process focused on obtaining unbiased estimates of treatment effect for the trial population and accurate estimates of associated standard errors.

Well-intended approaches to the development of a prior do not completely overcome challenges with the non-standardized and selective nature of this process. Ultimately, priors are not universal⁶ and may meaningfully differ among sponsors, regulators, payors, institutional review boards, patients, or data monitoring committees aiming to be fully objective and independent of sponsor-influence including prior beliefs⁷.

Robustness

Statistical methodologists historically have favored methods requiring fewer assumptions, especially when assumptions are untestable, difficult to verify, or easily violated, in order to maximize robustness. In contrast, many Bayesian analyses are model-based and validity of results is dependent on assumptions underlying those models.

Consider the strategy of using estimates of treatment effects in adults to generate prior distributions to be used in the analysis of pediatric RCTs intended to estimate treatment effects in children, an example described in the FDA’s guidance. Such extrapolation from adults to children is risky given the age-relatedness of physiological and developmental profiles. For example, the STARTS-2 trial evaluated sildenafil in children with pulmonary arterial hypertension and was conducted under an FDA pediatric written request. Sildenafil increased the death rate by 3.95-fold (95% CI, 1.46 to 10.65) in children, in high dose relative to low dose, although this effect was not seen in the adult populations⁸. Considerations around bridging from adults to children were addressed more rigorously during FDA’s ADEPT 7 Workshop, including with use of validated bridging biomarkers⁹.

Transparency and the Peer Review Process

The issues above highlight the importance of appropriate interpretation and evidence contextualization when reporting Bayesian analyses of clinical trials. Authors and journal editors must ensure transparency in design assumptions, construction of the prior and posterior, and the reduced reliability of results when randomization integrity is not preserved, in order for the impact on practice to be proper.

As mentioned above, construction of the posterior distribution that represents a post-trial state of belief regarding treatment effects begins with selection of a prior, the pre-trial state of belief regarding treatment effects. The prior is combined with data from the trial to produce the posterior. Transparency is enhanced by analyses that quantify the relative influences of the prior and the trial data on the posterior. The FDA draft guidance effectively discusses quantifying the influence of the prior and describes metrics to compare candidate priors during its selection. Analyses of clinical trial data without imposing priors, i.e., using frequentist methods and potentially Bayesian methods with non-informative priors, clarify the contributions of the prior and the trial data to the final results and should be standard supplementary analyses to Bayesian analyses.

Discussion

The FDA news release regarding the recent FDA Draft Guidance on Bayes Methods in Clinical Trials framed the guidance as “modernizing statistical methods” – implying that fixing outdated methods would address high costs and long timelines in drug development. On the contrary, we emphasize that currently used methods that preserve objectivity and benefits of randomization are not outdated, old school, or closed-minded. They are principled, based on scientific fundamentals for protecting integrity and ensuring robustness of scientific conclusions. The commitment to achieving these objectives should be protected, not replaced. Furthermore, compromising evidentiary standards threatens to slow, rather than accelerate, meaningful progress. More nuanced guidance and objective public messaging will help prevent mischaracterization and the propagation of misinformation, as well as inform appropriate selection of analytical methods including Bayesian methods or robust alternatives.

Fundamental goals of clinical research must include protecting and ensuring reliable, robust, and efficient evidence for disease treatment and prevention. Responding to pressures to reduce cost, sample sizes, and duration of clinical trials may yield studies with lower replicability, integrity, and reliability. In turn, since reliability is influenced by bias as well as precision, and increases in bias are not as readily discernable, improved precision should not be pursued at the expense of increased risk of meaningful bias.

FDA guidance that meaningfully informs how clinical trials should be designed and conducted to achieve replicability, integrity, and reliability of their results while balancing resource constraints, is of critical importance. This is especially true in late phase or confirmatory clinical trials where there is particular need to protect the integrity of randomization when it is implemented, to maximize objectivity of trial designs and to improve robustness by reducing dependence on assumptions. FDA guidance that provides substantive insights about the proper role and implementation of Bayesian methods, especially regarding use of informative priors in clinical trials intended to provide confirmatory evidence, is needed to ensure the most reliable and robust evidence to inform patient care.