Bayes and the Evidence Base: Reanalyzing Trials Using Many Priors Does Not Contribute to Consensus

Recent years have seen an increase in Bayesian reanalyses of clinical trials. Proponents argue that this helps the interpretation of trial results (1–3). Bayesian inference has also attracted skepticism, most of which is poorly justified (4). The stale criticism that subjectivism is at odds with the scientific method is misguided: conclusions based on the frequentist paradigm are equally dependent on the prior likelihood of hypotheses, albeit less transparently so (5). Nonetheless, we believe that some apprehension against Bayesian reanalyses is justified. Our concerns are not of a statistical nature, nor do they relate to studies that are designed around Bayesian methods from the outset. Rather, we are concerned that the many-priors mode of Bayesian reanalyses disregards the communal aspects of evidence-based medicine, in which consensus among clinicians is gradually developed through the accumulation of generalizable study data.

The Many-Priors Paradigm Frustrates Consensus Building

A key reason for conducting randomized controlled trials is to resolve uncertainty and disagreement in the clinical community (6). Equipoise derives not from the uncertainty of any individual but from the uncertainty in the community of expert clinicians: when some experts believe that a treatment is beneficial whereas others believe that it is harmful or wasteful, successful clinical trials forge consensus among reasonable parties who previously disagreed (7, 8).

Unfortunately, the results of many randomized controlled trials are not convincing. So investigators increasingly turn to Bayesian methods to reanalyze trials using a range of priors as representations of the a priori opinions in the expert community (9–11). Even though this can be insightful, there is no widely accepted framework to translate the spectrum posterior results into a consensus viewpoint about treatment efficacy. For example, a Bayesian reanalysis may indicate 98% probability of a clinically important benefit under strongly enthusiastic prior assumptions but also 37% probability of a negligible effect or harm under strongly skeptical prior assumptions (10). Parties with opposing suspicions about the treatment can each see their position strengthened by the results, especially if the intervention is costly or associated with severe adverse effects.

Bayesian methods are also used to derive probabilities for different treatment effect sizes. The product of evaluating multiple effect sizes using multiple priors is an often very large set of posterior probabilities. It is common for reports to contain as many as 30–80 distinct probabilities for a single primary endpoint (9, 10). This is highly transparent yet utterly perplexing to impartial readers. When there is no guiding principle to choose which of these many estimates is most relevant, it appears as if each can choose his own. The complexity is further increased by the heavy use of technical graphs and jargon, requiring readers to decipher probability density curves and priors expressed on the log-odds scale (12). Such representations of trial results are transparent only to those well versed in statistics.

Finally, the hyperquantitative way of presenting results distracts from the qualitative assessment that should be central to every trial interpretation. A skeptical prior (i.e., an expectation of small effects) does not provide a safeguard against bias or poor generalizability. Because bias distorts the very way in which the data are generated, a large and biased trial will easily overwhelm a strong skeptical prior.

In all, it is not surprising that the “pick your own prior” paradigm of Bayesian reanalyses appears vacuous to many. With strong data, no reasonable prior will make a material difference to the interpretation of a trial. With weak data, using different priors will only entrench disagreeing parties in their respective beliefs. In both cases, no progress toward consensus is made, while the potential points of disagreement grow exponentially in the cross-evaluation of multiple priors, multiple effect sizes, and multiple cutoff probabilities. As one critic has argued, it would be much simpler just to urge clinicians to enlist their own beliefs when faced with unconvincing data (13).

Yet the use of subjective priors is not necessarily at odds with the communal nature of evidence-based medicine. The idea of using priors that represent different beliefs in the community of experts (“a community of priors”) was born from the view that consensus arises when the data are strong enough to convince even a reasonable adversary (14–16).

The Adversarial Perspective

It was the perspective of the hypothetical other, not the quantification of one’s own beliefs, that motivated early proponents of Bayesian methods for clinical trials.

By using an adversarial perspective, a Bayesian reanalysis can be used to specifically assess whether the available data are strong enough to forge consensus among reasonable and well-intentioned experts (6). If the point estimate of a trial indicates benefit, regardless of the P value, consensus can arise only if the data are strong enough to convince those with reasonable pessimistic prior expectations. Conversely, if the trial results indicate harm or a negligible effect, it is the position of the reasonable optimists that requires careful evaluation (14–16).

Focusing Bayesian reanalyses squarely on the adversarial prior reduces the overwhelming amount of posterior probabilities in study reports and limits the potential points of disagreement to two important questions: is the prior representative of a “reasonable adversary” of the observed effect, and what amount of posterior uncertainty about the treatment effect is acceptable? The answers to both questions are rooted in factors such as the importance of the outcome, previous evidence, the costs and harms of the intervention, and a qualitative assessment of risk of bias and generalizability.

The adversarial principle may also help overcome an important asymmetry in the published literature. Bayesian methods are now used to reanalyze trials with borderline nonsignificant results, but reanalyses of trials with borderline significant results are suspiciously less prevalent. This one-sidedness is unnecessary and wasteful, as the interpretation of small trials with spectacular results may change most from an adversarial perspective (1).

Conclusions

Although personalized decision making is at the core of evidence-based medicine, the evidence base itself cannot be made contingent on personal beliefs. No critical care clinician can practice in isolation from his colleagues or without regard for cost-benefit considerations at the societal level. Yet the many-priors mode of reporting post hoc Bayesian analyses makes it seem as if each can choose his own treatment effect. This is confusing, overly complicated, and unnecessary. The discourse about the effectiveness of a treatment is better served by focusing on a single question: can the matter be considered settled, or is the evidence too weak to reach consensus?