From Efficacy to Effectiveness: Evaluating Psychedelic Randomized Controlled Trials for Trustworthy Evidence‐Based Policy and Practice

ABSTRACT

The recent review of a new drug application for MDMA‐assisted therapy for posttraumatic stress disorder by the United States' Food and Drug Administration (FDA) highlighted epistemological and methodological challenges for evidence assessments. Similar challenges will also be faced in reviews of other compounds in early‐ and late‐stage development, like psilocybin for depression. The regulatory demand for two successful phase 3 randomized controlled trials (RCTs) seems problematic, given a current lack of agreement on what constitutes “success”, particularly when psychoactive drug administration is concomitant with (psycho)therapy. These complex arrangements challenge the internal validity of estimated average treatment effect through comparison with conventional control conditions. This paper reviews the assumptions behind RCTs' current “gold‐standard” status in the hierarchy of evidence‐based medicine (EBM). Recapitulating known epistemic limits of randomization and blinding, it emphasizes the urgent need to avoid the extrapolation fallacy. The resulting argument is that the degree of trustworthiness that efficacy—reported in RCTs—will reliably predict effectiveness—in target populations outside RCTs—depends on what type of psychedelic treatments will be regulated. If “stand‐alone” drugs for large‐scale prescription and consumption, trustworthiness should be graded low. On the other hand, for regulation of drug‐assisted (psycho) therapies, the degree of trustworthiness can be considered high. The reason being that these two treatment approaches are based on different causal claims with distinct external validities. Therefore, careful assessment of support factors in each is recommended to prevent detrimental consequences, from potential rejection of effective therapies up to medical reversal of eventually approved drugs.

Abbreviations

ATE

average treatment effect

EBM

evidence based medicine

EBM+

evidence based medicine plus

EBPP

evidence based policy and practice

EMA

European Medicines Agency

FDA

Unites States Food and Drug Administration

PAP

psychedelic‐assisted psychotherapy

PAT

psychedelic‐assisted therapy

PTSD

posttraumatic stress disorder

RCT

randomized controlled trial

1. Introduction

After being largely dismissed during the last quarter of the 20th century, psychedelics were increasingly reappraised in clinical trials during the first decades of the 21st century. This generated hope amidst an innovation stall in clinical psychopharmacology, concomitant with high unmet medical need [1]. For example, esketamine was approved in many jurisdictions, including the USA and Europe [2, 3], while two phase 3 trials with “MDMA‐assisted therapy” for posttraumatic stress disorder (PTSD) were published [4, 5]. Additionally, multiple phase 3 trials are ongoing with psilocybin for the treatment of major and treatment‐resistant depression, and LSD for generalized anxiety disorder is in development. Finally, there is also an increasing number of phase 1 and 2 studies with similar compounds with a wide range of chemical, pharmacological, and psychoactive properties. This second biomedical wave, after initial research in the 1950s and early 1960s, unfolded in excessive enthusiasm and hype, especially as a for‐profit market of start‐ups, fueled by specific investment funds, started claiming intellectual property over these once discarded compounds [6, 7].

More recently, the enthusiasm has been balanced by ethical and safety concerns beyond the scope of the current paper [8, 9, 10, 11, 12, 13, 14], which focus on increasing criticism related to efficacy estimates. This focus on efficacy seems to have dominated two important events, the “Advancing Psychedelic Clinical Study Design” organized by the Reagan‐Udall Foundation for the FDA in January 31st and February 1st, 2024; and the second in April 16th and 17th, 2024, at the European Medicines Agency (EMA), entitled “EMA Multi‐Stakeholder Workshop on Psychedelics—Towards an EU Regulatory Framework”. In both events discussions were centered around safety, efficacy data and how to improve trial designs to account for interactions between (psycho) therapy and psychopharmacology. Finally, a few years after granting MDMA‐assisted therapy “breakthrough therapy” status, and following a psychopharmacology advisory committee meeting in June, the United States of America Food and Drug Administration (FDA) did not approve a new drug application (NDA) for MDMA‐assisted therapy for PTSD in August 2024, likely based on similar reasons regarding potentially biased efficacy estimates.

To better understand these controversies, I first review the criticisms of psychedelic randomized controlled trials (RCTs), and then examine the philosophical assumptions and epistemological premises usually left implicit in these criticisms, and also in regulatory guidance documents. This exercise, carefully conducted and based on philosophy of evidence in medicine, exposes how and why a reliable assessment of the current evidence‐base for clinical applications of psychedelics requires consideration of recent conceptual and practical advances in evidence‐based medicine (EBM) and evidence‐based policy and practice (EBPP). Particularly important is the evolution from the concept of a hierarchy of evidence to evidential pluralism in EBM+. Thus in EBM+, it is considered necessary to integrate evidence from clinical trials with mechanistic evidence for assessing causal claims and to reliably extrapolate results from trial samples to larger and different target populations.

This approach reveals that the two distinct treatment models currently advocated for—psychedelic‐assisted (psycho) therapy (usually referred to as PAT or PAP) on one end, and psychedelic drugs, on the other—make two different causal claims. The reliability of currently published RCTs and their assessment as sufficient or not for drug approvals depends on the purposes of each and the conditional epistemic premises they rely on.

1.1. Review of Current Criticism

With a few early papers [15], criticism regarding efficacy estimates from psychedelic RCTs has increased since 2020. The most frequent concern regards unblinding (or unmasking) of treatment allocation, and how expectancy can bias results (for a recent review, see [16]) (In spite of important calls to avoid the “blinding” terminology in respect for people with disabilities, I opted to keep with the most widely used term, and also to avoid confusion with the problem of counteracting mechanisms known as the problem of “masking.” [17]). This is because participants in experimental groups almost certainly realize they received a potent psychoactive compound, and therefore may have high—or positive—expectancy, while participants in the control groups most likely suspect they did not receive the drug, and may therefore be frustrated and have low—or negative—expectancy [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]. Nevertheless, it is also important to note that participants who guess they are in the experimental group may also then have negative expectations due to negative preconceptions about the drug, and therefore biases can go in both directions.

The most comprehensive critique was presented by Prof. Suresh Muthukumaraswamy from the University of Auckland, New Zealand. The main argument is that biases can happen in RCTs throughout planning, execution, analysis, interpretation of data and publication of results. Most of these are neither new nor specific to psychedelics, but are more likely to happen with potent psychoactive drugs whose effects are quite extraordinary and unmistakable by those who experience them, and also to observers. This is due to the drugs' characteristic psychoactive effects likely never going unnoticed by people who take them (especially in medium to high doses), and also by research staff due to characteristic overt (and covert) emotional and behavioral manifestations. A nonexhaustive list includes profusely smiling, laughing, screaming, crying and sobbing, as well as unusual and wide range of movements, bodily postures and facial expressions, etc. The main worry is that expectancy can, and likely did, inflate estimates of average treatment effects (ATEs), threatening the reliability of causal inferences. This would thus risk misinterpretation of underlying causes: “It is theoretically possible that the entire estimated effect size of these interventions could be accounted for by these confounds which would mean that they provide no evidence for efficacy” [32, 33].

1.2. From Hierarchy to Evidential Pluralism

Implicit in these criticisms is the larger context of EBM, a so‐called “paradigm” which emerged during the 1970s and 80s, gaining prominence in medicine during the last two decades of the 20th century [34], precisely when therapeutic uses of psychedelics were largely under‐researched. According to the highly influential principles of EBM, RCTs are considered “the gold standard” and are placed at the top of a hierarchy of evidence. Above RCTs there are other methods which strictly depend on them, such as systematic reviews and meta‐analyses. In EBM's parlance, RCTs are the only method capable of gathering unbiased evidence, and therefore policy decisions should be based, if not exclusively, predominantly on them, to the neglect of all other sources of evidence. This view has dominated for decades, as the following examples illustrate:

The only source of reliable evidence about the usefulness of almost any sort of therapy […] is that obtained from well‐planned and carefully conducted randomised […] clinical trials. Tukey, 1977 [35]

Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the ‘gold standard’. Sackett, 1996 [36]

The strength of an RCT is that […] the study's design controls for all the factors, other than the planned intervention, that might lead to different outcomes. This is terrifically powerful. Prasad and Cifu, 2019 [37]

Claims of this kind originate from common misunderstandings about RCTs' actual epistemic strengths, such as assuming that ATEs ensure causal inference [38]. In pharmacology this is usually misinterpreted as proof that the drug is the cause of the ATE in the trial sample because all other factors were supposedly controlled for by random allocation and blinding. However, this is not so. The high internal validity of RCTs is defined as “the validity of inferences about whether observed covariation … reflects a causal relationship” [39]. This is what gives RCTs their epistemic power, as the method is deductive. That is, if all the premises are met, the conclusion logically follows from the results, in the ideal scenario [40]. However, in practice, there are many premises that must be met for RCTs to actually approximate the ideal case. For example, Cook listed 26 assumptions that should be met for RCTs to grant causal inference, grouped as relating to the concepts of internal validity, statistical conclusion validity, construct validity and external validity. As it is very rare, or never the case, that all are met for any single trial, or even a small collection of trials, a gold standard status is unwarranted [41].

The first reason for this is because balance through random allocation (not to be confused with random sampling from a population) is not actually granted. Despite being frequently said to guarantee that the comparison groups in a single trial are (perfectly) balanced at baseline, and therefore equivalent except for the variable under study, single random treatment allocation cannot actually grant unbalanced groups to all relevant variables. By the laws of probability, balance would only be achieved with many repeated randomisations, indefinitely. For any single allocation, unbalance can exist, as throwing two groups of dice once, or a few times only, does not guarantee you'll get similar results. Only with many repetitions the tendency will be for balance [38, 42]. Medical research does not, and cannot, indefinitely repeat experiments, and therefore single randomized experiments, no matter how rigorous, are never entirely free of biases. The larger the number of potential confounders, the worse it gets. This is sometimes referred to as “unhappy randomization”, and rerandomization based on postrandomization verification of confounders is often promoted as a remedy. However, only known confounders can be checked for postrandomization, but unknown confounders must never be ruled out. Therefore, RCTs can always be biased in principle, and frequently are in practice [38, 42]. As Nobel Prize winner (2015) Angus Deaton and Nancy Cartwright, recipient of the Martin R. Lebowitz Prize for Philosophical Achievement (2017) and the Carl Gustav Hempel Award (2018) put it: “If we were to repeat the trial many times, the over‐representation of the unbalanced causes will sometimes be in the treatments and sometimes in the controls. The imbalance will vary over replications of the trial, and although we cannot see this from our single trial, we should be able to capture its effects on our estimate of the ATE from an estimated standard error. This was Fisher's insight: not that randomization balanced covariates between treatments and controls but that, conditional on the caveat that no postrandomization correlation with covariates occurs, randomization provides the basis for calculating the size of the error. Getting the standard error and associated significance statements right are of the greatest importance; therein lies the virtue of randomization, not that it yields precise estimates through balance” [38]. To summarize, the common assumption that any single RCTs yield unbiased results through balanced groups is wrong, and it would be just through many repetitions that randomization would approach unbiasedness. The attempt to solve these issues through meta‐analysis of RCTs is fraught with difficulties, first and foremost because meta‐analysis are not required by regulatory authorities for market authorizations. Further issues include, not limited to, massive publication bias, with just 53% of trials published in full [43], as well as epistemic and subjective issues on statistical choices in meta‐analytical procedures known for a long time and still unsolved [44, 45].

The second reason for imperfect internal validity is the caveat that after randomization imbalances can occur, and therefore other remedies may be necessary. That is what blinding aims at: controlling for postrandomization confounders. However, contrary to randomization, blinding has a poorly studied history, which contributes to it being rarely questioned [46]. Yet, inspection of the 10 most cited RCTs in biomedicine (each with > 6500 citations at the time of analysis), revealed that only two (20%) were successfully blinded [47]. In a sample of 200 randomly selected RCTs in five of the top medical and four of the top psychiatric journals, only 7% and 9%, respectively, were found to report blinding assessments, while blinding had been lost in 60% of those who tested for it [48]. Similarly, from a random sample of 300 studies from PubMed, only 24 (8%) explicitly reported risk of unblinding [49]. In a broader sample of the literature, randomly selecting 1599 studies, reporting of blinding assessments was found between 2% and 8% [50], while in the hundreds of antidepressant trials in the last two decades it was between 5% and 7% [51]. Finally, in a meta‐analysis of psychiatric disorders, it was reported that only 66 out of 2467 studies reported blinding assessments, and in this very small sample (less than 3%) blinding was preserved [52]. Philosopher John Worral thus seems correct and precise when concluding that blinding “is an ideal more often honoured in the breach than the observance” [53]. This is not to say that the blinding methodology and the confounders it aims to control for do not matter, but if the method frequently fails, besides poor practices it is possible that the underlying epistemic premises are weak. In the case of psychopharmacology and psychiatry, it seems clearly unreasonable to expect people to experience improvements in subjective feelings, especially through the use of psychoactive substances, without becoming consciously aware of these changes in themselves. Furthermore, as blinding became paramount to claims of validity (or lack thereof), its epistemic premises became pervasively misunderstood. As statistician Stephen Senn put it: “The whole point of a successful double blind trial is that there should be unblinding through efficacy. […] If the treatment is effective the guesses will distribute unequally between the arms of the trial and the trial will then be declared ‘not blind’” [54]. Therefore, the usual calls to invalidate studies due to unblinding suffer not only from the fact that reporting unblinding in biomedicine is rare (< 10%), but that demanding studies to be unblinded from beginning to end is epistemically flawed, particularly in cases of subjective symptoms. In the case of psychiatry and psychopharmacology, this is because the conditions are subjectively felt and because drugs specifically modulate functions of the central nervous system including interoception, proprioception, self‐awareness and consciousness. In these cases it is important not only to measure and report blinding assessments and rates of unblinding, but to clearly distinguish between malicious and benign unblinding, which are related to side effects and efficacy, respectively [55, 56].

Despite their important strengths, the weaknesses of randomization and blinding undermine the frequent overconfidence in RCTs' internal validity. And this is furthermore conflated with a second, and more widespread, problem: the assumption that an unbiased ATE estimate in a RCT, being causally determined due to supposed control of all other confounders, grants exportability or generalizability of this cause beyond the RCT [38]. This is the problem of external validity, defined as “the validity of inferences about whether the cause‐effect relationship holds over variation in persons, settings, treatment variables, and measurement variables” [39]. While internal validity refers to the degree of certainty that any outcome was causally determined, external validity refers to the degree of confidence that similar outcomes may be obtained elsewhere, in different people, historical times and/or circumstances (or the “truthfulness of the result” and the “applicability of the results”) [57]. As Cartwright put it: “For policy and practice we do not need to know ‘it works somewhere’. We need evidence for ‘it‐will‐work‐for‐us’ claims: the treatment will produce the desired outcome in our situation as implemented there. How can we get from it‐works‐somewhere to it‐will‐work‐for‐us?” [58]. In EBM and policy making based on it, it is quite common for internal validity to be explicitly prioritized while external validity is disregarded [59].

Discernment between these two concepts requires a careful distinction between efficacy, which is measured in RCTs, and effectiveness, or what needs to be estimated for populations beyond trial samples [60]. It is effectiveness, then, that matters most for EBPP. However, these points seem to be missing in the FDA's draft guidance “Psychedelic Drugs: Considerations for Clinical Investigations Guidance for Industry” (FDA, 2023. Psychedelic Drugs: Considerations for Clinical Investigations. Avaliable at https://www.fda.gov/regulatory‐information/search‐fda‐guidance‐documents/psychedelic‐drugs‐considerations‐clinical‐investigations [last accessed May, 1st, 2024]), where definitions are not provided and “efficacy” appears five times and “effectiveness” six times; and also in the EMA's draft “Guideline on clinical investigation of medicinal products in the treatment of depression” (https://www.ema.europa.eu/en/documents/scientific‐guideline/draft‐guideline‐clinical‐investigation‐medicinal‐products‐treatment‐depression‐revision‐3_en.pdf), which briefly mentions psychedelic drugs, using “efficacy” 60 times while never employing the term “effectiveness”.

The assumption that efficacy unproblematically translates to effectiveness, that is, that efficacy is generalizable to other patient populations in other settings and contexts, with different age, gender, race, comorbidities, cultures and socioeconomic status is known as fallacy of free extrapolation [61], extrapolator's fallacy [62] or fallacy of simple extrapolation [63]. This is intrinsically related to the policy challenges of how to appropriately use results from RCTs [38], which is a particular instantiation of the distinction of facts in particular situations from general truths [64]. Historically tragic examples of the extrapolation fallacy are the pervasiveness of biomedical studies with mostly male and white participants, which persists despite repeated calls from feminist and racial movements for decades [65]. RCTs with only white men have balanced groups regarding gender and ethnicity and can therefore have high internal validity, but do little or nothing to grant extrapolation of results to women and people of color. As Jon Williamson observed: “Any adequate causal epistemology needs to explain how extrapolation is possible and needs to clarify the logic of extrapolation.” [62].

Thus, RCTs are undeniably important methods in evidence assessments, especially for simple interventions where they are especially well suited, such as historical cases including streptomycin for tuberculosis, the polio vaccine, and treatments for rheumatic fever [66]. But RCTs also have important epistemic limitations, especially in prolonged and complex interventions [67, 68]. The first limitation is detecting harms, given that sample sizes calculated to assess efficacy (usually in the range of hundreds to thousands of patients) are usually too low to reliably and consistently detect rare and very rare adverse events (one in a thousand or one in hundreds of thousands, respectively). Secondly, because of the tradeoff between internal and external validity. Despite their ascendance to, and consolidation at, the top of EBM's rigid hierarchy, these limitations were clearly recognized by the early pioneers and proponents of RCTs:

Any belief that the controlled trial is the only way would mean not that the pendulum had swung too far but that it had come right off the hook. Sir August Bradford Hill, 1966 [69]

Between measurements based on RCTs and benefit … in the community there is a gulf which has been much under‐estimated. A L Cochrane, 1971 [69]

These epistemic limitations undermine common practices of relying solely on RCTs for causal inference, and led to serious challenges to their “gold standard” status and epistemic authority in evidence hierarchies and policy deliberations [47, 50, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]. These unfolded in the EBM+ movement (www.ebmplus.org), grounded in the Russo‐Williamson Thesis (RWT) and evidential pluralism, stating that to determine causality, evidence of mechanisms is needed in addition to evidence of differences from randomized clinical trials [74, 76]. This important development proposes that causal assessments in medicine require not only evidence of statistical correlations from (randomized) experiments, but also mechanistic evidence. In this expanded epistemology, making all evidence explicit and being explicit about their use in evidence evaluations is essential, and this includes evidence of mechanisms.

1.3. Towards Reliable EBPP for Psychedelic Therapies

The incorporation of evidence of mechanisms into medical assessments is contentious, given many known cases of problems in treatment implementation and scaling up because of flawed mechanistic reasoning, which were central to the establishment of the hierarchy of EBM privileging RCTs and leaving mechanistic reasoning at the bottom [34, 74, 75, 76]. However, the difference between mechanistic reasoning and mechanistic evidence matters. As Miriam Solomon summarized it: “Incompleteness is the consequence of there being […] complex interaction of multiple mechanisms in a chaotic and multidimensional system. There are possible hidden mechanisms everywhere in mechanistic stories, despite an easy impression of narrative or causal completeness” [76]. Extra care should also be taken with different meanings commonly attributed to mechanisms in neuroscience, with the distinction between reductive and nonreductive mechanisms being particularly important. In the nonreductive sense, mechanisms are to be found not only in the micro‐level phenomena of molecular and cellular biology, but can also include behavioral, psychological, relational and social factors [77].

Therefore, to avoid repeating pre‐EBM mistakes, mechanistic reasoning must not be confused with mechanistic evidence. Mechanistic evidence includes empirical data proving the existence of, and details on the operations of, biochemical, physiological, and behavioral processes [78]. As these can come from many sources, from basic in vitro and in vivo experiments up to human studies, the reliability of mechanistic evidence assessment requires careful procedures already available [75]. It is thus imperative to not entirely accept mechanistic reasoning as well as to not entirely dismiss mechanistic evidence [76]. Although relatively recent, evidential pluralism has roots in the nine famous Bradford Hill's criteria for assessing causal relationships, almost half of which are evidence of statistical associations and the other half are evidence of mechanisms [81, 82]. In pharmacology and regulatory drug approval, evidence of mechanisms already plays a critical role in many stages, including but not limited to pharmacokinetics, pharmacodynamics, pharmacogenetics, finding dosage regimens, biosimilarity, and bioequivalence. However, for market authorization, causality is assumed granted if proven through usually two phase 3 (“pivotal”) RCTs. However, mechanisms or their absence can provide support for or against RCT results, and this has been used in some cases in diverse medical fields. It is particularly important to note that evidence of mechanisms from early stages, such as preclinical and phase 1 and 2 studies, informs clinical trial design and interpretation of results [78].

To reliably extrapolate, it is not only necessary to ascertain that the study sample represents the target population, but also that they have similar causal structures. In other words, that the indispensable support factors will be present and that no overriding derailing factors (e.g., improper touch, judgmental attitudes or prejudices from therapists, lack of family support, etc.) will intervene to prevent the cause from obtaining in the target population. Thus, to safely extrapolate, Cartwright and Hardie proposed working with causal principles and support factors [79]. For any causal inference in RCTs, there are support factors that make that causal connection work in the study sample with a complex causal structure in place, which can't be overlooked because of assumptions related to balance through randomization and blinding. A trustworthy causal principle is essential to obtaining good arguments that what worked “somewhere” (in RCTs) will also work “here”, at each instance where that causal principle is being considered for implementation and scaling up in diverse target populations. A Causal Principle considerate that the variable under study may interact with multiple covariates was then formalized as [79]:

$$\begin{array}{c}\mathrm{CP}:y\left(i\right)=a_{\mathrm{1}}+a_{\mathrm{2}}{y}_{\mathrm{0}}\left(i\right)+a_{\mathrm{3}}b\left(i\right)x\left(i\right)+a_{\mathrm{4}}z\left(i\right)\hfill \end{array}$$

where CP stands for Causal Principle, y is the outcome of interest, x is the variable under study, b are support factors that interact with x, z are support factors independent of x, and a's are constants across individuals. Therefore, for each individual i, a causal outcome y(i) is the sum of a base level, y ₀ (i), with the variable being studied, x(i), modulated by all possible interacting support factors, b(i), plus additional effects from other support factors independent from x, z(i), each term in the equation having its relevant constant modulators, a ₁, a ₂, a ₃, and a ₄, which can be considered boost factors. For simplicity, only sums are shown, but nonlinear effects are possible.

The concept of support factors is essential, because in biology and medicine it is actually rare that diseases and treatments are caused by the classic deterministic description of causality as “same cause, same effect” [83, 84]. However RCTs are particularly poorly suited at identifying and studying support factors [38].

In the therapeutic applications of psychedelics, establishing reliable EBPP for subsequent treatment implementation in diverse healthcare systems in different jurisdictions, for different populations with different characteristics than those in RCTs, requires attention to support factors. Moreover, the complex nature of the human psychedelic experience and the inter‐relational processes that go on between patients and therapists (and research staff) should not be assumed to have all been easily controlled for in RCTs.

It is clear that there is a spectrum of different approaches to therapeutic uses of psychedelics, with stand‐alone drug administration on one extreme and PAT on the other end. These are not just two different clinical practices, but two distinct conceptual frameworks making different causal claims. Even if the hypothesis is that the drug only is the causal mechanism, support factors play roles. For psychiatric disorders these are likely more relevant, and PAT may be an extreme case where support factors are paramount. As Deaton and Cartwright remarked: “whether, and in what ways, an RCT result is evidence depends on exactly what the hypothesis is for which the result is supposed to be evidence, and that what kinds of hypotheses these will be depends on the purposes to be served” [38].

If in PAT the drug is one factor among many others related to efficacy, a causal principle would be contemplated by the full formalization of Cartwright and Hardie. Here the outcome might be changes measured through scores in validated questionnaires assessing depression or PTSD, for example. According to this mechanistic reasoning, these improvements would be caused not only by the drug effect in each patient, x(i), but crucially modulated, positively and/or negatively, by a series of support factors both interacting with the drug, a ₃ b(i), and also the ones independent from the drug, a ₄ z(i). Examples of known support factors include behavioral mechanisms such as the patient‐doctor relationship, therapeutic alliance, patient and therapist expectations, interpersonal relationships, psychotherapeutic interventions, goal‐oriented behaviors, language, self‐narratives, affective emotional expression, music, supportive or therapeutic touch, family support, socioeconomic status, etc. Interestingly, from this epistemic perspective, as the postulated causal principle includes these support factors as part of the mechanistic reasoning, classifying them as biases would be to reject, a priori, the postulated causal principle. In other words, it would be equivalent to rejecting a possible complex and multifaceted mechanism of action, including goal‐directed behaviors and social interactions. Without explicit logical and epistemic arguments for doing so, this becomes a philosophical bias [85]. A critical assessment must also take into account that risk of bias is not the same as an actualized bias, and thus risk of bias does not automatically invalidate results. Rather it calls for careful interpretation [53]. In PAT, end of trial blinding assessments likely include noticeable improvements and therefore benign unblinding through efficacy [55]. Invalidating such results would waste potentially effective treatments. Therefore the question for EBPP assessments is not only methodological but also socio‐epistemological, and involves assessing if such complex mechanistic reasoning is indeed supported by good quality mechanistic evidence.

In the case of psychedelics, mechanistic evidence of varied quality includes not only the more proximal and physiological receptor binding and intracellular signaling pathways in vitro and in vivo and downstream neural plasticity [86, 87, 88, 89], but also behavioral mechanisms such as reopening of social reward learning periods [90], empathy‐like [91] and prosocial behaviors [92] in mice; social behavior in cephalopods [93] and sociality in humans [94]. Research in humans has also evolved a lot in understanding the phenomenology of the psychedelic experience. These drugs trigger a variety of subjective experiences conceptualized as emotional breakthrough [95], connectedness to self, others, and world [96], ego‐dissolution [97, 98, 99], oceanic‐boundlessness [100] and peak affective and cognitive experiences contentiously framed as being mystical or spiritual [101, 102, 103], as well as intensely challenging experiences [104]. Furthermore, some of these subjective experiences can be mapped to neurobiological circuits and neurochemistry of different compounds [105, 106], corroborating their evidential relevance as mechanisms. Finally, some PAT clinical studies showed that qualitative and phenomenological aspects of lived experiences [107, 108, 109, 110, 111] and interpersonal relationships [112] correlated with treatment outcomes, which can corroborate causal claims that these are part of the therapeutic mechanism of action, or support factors. How much each feature contributes, which are indispensable for different disorders, and the interactions among them are still open for further research, but the appreciation of a complex causal model depends on differing philosophical commitments regarding the experiential domain and lived experiences in psychedelic research [113, 114]. Given EBM's positivist epistemology and practices [115], it is particularly important to emphasize that individuals are constituted through lived experiences and that phenomenology constitutes a basic aspect of the science of psychiatry [116]. Incorporating these in evidence assessments might have, for example, helped the FDA's psychopharmacology advisory committee deliberating on the role of therapy in MDMA trials.

On the other hand, for postulated mechanistic reasoning such as “[t]he effects observed thus far in the best controlled studies of psychedelic treatment must be attributed to the drug itself and not to psychotherapy”, where poorly defined “psychological support” is claimed to be for safety reasons only [115], the formalization of the causal principle might look like (adapted from [79]):

$$\mathrm{CP}:y\left(i\right)={\mathit{\text{a}}}_{\mathrm{\text{1}}}+a_{\mathrm{2}}{\mathit{\text{y}}}_{\mathrm{\text{0}}}\left(i\right)+x\left(i\right)$$

where support factors related to, and unrelated to, the treatment variable, a ₃ b(i) and a ₄ z(i), respectively, were supposedly controlled by randomization and blinding and therefore excluded from the causal principle. For this kind of causal claim, however, the epistemic weaknesses of randomization and blinding are paramount, as in psychedelic trials correct guesses in treatment groups approach 100%, at least in the 17% of recent studies reporting it [28], and lower rates would be very hard to obtain in experimental groups. In this case, the mechanistic evidence to support this kind of mechanistic reasoning would necessarily include actual evidence that participants were functionally blinded and that other subjective and interpersonal factors were not statistically related to efficacy outcomes. Evidencing such claims requires explicitly reporting and evaluating how much risk of bias was actually present or not and publishing blinding assessments.

As in evidential pluralism all evidence must be made explicit and explicitly discussed, for the two types of causal claims related to psychedelics, measuring and reporting blinding assessments, expectations, subjective and phenomenological aspects, and therapeutic alliance are necessary. However, in PAT unblinding does not threaten internal validity, given that these factors are here postulated as mechanisms of action. Furthermore, unblinding does not mean that the drug does not contribute to efficacy. It means that the drug is not the only factor related to efficacy. This is why theoretical assumptions such as Muthukumaraswamy's [32, 33] are extremely unlikely. Finally, unblinding and therapy do not seriously threaten external validity in assisted‐therapy models, because if the extrapolation aims for other assisted‐(psycho)therapy practices and contexts, a similar causal structure can be more reasonably expected. This should be, however, a priority area for funding and more fine‐grained research to better assess what are the indispensable contributing factors in different therapeutic approaches, and which derailing factors can undermine extrapolation, for example different socio‐economic and cultural variables. On the other hand, for causal claims that efficacy is caused solely by the drug, internal validity is threatened by unblinding, as efficacy may have been strongly influenced by factors other than the drug itself. In addition, external validity is seriously undermined, given that the “drug only” mechanistic reasoning may not be supported by sound mechanistic evidence. Therefore, the causal structure in each RCT may have included many other factors which can't easily be controlled for, making it likely that the causal principle in RCT samples will not be obtained in a variety of target populations with distinct causal structures.

2. Discussion

As current regulatory practices are still highly reliant on EBM, despite its epistemological shortcomings, the eventual implementation of psychedelics as therapeutics requires serious consideration of limitations in current evidentiary norms [12, 117]. If these considerations continue to be overlooked during assessments of psychedelic therapies, there are increased risks of rejection of effective treatments, as well as scaling up impoverished and riskier practices [118]. This can ultimately lead to a medical reversal: approved treatments later abandoned by being perceived as ineffective and/or harmful. In the case of psychedelic therapy, however, an eventual medical reversal would not be caused by lack of rigor in RCTs, but by blindness to the extrapolation fallacy obscuring the study of the myriad support factors and complex causal structures of therapies during nonordinary states of consciousness [119].

Author Contributions

The author takes full responsibility for this article.

Conflicts of Interest

The author declares no conflicts of interest.

Data Availability Statement

No data was used in this article.