History repeating: guidelines to address common problems in psychedelic science

Abstract

Research in the last decade has expressed considerable optimism about the clinical potential of psychedelics for the treatment of mental disorders. This optimism is reflected in an increase in research papers, investments by pharmaceutical companies, patents, media coverage, as well as political and legislative changes. However, psychedelic science is facing serious challenges that threaten the validity of core findings and raise doubt regarding clinical efficacy and safety. In this paper, we introduce the 10 most pressing challenges, grouped into easy, moderate, and hard problems. We show how these problems threaten internal validity (treatment effects are due to factors unrelated to the treatment), external validity (lack of generalizability), construct validity (unclear working mechanism), or statistical conclusion validity (conclusions do not follow from the data and methods). These problems tend to co-occur in psychedelic studies, limiting conclusions that can be drawn about the safety and efficacy of psychedelic therapy. We provide a roadmap for tackling these challenges and share a checklist that researchers, journalists, funders, policymakers, and other stakeholders can use to assess the quality of psychedelic science. Addressing today’s problems is necessary to find out whether the optimism regarding the therapeutic potential of psychedelics has been warranted and to avoid history repeating itself.

Introduction

Psychedelics are mind-altering substances and include serotonergic hallucinogens [e.g. psilocybin, lysergic acid diethylamide (LSD), and Dimethyltryptamine (DMT)], entactogens [e.g. 3,4-methylenedioxy-methamphetamine (MDMA)], and dissociatives (e.g. ketamine). In the last decade, we have witnessed increased enthusiasm regarding the clinical application of psychedelics. Preliminary results from clinical trials using psychedelic therapy appear to show potential for the treatment of a wide variety of mental disorders, including major depressive disorder (MDD), ¹ end-of-life-anxiety, ² and addiction. ³ Results from phase II and phase III studies indicate that MDMA-assisted psychotherapy may be efficacious in the treatment of post-traumatic stress-disorder (PTSD). ⁴ And ketamine is increasingly being used in patients with treatment-resistant depression or suicidal ideation.^5,6

Research also appears to show that psychedelics have beneficial effects in healthy volunteers. Participants in experimental research have rated psychedelic trips among their most meaningful life-events ever,^7,8 and psychedelic experiences in turn appear to cause other positive outcomes such as a healthier lifestyle, ⁹ increased mindfulness, ¹⁰ enhanced creativity and problem-solving, ¹¹ pro-environmental behavior, ¹² and feelings of connectedness. ¹³

Taken together, the apparent benefits of psychedelics have led to significant changes in the mental health landscape. In the United States, more than 500 specialized ‘ketamine-clinics’ have emerged ¹⁴ that offer ketamine infusions to patients. ¹⁵ MDMA-assisted psychotherapy is awaiting approval by the FDA as medicine for the treatment of PTSD, ⁴ and the nonmedical use of psilocybin has already been legalized in Oregon ¹⁶ ; other states are to follow suit. ¹⁷ Centers in Portugal, the Netherlands, and South America organize legal psilocybin or ayahuasca retreats for the treatment of MDD ¹⁸ and Australia officially recognized psychedelics as medicines in early 2023. ¹⁹ Pharmaceutical companies have made considerable investments into clinical research and filed patents on the production of psychedelics (e.g. for synthesizing a polymorph of psilocybin) as well as the therapeutic processes, for example, the training of therapists; cf. Smith and Appelbaum. ²⁰ This has resulted in a flurry of different clinical applications, which can be broadly grouped as ‘psychedelic therapy’, of which psychedelic-assisted psychotherapy is the specific application of psychedelics embedded within a psychotherapeutic context.

Threats to validity in psychedelic research

However, we see many reasons to be seriously concerned about this optimism. In this paper, we highlight 10 key challenges that psychedelic science is currently faced with. These challenges threaten four types of validity^21,22 and cast serious doubt on the inferences that have been drawn in research carried out within the last decade. Threats to internal validity are best thought of as rival explanations, and raise doubts about whether a particular psychedelic intervention, rather than other factors, explains the results of a study. External validity refers to the generalizability of the research findings from the studied sample to the population of interest. Construct validity concerns the question how constructs in a study are operationalized, and what the exact working mechanisms are. Finally, statistical conclusion validity concerns the extent to which the conclusions, based on the data and statistical analyses, are warranted.

In psychedelic studies, internal validity is commonly threatened by the lack of control groups, the breaking blind problem, and placebo effects. The main sources of threats to external validity are low-powered studies and a strong selection bias in the inclusion of participants. Threats to construct validity include measurement problems as well as the lack of long-term treatment effects and mechanisms of action. Statistical conclusion validity is threatened by the multiple comparisons problem, conflicts of interest (COIs), outcome switching, and other questionable research practices. Of note, these validity threats are well-understood, ²¹ and it is therefore surprising that psychedelic science appears to ignore many of the lessons learned from decades of research. Much of the recent work is history repeating itself.

Previous reviews on psychedelic therapy have focused on the lack of sufficient safety data, ²³ the lack of open science practices, ²⁴ the breaking blind problem, ²⁵ and the role of expectations and placebo effects. ²⁶ In this paper, we take a broader perspective. First, we highlight the 10 most pertinent challenges and discuss how these problems threaten valid inferences. We conclude that because these validity threats often co-occur in psychedelic studies, it is premature to draw firm conclusions about the safety and efficacy of psychedelic therapy. Second, because solutions for some of these problems are more readily available than for others, we classify the problems as easy, moderate, or hard, depending on the methods and resources required to solve them. Third, we suggest how the problems can be mitigated, and consider solutions in terms of both scientific methods (e.g. more rigorous measures, statistics, and reporting) and resources (e.g. more funding, more therapists, more time, more participants). Finally, we summarize concrete recommendations for researchers and clinicians to move forward (see Table 1) and provide a resource list for reviewers, journalists, funders, and policymakers to facilitate the accurate assessment of the quality and rigor of psychedelic studies (see Table 2).

Table 1.

Easy, moderate, and hard problems in psychedelic science – and how to address them.

Problems	Solutions
Easy problems
Invalid statistical inferences	Involve independent statistical experts in peer-review process
	More transparency through open science practices, including:
	Publish (anonymous) reviews alongside scientific papers
	Disclose all measures and statistical analyses
	Transparent preregistration of all studies
	Publish null results
	Investigate robustness of results (e.g. via multiverse analyses or many-analyst approaches)
	Share de-identified data
Conflicts of interest	Transparent reporting of all COIs in publicly accessible ways (i.e. not paywalled)
	Systematic instead of narrative reviews
Safety and adverse events	Report adverse events transparently and systematically
	Independent arbiters assess whether adverse events are related to the treatment
	Include safety/adverse events as primary or secondary outcomes
Moderate problems
Lack of control groups	Fund and carry out studies with control groups; team science/multi-center collaborations can mitigate costs
Sample sizes	Fund and carry out properly powered studies; team science/multi-center collaborations can mitigate costs
Selection bias	Transparently disclose inclusion and exclusion criteria and full recruitment procedure
	Assess and report patient background characteristics that may influence treatment efficacy
	Use more representative samples
Study duration	Include long-term outcomes (⩾12 months)
	Use larger samples to mitigate attrition
Hard problems
Breaking blind problem	Assess and report blinding efficacy
	Include an active control condition
	Recruit participants without prior psychedelic experiences
	Control for the breaking blind problem statistically
Placebo effects	Include a third study arm without any intervention to test against the placebo effect
	Measure expectations of patients and therapists
	Convey more realistic expectations (stop the hype)
Mechanisms of action	Carry out independent and high-powered replication studies
	Assess and report therapeutic alliance, efficacy of psychoplastogens, and efficacy of other techniques that induce an altered state

COIs, conflicts of interest.

Table 2.

Checklist for assessing the quality and scientific rigor of psychedelic studies for mental health problems.

Criterion	Description
1. Valid Inferences	Is there sufficient transparency around data collection and statistical analyses, and are the conclusions supported by the evidence? Is there evidence that the treatment and not other factors (e.g. breaking the blind) explain the difference between the intervention and control group? Have independent reviewers with the relevant statistical expertise been assigned to review the manuscript? Are the reviews publicly available?
2. Conflicts of interest	Are potential COIs reported transparently in the paper? What is the nature of these COIs, and, in the presence of severe COIs, are there sufficient safeguards in place so that the findings can be considered trustworthy (e.g. preregistration of primary outcomes and statistical analyses)? Are all included measures fully disclosed and reported?
3. Safety and adverse events	Is it easy to find all relevant information regarding adverse events in the study? Is there an independent arbiter to decide whether an adverse event is related to the treatment? Is the psychotherapy component of the study standardized and fully described? Were trained therapist used to carry out treatments?
4. Control group	Is a control group included to address common validity threats such as placebo effects, expectancy effects, and regression to the mean? If no control group is included, are interpretations sufficiently careful?
5. Sample size	Is a power or sensitivity analysis provided, and does it include a justification of the minimum effect size of interest? Is the study sufficiently powered to detect a difference between intervention and control group (rather than powered against no effect at all)?
6. Selection bias	Does the studied sample differ from the population of interest? Is a statement about constraints on generalizability included? Is demographic (e.g. gender, age, socioeconomic background) and clinical (e.g. severity, comorbidities) information provided?
7. Study duration	Do scientists follow the patients for a sufficient time frame to justify the conclusion that successful treatment took place, that is, that people have returned to a normal level of symptom load, wellbeing, and functioning?
8. Breaking blind problem	Have efforts been made to minimize the risk of unblinding (e.g. by using active placebos)? Was masking efficacy (i.e. if blinding succeeded) assessed and reported?
9. Placebo effects	Does the study design account for placebo effects, for example, by comparing the intervention arm against a control group? Does the study include measures to assess patient and therapist’s expectations about treatment outcomes both at the beginning and during the treatment?
10. Mechanisms of action	Are inferences regarding potential mechanisms of action supported by evidence? Are the data and materials available in a repository for replication and secondary analyses?

COIs, conflicts of interest.

The easy problems

Problems that are relatively easy to address within psychedelic science include (1) invalid statistical inferences and questionable research practices, (2) COIs, and (3) safety and adverse events.

Invalid statistical inferences and questionable research practices

A conclusion is valid if the inferences follow from the evidence presented. There are two tiers of challenges here. First, authors in the psychedelic literature have regularly drawn conclusions that are either misleading or clearly contrast with the data. Four examples follow to showcase that this problem is common practice. Abbar et al. ²⁷ found in a randomized controlled trial (RCT) comparing ketamine against placebo that there was no persistent benefit of ketamine over placebo at the exit timepoint of the trial in week 6, but concluded in the abstract that ‘ketamine [. . .] has persistent benefits for acute care in suicidal patients’. Ionescu et al. ²⁸ found in an open-label ketamine study that only 2 of 14 patients show sustained improvement at 3-month follow-up (which may well be due to the placebo effect or other factors), but the title of the paper reads ‘Rapid and Sustained Reductions in Current Suicidal Ideation’ (our highlight). Palhano-Fontes et al. ²⁹ concluded in their ayahuasca study (n = 14 treatment, n = 15 placebo) that ‘blindness was adequately preserved’, when all participants in the treatment group said they believed they had received ayahuasca, but less than half of participants in the placebo group said so. And Daws et al. ³⁰ compared two treatment arms, including one using psilocybin-assisted psychotherapy, against each other, concluding that one treatment outperformed the other despite the lack of a statistically significant interaction term between the treatments. Journals, reviewers, funders, and scientific institutions need to hold authors accountable for such inferences.

The second tier of challenges is when questionable research practices – that usually go hand in hand with a lack of transparency – raise doubts about the validity of conclusions.^{31 –33} One common practice concerns flexibility regarding the analysis of primary outcome measures, which is common in psychedelic science. A recent phase II trial ³⁴ administering psilocybin for treatment-resistant depression funded by a pharmaceutical company registered their primary outcome on clinicaltrials.gov for the time frame ‘up to 12 weeks’, which allows for degrees of freedom in obtaining statistically significant results and severely threatens valid inferences. Another study, using ketamine to treat depression ³⁵ switched a secondary to a primary outcome a year after data collection started, and the publication contains no results at the time point of 2 weeks that can be found in the clinicaltrials.gov registration (https://osf.io/uhdrp). Yet another psilocybin trial for depression ³⁰ analyzed a different outcome than registered on clinicaltrials.gov. ³⁶ In all these cases, doubts remain whether equally positive results had occurred if stricter procedures would have been in place.

Another concern is related to multiple testing. Psychedelic studies often contain a flurry of different outcomes, including physiological, neural, cognitive, and self-report measures. This dramatically increases the chances of false positive findings when not dealt with appropriately. For instance, a recent paper on the efficacy of psychedelic therapy concluded that several secondary outcomes clearly favored psilocybin over escitalopram, but the lack of correction for multiple testing raises doubts about the validity of these findings. ¹ In our own research on psilocybin microdosing, we included six different tasks, with multiple subcomponents and measures per task, ³⁷ totaling the number of dependent measures to more than 20. We found effects of psilocybin microdosing on two outcomes. But because we had all outcomes preregistered transparently, and these two findings did not survive correcting for multiple testing, they likely reflected chance findings.

There are further problems, such as selectively removing outliers that result in significant findings, interpreting nonsignificant p-values in small samples as ‘trends toward significance’ (but not interpreting barely significant p-values as trends toward nonsignificance), and using one-sided tests over two-sided tests to obtain desired results, ³⁰ discussed in Love. ³⁸ All these problems jointly threaten both internal validity and statistical conclusion validity, as it remains unclear whether the conclusions are supported by the data and if other explanations might be more plausible for the effects observed. These concerns go hand in hand with evidence for substantial publication bias in the clinical trial literature for both pharmacological ³⁹ and psychotherapeutic interventions. ⁴⁰ This means that significant findings in trials are much more likely to be published in scientific journals, whereas nonsignificant findings are often not published, inflating meta-analytic effect sizes and threatening statistical conclusion validity.

We see two main ways forward. First, journals need to evaluate research papers more rigorously. This includes vetting by statistical and clinical trial experts without conflicts of interest. For the Daws et al. ³⁰ paper referenced above, we applaud one of the reviewers for openly acknowledging that they did not catch a statistical problem, which may have impacted their review of the paper, and thereby the main conclusions of the paper. ³⁸ To foster transparency and accountability, reviews should be published alongside papers, giving the research community insights as to the rigor of the review process. Journals must also compare the manuscript with the clinical trial protocol and statistical analysis plan. Papers in which conclusions stand in contrast to the presented evidence should not be considered for publication. Second, researcher degrees of freedom and lack of transparency that foster questionable research practices can be reduced by adherence to best practices. These include publishing null-findings, for instance in the format of a null results in brief section ⁴¹ ; reporting on all measures and tasks, preferably in the main paper rather than somewhere in Supplemental Materials; and preregistering research studies (including outcome measures and statistical analyses) to distinguish between exploratory and confirmatory findings. ⁴² The registered reports publication format offers a useful tool to increase transparency, by providing authors with peer-review feedback prior to conducting a study and (if accepted) a guaranteed publication even in case of null results. ⁴³ Of note, there are already well-established best practices that have shown to increase transparency in the reporting of clinical trials, ⁴⁴ such as the Consolidated Standards of Reporting Trials (CONSORT) reporting guidelines specifying 25 criteria. ⁴⁵ Unfortunately, guidelines are not always taken as seriously as they should – for example, an audit of the clinical trial literature in clinical psychology indicated that (pre-)registrations were often incomplete and lacked sufficient information to be reproducible. ⁴⁶ Overall, we deem solutions to these problems readily available; they primarily require implementing more rigorous scientific and statistical methods and transparent evaluating and reporting.

COIs with sponsored studies

Early research on psychedelics was often initiated by academic centers and supported by philanthropic organizations, for example, the Multidisciplinary Association For Psychedelic Studies (MAPS); cf. Mitchell et al. ⁴ More recently, pharmaceutical companies have become involved, ⁴⁷ leading to financial COIs for some funders and authors. COIs by themselves are not a red flag, but conclusions from papers with COIs should be interpreted with extra care, given that they tend to go together with problematic practices. For instance, an analysis of 397 clinical trials in psychiatry has shown that studies reporting a COI were five times more likely to report positive results. ⁴⁸ COIs are also pronounced in the psychedelic literature: a recent opinion paper on ketamine for treatment-resistant depression featured a COI section nearly five times as long as the paper’s introduction section ⁴⁹ : of 25 authors, 19 declared COIs, including patents for treating depression with ketamine. We are not convinced that the research community always takes best COI practices sufficiently seriously. For example, while the organizers for the psychedelic science 2023 conference (with over 300 speakers) originally followed recommended disclosure protocols, ⁵⁰ by asking speakers to declare COIs in a dedicated presentation slide, slides were removed by the organizers before the talks and moved to the conference mobile app.

Moving forward, policy decisions should rely on systematic rather than narrative reviews, given that narrative reviews allow for more degrees of freedom to obtain desired results. ⁵¹ Industry-independent experts should be involved in all stages of the clinical trial process, including study design, data collection, analysis, writing, and peer review. Furthermore, we encourage all journals to publish COIs in a publicly accessible way, rather than putting them behind paywalls. COIs should be clearly communicated in presentations at scientific conferences as well, instead of being skimmed over or hidden in online apps. Transparency and open science tools provide scientific solutions and help safeguard against COI. These tools include: preregistration; blinding of group membership when conducting analyses; multiverse analyses, a method for assessing the robustness and boundary conditions of a specific effect ⁵² ; and many-analyst approaches see for instance. ⁵³ These solutions are all readily available and implementing them will increase statistical conclusion validity of psychedelic science. Making more public funding available for conducting independent research is another prerequisite to safeguard research against COIs.

Safety and adverse events

Sufficient information on potential risks, dangers, and adverse events is missing to draw conclusions that psychedelics are safe to use in the context of mental health treatments. For example, although esketamine (in the form of a nasal spray) was approved by the FDA for treatment-resistant depression in 2019, there is evidence that the approval process overlooked red flags, ¹⁴ and a meta-analysis from 2021 concluded that there was insufficient data regarding the long-term safety of ketamine. ⁵⁴ A recent analysis found a systematic underreporting of serious adverse events studies on the safety and efficacy of esketamine in depression ⁵⁵ : 41.5% of serious adverse advents that were found on clinicaltrials.gov were not reported in published articles, and nearly all of them (88 out of 94) occurred in esketamine treatment arms, not placebo arms.

We see similar issues for other drugs now. Adverse events for therapy with MDMA and classical serotonergic hallucinogens are not systematically assessed and reported. ⁵⁶ A straightforward definition of an adverse event in psychedelic research is missing, and standardized measurements and transparent reporting of adverse events are lacking. Most studies rely only on spontaneous reporting by patients or therapists, which in turn require a careful process of interpretation to assess whether the adverse event can be attributed to the psychedelic therapy specifically.

And adverse events do occur. ⁵⁷ A recent clinical trial using psychedelic therapy ³⁴ reported increased suicidal ideation and intentional self-injury in the 10 and 25 mg psilocybin group, whereas the control group who received 1 mg of psilocybin remained unaffected. In the same study, suicidal behaviors were observed in three patients, but authors concluded in a media report that ‘these cases were probably random events and unrelated to the dose of psilocybin, which would have been fully cleared from the patients’ bodies’. ⁵⁸ This is not the most careful interpretation of the data, and we note that the opposite rationale is applied to successful treatments: people get better despite the drug being fully cleared from the patients’ bodies. Sometimes, conclusions are radically opposed to presented evidence. In a placebo-controlled study using ayahuasca as an intervention for depression, 4 of 15 participants in the experimental group (i.e. ~27%) had to be hospitalized for 1 week ‘due to presenting a more delicate condition’. ²⁹ Nonetheless, authors concluded that the ‘study brings new evidence supporting the safety and therapeutic value of psychedelics’ (our highlight).

Psychedelic experiences can have short- and long-term adverse consequences. Short-term risks include the destabilizing effects that psychedelics can have through the experiences they can trigger, which can be difficult to handle both for the person and their therapist ⁵⁹ – experiences that can result to everything from increased acute agitation to prolonged emotional dysregulation. ⁶⁰ Many people also report adverse after-effects including recurrent hallucinations, increased anxiety, and physiological discomfort. ⁶¹ In the long term, such experiences can also cause ontological shocks, resulting in a dramatic shift in one’s religious and spiritual worldviews. ⁶² In a recent article, a patient who had participated in a psilocybin study described their state of confusion, anxiety, and distress, ⁶⁰ resulting in a long and desperate search including the use of spiritual practices, meditation techniques, and theology. This illustrates the dramatic effects that psychedelic therapy may have on some patients, and the need for careful spiritual, existential, religious, and theological support, ⁶³ long after psychedelic sessions. ⁶⁴

Unfortunately, such support is typically not included in the current clinical trials which – often funded through pharmacological companies – strive for cost- and time-effective interventions. This is perhaps best summarized in the words of a patient receiving MDMA-assisted psychotherapy: ‘(. . .) they tore open my chest, and they repaired the little damage in the heart there but then everyone just walked away from the table and my chest was still wide open’.

Another obvious risk of psychedelic therapy concerns safety in the therapy room. Some proponents of psychedelic therapy have argued that clients regress during trips and require close physical content, ⁶⁵ and many protocols include ‘nurturing touch’ as an important component during therapy. The MAPS study protocol, for example, writes that ‘mindful use of touch can be an important catalyst to healing’. ⁶⁶ But already since decades there is ‘a lack of consensus about the use of touch and the complex ethical and clinical issues surrounding its use’, ⁶⁷ a crucially important topic, especially in psychedelic therapy given the highly suggestible and vulnerable state clients are in. Overall, manuals provide little guidance, leaving what is acceptable to the interpretation of the therapist. ⁵⁷ A patient who took part in the phase II trial for MDMA-assisted psychotherapy for the treatment of PTSD (MAPS) reported several inappropriate encounters, including sexual abuse by the therapist, ⁶⁸ casting doubt whether a safe and supportive therapeutic environment can currently be guaranteed. ⁵⁷

Overall, problems related to safety and adverse events are considerable, and without systematic and transparent reporting, it remains unclear for which patients psychedelic therapy provides a safe alternative and for whom it imposes a potential risk. More systematic and large-scale studies about the prevalence and persistence of adverse effects are needed before psychedelics can be considered safe and efficacious treatments for mental health problems. Without such data, we have history repeating all over again, for example, with respect to the adverse effects of mindfulness-based interventions, ⁶⁹ opioids, ⁷⁰ or ketamine. ⁵⁶ Given the profound consequences that psychedelics can have, researchers, funders, and regulators should consider including safety and adverse events as core outcome measures in clinical studies, rather than reporting them (if at all) in Supplemental Materials. Furthermore, instead of having the researchers decide whether an adverse event is related to a drug or not, such decisions should be made by independent arbiters. Finally, the psychotherapy and its components (e.g. acceptance and commitment approach; use of ‘nurturing touch’) applied within psychedelic therapy need to be standardized (see also section ‘Mechanism remains unclear’) and rigorously evaluated using the same stringent criteria for assessing safety and efficacy. ⁵⁷

These solutions are also relatively easy to implement and primarily require more rigorous scientific reporting.

The moderate problems

Whereas the easy problems should be relatively simple to address, the moderate problems – (1) the lack of (good) control groups, (2) sample sizes, (3) selection bias, and (4) lack of long-term follow measurements – require more effort. Solving the problem of sample size primarily requires more financial resources; for the other problems, scientific solutions will have to play a role as well.

Lack of control groups

Many clinical studies on the potential utility of psychedelics are open label, that is, without a control group. This is a typical design to start investigations into potential new treatments, but strongly limits what can be learned about the potential efficacy of interventions. ⁷¹ For instance, a recent open-label study followed 27 patients with MDD after two doses of psilocybin with supportive psychotherapy. ⁷² A year later, around 60% of the participants were no longer depressed. The authors concluded that there were ‘substantial antidepressant effects of psilocybin-assisted therapy’. But without a control group, we don’t know if the recovery was due to the intervention. To put the 60% into context, an analysis of 19 studies showed that the expected 12-month recovery rate of people with MDD who do not receive any treatment is around 53%. ⁷³

Demonstrating that mental disorder severity or diagnostic prevalence decline during treatments without comparing recovery rates to placebo does not establish evidence that treatments are efficacious. ²⁵ This inference threat that is well-established for pharmacological and psychotherapeutic treatments of mental disorders is even more pronounced in psychedelic research, especially for the many naturalistic field studies conducted at psychedelic retreats.^74,75 In these studies, strong expectations, peer pressure, and the effect of a charismatic leader may further increase the placebo effect see also Plesa and Petranker. ⁷⁶

One related challenge that highlights the importance of a control group is regression to the mean, which ‘decision-makers should always consider [. . .] to be a viable explanation for the observed change in an outcome in a pre-post study’. ⁷⁷ Regression to the mean is a decrease in scores over time due to selecting people based on extreme values (such as the threshold on depression severity) when entering studies. This problem is common in clinical trials and psychedelic studies are no exception, where a particular score on mental health problems is often required for enrolment.

Taken together, these problems imply that an improvement of patients in psychedelic open-label studies cannot be causally attributed to the treatment itself. Including a control condition is a first, crucial step to increase internal validity. Several studies have implemented control groups, including active placebos, for example, diphenhydramine for the treatment of alcohol abuse disorder ³ ; low-dose psychedelics, ² or treatment as usual, for example, selective serotonin reuptake-inhibitors (SSRIs) for the treatment of major depression; cf. Carhart-Harris et al. ¹ This design enables comparing two treatments over time (reflecting differences of treatments), rather than just the improvement of one group (confounded by, for example, placebo response and regression to the mean), although threats to internal validity still need to be managed, for example, the amount of contact a person has with a therapist should be the same in both treatment arms; cf. Kazdin. ²¹ The question about what constitutes a good control condition in psychedelic therapy is also directly related to the hard problem of ‘breaking blind’, discussed in section ‘The breaking blind problem’. Adding a control group will increase costs and time, which can be mitigated via team science, slow science, and multi-center studies, that is, different teams pooling resources to design and carry out more rigorous and better controlled studies. This solution can reduce many threats to internal validity and will make findings of psychedelic studies more robust.

Sample size

We would not finance or conduct a poll about who will win the next US presidency in 20 or 50 participants because such samples are not sufficiently informative regarding the general population we want to learn about. The same applies to clinical trials: researchers do not conclude that Alice and Bob responded well to ketamine, but that ‘Ketamine is a safe and efficacious treatments for MDD’, ⁷⁸ requiring randomly drawn samples that sufficiently large to represent the target population of interest (in the case of the study of Bahji et al., people diagnosed with depression). That is currently not the case, and problems of small samples (a threat to external validity) are further exacerbated by highly selective groups of patients (a threat for internal and construct validity).

The most recent meta-analysis of the seven available RCTs for psychedelic therapy for MDD and anxiety-related disorders showed that there was a significant reduction in symptoms up to 5 weeks following the intervention. ⁷⁹ However, the number of participants per treatment arm was small, with more recent studies having somewhat higher power than early studies. ⁸⁰ As underpowered studies typically result in imprecise and inflated effect sizes, ⁸¹ especially when questionable research practices and publication bias come into play, it is plausible that the initial efficacy of psychedelic therapy was overestimated. When larger preregistered studies will be conducted, it is likely that the resulting effect sizes will be lower than previously reported.

This is also relevant for understanding working mechanisms regarding the acute action of psychedelics, given that studies about neuroscientific theories are based on about a dozen neuroimaging datasets with very small samples (see also section ‘Mechanism remains unclear’). Some of the most widely cited papers do not contain more than 10 participants, and a particular dataset by Carhart-Harris et al. ⁸² has been re-analyzed in numerous ways across 10 papers ⁸³ ; to the authors’ credit: they were one of the few groups that actually openly shared their data thereby allowing other researchers to conduct secondary analyses. Threats to external validity are exacerbated by the large degrees of freedom in the analysis of neuroimaging data and the lack of a standardized analysis pipeline, ⁸⁴ likely leading to many false positive results in the literature on neural mechanisms underlying psychedelics.

Small sample sizes primarily threaten external validity and statistical conclusion validity, as it remains uncertain whether findings generalize to the broader population. Although the problem of underpowered studies has been recognized for a long time, there are still strong incentives for conducting studies focused on discovery and explanation rather than study designs that critically evaluate the intended effects of a given therapy. ⁷¹ This can be mitigated somewhat if findings from small studies are reported on properly and if open science practices are implemented to prevent obtaining false positive results in underpowered samples.

Addressing these threats requires larger samples, which in turn will cost more resources. Larger samples will also help addressing the important question of what treatments work for whom. It could well be that psychedelic therapy selectively works for specific patient groups, such as patients primarily coping with depression triggered through a life-threatening illness, ⁸⁵ and current samples are not sufficiently large to find such effects: in a recently pooled moderation analysis of 17 ketamine studies including more than 800 patients; however, ⁸⁶ no consistent moderators were identified. This, in turn, highlights the need for more team science, multi-center and multi-lab studies to conduct few high-powered, preregistered studies, rather than many underpowered open-label studies that can fall prey to publication bias and other issues.

Selection bias

Participants learn about clinical trials from media reports, trial advertisement, and clinician referral (Muthukumaraswamy et al. ²⁵ ). This, along with prior experiences, introduces different types of selection biases. For example, inclusion criteria of a recent study were right-handedness, mild-to-moderate depressive symptoms without psychotic features, abstinence from medication, drugs and alcohol (including tapering-off antidepressant medication, a challenge for many patients), good physical health, and no suicide risk. ⁸⁷ Consequently, only very selective groups of patients are eligible for participating in these studies. The recent phase III study initiated by the MAPS included only 26% of 345 treatment-seeing participants, ⁴ and the rules by which they were picked across all the participating sites are not fully transparent.

What consequence can selection have? Compared to the general population of treatment-seeking individuals, participants in small psychedelic studies may be easier to treat, a common phenomenon for psychotherapeutic and pharmacological treatments as well. For example, when applying exclusion criteria of 161 antidepressant efficacy trials to 1271 in patients diagnosed with depression, between 76% and 99% of participants would have been removed, largely due to factors resulting in more severe psychopathology, for example, suicidal ideation, comorbidities.^88,89 This is important because cases with more severe or complicated psychopathology have worse treatment outcomes. ⁹⁰ Participants enrolled in psychedelic studies are potentially also more motivated than the average treatment-seeking population, given that they often actively reach out to investigators such as in the MAPS study. Furthermore, participants who have prior positive experiences with psychedelics are more likely to participate in psychedelic trials. Prior experiences with psychedelics increase the likelihood of breaking blind cf. Aday et al. ²⁶ and Carbonaro et al., ⁹¹ which may in turn amplify observed treatment efficacy and reduce adverse events.

These and other selection biases lead to an overestimation of the psychedelic treatment effect, producing a substantial threat to internal and external validity. A related threat is the focus of psychedelic studies on White, Educated, Industrialized, Rich, Democratic (WEIRD) samples.^{92 –94} As a result, it remains unclear to what extent obtained findings generalize to the population at large. It could well be that psychedelics can help depressed patients to better deal with existential and meaning-related issues, but only if the basic needs for social security and safety are fulfilled. Furthermore, the clinical-medical approach to psychedelics may not fit well with the worldview and practices among some indigenous people and ethnic minorities; theories about the potential psychoplastogenic effects of psychedelics showcase a medical-mechanistic framing that can clash with traditional beliefs around psychedelics as plant medicines and a way to communicate with the divine. ⁹⁵

Despite common claims in the literature that psychedelic drugs work, for example, ‘for MDD’ ⁷⁸ or ‘for PTSD’, ⁴ the combination of small samples and selection bias does not allow for such conclusions. Addressing selection biases requires better science: researchers need to measure and report more. In addition to clinical information such as diagnoses, severity, and comorbidities, other potential factors should be assessed and reported, including motivation and prior experience with psychedelics. Researchers also need to be more transparent about inclusion and exclusion criteria and should disclose the full study protocol, screening instruments, and the number of participants included and excluded. The problem of selection bias is also related to availability of resources; with more funding and personnel, it would be possible to recruit larger and more representative samples, including more difficult-to-reach populations.

Lack of long-term follow-up

Proponents of psychedelic therapy argue that one of the most important breakthroughs would be to provide efficacious treatments to people with severe and chronic mental health problems for whom gold-standard treatments have not been successful.^96,97 This explains why many studies have been conducted in treatment-resistant populations. ⁷⁹ However, follow-up times in these studies cannot support claims of successful treatments. For example, a 2016 paper using ayahuasca as treatment for treatment-resistant depression ⁷⁴ followed 29 participants for a total of 7 days, concluding that their research supports ‘the safety and therapeutic value of psychedelics’. Participants had suffered from depression on average for 11 years, and a 7-day follow-up period is entirely insufficient to determine treatment efficacy. In the most recent meta-analysis on the effects of psychedelic therapy for depression and anxiety, ⁷⁹ only three studies reported treatment outcomes up to 5–8 weeks following the intervention.

Studies using ketamine face similar challenges and have often measured outcomes 4 h, 24 h, and a few days after infusion.^98,99 It is therefore not surprising that a recent meta-analysis concluded that ‘long-term safety and efficacy [. . .] are yet to be investigated’. ¹⁰⁰ We note that this reflects the state of the scientific literature after the FDA approved this drug for treating depression.

In summary, most studies in the psychedelic literature have at best demonstrated short-lived symptom relief, rather than successful treatment, contrary to claims popular in this literature. Given that over half of all cases of untreated depression remit spontaneously within 1 year ⁷³ – depression is an episodic disease – it cannot be concluded without longer-term follow-ups that psychedelic therapy offers promising treatments for mental disorders. Australia now recognizes psychedelics as medicines, despite warnings from researchers in the field, including those involved in the largest trials: ‘These treatments are not well established at all for a sufficient level of broad-scale implementation [. . .]. We’ve got no data on long-term outcomes at all, so that worries me a lot’. ¹⁹

This challenge primarily threatens construct and internal validity, as it remains unclear whether psychedelic therapy directly targets mental health symptoms or whether there are rival explanations, for example, a positive afterglow following psychedelic experiences. ¹⁰¹ Therefore, studies need to implement more long-term follow-up measurements, preferably up to 1 year following the intervention, and larger samples are required to mitigate (selective) attrition. Such studies would be more expensive to conduct, but they are required to conclude with any confidence that psychedelic treatments add to the landscape of available treatments and are worth the associated risks.

The hard problems

The hard problems in psychedelic science include (1) the breaking blind problem, (2) placebo effects, and (3) the lack of a clearly specified mechanism. Solving these problems will require both rigorous scientific work and innovation, as well as more financial resources.

The breaking blind problem

RCTs are considered the gold standard in clinical psychology and psychiatry. They are usually double blind, meaning that neither participants nor researchers are aware of group membership. Unfortunately, blinding of participants and researchers is, depending on the particular psychedelic substance, either difficult or impossible. This is because the psychoactive effects of an active dose of most psychedelic substances become obvious to both the participant and the experimenter or clinician in about 30–60 min after intake. The breaking blind problem is therefore the rule rather than the exception in psychedelic trials. ²⁵ For instance, in a recently published placebo-controlled study, 15 of 19 participants correctly guessed they had received a placebo, whereas 12 of 15 participants correctly guessed they receive an active dose after a medium- to high-dose psilocybin session; ¹⁰² unfortunately they did not report the clinicians’ guesses about treatment allocation. A recent systematic review indicated that although most studies on psychedelic therapy were nominally blinded, participant blinding was only assessed in 8 out of 81 studies. ¹⁰³

This threatens internal validity and the conclusions that can be drawn based on clinical studies for several reasons. First, it is a threat to valid measurement on the side of the clinical interviewer because raters who know about group membership may no longer be unbiased when determining whether patients improved, for example, in their PTSD symptoms. This bias may be exacerbated when interviewers have personal positive experiences with psychedelics. ¹⁰⁴ Second, breaking the blind adds a confound regarding attention and care: given that psychedelic psychotherapy is conceptualized as synergistic process that is catalyzed by the psychedelic substance, researchers and clinicians may implicitly provide more attention and put in more effort when they come to realize that the patient received an active dose compared to a placebo. ²⁶ Third, it is a threat to valid measurement on the side of the patient, as it boosts expectancy and therefore placebo effects.

Despite these threats, the breaking blind problem is largely ignored in the literature, as evidenced by the fact that researchers rarely report on masking efficacy, that is, the extent to which blinding was preserved for systematic reviews, see Muthukumaraswamy et al. ²⁵ and Nayak et al. ¹⁰³ For example, the very recently published, first placebo-controlled, double-blind RCT for psilocybin-assisted therapy in MDD ⁸⁷ went through considerable efforts to mask treatment allocation, but did not assess or report whether blinding was successful. However, given that the authors’ analyses reveal massive differences between groups in their subjective experiences (including oceanic boundlessness, anxious ego dissolution, auditory alterations, impaired cognition and control, and disembodiment), the study is clearly not double blind, and should not have been published as such. Even in the most recent phase III MDMA study for PTSD ⁴ blinding was not formally assessed, and the authors only report ‘anecdotal’ data, showing that participants guessed group membership correctly with over 90% accuracy.

Overall, we strongly recommend to formally assess masking efficacy in modern clinical trials, given that the breaking blind problem is a severe threat to valid inferences, and it only takes very little effort to assess and report. The problem cannot be easily solved and other treatments in which experiences form an integral part of the therapy (e.g. exposure therapy or music therapy) face similar challenges. ¹⁰⁵ Different solutions have been proposed to reduce the risk of breaking blind, including the use of active placebos that induce physiological, for example, such as methylphenidate; see for instance Griffiths et al. ⁸ or psychological effects, for example, comparing MDMA with psilocybin; cf. Muthukumaraswamy et al. ²⁵ Recruiting psychedelic- naïve participants for clinical trials may reduce the risk of breaking blind somewhat for low-dose trials, as people without prior experience will find it more difficult to correctly guess their condition assignment.^26,91 Using different dosing conditions of the same substance (e.g. comparing 10, 20, and 30 mg of psilocybin) appears an effective way to avoid the breaking blind problem.^25,106 This design rests on the assumption that a low dose is not or less therapeutically effective, and it allows directly assessing the dose–response relationship. Selective or partial disclosure of information prior to the study could be another way to reduce the risk of breaking blind, although it faces ethical challenges. ²⁶ For instance, providing instructions that the study consists of multiple arms (instead of two) with different dosing regimens reduces chances of breaking blind. Patients could also be informed about all possible side effects of all potential study drugs included in the clinical trial (i.e. rather than the side effects of each specific substance they are about to receive), to further minimize confidence about treatment allocation. ¹⁰⁷ Alternatively, using a balanced placebo-informed design, drug information, and actual drug administration could be independently manipulated to allow a direct comparison of the effect of expectations about condition assignment and the actual psychedelic substance. ²⁵

A radical but ethically challenging idea was proposed by Nautiyal and Yaden ¹⁰⁸ : in order to completely rule out the breaking-blind problem, patients could be anesthetized prior to the administration of psilocybin or placebo. In line with this suggestion, a recently registered study protocol intends to combine psilocybin with the benzodiazepine midazolam to induce amnesia for the psychedelic experience. ¹⁰⁹ Similarly, in a recent study using a triple-masked randomized placebo-controlled design, MDD patients received a ketamine infusion or a placebo while being anesthetized, thereby effectively avoiding participants from breaking blind. ¹¹⁰ However, both the ketamine and the placebo group showed an antidepressant effect, which might be related to a general placebo response in both groups and/or the antidepressant effects of being anesthetized .

At the minimum, all clinical trials should always assess and disclose the rate of breaking blind: transparency is a first crucial step to get an idea of how severe the problem is. Another easy-to-implement solution is to use blinded analysis of the primary outcome measures, ¹¹¹ or to keep raters blind to the full design of the study including which compound is being tested for further recommendations, see Even et al. ¹¹² We see some progress in this area, given that new statistical tools are under development aiming to ‘control for’ participants who broke the blind during the study, and a new scale has been developed to formally assess whether participants broke blind. ¹¹³

Placebo effects

Prior expectations about the efficacy of treatments have strong effects on participant experience and outcome reporting. ²⁶ In the realm of psychedelic drugs, both the set (i.e. the person’s expectations, bodily, and mental state) and the setting (i.e. the physical context and environmental factors) strongly contribute to the psychedelic experience, amplifying the placebo response.^114,115 This effect may be further enhanced because psychedelics increase suggestibility ¹¹⁶ and thus potentially function as a super placebo. ¹¹⁴ A recent meta-analysis indicated that efficacy of cannabis-based treatments for pain does not exceed placebo effects, and that media-reporting about the efficacy of cannabis is often inaccurate and overly positive, leading to exaggerated expectations by patients. ¹¹⁷ Similarly, given the current hype around psychedelics, the so-called Pollan-effect reflects the exaggerated expectations that have been caused by selective media-reporting about the beneficial effects of psychedelics. ²⁶ We see several challenges posed by this hype.

First, when participants in the placebo group know about the group membership (e.g. because they do not experience the psychedelic experiences they expect), this can lead to demotivation, nocebo effects, and dropout. ¹¹⁸ In the face of disappointment, an intervention group may outperform a placebo group not because the treatment works better than placebo, but because the control group works worse than placebo. Second, broken expectations can lead to false statements that are under-researched. Interviews with participants of the MAPS study have indicated that even though some patients felt worse following the treatment, they ended up reporting improvements. Participants felt immense pressure to report positive outcomes because investigators and media outlets had touted MDMA-assisted psychotherapy as highly efficacious treatments. Some participants said they were worried that honestly reporting their experiences might jeopardize approval of treatments that they understood to be potentially lifesaving for many others. ¹¹⁹

Placebo effects threaten internal validity of treatments, as it remains unclear whether symptom improvement relies on the placebo effect or can truly be attributed to the effects of the psychedelic therapy. Including a third study arm (next to an active placebo), consisting of an additional control condition that receives no treatment at all, allows a formal quantification of the size of the placebo effect and a control for treatment nonspecific factors. ¹²⁰ All studies should assess patients’ and therapists’ expectations prior to and during trials, which provides information necessary to control for expectation and placebo effects. ²⁶ Another solution is to induce more realistic expectations for clinicians and patients by providing explicit information about the current uncertainty regarding the efficacy of psychedelic therapy in the informed consent of studies. More generally, we hope to see more nuanced conclusions in scientific studies, university press releases, and media reports; the current hype directly contributes to validity threats for psychedelic science and does a strong disservice to treatment-seeking populations.

Mechanism remains unclear

Several different working mechanisms have been proposed to account for the effects of psychedelics, including increases in neuroplasticity, neural entropy, or psychological flexibility. ¹²¹ Although some have argued that the subjective experiences are the primary mechanism of action, ¹²² others have suggested the neurochemical effects instead explains efficacy, such as potential neuroplastic effects of psychedelics, that is, the so-called neuroplastogen model. ¹²³ A third model posits that psychedelics loosen maladaptive prior beliefs and increase sensitivity to bottom-up prediction error signaling. ¹²⁴ On this account, the hallmark of most psychopathological disorders is thought to be rigidity in thinking, emotions and behavior, and psychedelics supposedly counter this rigidity, reflected in increased brain entropy following psychedelic therapy. ³⁰

But psychedelic therapy also consists of a psychotherapy component. This component is under-researched, with only 3 out of 21 studies having evaluated the effect of providing psychological support compared to a minimally supportive condition. ¹⁰³ A related practical challenge for evaluating the efficacy of treatments is the wide variety of different therapeutic approaches used interchangeably in psychedelic therapy, often in the same study. This includes emotion-focused, psychodynamic, transpersonal, existential, or nondirective therapies, ¹²⁵ but the MAPS protocol also lists internal family systems, voice dialog, psychosynthesis, Hakomi, sensorimotor therapy, and holotropic breathwork. ⁶⁶ This in turn relates to the lack of standardized training and requirements for therapists: there are currently no formal minimum requirements for professional psychiatrists or psychologists to apply psychedelic therapy.

All proposed mechanisms and therapies have in common that there is a high degree of uncertainty and a lack of strong empirical evidence.^121,126 A parallel may be drawn with research on antidepressants: the serotonin hypothesis remains one of the most widely accepted mechanisms underlying the efficacy of SSRIs, but a recent systematic review indicated that there is little empirical support for this hypothesis as core working mechanisms for SSRIs. ¹²⁷ Much more rigorous and fundamental research is necessary to establish the causal pathways through which psychedelics exert their potential effects to avoid the situation that the SSRI literature finds itself in, where proposed working mechanisms may have been held up in part by false positive findings, an over simplistic narrative convenient for some stakeholders, and potential conflicts of interests.

Together, the lack of a clearly specified mechanism poses a threat to construct validity. Independent and high-powered replication studies are needed to test central predictions from promising neurobiological models specifying the causal-mechanistic pathways of psychedelics, ¹²¹ and work on the interaction between psychedelic use and psychotherapeutic interventions is crucial to move the field forward. In line with the observed importance of therapeutic alliance for predicting clinical improvements, ¹²⁸ including an assessment of the relationship between the client and the therapist will provide insight in the psychotherapeutic component as a crucial mediating factor in psychedelic-assisted psychotherapy. The development of new classes of psychedelics, so-called psychoplastogens, ¹²⁶ which induce neurobiological plasticity without the accompanying psychedelic effects, will allow assessing the importance of the different pharmacological and neural pathways that might contribute to therapeutic efficacy. And clinical research using other methods, such as meditation, sensory deprivation, or breathwork exercises, will shed light on the therapeutic importance of subjective experiences in the therapeutic process.^108,122

Connecting the dots

Validity threats are problematic when they occur in isolation, but valid inferences become exponentially more difficult when validity threats interact with each other. In most psychedelic trials, the lack of appropriate controls, the breaking-blind problem, as well as expectancy and placebo effects typically tend to co-occur. ²⁶ These problems become even more challenging in small and selective samples relying on short-term follow-ups, and on top of that, conflicts of interest come into play, which interact with the lack of proper scientific practices such as transparent preregistration of all measures and analyses, and publishing all findings irrespective of whether they are positive or not. These challenges, together with the lack of safety data, such as reliable and transparent reporting of adverse events, lead to the conclusion that it is too early to draw any firm conclusions regarding the efficacy and safety of psychedelic therapy.

These problems are not unique to psychedelic science and tend to affect treatments in clinical psychology and psychiatry as a whole. But the challenges have been recognized for a long time see for example Sterling et al., ¹²⁹ and numerous solutions are readily available, which as of yet have not been implemented by the psychedelic research community. Psychedelic science is history repeating, and little concerted action has been taken to reduce bias in reporting and publishing of results.

Conclusion: A roadmap for psychedelic science

To improve the rigor and credibility of psychedelic clinical science, we need to set up studies aiming to address as many validity threats as possible. In Table 2, we provide a brief checklist that can be used by researchers, funders, reviewers, and policymakers to vet the quality of psychedelic studies see also Schiavone et al. ¹³⁰ Such criteria can help determining the value of previously conducted work but can also shape the future. Researchers, funders, and policymakers can use this list to assess if planned studies are fit to meet the needs of psychedelic science moving forward. Note that these criteria need to be interpreted in the context of the study, of course; meeting all criteria does not necessarily make a study rigorous, and many studies not ticking all boxes will be valuable, too. NIH recently developed guidelines for funding psychedelic research, concluding that studies that lack ‘basic quality controls and methodological rigor’ should be considered as ‘low priority’. ¹³¹ In our reading, this renders nearly all work in this field as ‘low priority’.

Psychedelic science can move forward by including appropriate control conditions, more participants, more diverse samples, long-term follow-up measurements, and by being a lot more transparent about measures and methods. The hard problems require more rigorous scientific work, but even here, transparency is a necessary first step: the number of people breaking blind, expectations about the treatment, adverse events, and other information we discussed should be collected and reported.

Given the current state of research, strong caution is warranted regarding the hype around psychedelics as treatments: there is not enough robust evidence to draw any firm conclusions about the safety and efficacy of psychedelic therapy. Our hope is that new studies may find credible evidence that psychedelic therapy can be a useful tool for the treatment of specific groups of patients. Until that time, we urge caution repeating the history of so many hyped treatments in clinical psychology and psychiatry in the last century. For psychedelic research in particular, we are not the first to raise concerns and can only echo the warning expressed more than half a century ago:

To be hopeful and optimistic about psychedelic drugs and their potential is one thing; to be messianic is another. Both the present and the future of psychedelic research already have been grievously injured by a messianism that is as unwarranted as it has proved undesirable. ¹³²