Standards for randomized controlled trials of efficacy of psychological treatments

The randomized controlled trial (RCT) is the gold standard of evidence for health interventions, including psychological treatments such as psychotherapies. Indeed, an evidence‐based treatment is defined as having two or more high‐quality supporting RCTs.

RCTs of psychological treatments share many of the basic methods with pharmacological RCTs, including clear definition of the participant sample, randomization to an experimental or control condition, and standardized assessment that is consistent across treatment arms. However, there are some unique challenges to RCTs of psychological treatments, including specifications for the control condition, treatment integrity procedures, and issues related to trials of digital interventions.

The control condition aims to remove the effects of threats to internal validity, such as the natural course of the target condition, the effects of research procedures, and statistical artifacts such as regression towards the mean, thereby enhancing confidence that the outcome of the trial can be uniquely attributed to the experimental condition. While the pill‐placebo is an elegant standard for pharmacological RCTs, there is no predominant solution for RCTs of psychological treatments.

There are many forms of control conditions for these latter RCTs, including no‐treatment and waitlist conditions, treatment‐as‐usual, conditions that provide a portion of the experimental treatment such as attention controls, and active comparators that provide another validated treatment. The choice of control condition is guided first and foremost by the scientific question. The RCT evaluates those treatment elements that are unique to the experimental condition, since the treatment elements common to both the experimental and control conditions are washed out through randomization. If the question is whether or not a treatment can improve outcomes relative to what is currently done, treatment‐as‐usual may be appropriate, if defined correctly ¹ . If the question is whether a specific set of treatment procedures is effective, an active treatment that controls for all other treatment processes may be more appropriate.

One challenge is that control conditions themselves have different effects on the outcome, some obvious and some less obvious ² . The larger the effect of the control condition, the smaller the difference between the control and experimental treatments. For example, control conditions that include an active treatment (e.g., attention control or alternative treatment) will likely produce large effects, and consequently smaller differences between the experimental and control treatment. Control conditions with little or no treatment (e.g., no‐treatment control) produce smaller changes in outcome, thereby resulting in larger between‐treatment effects.

However, control conditions may also produce inadvertent or unexpected effects. For example, two meta‐analyses have found that waitlist controls produce significantly smaller improvements compared to no‐treatment conditions, and thereby larger effects for the experimental treatment relative to the control condition ² , ³ . One possible explanation for this is that participants assigned to a no‐treatment condition recognize that they will not get treatment and search for other solutions to their problems, while waitlist control conditions shut down natural help‐seeking behaviors. Participants assigned to a waitlist condition expect that treatment will come and consequently do what the researchers ask – they wait. Thus, seemingly similar control conditions can produce dramatically different results.

Several decision‐making frameworks for control condition selection have been proposed ⁴ , ⁵ . Generally, these frameworks suggest that earlier phase RCTs that are piloting an intervention should use control conditions that have less of an effect, because the greatest threat to the public good in early phases is killing off the innovation of promising treatments. Later phase RCTs should be more rigorously controlled, as it is critical to protect patients, providers and payers from ineffective or dangerous treatments.

The experimental treatment should be clearly defined by a specific manual. Treating clinicians should be trained and supervised, and the fidelity of treatment administration should be monitored. While these treatment integrity procedures are usually applied to experimental interventions, many RCTs do not apply them to control treatments. A meta‐analysis found that RCTs that do not manualize the control treatment, provide less training and supervision to control treatment therapists, or do not conduct fidelity monitoring of the control condition, produce significantly larger between‐treatment effect sizes than RCTs that apply all of these procedures equally across treatment arms ² . Thus, all research procedures should be applied equivalently across treatment arms, including treatment integrity procedures.

The outcomes achieved in rigorously controlled RCTs are usually diminished in clinical practice. This phenomenon, referred to as “voltage drop” or research‐to‐practice gap ⁶ , is common across medicine, but has some unique considerations in psychological treatments. Many of the research procedures necessary to ensure internal validity reduce generalizability. For example, treatment integrity processes are quality controls that can strengthen treatment potency but are not processes that commonly exist in real‐world settings. Subsequent implementation trials are necessary to evaluate the effects of a treatment under real‐world conditions ⁷ .

Over the past two decades, there has been a dramatic growth in the number of RCTs evaluating digital mental health interventions, delivered via smartphone apps or websites. These interventions can be deployed as fully automated (without any human support), coached or guided (including some low‐intensity human support from a therapist or coach), or as an adjunct to standard treatment. Because a digital mental health intervention is clearly defined through the software code, treatment integrity measures are not needed for the digital portion of the intervention. While treatment integrity procedures are still recommended for the human support components, this support is typically much less complex than psychotherapy, and therefore simpler to manage and potentially more generalizable. Control treatment definition and selection poses similar challenges to those described above, with additional possible control elements including receipt of a sham or alternative app.

Traditional RCT methodology requires that the treatment be held constant throughout the trial, so that it is clear exactly what is being evaluated. In contrast to medications, which do not change at all, and psychotherapies, in which the treatment manuals remain constant, apps change frequently. Once an RCT is completed, an app is likely to be changed before being released and will continue to be modified thereafter. Thus, we argue that RCTs should test the treatment methods and principles of digital mental health interventions rather than the products themselves, as the product will continue to evolve as long as it is available and supported.

Trials of intervention principles (TIPs) ⁸ allow the experimental app to be iteratively improved during the trial by incorporation of new learning, thus reflecting the dynamic nature of digital technologies. A principle statement is articulated a priori, and is used to constrain any iterative changes made during the trial, thereby maintaining internal validity. In accordance with the CONSORT‐EHEALTH reporting guidelines, any such changes to the app should be documented and reported, allowing transparency ⁹ . While TIPs articulate a solution to challenges posed in the evaluation of digital mental health interventions, how evidence is defined for digital products that continually evolve remains an area of debate.

In summary, well‐controlled RCTs of psychological interventions are necessary for the protection of all stakeholders, including patients, from ineffective treatments. Considerations unique to RCTs for psychological interventions include the definition of control conditions and ensuring that treatment integrity procedures are consistent across treatment arms. With the growing number of RCTs for digital mental health interventions, RCT methodologies must further adapt to take into consideration the evolving nature of digital technologies, which may include methods allowing iteration of the experimental app during the trial.