Intervention optimization: A paradigm shift and its potential implications for clinical psychology

Abstract

To build a coherent knowledge base about what psychological intervention strategies work, develop interventions that have positive societal impact, and maintain and grow this impact over time, it is necessary to replace the classical treatment package research paradigm. The multiphase optimization strategy (MOST) is an alternative paradigm that integrates ideas from behavioral science, engineering, implementation science, economics, and decision science. MOST enables optimization of interventions to strategically balance effectiveness, affordability, scalability, and efficiency. In this review we provide an overview of MOST; discuss several experimental designs that can be used in intervention optimization; consider how the investigator can use experimental results to select components for inclusion in the optimized intervention; discuss the application of MOST in implementation science; and list Future Issues in this rapidly evolving field. We highlight the feasibility of adopting this new research paradigm, and its potential to hasten the progress of psychological intervention science.

1. Introduction

Many psychological interventions are made up of multiple components, each of which is expected to contribute to effectiveness. For example, Watkins et al. (2023) hypothesized that seven distinct components (e.g. functional analysis, concreteness training, self-compassion training) would contribute to the success of a cognitive behavioral therapy (CBT) intervention for treating depression. In this review we take a broad view of multicomponent psychological interventions, including interventions that aim to improve mental and/or physical health. From this perspective, psychological interventions are remarkably diverse. The components may all be behavioral or psychological; or the set of components may include one or more medical elements, such as an antidepressant. They may involve participants at any point along the life course, and they may be aimed at the individual or at an aggregate unit such as dyad (e.g. patients and their care partners), family, neighborhood, medical practice, school, faith congregation, or other organization or entity.

This review is based on the premise that the goals of psychological intervention science are to (a) build a coherent knowledge base about what intervention strategies work, why, for whom, under what circumstances; (b) leverage this knowledge to develop interventions that have positive societal impact; and (c) make ongoing improvements in intervention strategies to achieve ever-greater positive societal impact on mental and physical health. To date these goals have been pursued primarily within the classical treatment package research paradigm, in which the emphasis is on establishing whether an intervention as a package of components demonstrates a statistically and clinically significant effect when compared to a suitable control. Within this paradigm, establishing that a treatment package has a statistically and clinically significant effect—i.e., intervention evaluation—is often considered the logical first step in the process of arriving at an effective and implementable evidence-based intervention. This emphasis on interventions as a package has persisted despite calls for the clinical psychology field to attempt to study individual intervention components (Kazdin 2000).

We agree that intervention evaluation is essential. However, for several reasons, we question whether in the long run the classical treatment package paradigm is the best approach to advancing the three goals of psychological intervention science. First, the expectation that the effect of a treatment package will be demonstrated before examining the performance of any of its components has the unintended consequence of incentivizing the development of complex, expensive, and inefficient treatment packages that may be too impractical to implement. Investigators have few reasons to limit the number of components; guard against including weak, inactive, or even potentially counterproductive components; or consider how resource-intensive the resulting intervention will be, and they have every reason to include as many components as they can think of to guard against null results in an evaluation randomized controlled trial (ERCT). Second, if the intervention package produces a statistically and clinically significant effect but is not practical to implement, under the classical treatment package paradigm it is unclear what to do to make it implementable. Making the intervention less resource-intensive by removing some components, or setting some to less resource-intensive levels, risks reducing or even eliminating the intervention’s effect unless the decisions can be based on information about individual component performance—information that is generally not obtainable within this paradigm. Third, the classical treatment package paradigm provides little data useful in identifying the strong and weak parts of an intervention, and therefore little systematic basis for ongoing improvement.

In this article we review the multiphase optimization strategy (MOST; Collins 2018; Collins et al. 2021), a different way of thinking about the science of intervention development, evaluation, and implementation. MOST offers an alternative to the classical treatment package paradigm for pursuing the goals of psychological intervention science. Integrating ideas from behavioral science, industrial and control engineering, implementation science, health economics, and decision science, MOST places intervention optimization as logically precedent to intervention evaluation. The purpose of optimization is to enable the investigator to identify the combination of components that best meets the objective at hand. If the objective is to identify the most effective intervention, optimization can be used to eliminate counterproductive components. However, the objective may not necessarily be to select the most effective intervention in an absolute sense; instead, it will often be to identify the most effective intervention that is practical to implement. Implicit in intervention optimization is acknowledgement that placing limits on resource requirements to make an intervention implementable may take a toll on effectiveness to some degree. Strategic trade-offs are often necessary to achieve positive societal impact. To take an extreme, yet not uncommon, example, a highly effective intervention will have little societal impact if its high resource demands prevent its adoption.

Optimization of an intervention typically requires an optimization randomized controlled trial (ORCT). An ORCT uses any of a variety of experimental designs that are appropriate and efficient for this purpose. ORCTs are designed to systematically assess the performance of intervention components. The experimental results, along with cost and other information, are then used as a basis for identifying the set of components and component levels that represents the optimized intervention according to a clearly specified optimization objective. Only after optimization is the intervention evaluated as a package in an ERCT.

The information gained from ORCTs is useful in building a coherent knowledge base about what intervention strategies work and how it can be improved. The investigator can then conduct subsequent ORCTs to examine new or revised components and re-optimize, in an ongoing cycle. This continual optimization enables incorporation of new intervention and implementation strategies suggested by the research literature, as well as adaptation to changing societal, public health, economic, and implementation circumstances. This has the potential to maintain and even incrementally increase an intervention’s positive societal impact over time.

The purpose of this review is to introduce MOST as a paradigm shift in psychological intervention science, discuss its potential contributions to the field of clinical psychology, and provide enough basic information and suggestions for additional literature to enable readers to consider employing this paradigm in their own research. We begin with an overview of MOST (for a comprehensive introduction to MOST, see Collins 2018). We then go into more detail about the factorial ORCT, focusing on the particularly efficient 2^k design. This is followed by a discussion of decision-making based on the results of a factorial ORCT. Next, we consider optimization of adaptive interventions and two ORCT designs that are useful in this context, the sequential multiple assignment randomized trial (SMART) and the micro-randomized trial (MRT). We briefly review some other ORCT designs. We then take a closer look at the application of MOST in implementation science. We end by offering some ideas for future directions in the rapidly developing field of intervention optimization, and some brief conclusions. Throughout we highlight the feasibility of adopting this new research paradigm, and its tremendous potential to hasten progress toward the three goals of psychological intervention science.

2. Overview of intervention optimization via the multiphase optimization strategy (MOST)

2.1. Definition of optimization

Optimization is a broad concept that has different definitions in different fields and applications. We use a definition adapted from industrial and control engineering for application to intervention science: Optimization is the process of achieving a strategic balance of intervention effectiveness,¹ affordability, scalability, and efficiency (Collins et al. 2021). MOST is a principled approach to intervention optimization—in other words, to development of interventions that achieve this strategic balance, which we call intervention EASE.

Here affordability is defined as the “extent to which the intervention is deliverable within budget, and offers a good value” (Collins et al. 2021, p. 2000). Scalability is the “extent to which the intervention is implementable in the intended setting with no need for ad hoc modifications” (Collins et al. 2021, p. 2000). In practice many factors, such as community support, determine whether an intervention ultimately goes to scale. Thus, intervention optimization does not guarantee that an intervention will be scaled up, but instead can enable the investigator to balance competing considerations to identify an intervention that is practical to scale up. Efficiency is the extent to which the intervention achieves the desired outcome(s) with minimal resource demands, relative to the alternatives. Efficient interventions do not demand participant effort that does nothing to further the intervention’s goals, and therefore may be more ethical than less efficient alternatives. Efficient interventions may also be more effective if they prevent participant fatigue and overwhelm, thereby making it more likely that engagement will be maintained for the duration of the intervention. Affordability, scalability, and efficiency are related but distinct; for example, an intervention could be affordable if it costs less than the upper limit of an implementation budget, but inefficient if it contains inactive components.

2.2. The concept of intervention components

Because intervention optimization using MOST is based on the idea of assessing the individual and combined contributions of intervention components, it is necessary for the investigator to identify intervention components. Intervention components must be conceptually and practically distinct, and although they may, and often will, interact, it must be possible to manipulate them independently in an ORCT. Components may be defined at any level of granularity, from relatively macro (e.g. text messaging as an overall strategy) through relatively micro (e.g. each individual text message). Nearly any aspect of an intervention can be considered a component, as long as it is useful and feasible to separate it out for empirical study. For example, a component may correspond to an element of the treatment itself; or a feature included solely to promote adherence to or engagement in an intervention; or a feature designed to affect the implementation of the intervention, such as specialized training offered to program delivery staff or psychotherapists.

Investigators may find frameworks such as the Behaviour Change Wheel (Marques & Guastaferro 2022, Michie et al. 2014) and Human Centered Design (O’Hara et al. 2022) helpful in identifying distinct intervention components.

2.3. Brief overview of MOST

As Figure 1 illustrates, MOST comprises three phases: preparation, optimization, and evaluation. In the preparation phase the investigator lays the groundwork for intervention optimization through a series of specific activities (Landoll et al. 2022). An essential part of this preparation is deriving or revising a detailed conceptual model. This is a model of the behavioral process to be intervened upon, showing where and how each intervention component is hypothesized to have an effect on the process, and specifying any important moderators hypothesized to alter component effects. It is possible that not all of the components in the conceptual model will be investigated in a given ORCT. Some components may already have been selected for inclusion in the intervention; for example, their effectiveness may have been established in a prior ORCT, or they may represent an established standard of care. The components that will be examined in the ORCT are called candidate components. The conceptual model is theory-based wherever possible, and may be informed by more than one theory. Detailed examples of conceptual models can be found in Gwadz et al. (2017) and Kugler et al. (2018).

Figure 1.

The multiphase optimization strategy (MOST), consisting of three phases: preparation, optimization, and evaluation. Adapted with permission from Collins (2018).

^aORCT = optimization randomized controlled trial.

^bERCT = evaluation randomized controlled trial.

Suppose an investigator who wishes to use MOST to optimize an intervention for depression based on CBT plans to examine the following four components (adapted from Watkins et al. 2016): functional analysis (systematic reduction of undesired behaviors and increasing of desired behaviors), relaxation (e.g. progressive muscle relaxation), challenging negative thoughts (e.g. psychoeducation), and activity scheduling (e.g. reducing avoidance). Any pilot work in preparation for the ORCT, whether it is done to test the acceptability, feasibility, or practicality of these intervention components, to pilot the ORCT itself, or for some other reason, is conducted in the preparation phase of MOST.

In the preparation phase the investigator also identifies the optimization objective (previously called the optimization criterion). This is the investigator’s operational definition of intervention EASE. It contains a statement of the desired outcome(s) of the intervention and the salient constraints. For example, if it is known that no more than $500 per participant will be available to implement the intervention, the optimization objective might be to arrive at the best expected outcome (expressed in terms of the outcome variables) that can be obtained within an upper limit of an implementation cost of $500 per person. Or, it may be determined that staff can spare no more than two hours to implement the intervention. Then the optimization objective might be to arrive at the best expected outcome that can be obtained in two hours or less of implementation time. Our hypothetical intervention scientist wishes to identify a value-efficient intervention, which balances predicted effectiveness and implementation costs without placing an upper limit on costs. There are many different possible optimization objectives. In every case the idea is to arrive at an intervention that represents a realistic and acceptable balance of effectiveness, affordability, scalability, and efficiency.

The ultimate purpose of the next phase of MOST, optimization, is identification of the optimized intervention. Optimization is always completed prior to the evaluation phase, and typically involves assessing the individual and combined performance of the candidate intervention components by means of an ORCT. In the hypothetical CBT example, the intervention scientist would conduct an ORCT on the four candidate intervention components. Figure 1 lists three popular ORCT designs, all of which are efficient ways of examining individual components under consideration for inclusion in an intervention: the factorial experiment (Fisher 1946), and two other designs drawn from the family of factorial experiments, namely the sequential multiple-assignment randomized trial (SMART; Kidwell & Almirall 2023, Murphy 2005), and the micro-randomized trial (MRT; Liao et al. 2016, Qian et al. 2022). By contrast, for research in the evaluation phase, MOST uses an ERCT, which may be a standard two-arm clinical trial or other design from this family. Both ORCTs and ERCTs meet the definition of a clinical trial maintained by the United States National Institutes of Health (National Institutes of Health 2017).

Once the ORCT has been conducted, our hypothetical investigator will use the results to decide which combination of the four components will make up the optimized intervention. As mentioned above, this investigator’s optimization objective is to identify a value-efficient intervention, in other words, an intervention that offers good potential effectiveness in relation to anticipated implementation resource demands. This can be a complex decision-making process, particularly if there are interactions among the components or there are multiple outcomes to be considered. Recent work in decision-making for intervention optimization is reviewed later in this article.

Two fundamental principles of MOST appear in Figure 1. One is the continual optimization principle, which states that the investigator has the opportunity to improve the intervention in an iterative fashion over time. This principle suggests that the hypothetical investigator will immediately return to the preparation phase after the completion of the evaluation phase, armed with information about the intervention’s strong and weak points gained during the optimization phase. The other is the resource management principle, which suggests that the investigator must make efficient use of research resources. For example, in Figure 1 immediately after the optimization phase there is a diamond, indicating that at this point the investigator considers whether or not to proceed to the evaluation phase. If the ORCT data suggest the optimized intervention is reasonably likely to demonstrate a statistically detectable and clinically significant effect, then the investigator is justified in devoting resources to an ERCT. On the other hand, if the optimized intervention does not appear potentially strong enough to warrant an ERCT, the resource management principle suggests returning to the preparation phase. However, this is markedly different from starting over after a nonsignificant effect in an ERCT. The data from the ORCT indicate which components performed satisfactorily and which did not, providing a firm basis for continuing to make progress toward an intervention with high societal impact. The investigator can reconsider the conceptual model, identify the components that must be revised or replaced, and conduct another ORCT.

The resource management principle also suggests that the investigator must select an ORCT design carefully, so as to make the best use of research resources and collect the highest-quality and most pertinent scientific data. We turn now to one particularly efficient experimental design, the 2^k factorial.

3. The 2^k factorial ORCT

3.1. Introduction to the factorial ORCT

In a factorial ORCT, each candidate component is represented by an independent variable, called a factor. The factors are manipulated simultaneously in a coordinated fashion. In a 2^k factorial experiment, there are k factors, each of which has two and only two levels.

Recall that our hypothetical investigator wishes to examine the performance of four components that are candidates for inclusion in a CBT intervention to treat depression: functional analysis, relaxation, challenging negative thoughts, and activity scheduling. Table 1 illustrates a factorial ORCT to investigate the performance of these four components. Each component is represented by a factor, referred to as FA, RE, CN, and AS, respectively. In this experiment each factor has two levels (i.e., a 2×2×2×2, or 2⁴ factorial experiment), and there are 16 experimental conditions (see Gallis et al. 2019, for a helpful discussion of approaches to conducting random assignment). For factors FA, CN, and AS the factor levels are yes (i.e., offered; shaded) and no (i.e., not offered). For RE the levels are relaxation training (shaded) and video. Factorial experiments have a balanced structure, evident in Table 1, in which each level of a factor appears the same number of times at each level of every other factor. For example, consider the no level of FA. As the table shows, RE is set to video four times (conditions 1 through 4) and relaxation training four times (conditions 5 through 8). Now consider the yes level of FA. RE is again set to video four times (conditions 9 through 12) and relaxation training four times (conditions 13 through 16). A perfectly balanced factorial experiment has this characteristic structure, and in addition has equal sample sizes across experimental conditions.

Table 1.

Illustration of 2⁴ factorial experiment

Condition number	Factor
	FA	RE	CN	AS
1	No	Video	No	No
2	No	Video	No	Yes
3	No	Video	Yes	No
4	No	Video	Yes	Yes
5	No	Relaxation training	No	No
6	No	Relaxation training	No	Yes
7	No	Relaxation training	Yes	No
8	No	Relaxation training	Yes	Yes
9	Yes*	Video	No	No
10	Yes	Video	No	Yes
11	Yes	Video	Yes	No
12	Yes	Video	Yes	Yes
13	Yes	Relaxation training	No	No
14	Yes	Relaxation training	No	Yes
15	Yes	Relaxation training	Yes	No
16	Yes	Relaxation training	Yes	Yes

^*Shading highlights the different levels of the factors. The main effect of each factor is the mean of the shaded experimental conditions minus the mean of the unshaded experimental conditions. For example, the main effect of FA is the mean of experimental conditions 9 through 16 minus the mean of experimental conditions 1 through 8.

The logical underpinnings of any factorial experiment are different from those of an ERCT. When properly analyzed using factorial analysis of variance (ANOVA), a 2^k factorial experiment enables estimation of the main effect of each of the k factors (where k=4 in the example), and interactions involving two or more factors up to the k-way (in the example, 4-way) interaction. When, as we recommend (Kugler et al. 2018), the analysis is based on effect coding (−1,1 in the case of two levels in a factor), two different types of quantities are estimated. The main effect of a factor is an estimate of that factor’s effect averaged across the remaining factors. The interaction between, say, Factor A and Factor B reflects the degree to which the effect of Factor A varies depending on the level of Factor B, averaged across the remaining factors. (An ANOVA based on dummy (0,1) coding yields the same omnibus F but estimates of different quantities; see Kugler et al. 2018.)

Effect estimates based on an ERCT are obtained by direct comparison of the means of experimental conditions. By contrast, main and interaction effect estimates based on a factorial experiment are obtained by comparing the means of specific combinations of experimental conditions. This can be illustrated by returning to Table 1. The main effect of the factor FA is estimated by comparing the mean of the eight conditions in which FA is set to yes (shaded dark in the table) to the mean of the eight conditions in which FA is set to no. Thus, the main effect of FA is defined as the effect of FA averaged across the remaining factors. In this 2⁴ experiment, data from all participants contribute to this effect estimate. Similarly, the main effect of the factor RE is estimated by comparing the mean of the eight conditions in which RE is set to yes (shaded dark) to the mean of the eight conditions in which RE is set to no. Again, data from all participants contribute to this effect. The same logic holds for the estimate of the main effect of CN and the main effect of AS. An interaction involving, say, FA and RE is said to occur if the effect of FA differs across the levels of RE, averaged across all levels of the other factors in the experiment.

Because in a properly conducted factorial ANOVA each participant contributes to the estimation of multiple effects, factorial experiments typically use many fewer participants than alternative approaches that have been used in clinical psychology, such as dismantling studies and other component designs. As Papa and Follette (2015) explained, “Component designs include either adding or subtracting specific elements of a treatment protocol to assess the relative effects of the component in the context of the other elements in the design” (p. 1). Component designs involve direct comparison of individual experimental condition means and therefore have statistical properties similar to the ERCT. Consider a component design comprising five experimental conditions: one control condition, and four experimental conditions each setting one of the components to yes and the others to no. As compared to a factorial experiment, the component experiment would require 2.5 times the participants to achieve the same level of statistical power. A separate experiment for each component would require four times as many participants. In both cases the quantity being estimated confounds main effects and interactions (Collins et al. 2009). The SMART and the MRT, reviewed below, are in the general family of factorial experiments, and similarly make use of the idea of aggregating experimental conditions in a principled and efficient fashion.

We wish to highlight two related characteristics of the factorial experiment, some of which may initially appear counterintuitive to those trained primarily in the ERCT. First, because effects are estimated based on combinations of experimental conditions, the power associated with a factorial experiment is only indirectly related to the per-condition sample size. Instead, power is directly related to the sample size per level of a factor. For this reason, factorial experiments can have very small per-condition sample sizes and nevertheless be well-powered if the aggregate sample size per level is sufficiently large. Second, it is often possible to add one or more factors to a factorial experiment without needing to add more participants to maintain power, if the additional effects to be estimated are no smaller than the ones upon which the original power analysis was based (assuming the results are to be interpreted from a decision-priority perspective; we return to this idea below). This is because although adding one or more factors increases the number of experimental conditions and therefore results in a smaller per-condition sample size (and potentially adds logistical complexity), if the sample size per level of a factor is identical, power will not be appreciably diminished.

The factors in a factorial experiment can, in theory, have any number of levels. However, as we have mentioned, the 2^k experiment is particularly efficient. Consider a factorial ANOVA conducted within the regression framework and using effect coding. In this case, to power a 2^k factorial experiment it is necessary only to identify the smallest regression coefficient of scientific interest (power for factorial experiments is usually based on main effects) and power the experiment to detect that one. Any regression coefficient equal or greater in magnitude will be detectable with at least the selected level of power (Cohen 1988). As compared to a 2^k design, a factorial design including more than two levels of even one factor can require a substantial sample size increase, often ≥50%.

3.2. A few words about interactions

In our view, understanding interactions is essential for building a coherent intervention science knowledge base, and for ongoing improvement of psychological interventions. Only the factorial family of experiments enables estimation of interactions among components. As mentioned above, in the particularly efficient balanced 2^k experiment, all else being equal, statistical power is identical for detection of any regression coefficient of a given size. This holds whether the regression coefficient corresponds to a main effect or an interaction of any order (again assuming effect coding is used in the analysis; for more about statistical power for detection of interactions, see Chapter 4 in Collins 2018). For readers tempted to avoid complexity by choosing experimental designs that do not feature interactions, we wish to point out that if an interaction is present it will manifest itself in data no matter what experimental design is used, but only the family of factorial designs enable disentangling the interaction from other effects (Collins et al. 2009).

3.3. The concept of control in a factorial experiment

The concept of control is different in a factorial experiment as compared to an ERCT. In an ERCT there is typically a single control arm, whereas in a factorial experiment there is no single experimental condition that serves as “the” control condition. Instead, one level of each factor is the designated control for that factor. In Table 1, the conditions in which the factor is set to the no level serve as the control for that factor. For the FA factor, the aggregate of conditions 1 through 8 serves as the control; for RE the aggregate of conditions 1—4 and 9—12 serves as the control; and so on. The factors may have different types of controls; in this example, three of the factors use a “no” level as the control, whereas a video is used as the control for the RE factor.

3.4. Conclusion-priority vs. decision-priority perspectives

There are two related but distinct perspectives an investigator can take on the data from a 2^k experiment, or indeed any ORCT (Collins 2018). From a conclusion-priority perspective, the purpose of an experiment is to draw scientific conclusions about a hypothesized effect, under the scrutiny of fellow scientists. This perspective requires, for example, adhering to well-established conventions about Type I (α ≤ .05) and Type II (β ≤ .20) error rates. By contrast, from a decision-priority perspective, the purpose of an experiment is to use available resources efficiently to provide information that will enable the investigator to decide which components and component levels will constitute the optimized intervention, which may then be evaluated further. From this perspective, the investigator is less constrained by scientific conventions, and instead uses the experimental data in any reasonable and transparent manner. The investigator may, for example, wish to raise the Type I error rate above the conventional level in order to reduce the Type II error rate, and accept an increased probability of mistakenly including an inert component to reduce the probability of overlooking a potentially useful one.

The same data may be approached from either perspective, depending on the scientific objective. For example, we do not advocate using a Bonferroni or similar adjustment to maintain a specific Type I error rate when using the results of an ORCT for decision-making purposes. When conducting analyses intended to test hypotheses about whether particular components have a statistically detectable effect, or when testing mediation (Piper et al. 2017, Strayhorn et al. 2022) or moderation (Piper et al. 2017) hypotheses with data from an ORCT, such an adjustment could be considered. A conclusion-priority perspective is typically appropriate when conducting an ERCT.

4. Decision-making and integration of resource considerations

Suppose our hypothetical investigator has conducted the 2⁴ factorial ORCT illustrated in Table 1. The 16 experimental conditions represent different combinations of components, each of which is an alternative version of the intervention, and the goal of the optimization process is to select the alternative that best meets the optimization objective. How can the investigator accomplish this?

Collins et al. (2014) recommended a component screening approach for identifying the optimized intervention based on the results of a factorial experiment. This approach is a systematic way of sorting through the many effect estimates provided by a factorial ANOVA to screen out underperforming components. However, a component screening approach has important limitations; for example, it cannot accommodate more than one outcome variable of interest; it can be laborious; and complex patterns of interactions can make decision-making difficult. In this section we summarize some very recent work (Strayhorn et al. 2023a,c) proposing a new and demonstrably better approach for selecting an optimized intervention based on the results of a factorial experiment. The new approach to decision-making relies on relative comparisons among the posterior expected outcomes associated with each alternative intervention under consideration (Strayhorn et al. 2023a).

When the optimization objective involves simply maximizing expected effectiveness on a single outcome variable of interest, choosing an optimized intervention can be relatively straightforward. Suppose the outcome variable in the hypothetical ORCT is a mental health inventory, where a higher score indicates better mental health. Strayhorn et al. (2023a) recommended conducting a factorial ANOVA by fitting a regression equation using a Bayesian approach, thereby producing a posterior expected outcome, in other words, a $\widehat{Y}$, for each experimental condition. In our example, the optimized intervention would be the combination of components and component levels associated with the best expected outcome—in this hypothetical case, the largest $\widehat{Y}$ on the mental health inventory. Strayhorn et al. (2023a) demonstrated via statistical simulation that this vastly easier approach works better than the component screening procedure. If instead the optimization objective is to select, say, the combination of components that produces the best expected outcome subject to an upper bound on implementation cost, then the most favorable $\widehat{Y}$ corresponding to an alternative under that limit can be selected.

However, recall that our hypothetical investigator wishes to include cost in decision-making. Suppose per-person implementation costs for each component level are as follows: functional analysis, $400; relaxation, $100 for the video, $200 for relaxation training; challenging negative thoughts, $300; and activity scheduling, $500. The alternative intervention costs range from $100 for the version where only the relaxation video is provided, to $1400 for the version where all four components, including relaxation training, are provided. As mentioned above, this hypothetical investigator’s optimization objective did not specify an upper limit on cost, but instead states the desire for a value-efficient intervention. In other words, this investigator’s definition of intervention EASE is a strategic balance of anticipated effectiveness (defined in terms of the mental health measure) against resource demands (defined in terms of per-person implementation cost).

This now becomes a multi-criteria decision problem, in which the criteria are the outcome variable—the mental health inventory—and cost. In any multi-criteria decision problem, the alternative interventions under consideration are likely to differ in their performance on the various criteria, often in ways that complicate the selection of an optimized intervention considerably. Because it may not be possible to simultaneously achieve the “best” on every criterion that matters to investigators, selecting one optimized intervention can require that investigators confront challenging tradeoffs among criteria, such as the not-unusual tradeoff between effectiveness and cost.

Decision analysis for intervention value efficiency (DAIVE; Strayhorn et al. 2023c) is an emerging decision-making framework that facilitates this sort of complex multi-criteria decision-making in the selection of an optimized intervention. To facilitate decision-making, DAIVE (1) plots the alternative interventions according to their expected outcomes and their costs on an x-y plane and (2) highlights the “value-efficient” alternatives, those that cost less than alternatives while producing the same or better expected outcomes. Figure 2 shows this plot for our hypothetical example (based on artificial data). Each plot point represents the expected outcome (normed to range between 0 and 1) on the X axis and anticipated per-person delivery cost on the Y axis for one of the 16 combinations of components (i.e., alternative interventions) included in the 2⁴ ORCT. The subset of value-efficient alternatives (highlighted and connected with a solid line) makes up the “value efficiency frontier.” The value efficiency frontier by definition begins with the alternative that costs the least and ends with the alternative that is associated with the best expected outcome. Each intervention on the value efficiency frontier could technically be chosen as the optimized intervention, depending on the decision-maker’s willingness to commit additional resources in order to pay for improved effectiveness.

Figure 2.

Graph showing expected outcome (x-axis) and anticipated per-person delivery cost (y-axis) of each of a set of alternative versions of an intervention to deliver cognitive behavioral therapy (CBT). Each point represents a combination of candidate components. The green line shows the value efficiency frontier. Based on artificial data.

In Figure 2 there are three alternative interventions on the value efficiency frontier: the minimal intervention, which consists only of the relaxation video; the intervention consisting of relaxation training only; and the intervention consisting of functional analysis, relaxation training, and challenging negative thoughts. This last intervention is associated with the best expected outcome, which makes it a great option for the optimized intervention. However, at $900 it is also the most expensive alternative on the value efficiency frontier. The alternative that contains relaxation training only delivers an expected outcome that is relatively close to the best expected outcome—and at considerably lower cost ($200). An investigator open to strategically “trading” some effectiveness for a lower delivery cost in this way might tentatively select the intervention made up of relaxation training only, rather than the one expected to be more effective but also costlier. Before a definitive selection is made, results would be further contextualized using examination of main and interaction effects and, in some cases, additional steps to determine whether continuing on to the evaluation phase of MOST and an ERCT would be justified.

The above example offers one simple illustration of DAIVE, based on a single intervention outcome (the mental health measure) and one particular cost (monetary per-person cost of intervention delivery). One of the exciting advantages of DAIVE is the flexibility to accommodate optimization objectives in which effectiveness is defined in terms of multiple outcomes (Strayhorn et al. 2023c), e.g., several health outcomes, or one health outcome and one empirical implementation-related outcome variable, and so on. In these cases, the multiple outcomes are combined using a value function carefully selected to reflect the investigators’ preferences, including which outcomes, if any, matter more than others, and under what circumstances. Selection of the optimized intervention now involves relative comparisons among the expected values associated with each alternative intervention, based on the selected value function. Furthermore, because “cost” in DAIVE can be defined in terms of any limited resource that is relevant for the implementation of an intervention, efficiency can be considered in any number of ways. For example, DAIVE could be applied to strategically balance anticipated effectiveness on one or more outcomes with efficiency in the use of provider time.

5. Optimization of adaptive interventions

5.1. Fixed and adaptive intervention strategies

It is helpful to distinguish between fixed and adaptive (Almirall & Chronis-Tuscano 2016, Collins et al. 2004) interventions, because this can be a critical consideration in choice of ORCT design. Fixed interventions, like the hypothetical CBT example above, call for all participants to be offered the same treatment in a uniform manner. There may in practice be variability in treatment, as when a participant does not engage in every aspect of treatment, but this unplanned variability is not a part of the treatment strategy. By contrast, an adaptive intervention calls for the treatment to be varied in a prespecified and principled fashion across individuals and within individuals over time. The decisions about what treatment to offer an individual are made according to a set of decision rules at key decision points, based on measured tailoring variables. Adaptive interventions vary in terms of the timescale of adaptation (Montgomery et al. 2011; Nahum-Shani et al. 2018b, 2022b). Interventions with slower timescales of adaptation have fewer decision points—that is, fewer opportunities to consider changing the treatment—spaced farther apart in time, whereas interventions with faster timescales of adaptation have more decision points spaced closer together.

There are two key reasons to consider employing an adaptive rather than a fixed treatment strategy. One is to attempt to produce better outcomes, for example by providing additional treatment for participants who do not obtain a satisfactory result with an initial treatment. A second reason is to attempt to conserve resources and increase efficiency by providing no more treatment than needed; this has the potential to enhance intervention affordability, scalability, and efficiency, as well as to produce better outcomes by minimizing participant burden and habituation.

As will be demonstrated shortly, in addition to all the types of components that are investigated in optimization of fixed interventions, in optimization of adaptive interventions the tailoring variable, the timing or number of decision points, or any aspect of the decision rule can be considered a component and investigated in an ORCT.

5.2. Optimization of adaptive interventions with relatively slow timescales of adaptation: The sequential multiple assignment randomized trial (SMART)

The SMART is an ORCT used for optimization of adaptive interventions with relatively slow timescales of adaptation. Consider a hypothetical adaptive intervention aimed at encouraging adherence to HIV treatment, consisting of taking a daily pill. Compliance is measured daily via an electronic pill bottle cap. The outcome is adherence, operationalized as number of daily pills taken during the final month of the intervention, days 90-120. All participants are offered encouraging text messaging, peer mentoring, and behavioral skills training extending over an initial 45-day period. At 45 days, adherence is assessed in terms of number of daily pills taken during this time. Those who are designated adherent, according to a prespecified cut-off score (discussed further below), are provided with encouraging text messaging going forward, and no other treatment. Those who are designated non-adherent continue to be provided with encouraging text messaging and peer mentoring, and in addition are offered motivational interviewing. In this simple adaptive intervention, there is a single decision point, occurring at 45 days. The measure of adherence serves as the tailoring variable. The decision rule is: “If adherent, step down to text messaging only. If non-adherent, step up by adding motivational interviewing.”

Now suppose an investigator wishes to optimize an adaptive intervention similar to the one just described. The investigator has identified the following research questions:

1. Which operational definition of the tailoring variable results in more daily pills taken during days 90-120: adherence defined as taking the pill (a) ≥ 35 times or (b) ≥ 40 times over an initial 45-day period?

2. For those participants designated as adherent at Day 45, which is a better second-line treatment: (a) step down to text messaging only or (b) continue with text messaging and peer mentoring?

3. For those participants designated as non-adherent at Day 45, which is a better second-line treatment: (a) continue with text messaging and peer mentoring, or (b) step up by adding motivational interviewing?

Thus, three candidate components are being investigated: the operational definition of the tailoring variable, adherence; the second-line treatment strategy for adherent participants; and the second-line treatment strategy for nonadherent participants.

These three research questions can be addressed in the SMART illustrated in Figure 3. In this experiment, initially all participants are offered text messaging, peer mentoring, and behavioral skills training; this is a “given” and not examined in the experiment. In the first randomization on Day 1, each participant is assigned to one of two operational definitions of adherence. The next randomization, on Day 45, is conditional on whether the participant is designated adherent or non-adherent based on their assigned operational definition. Those designated adherent are randomly assigned to one of two second-line treatments; similarly, those designated non-adherent are randomly assigned to one of two second-line treatments. This design includes eight experimental conditions, labeled A—H on Figure 3.

Figure 3.

Illustration of a sequential multiple-assignment randomized trial (SMART). Each participant is randomized twice. On Day 1 each participant is provided with the same intervention and randomized to an operational definition of the tailoring variable, adherence. On Day 45, each participant is randomized to second-line treatment, conditional on whether they are designated adherent or non-adherent based on their assigned definition of the tailoring variable.

The SMART is a type of factorial design, and uses the same logic of comparing means based on aggregates of experimental conditions. However, in a SMART some research questions pertain to all participants, whereas others pertain only to certain subgroups, reducing the sample size available for those questions. Question 1, the comparison of the two operational definitions of the tailoring variable, pertains to all participants and can be addressed by comparing the mean outcome across conditions A—D to the mean outcome across conditions E—H. Question 2 pertains only to participants designated as adherent. It can be addressed by comparing the mean outcome of conditions A and E to that of conditions B and F. Similarly, Question 3, which pertains only to participants designated as non-adherent, can be addressed by comparing the mean outcome of conditions C and G to that of conditions D and H. Because some research questions will be addressed based on a subset of participants, careful planning is required to ensure sufficient power in a SMART (Dziak et al. 2023; Kidwell et al. 2018; Yap et al. 2023).

5.3. Optimization of adaptive interventions with relatively fast timescales of adaptation: Just-in-time adaptive interventions and micro-randomized trials

Just-in-time adaptive interventions (JITAIs; Nahum-Shani et al. 2018b) have relatively fast timescales of adaptation—daily, hourly, even minute-to-minute. They use dynamically changing information about the individual’s internal state (e.g., craving, stress), and context (e.g., physical location), usually obtained from self-reported and/or sensed data from wearables and smart devices, to adapt intervention delivery to the individual’s state and context in daily life. JITAIs have particularly high potential to minimize burden and avoid habituation, because technology enables them to deliver an intervention only when necessary (i.e., only when states of vulnerability and/or opportunity occur) and only when the individual is receptive and likely to engage with the intervention, where engagement is defined as the investment of energy in the intervention (Nahum-Shani et al. 2022c).

For illustrative purposes, consider a simple hypothetical JITAI designed to increase self-monitoring of dietary intake in overweight and obese adults. Obese/overweight adults who are motivated to lose weight are asked to specify three windows of time in which they typically eat their main meals (breakfast, lunch, and dinner). Then, participants receive a weight loss app for their mobile devices. The app enables the participant to record their dietary intake and to view relevant graphical feedback. At three decision points per day, at one hour after the person’s pre-specified mealtime (for breakfast, lunch, and dinner), if dietary intake for the current meal was not recorded in the app, then a prompt is delivered immediately via the app (if the person is not driving) encouraging the individual to self-monitor their dietary intake. Otherwise, no prompt is delivered.

The MRT is an ORCT that is used in optimizing a JITAI. Like the SMART, the MRT involves sequential randomizations. However, as compared to a SMART, in an MRT there are many more randomizations, and they occur much more rapidly: each individual is randomized at each of many decision points over the course of the study, and because of the fast timescale of adaptation that characterizes the JITAI, the length of the time interval between randomizations is relatively short (e.g., hours).

Suppose investigators developing a weight loss intervention would like to answer the following question about a JITAI for promoting dietary intake self-monitoring: Does sending (vs. not sending) a reminder one hour before a meal increase the likelihood of self-monitoring in the next two hours? To answer this question, the investigators design the following (hypothetical) MRT: At the beginning of the study, obese/overweight adults are provided with the weight loss app as described above, and asked to specify the windows of time in which they typically eat their main meals. During a 12-week period, the app passively collects information about the participant’s self-monitoring behaviors. Three times daily, one hour before the participant’s pre-specified mealtime, they are randomly assigned (with 0.50 probability) to receive either a text message (again, if they are not driving) encouraging them to self-monitor their dietary intake, or no message. The proximal outcome, whether or not the participant self-monitored their dietary intake during the following two hours (binary outcome), is assessed following each randomization.

The hypothetical MRT discussed above includes a single experimental factor that can take either of two levels—prompt vs. no prompt—on each of three decision points per day over the entire 12-week period. The question motivating this MRT can be addressed by pooling outcome data across all 252 decision points (3/day, 7/week, 12 weeks) and N individuals. Specifically, this will involve comparing the probability of self-monitoring in the next two hours following all the decision points in which a prompt was delivered (across all individuals) to the probability of self-monitoring in the next two hours following all the decision points in which a prompt was not delivered (across all individuals). Because MRTs enable investigators to answer scientific questions about intervention effects based on a combination of between-person contrasts and within-person contrasts in the proximal outcome, they typically require fewer experimental participants relative to comparable between-subject experiments (Qian et al. 2022).²

This hypothetical example includes a single factor, but MRTs can include multiple factors. Whereas in a standard factorial design like the 2^k discussed above, the main effect of a factor is computed by averaging over the levels of the other factors, in MRTs the main effect of a factor is computed by averaging over the levels of the other factors and all decision points (Qian et al. 2022). The meaning of main effects in an MRT is different from standard factorial designs. Because MRTs involve many decision points in which individuals are randomly assigned to intervention options, main effects are interpreted in reference to how intervention options were delivered prior to a given decision point (for more details see Qian et al. 2022). In the current example, the schedule of intervention delivery prior to each decision point involves random assignment with 0.50 probability to either sending or not sending a reminder at all prior decision points. Hence, the main effect of sending (vs. not sending) a reminder at a given decision point assumes that the reminder was sent about 50% of the times at previous decision points. This effect would not generalize well to settings in which a reminder was sent 10% or 90% of the times. Thus, the randomization probabilities in an MRT shape the meaning of the main effect; this is different from standard factorial designs where the randomization probabilities concern the design of the trial and are mainly guided by considerations related to power for detecting the effects of interest. In MRTs the randomization probabilities are not only part of the experimental design, but they also hold scientific meaning and practical implications for intervention design.

In an MRT, as in the SMART, random assignment may be restricted to specific conditions, conditional on a measured variable. Restrictions may be informed by prior empirical evidence or practical considerations suggesting that certain JITAI options should not be considered under specific conditions. For example, individuals may not be eligible to be randomized to be sent/not sent a reminder if they are driving a car or if they activate the “do not disturb” mode on the smartphone (Nahum-Shani et al. 2021).

In many ORCTs and ERCTs the primary outcome is distal, meaning that it is intended to capture phenomena that unfold or change over a relatively long period of time. Thus, the primary outcome is typically measured at multiple later time points following randomization (e.g., weight measured at months 3, 4, 5 and 6). However, the primary outcome in an MRT is proximal, meaning that it is intended to capture phenomena that unfold or change over a relatively short period of time. In the hypothetical example above, the intervention was aimed at distal weight loss, but the proximal outcome was self-monitoring. Thus, the primary outcome in an MRT is always measured soon after each randomization and is measured repeatedly because there are repeated randomizations.

6. Other experimental designs useful in intervention optimization

6.1. Fractional factorial designs

Fractional factorial designs are used to address the same scientific questions as corresponding standard factorial designs, but using only a reduced number, or fraction, of the experimental conditions, while requiring the same sample size. Fractional factorial designs are worth considering when (a) reducing the logistical complexity of implementing a large number of experimental conditions would increase efficiency, or (b) cluster randomization must be used (see next section), and the overall sample size provides adequate statistical power but the number of clusters is insufficient to populate the experimental conditions (Dziak et al. 2012). In selecting a fractional factorial design, the investigator explicitly trades precision of effect estimates for efficiency, because fractional factorial designs result in some effects being combined with others (this phenomenon, called aliasing, occurs in designs from the ERCT family too; see Collins et al. 2009). Examples of fractional factorial ORCTs include Collins et al. (2011), Gwadz et al. (2017), Watkins et al. (2016), and McClure et al. (2012). An introduction to fractional factorial designs can be found in Collins (2018), and Wu and Hamada (2011) provide a comprehensive overview. Before making a final selection of a fractional factorial design it is a good idea to seek input from an expert on experimental design.

6.2. ORCTs with cluster randomization

Cluster randomization is frequently necessary in ORCTs in school-based research (Caldwell et al. 2012), implementation science (Kilbourne et al. 2014), and other areas, because (1) frequent interaction between participants within a cluster would lead to contamination between experimental conditions, or (2) outcomes at both the individual and cluster level are of interest. Clusters may be naturally occurring (e.g. schools) or experiment-induced (e.g. therapy groups). As compared to randomization of individual participants, cluster randomization in an ORCT or ERCT usually requires more participants to achieve the same level of statistical power (Murray 1998). Because ORCTs tend to have many experimental conditions, there may be enough individual participants (e.g. students) to achieve a desired level of statistical power, but not enough units (e.g. schools) to populate the experimental conditions. In this case, a fractional factorial design may be a good choice. More about cluster randomization in ORCTs can be found in Dziak et al. (2012), Nahum-Shani & Dziak (2018), Almirall et al. (2018), and Nahum-Shani et al. (2018a).

6.3. Evaluation-optimization trials

The continual optimization principle of MOST suggests that investigators should be thinking about further improvements to their optimized interventions on a continual basis. Suppose an investigator’s next step is an evaluation of an optimized intervention, and at the same time a set of components that are candidates to be added to the optimized intervention has been identified. A time-saving alternative to conducting a separate ERCT and ORCT is the evaluation-optimization trial (E-ORCT), which enables evaluation of the optimized intervention along with examination of the performance of a set of candidate components. For example, Tanner et al. (2021) had previously optimized itMatters, an intervention aimed at prevention of excessive alcohol use and risky sex in college students (Wyrick et al. 2020). In the process of designing the ERCT, the investigators learned college administrators who would ultimately adopt the intervention were keenly interested in adding a component on prevention of sexual victimization. As a response, Tanner et al. (2021) conducted a 2² experiment rather than a traditional two-arm ERCT. One factor examined the performance of the optimized intervention, with levels optimized intervention or control; the other examined the performance of the sexual victimization component, with levels included and not included. It would have been arguably more efficient to include additional factors corresponding to additional candidate components. The E-ORCT is appropriate only when the candidate components are to be added to the optimized intervention, not when the candidate components are under consideration as substitutes.

In the E-ORCT, the ERCT is one of the factors in a factorial experiment, i.e. it is crossed with all remaining factors. This is emphatically not the same as adding a separate control “arm” to an ORCT with the objective of conducting an ERCT by comparing its mean to that of whichever experimental condition ultimately corresponds to the optimized intervention. We do not in general advocate adding a separate control “arm” to any ORCT, because it usually wastes resources; an efficient ORCT is unlikely to contain enough participants in any one experimental condition to provide adequate power for an ERCT.

6.4. Multiple-timescale ORCT designs for optimization of adaptive interventions

Adaptive interventions often involve multiple timescales. In some settings, decisions about certain components can be made only once (i.e., at a single decision point), whereas decisions about other components can be made several times during the intervention (i.e., at multiple decision points). The hybrid experimental design (HED) is a new experimental approach developed to help investigators answer scientific questions about the delivery and adaptation of multiple intervention components on multiple timescales—slow and fast (Nahum-Shani et al. 2022a). The HED involves sequential randomizations on multiple timescales corresponding to the proper timescale at which each component should be considered. The HED is highly flexible and can take various forms depending on the scientific questions motivating the study. For example, suppose that investigators would like to answer the following questions about the development of a 12-week weight loss intervention that integrates a mobile app and coaching:

Question A: Does sending (vs. not sending) a reminder before each main mealtime benefit self-monitoring in the next 2 hours?

Question B: Does delivering (vs. not delivering) a brief coaching session at weeks 0 (program entry), 4 and 8 amplify the proximal benefits of sending (vs. not sending) a reminder before each main mealtime?

These questions and others can be answered via an HED in which participants are randomly assigned three times daily, before each main mealtime, to either sending a text message that encouraged them to self-monitor their dietary intake or no message (with 0.50 probability). Additionally, the HED will randomly assign participants at weeks 0, 4, and 8 to either a brief coaching session or no coaching session (with 0.50 probability). Here, randomizations are employed at two timescales: every few hours and every four weeks. There are many other variations of the HED, depending on the scientific questions motivating the study (Nahum-Shani et al. 2023).

6.5. System identification

As compared to the other approaches discussed in this review, system identification represents a different perspective on intervention optimization, one that has emerged from control systems engineering (Guo et al. 2022; Rivera et al. 2018). From this perspective, human behavior can be viewed as a dynamical system, and adaptive interventions with fast timescales of adaptation can be viewed as engineering controllers built to influence the system to achieve a specified outcome. System identification, which typically requires collection of intensive longitudinal data by methods such as ecological momentary assessment (Shiffman et al. 2008) and frequently involves experimentation, forms the basis for modelling the dynamical system. Standard engineering approaches, such as model predictive control, can be used to develop a controller. Examples of application of this approach include development of an adaptive intervention to moderate gestational weight gain (Downs et al. 2021) and identification of the appropriate intervention dose (Voils et al. 2014).

7. Optimization and implementation science: Wraparound interventions

7.1. MOST and implementation science

As applied in the field of clinical psychology, implementation science aims to determine how psychological interventions can fulfill their potential for positive societal impact (Stirman & Beidas 2020), for example by the study of how to ensure fidelity of intervention delivery or expand an intervention’s reach. MOST intersects with the field of implementation science in at least two ways. First, as discussed above, MOST can be used to develop and evaluate an intervention that has the potential to achieve a positive societal impact because it is designed to meet the demands and constraints of the intended real-world setting from the outset. This can be accomplished by using an optimization objective that includes considerations such as implementation budget, demands on staff time, and so on in decision-making. Second, MOST can be used to optimize an implementation strategy to facilitate an intervention’s adoption, implementation, dissemination, and scale-up (Curran et al. 2012, Leeman et al. 2017, Proctor et al. 2013). In this section we highlight this second way MOST intersects with implementation science.

7.2. Implementation strategies as “wraparound” multicomponent interventions

It is widely recognized that appropriate multicomponent implementation strategies are necessary to achieve desired implementation outcomes (Aarons et al. 2011, Baker et al. 2010, Perry et al. 2019, Powell et al. 2017, Weiner et al. 2012). In fact, more than 70 implementation strategies have been identified, falling into six categories: planning, educating, financing, restructuring, managing quality and attending to policy (Powell et al. 2012, 2015). Implementation frameworks, such as the Consolidated Framework for Implementation Research (CFIR; Damschroder et al. 2009, 2022), may be used to identify barriers that implementation strategies must overcome to achieve the goals of adoption, implementation, or dissemination. However, there has yet to be much empirical examination of individual implementation strategies, so it is unknown which implementation strategies are most effective and whether (or how) they may interact.

Optimizing multicomponent implementation strategies is an important next step for the fields of intervention optimization and implementation science. Consider a psychological intervention that has been optimized, evaluated, and deemed ready for implementation. Now, to achieve a high degree of positive societal impact, an effective, affordable, and efficient multicomponent strategy for implementing this intervention must be developed. MOST can be applied to optimize a multicomponent implementation strategy, in much the same way other interventions are optimized, if the implementation strategy is considered an intervention itself—a “wraparound” with respect to the psychological intervention (Guastaferro & Collins 2021).

7.3. Fixed and adaptive wraparound interventions

A wraparound intervention may be fixed or adaptive. For example, Feinberg and colleagues are optimizing a fixed wraparound intervention for the implementation of Family Navigation (R01 MH117123). Family Navigation, an evidence-based care management strategy designed to reduce disparities in care, is a complex intervention comprised of multiple strategies (i.e., motivational interviewing, problem-solving, education, care coordination) that can be implemented in multiple ways (Broder-Fingert et al. 2019). The investigators are conducting a 2⁴ factorial experiment to identify which implementation strategies, independently and in combination, contribute to a family’s achievement of a behavioral health service goal in the shortest period of time (i.e., within 90-days of randomization). At this writing, the project is ongoing.

Kilbourne et al. (2014) used a SMART to design an adaptive wraparound intervention to improve the success of Replicating Effective Programs (REP). REP is an effective implementation strategy designed to support implementation of evidence-based programs in care settings. However, when REP was applied to the implementation of a psychosocial evidence-based program for mood disorders, there was low uptake (< 50%) across sites, which led to minimal improvements in clinical outcomes (e.g., depression symptoms). It was hypothesized that REP needed to be enhanced with other implementation strategies in some settings. The SMART enabled investigators to compare the effectiveness of enhanced implementation strategies among clinics not responding to initial implementation support (i.e., REP). Results indicated that patients from clinics that were randomized to receive the more intensive implementation support did not have significantly improved clinical outcomes compared to patients at clinics randomized to receive a lower intensity implementation support (Smith et al. 2019). For other examples of SMARTs being used to develop adaptive implementation strategies, see Smith et al. (2022) and Fernandez et al. (2020).

8. Future directions

In this section we list a number of interesting future directions in this rapidly evolving field.

Much of the literature on intervention optimization has drawn a sharp distinction between fixed and adaptive interventions, but in practice many interventions incorporate elements of both. Moreover, many adaptive interventions involve a variety of timescales of adaptation. More ORCT designs are needed to enable efficient optimization of such interventions. In addition, methods are needed to enable inference from ORCTs conducted in smaller-sample situations. One approach would be to extend N-of-1 trial methods to ORCTs. Another would be to make more effective and efficient use of experimental information via Bayesian methods, including sequential analysis and the use of empirical priors based on trials conducted in similar circumstances.

The concept of the optimization trial could be expanded beyond ORCTs. Causal inference procedures could be developed to enable conclusions about individual component effects to be drawn based on nonexperimental data, or data from an ERCT, with increased confidence.

Decision-making to select the optimized intervention based on the results of an ORCT is very much an open area. At this writing, DAIVE has been applied only to results obtained using a factorial or fractional factorial ORCT; extensions are needed to the SMART, MRT, and other ORCTs. Determining the best use of the uncertainty estimates that are readily obtainable with a Bayesian approach is another priority. An intriguing possibility for future work involves using DAIVE to optimize based on two or more very different types of outcomes, such as a desirable health outcome and an undesirable side effect, or a health outcome and an implementation outcome. There are also many open questions pertaining to how to identify who should be engaged in optimization of an intervention; how to incorporate their preferences into the decision-making process; how to elicit value functions and weights so that they accurately reflect these preferences; and who ultimately selects the optimized intervention.

The concept of intervention EASE could be extended to include equitability, where equitability is defined as “the extent to which the health benefits provided by an intervention are distributed evenly, such that all participants have a fair and just opportunity to achieve the desired outcome of the intervention” (Strayhorn et al. 2023b). This could enable decision-makers to determine, for example, how to design an intervention to reduce health disparities, or what level of resources is needed to maintain effectiveness while achieving equitability.

The barriers that affect implementation may vary across settings. Assuming the results of a particular ORCT can be generalized across settings, it is possible to use its results to arrive at optimized interventions suitable for different settings, by applying different optimization objectives as appropriate (Guastaferro & Collins 2021). For example, if one setting can afford to spend $300 per participant and another can afford only $200, two different optimized interventions can be produced. To our knowledge this has not yet been attempted.

MOST is currently framed as a sequence of discrete phases, but a more dynamic and flexible approach, based on a Bayesian perspective on data collection and analysis (Vanness et al. 2023) may be more efficient. The phases of MOST would then be a seamless sequence of steps, with more potential for back-and-forth, and flow among them determined by critical decisions at multiple points. For example, data would be collected for an ORCT only while the information has sufficient value (e.g. until the costs of collecting additional information exceed the benefits of reducing uncertainty), at which point the immediate next step would be either (a) proceed to an ERCT; (b) return to the preparation phase; or (c) if the results of the ORCT are sufficiently strong, implement the intervention without conducting an ERCT. Such an approach has the potential to speed up the optimization process and increase the efficiency of its use of research resources. Arriving at this more flexible version of MOST will require considerable methodological work, as well as development of software to make the framework practical for application by intervention and implementation scientists.

9. Conclusions

We have proposed that the field of clinical psychology can make considerable progress toward its goals by shifting to a new research paradigm, intervention optimization via MOST. We hope that this article has stimulated readers to consider how MOST can be useful in their research. Readers who wish to keep abreast of developments in this rapidly-developing field may be interested in the websites of the Data Science for Dynamic Intervention Decision-Making Center (d³c) at the University of Michigan https://d3c.isr.umich.edu/ and the Center for Advancement and Dissemination of Intervention Optimization (cadio) at New York University https://cadio.org/.

We predict that as more scholars in the area of clinical psychology adopt the MOST paradigm, and as this paradigm develops, a substantial and coherent body of evidence will accumulate about what intervention approaches work, for whom, and when; interventions will continually become more effective and more readily implementable; and clinical psychology will achieve increasingly positive societal impact.

The following material is intended for a highlighted box titled Summary Points.

1. To achieve the goals of psychological intervention science, a new research paradigm is needed: Intervention optimization via the multiphase optimization strategy (MOST).

2. In MOST, optimization is defined as the process of empirically selecting components/component levels to achieve intervention EASE: a strategic balance of intervention effectiveness, affordability, scalability, and efficiency.

3. Evaluation randomized controlled trials (ERCTs) are experiments designed to be efficient for estimation of the effect of an intervention as a package.

4. Optimization randomized controlled trials (ORCTs) are experiments designed to be efficient for assessment of the performance of intervention components.

5. There are numerous ORCT designs to choose from, with more being developed.

6. The logical underpinnings of ORCTs are different from those of the ERCT; e.g., they are powered differently.

7. MOST can be used to optimize fixed or adaptive interventions, or interventions that have both fixed and adaptive aspects.

8. Decision-making—based on the results of an ORCT and potentially other information—for arriving at the set of components/component levels that will make up the optimized intervention is a critical part of MOST.

9. MOST can be used in implementation science in two ways: to develop interventions that are both effective and implementable, and to optimize implementation strategies.

10. Use of the MOST paradigm will enable intervention science to accumulate a coherent base of knowledge; leverage this knowledge to develop high-impact interventions; and continue to increase the positive impact of these interventions.

The following material is intended for a highlighted box titled Future Directions.

1. Develop additional ORCT designs

2. Extend optimization trials beyond randomized experiments

3. Extend decision-making methods and determine how best to incorporate input from interested parties

4. Extend intervention EASE to include equitability as one of the criteria in achieving a strategic balance

5. Extend MOST to situations in which resources, constraints, and barriers vary across settings

6. Develop a more dynamic and flexible version of MOST