Developing more efficient, effective, and disseminable treatments for eating disorders: an overview of the multiphase optimization strategy

Abstract

The present manuscript describes the multiphase optimization strategy (MOST) and its potential applications to treatments for eating disorders (EDs). The manuscript describes the three phases of MOST, discusses a hypothetical case example of how MOST could be applied to developing a disseminable ED treatment, and reviews the pros and cons of the MOST approach. Outcomes from treatments for EDs leave room for improvement. However, traditional methods of treatment development and evaluation (i.e., the treatment package approach) make it challenging to determine how best to improve ED treatments. For example, testing full treatment packages in open trials and RCTs without systematic testing of each component is inefficient (as it is unknown which components are effective), and often does not provide concrete future directions for optimization of the treatment. Much stands to be gained by optimizing treatments in the early stages before testing them in open trials or RCTs. MOST is an alternative, engineering-inspired research framework that is well-suited to address the issues of inefficiency associated with the treatment package approach. MOST entails identifying the most promising treatment components for inclusion in interventions, then eliminating or deemphasizing less efficacious/inert components. This strategy results in a treatment comprised of only effective components that can then be tested via RCT. Though the MOST approach has limitations, it has the potential to greatly benefit ED treatment research and is worthy of application in the field.

Improving eating disorder treatment research

Existing treatment approaches for eating disorders (EDs) are effective for many patients, but there is room for improvement, as many individuals remain partially or fully symptomatic after treatment ends [1, 2]. Recent meta-analyses have found that as many as 50 or 60% of eating disorder patients do not fully respond to treatment [3, 4]. A recent meta-analysis of the efficacy of cognitive-behavioral therapy (CBT) for EDs found it to be the most empirically supported treatment, however, the review highlighted some limitations, e.g., the lack of evidence for long-term efficacy and durability of treatment effects and low quality of existing treatment studies [5]. Compounding this problem, treatments for EDs are often intensive and expensive, thus necessitating that treatments be made more efficient and disseminable [6].

While limited efficacy and disseminability of ED treatments are problematic, as described below, current methods of treatment development and evaluation make it challenging to identify how to improve ED treatments, both in terms of improving efficacy and disseminability. The traditional approach (hereby referred to as the “treatment package approach”) to developing and evaluating treatments for EDs consists of construction, testing using an open trial, and then a randomized controlled trial (RCT) that compares the treatment package to a control [7]. RCTs are an indispensable facet of clinical research [8–10]. Yet, researchers recently have argued that the traditional treatment package approach can be inefficient and costly [10] and much stands to be gained by optimizing treatment before testing them using RCTs [7, 11].

Several factors contribute to the inefficiencies and cost of the treatment package approach. First, it is common for large, “kitchen sink” multi-component packages to be tested, preventing delineation of which components are efficacious, incrementally beneficial, inert, or iatrogenic [12]. Second, researchers often move directly from treatment development to open trials or RCTs, without devoting attention to initial pre-efficacy testing (e.g., examination of whether certain components are efficacious). Instead, treatment packages are tested in a full-scale study when instead they could be first refined and retested (e.g., certain components eliminated) before devoting substantial resources [13]. Third, the treatment package approach does not allow for the best possible evaluation of treatment mediators and moderators. Treatment mediators are possible mechanisms through which a treatment achieves its effects. Traditional post-hoc evaluation of “active ingredients” in treatment packages via mediation analyses can suggest a relationship between certain components within an intervention and treatment outcomes, but true causal conclusions are limited as participants cannot be randomly assigned to individual intervention components [7], changes in a process variable cannot be attributed to a specific component, and difficulties with self-report measurement preclude confidence in existing findings. Even with sophisticated statistical analyses such as structural equation modeling (SEM) that allow for the evaluation of complex mediational models, SEM fit indices are sometimes insensitive to omitted variables, SEM still relies on highly sensitive temporal measurement to derive the impact of specific treatment components, and recommended sample sizes for SEM are well above the typical ED trial (e.g., n ≥ 200) [14]. Identification of treatment mediators is especially important in ED treatment given that extant interventions are fairly complex and multi-faceted. As argued by prominent researchers, being able to infer causal mediation would allow us to develop treatments that yield larger effect sizes or the same effect at a lower cost [15]. Treatment components deemed as active could be intensified and inactive/redundant components discarded. Treatment moderators specify for whom or under what conditions the treatment works [15], and cannot be ideally evaluated through treatment packages. Sophisticated moderator analyses would allow for identification of certain subpopulations who respond to certain specific components (or combination of components), or identification of treatment components that are synergistic or antagonistic to one another. Despite the critical importance of identifying treatment mediators and moderators, ED research has struggled to do so [16, 17].

There are numerous examples of these problems leading to stalls in ED treatment research specifically. For example, CBT is widely accepted as the treatment of choice for adult binge eating spectrum disorders. Innovations on this treatment have taken several forms, such as an enhanced version of CBT (CBT-E). However, despite over 60 trials of CBT, few consistent mediators have emerged [16], providing little direction on which treatment ingredients are active and could be expanded upon. Some creative research regarding mechanisms of ED treatment has been conducted [18–27], but the implications of mediational findings are often difficult to translate into concrete next steps. For example, some research suggests that reductions in dietary restraint early in CBT (during which a small suite of behavioral strategies, such as self-monitoring, regular eating, and weekly weighing are provided) are likely responsible for a large proportion of the treatment’s effects [28]. As such, components of self-monitoring and regular eating are likely efficacious components that should be retained in an optimized treatment. However, because of the way the research was conducted, we have very little knowledge regarding the incremental efficacy of other CBT or CBT-E components. For example, it is unknown whether the mood modulatory module of CBT-E, in fact, leads to improvements in acquisition and utilization of mood modulatory skills, whether that component has incremental benefit, and/or whether it is synergistic or even antagonistic with another component in the treatment. Lack of knowledge regarding the effects of specific treatment components impacts both the ability to the efficacy of existing treatments and increasing disseminability and access to treatment. In the case of improving overall efficacy, component analysis allows for eliminating inert components and for adding components that improve efficacy (which may be warranted even if the treatment is made more complex and/or expensive). In the case of improving disseminability, understanding the efficacy of treatment components is necessary for stripping interventions to their simplest and least expensive (yet still efficacious) parts to facilitate widescale delivery.

The treatment package approach also impedes deriving crucial information from RCT findings. For example, integrative cognitive affective therapy (ICAT) for BN showed equivalence to CBT-E in the treatment of BN [29]. However, from the design of the trial, it is impossible to surmise if certain components of ICAT are especially powerful (and thus worth retaining/further testing) or whether another component of ICAT could be iatrogenic (or dampening the effect of other components). While mediator analyses can be helpful, it is sometimes nearly impossible to isolate the relative impact of specific components on hypothesized mediators due to the presence of the other treatment components and difficulties with establishing temporal precedence (i.e., that change in a specific mechanism was responsible for subsequent change). It is additionally possible (and perhaps likely), that a not-yet-tested combination of already-existing treatment components (e.g., self-monitoring, regular eating, with another component not currently included in CBT or CBT-E, such as medication, neurocognitive training, or smartphone-delivered interventions) could be delivered together as a highly efficacious treatment superior to our current gold-standard. Continuing with extant methods of treatment development and evaluation (construction of treatment packages and testing only in their fullest form) may mean that even 20 or 30 years of work will yield little progress identifying efficacious combinations of treatment components, and thus developing a more effective or optimized treatment for EDs. The path towards developing more efficacious treatments for eating disorders may hinge upon taking a different approach.

Multiphase Optimization Strategy (MOST), an alternative research framework, is well-suited to address many of the issues of inefficiency and cost that stem from solely using the treatment package approach. In the remainder of this manuscript, we (1) describe MOST; (2) discuss a hypothetical case example of how MOST could be applied to developing a disseminable treatment for EDs, and (3) discuss the pros and cons of the MOST approach, including the types of interventions that are either well- or ill-suited for development and testing with the MOST approach.

Multiphase Optimization Strategy (MOST)

Multiphase Optimization Strategy (MOST) describes a comprehensive, principled, engineering-inspired framework for optimizing and evaluating multicomponent behavioral interventions [7, 30–32]. Compared with the treatment package approach, MOST is a more efficient approach to treatment development and refinement. While MOST encourages the eventual use of traditional RCTs for intervention evaluation, it emphasizes the necessity of first identifying and testing the most promising individual treatment components for inclusion in multi-component interventions. For example, a MOST approach would have resulted in testing the main and interactive effects of the novel components of CBT-E (e.g., mood modulatory skills, interpersonal skills) before conducting an RCT (see Phase II, described below). MOST has been applied in other areas of behavioral treatment research [33] and is consistent with initiatives in the broader field. For example, the experimental therapeutics approach to clinical trials by the National Institute of Mental Health [34] supports such research approaches that enable isolation of treatment components, identification of treatment mechanisms, and a more systematic construction of treatment packages.

Utilizing MOST will result in a treatment that is comprised only of effective components, and also allows for identifying components that are complementary or synergistic with each other (e.g., perhaps regular eating only is effective when self-monitoring is also present). For example, if an existing 20-h treatment protocol consists of 8 h of an inert component, we may be able to shorten it to 12 h to increase efficiency disseminability, or increase the amount of time spent on more effective components to increase overall impact. Designing treatments that are powerful, but still relatively less intensive, is all the more critical as the field increasingly attempts to develop treatments that can be disseminated widely, at low cost and with relatively low training burden. Furthermore, precise identification of specific treatment components (or combination of components) that are effective for certain subgroups could lead to a more systematic, evidence-based method of selecting and sequencing treatments based on initial client presentation. Previous approaches attempting to identify powerful treatment elements certain subgroups (e.g., Distillation and Matching) rely on novel methods of aggregating and analyzing data and treatment elements across many trials [35, 36]. While highly innovative and significant, deriving concrete implications from the Distillation and Matching approach can prove difficult due to significant heterogeneity across trials. MOST allows for experimental evaluation of the impact of specific treatment components and under which conditions certain components or combination of components are most impactful.

A recent example from extant MOST literature is that of a study which aimed to identify intervention components that help smokers increase long-term abstinence [37]. The study examined five treatment components, but found that two (i.e., extended medication, maintenance counseling) worked well together and demonstrated promise as the strongest combination of components, while other components (i.e., medication adherence counseling, automated adherence calls, electronic medication monitoring) did not further boost abstinence rates—and, in some cases, lowered them. These results demonstrate how effort and resources may be better spent on some aspects of treatment than others to achieve more favorable outcomes.

MOST should occur prior to conducting investigations of the superiority of a particular treatment package to avoid extensive testing of a treatment package that may contain ineffective components or inadequate emphasis on others. MOST occurs in three phases (Table 1): Phase I, Preparation, during which identification of treatment components to test occurs (via the development and/or parsing of a theoretical model and pilot/feasibility testing and iteration of a new intervention component, if applicable); Phase II, Optimization, during which treatment components are empirically evaluated, often via an efficient randomization experiment (e.g., a factorial design) so that their efficacy can determine inclusion or exclusion from an optimized treatment package; and Phase III, Evaluation, during which the optimized intervention (with only efficacious components) is tested against the current standard (typically via a traditional RCT). Using MOST to develop interventions composed of only effective treatment components, existing treatments can be refined so as to identify the most efficacious, cost-efficient, and briefest version of the treatment. Below, we describe each phase of MOST in more detail.

Table 1.

Purpose of each phase of Multiphase Optimization Strategy (MOST)

MOST phase	Purpose
Phase I: preparation	Form a theoretical model; selection of intervention components to test
Phase II: optimization	Identification of efficacious treatment components to include in optimized intervention
Phase III: evaluation	Testing of optimized intervention against best-available intervention

MOST phase I: identifying treatment components

The preparation conducted in Phase I lays the foundation for optimization of the intervention in question via forming a theoretical model, selecting which intervention components to examine, and identifying an optimization criterion [32]. Using existing research, previous treatment research, theory, secondary analyses of existing data, pilot testing, and other relevant sources, a theoretical model is formed to select the components that have high potential for true efficacy [32]. Components can be derived from an existing treatment package, promising novel interventions that have been pilot tested, or can take the form of new modes or varying dosages of delivery [7]. Such an approach allows researchers to use theory and empirical findings to build a novel intervention (or combination of intervention components) from the ground up. Pilot and feasibility testing, as well as iteration of a treatment component can also occur during Phase I. Although the researcher should and can take inspiration for treatment components from analog studies and theory, likely the researcher is following a body of treatment researchers conducted by many who have not followed the MOST approach. Therefore, the researcher should not ignore data (e.g., mediating effect of reducing dietary restraint) obtained from previous “kitchen sink” treatment trials. An additional essential task for Phase I of MOST is to identify an optimization criterion—an operational definition of the goal to be achieved by optimizing the intervention. A common example of such a criterion is that the intervention should have no inactive components [32], but can also refer to other ways of optimizing an intervention e.g., identifying the most efficacious intervention given a specific cost limit [e.g., the most efficacious intervention for under $2,000 per patient], identifying the most efficacious intervention given a maximum time limit (e.g., most efficacious intervention that takes less than 5 h). Especially in the case of optimizing a treatment in terms of cost and disseminability, refining the theoretical model may entail identifying novel delivery methods, different dosages, or certain combinations of already-existing treatment components that could yield sufficient efficacy. However, a researcher should not feel confined by of an extant treatment model or package, although a researcher may choose to retain likely essential treatment components as a “constant component” in Phase II (e.g., regular eating; see “MOST Phase II: Optimization”).

As an example of how Phase I may be applied in practice (outside the ED field), consider an ongoing trial [12, 38]—the first to use MOST to develop a behavioral weight loss treatment. The investigators aim to identify which of five treatment components contribute to weight loss outcomes and yield the best results attainable for $500 or less per patient [12]. Based on prior studies and using a conceptual model intended to target social-cognitive mechanisms—specifically, supportive accountability to bolster treatment adherence—the investigators decided upon five treatment components that manipulate supportive accountability: treatment intensity (12 versus 24 coaching calls), reports sent to primary care physician (no versus yes), text messaging (no versus yes), meal replacement conditions (no versus yes), and training of participants’ self-selected support buddy (no versus yes) [12]. (Of note, a mistake was made in the implementation of this trial, which is described in the Other potential downsides of the MOST approach later in this manuscript). In summary, Phase I allows for the researchers to build a theoretical model for what needs to be addressed in treatment and choose components to test whether modifying such factors is efficacious.

MOST phase II: optimization

Phase II of MOST consists of evaluating the efficacy of the components (i.e., a component selection experiment) identified in Phase I. Factorial designs represent an especially efficient way to conduct a component analysis because consolidation of all cells containing (“On”)/not containing a component (“Off”) allows a comparison drawing on the full sample and thus yields the same statistical power as a two-arm treatment–control design [39] (see Fig. 1). Alternatively, a factor could provide an intervention at different levels to identify necessary dosage (e.g., coaching calls 1x/per week versus 3x/week). Although a factorial experiment is not a requirement of Phase II of MOST, a factorial experiment facilitates assessing the effects of individual components of an intervention and allows for an increased number of experimental conditions [32]. By examining combinations of experimental conditions and using mean main effect comparisons, conservation of resources (e.g., sample size, costs) is made easier [32]. While an individual experiment (e.g., randomizing participants to one condition of the several being studied) could also be conducted, such an approach requires many more participants, and does not allow for examination of synergy/antagonistic effects between components [40].

Fig. 1.

Factorial design randomization and generation of statistical comparisons

Moreover, the component analysis made possible through factorial designs is a viable alternative to traditional mediation analyses conducted in solely RCT designs. Using a factorial design, it is possible to randomly assign participants to specific intervention components and examine the components’ effects in isolation [7]. Factorial designs allow for examination of additive and/or synergistic effects of treatment components without a need for an increased sample size. For example, using a factorial design, researchers can examine whether the efficacy of one component depends on the presence of another (e.g., whether the impact of regular eating is dependent on the presence of self-monitoring). Additionally, based on the optimization criterion, additional data, such as the cost or time required for each component, can be collected during this phase of MOST. The makeup of the “optimized” intervention is then chosen based on the outcome of the component selection experiment and the optimization criterion, and weak or iatrogenic components are not included in the optimized intervention.

MOST phase III: evaluation

Identifying effective and non-effective components in Phase II will position researchers for constructing and evaluating an optimized treatment in which all available treatment time is devoted to effective components, or is optimized in some other way, e.g., is made shorter or cheaper. Phase III is expected to evaluate a treatment that may have superior efficacy (because 100% of its components are efficacious) and/or disseminability (because fewer sessions and fewer types of training are necessary). The most typical design for implementing MOST Phase III is an RCT of the new, optimized treatment versus the standard/traditional preferred treatment.

Illustrative example: building a disseminable treatment for EDs

To illustrate the potential benefits of a MOST approach, we walk through hypothetical MOST Phases I, II, and III of developing a disseminable intervention for bulimia nervosa (BN).

Building a disseminable intervention for BN: MOST phase I

MOST Phase I, depending on the optimization criterion, would conduct a wide-scale review of theoretical maintenance/treatment-interfering factors for EDs and components that target such factors (e.g., self-monitoring, regular eating emotion regulation skills, medications, methods for manipulating neurocognitive processes such as inhibitory control trainings). This approach allows for “starting from scratch” to develop a novel intervention (or intervention made of a novel combination of components). For example, an ED researcher may review recent findings from neuroimaging and ecological momentary assessment literature that suggest there may be factors (e.g., mood disturbance, poor inhibitory control), that should be targeted in treatment. However, if a certain novel treatment component has never been subjected to feasibility/acceptability testing, such an evaluation should take place before the Phase II experiment, i.e., a treatment component should be determined feasible/acceptable in Phase I before being chosen for a Phase II experiment. However, it is not necessary that a treatment component have empirical evidence pointing to its efficacy to be included in the theoretical model and tested in Phase II (see Table 2). Another researcher seeking to develop the most cost-effective intervention or the most effective intervention under a specific limit on cost would conduct a review of costs associated with existing interventions. For an example of a hypothetical table of theoretical factors and treatment components that could be created from Phase I, see Table 2; this table is not meant to be comprehensive, but simply to serve as an example of how a review of theory and treatment components could be conducted.

Table 2.

Partial example of Phase I literature review of treatment components

BN maintenance factor	Specific possible treatment component(s)	Evidence for efficacy of specific treatment component?
Dietary restriction	Self-monitoring (CBT)	Yes (strong)
	Regular eating (CBT)	Yes (strong)
Overvaluation of shape and weight	Reducing shape checking, decreasing overvaluation of shape and weight (CBT/E)	No
Maladaptive responses to cues	Cue exposure training	No
Attention bias to food cues	Attention bias modification	Some (preliminary)
Fear of weight gain	Exposure treatment	No
Emotion dysregulation	Emotion regulation skills (from dialectical behavioral therapy); Mood modulatory skills (CBT)	No
Mood disturbance	Anti-depressant medication (e.g., SSRI)	Some
Impulsivity	Computerized inhibitory control training	No

One obvious direction in ED research is to dismantle the components of CBT (e.g., self-monitoring, regular eating, alternative activities) and include them in a factorial design (i.e., testing their main and interacting effects). While such a study would be of high value to the field, and somewhat similar to MOST Phase II in that a dismantling study can fall under the umbrella of a component selection experiment, we have chosen to describe the process of building a new treatment package, using all three phases of MOST, for the purposes of an illustrative example. Additionally, conducting a factorial experiment with an in-person psychotherapy has special considerations that are discussed in “Considerations and challenges with using the MOST approach”. As such, for the illustrative purposes of this example, our operational definition of the ultimate (Phase III) goal of the intervention being developed and tested is a highly disseminable, treatment for BN without any inactive components. In this illustrative example, all treatment components will be delivered in the absence of a therapist (e.g., via a web-based self-help module, online computerized training, or provision of a medication) and will focus on creating the most efficacious version of a fully remote treatment. As such, in Phase I lies the opportunity to develop a theoretically and empirically driven novel intervention for EDs, the components of which will be tested in Phase II. Researchers could choose to test certain components for several different reasons. If a treatment component is already in use but the efficacy of the specific component is unknown (or its delivery via a different modality is unknown), it may be of scientific interest to include it in the experiment to evaluate whether the component is necessary. Or, if a component has theoretical support from basic research and shows promise in initial analog designs, it may be of interest to examine what dose is most appropriate. However, for feasibility, some selection criteria must be applied to limit the number of components tested in any given study. For the purposes of the current example, we will walk through the hypothetical example of testing four components that target four different maintenance factors supported by the literature (see Fig. 2): reducing overvaluation of shape and weight (via web module; levels = on or off), increasing adaptive responses to cues (via web module; levels = on or off), computerized daily inhibitory control training [implemented via the internet; levels = on (active) or off (sham)], and SSRI [levels = on or off (placebo)]. Alternatively, the researcher could modify the levels to reflect different dosages rather than “on” and “off” levels (e.g., daily versus 3x week inhibitory control training). The simple model derived from Phase I integrates findings from ED treatment that show overvaluation of shape and weight maintains ED behavior [41], individuals with ED behavior are cued by internal and external triggers [42], and that impulsivity and mood disturbance drive ED behavior [43]. In this theoretical example, a researcher may choose components that he or she believes are amenable to delivery in a remote format. The empirical and theoretical case could be made for many different components to include in a factorial experiment; we chose these components to illustrate the method in this manuscript.

Fig. 2.

Example conceptual model of treatment for BN

Building a disseminable treatment for BN: MOST phase II

Given that it is well established that certain treatment components are crucial to treatment success, the design of this example study includes a “constant component” treatment condition of self-monitoring and regular eating (i.e., all participants will, at a minimum, receive a web module containing content that teaches the patient about self-monitoring and regular eating).

A full 2 × 2 × 2 × 2 factorial design with a “constant component” (i.e., a component/intervention that every person receives, which in this case is self-monitoring and regular eating) would include 16 possible conditions (testing 15 effects: four main effects, 6 two-way component interactions, 4 three-way interactions, and 1 four-way interaction; see Table 3). Constant components are often used when there is a “minimal necessary” intervention that should be delivered to all participants (e.g., self-monitoring and regular eating is considered an essential part of treatment), or when one desires to test an add-on to an existing intervention. When a constant component is utilized, it must be assumed that the efficacy of the experimental components is dependent on the provision of the constant component (i.e., any of the experimental components must be delivered with the constant component to yield the effect).

Table 3.

The 16 conditions of a full (2 × 2 × 2 × 2) factorial design

Cell #	Treatment condition	OV	AR	ICT	Med
1	[RE+] OV + AR + ICT + Med	On	On	On	On
2	[RE+] OV + AR + ICT	On	On	On	Placebo
3	[RE+] OV + AR + Med	On	On	Sham	On
4	[RE+] OV + ICT + Med	On	Off	On	On
5	[RE+] AR + ICT + Med	Off	On	On	On
6	[RE+] OV + AR	On	On	Sham	Placebo
7	[RE+] OV + ICT	On	Off	On	Placebo
8	[RE+] OV + Med	On	Off	Sham	On
9	[RE+] AR + ICT	Off	On	On	Placebo
10	[RE+] AR + Med	Off	On	Sham	On
11	[RE+] ICT + Med	Off	Off	On	On
12	[RE+] OV	On	Off	Sham	Placebo
13	[RE+] AR	Off	On	Sham	Placebo
14	[RE+] ICT	Off	Off	On	Placebo
15	[RE+] Med	Off	Off	Sham	On
16	RE only	Off	Off	Sham	Off

RE self-monitoring and regular eating, OV reducing overvaluation of shape and weight, AR adaptive responses to cues, ICT inhibitory control training, Med anti-depressant medication

As demonstrated in Table 4, a full factorial design allows for evaluation of all main and interaction effects of each treatment component using 100% of the sample (i.e., each comparison has the same power that a 2-arm RCT would have). For example, to compare the independent efficacy of the Reducing Overvaluation component as an addition to regular eating/self-monitoring, we would compare mean change in BN symptoms of all the cells that contain this component (cells 1–4, 6–8, and 12 in Table 4) to the mean change in BN symptoms of all the cells that do not (cells 5, 9, 10, 11, 13–16 in Table 4). To evaluate the efficacy of the Adaptive Responses component as an addition to regular eating/self-monitoring, mean change in BN symptoms of cells 1–3, 5, 6, 9, 10, 13 (cells that contain Adaptive Responses) are compared to mean weight loss of cells 4, 7, 8, 11, 12, 14, 15, 16 (cells that do not contain Adaptive Responses). Moreover, interaction effects can also be tested using all participants. For example, to test whether the effect of Inhibitory Control Training varies depending on whether Medication was provided, we would group cells in which both components are either both on or both off (1, 4, 5, 6, 11, 12, 13, 16) and cells in which one of the components (but not both) is present (2, 3, 7, 8, 9, 10, 14, 15). In this comparison, an interaction effect is indicated if there is a difference in the effect of Inhibitory Control Training on BN symptoms when Medication is on versus when Medication is off, collapsing over levels of Adaptive Responses and Reducing Overvaluation. In other words, each group of cells has equal dosage of Inhibitory Control Training and Medication being delivered (and equal Adaptive Responses and Reducing Overvaluation), and thus if cells 1, 4, 5, 6, 11, 12, 13, 16 show increased effect over 2, 3, 7, 8, 9, 10, 14, 15 it can be attributed to Inhibitory Control Training and Medication being delivered simultaneously. Table 4 shows the group comparisons that test each main and interaction effect (i.e., whether the effect of one component depends on the presence of another). With this design, it is also possible to identify whether components are synergistic (i.e., the effect of the components together exceeds their additive benefit) versus antagonistic (i.e., that the combined effect of the components is less than what would be expected based on the two main effects alone). If the researcher were interested in the effect of each component and two-way interactions, but did not have any hypotheses regarding higher-order interactions, he or she could implement a fractional factorial design (a type of incomplete factorial design), described below.

Table 4.

Group comparisons to test each effect

Main or interaction effect to evaluate	Comparison		% of total sample
	Cells in Group 1	Cells in Group 2
OV	1, 2, 3, 4, 6, 7, 8, 12	5, 9, 10, 11, 13, 14, 15, 16	100
AR	1, 2, 3, 5, 6, 9, 10, 13	4, 7, 8, 11, 12, 14, 15, 16	100
ICT	1, 2, 4, 5, 7, 9, 11, 14	3, 6, 8, 10, 12, 13, 15, 16	100
ICT + Med	1, 4, 5, 6, 11, 12, 13, 16	2, 3, 7, 8, 9, 10, 14, 15	100
AR + Med	1, 3, 5, 7, 10, 12, 14, 16	2, 4, 6, 8, 9, 11, 13, 15	100
AR + ICT	1, 2, 5, 8, 9, 12, 15, 16	3, 4, 6, 7, 10, 11, 13, 14	100
OV + Med	1, 3, 4, 8, 9, 13, 14, 16	2, 5, 6, 7, 10, 11, 12, 15	100
OV + ICT	1, 2, 4, 7, 10, 13, 15, 16	3, 5, 6, 8, 9, 11, 12, 14	100
OV + AR	1, 2, 3, 6, 11, 14, 15, 16	4, 5, 7, 8, 9, 10, 12, 13	100
AR + ICT + Med	1, 5, 6, 7, 8, 13, 14, 15	2, 3, 4, 9, 10, 11, 12, 16	100
OV + ICT + Med	1, 4, 6, 9, 10, 12, 14, 15	2, 3, 5, 7, 8, 11, 13, 16	100
OV + AR + Med	1, 3, 7, 9, 11, 12, 13, 15	2, 4, 5, 6, 8, 10, 14, 16	100
OV + AR + ICT	1, 2, 8, 10, 11, 12, 13, 14	3, 4, 5, 6, 7, 9, 15, 16	100
OV + AR + ICT + Med	1, 6, 7, 8, 9, 10, 11, 16	2, 3, 4, 5, 12, 13, 14, 15	10%

RE self-monitoring and regular eating, OV reducing overvaluation of shape and weight, AR adaptive responses to cues, ICT inhibitory control training, Med anti-depressant medication

Of note, comparisons in the factorial design utilize a group of 4 cells compared to another group of 4 cells; i.e., a factorial design with 8 cells of 15 participants has the same power as a traditional RCT with 2 cells of 60 participants, and yields a much larger amount of information (e.g., efficacy of each component, and whether there are synergistic relationships between components). As can be seen in Table 4, each comparison utilizes the full sample and is essentially a “mini-RCT” for testing each component or combination of components. With this design, it is possible to determine whether any of the components yield any efficacy. For the measurement of mechanisms, all patients would be provided the same assessment battery; the factorial design allows for specification of which components uniquely impact which mechanisms (if any), and allows for evaluation of whether certain components, or combinations of components, impact unexpected mechanisms. For example, we would be able to determine whether Reducing Overvaluation uniquely impacted overvaluation of shape and weight, or if another component also impacted over-valuation of shape and weight. Such conclusions are often not able to be drawn from assessments of mechanisms in “kitchen sink” trials of intervention.

In some cases, to further conserve resources, a fractional factorial design may be implemented in place of a complete factorial design (i.e., contrasted with incomplete/reduced factorial designs, a design approach that involves manipulation of multiple independent variables but includes fewer experimental conditions than a full factorial design [44]). Fractional factorials require that investigators choose which effects/conditions are of primary scientific interest (likely main effects and two-way interactions), and which are not of interest and likely to be negligible in effect size (e.g., higher order interactions). By removing a carefully selected fraction (e.g., 1/2 or 3/4) of experimental conditions (e.g., all conditions with more than four components), the factorial design can be made even more efficient and cost-effective (e.g., 16 experimental conditions instead of 32) as a fractional factorial without necessarily sacrificing the ability to make scientific conclusions [32]. A fractional factorial is a special type of incomplete factorial design; other incomplete factorial designs may be full factorial designs but with some combinations of interventions being left out (e.g., if there is a reason to believe that delivering two of the interventions at the same time could be harmful to participants) [32, 40, 44].

Power analysis and statistical considerations of factorial experiments

Power analyses for a factorial experiment can be conducted using ready-made software, e.g., PROC POWER or FactorialPowerPlan macro in SAS. Effect coding (e.g., − 1, 1 for a factor with two levels) or dummy coding (0, 1) may be used, which may impact power obtained [45]. Effect coding is recommended as it allows for equal power for detecting main effects and interactions in a factorial experiment (see Table 5 for effect codes in our illustrative example) and effects that are uncorrelated when there are equal ns in each experimental condition. All effects (main and interaction) include a comparison of four treatment conditions against four other treatment conditions, and as such, factorial designs have identical power to detect effects as typical, two-arm RCTs with the same number of participants.

Table 5.

Effect coding for illustrative example

Treatment condition	OV	AR	ICT	Med
[RE+] OV + AR + ICT + Med	1	1	1	1
[RE+] OV + AR + ICT	1	1	−1	−1
[RE+] OV + AR + Med	1	1	−1	1
[RE+] OV + ICT + Med	1	−1	1	1
[RE+] AR + ICT + Med	−1	1	1	1
[RE+] OV + AR	1	1	−1	−1
[RE+] OV + ICT	1	−1	1	−1
[RE+] OV + Med	1	−1	−1	1
[RE+] AR + ICT	−1	1	1	−1
[RE+] AR + Med	−1	1	−1	1
[RE+] ICT + Med	−1	−1	1	1
[RE+] OV	1	−1	−1	−1
[RE+] AR	−1	1	−1	−1
[RE+] ICT	−1	−1	1	−1
[RE+] Med	−1	−1	−1	1
RE only	−1	−1	−1	−1

For a full factorial experiment, a classic factorial ANOVA (with each treatment condition as a factor) or more advanced regression models (e.g., multilevel models) can be conducted to evaluate the main and interacting effects of each component. Methods for handling clinical trial issues such missing data, adherence, and dropout can be handled in similar ways to a typical, non-factorial RCT (e.g., using multiple or maximum likelihood imputation). There is no one “optimal” method for making final decisions regarding component selection for the optimized intervention. One approach involves making preliminary selections based on main effects that exceed a predetermined criterion for statistical significance or effect size, which may be re-evaluated based on detection of any interaction effects [46]. For example, it may be that there was no main effect of Reducing Overvaluation, so it would be left out based on preliminary examination of main effects. However, if there was a sufficiently large Reducing Overvaluation × Adaptive Responses interaction such that the effect of Overvaluation was present but only in the presence of Adaptive Responses, the researcher could choose to retain both components, depending on the optimization criterion set ahead of time. A “sufficiently large” effect should be set a priori by the experimenter for each component (i.e., the difference in effect between treatments with the component present or absent), such as exceeding a specific statistical significance threshold, or a meaningful raw difference (e.g., an additional decrease of × binge episodes).

Building a disseminable treatment for BN: MOST phase III

The design of the MOST Phase III trial for this example, assuming any of the components appear to yield efficacy, would use a traditional RCT to compare the most efficacious/optimized version of the intervention to a comparison group. In this illustrative example, one could compare the treatment to another available treatment that can be delivered fully remotely (e.g., a self-help book such as Overcoming Binge Eating) to compare efficacy, or a first-line treatment such as CBT to investigate non-inferiority.

Considerations and challenges in using the MOST approach

There several important challenges worthy of consideration when considering the MOST approach or factorial designs more broadly. First, a key assumption of MOST is that treatment components can be isolated from one another. In the illustrative example above, each of the components are completely distinct from one another. Many treatment components, especially those that are a part of traditional psychotherapy, are much more difficult to separate from one another. For example, if we were examining the effect of separate components from Acceptance and Commitment Therapy (ACT), we could identify Psychological Acceptance and Mindfulness as two potentially efficacious components. However, in ACT, mindful awareness and psychological acceptance are highly related concepts, as mindful awareness includes becoming conscious of one’s inner experiences (e.g., thoughts and feelings), and acceptance involves a stance of nonjudgment towards these inner experiences. As such, isolating each of these components may be challenging in terms of manual development and potential contamination by therapists. Relatedly, another challenge of isolating components is the scenario when the success of a treatment component is contingent upon mastery of another. For example, perhaps utilization of effective coping skills is contingent on emotional awareness. Therefore, it could be challenging to separate the two components. As such, a major limitation of MOST designs is its inability to account for overlapping treatment components.

A second crucial challenge in using factorial designs, especially if testing components of traditional psychotherapy, is the potential confounding factor of varying amounts of treatment time between treatment conditions. For example, if we were to examine the effects of three components of dialectical behavior therapy skills (DBT; emotion regulation—ER, distress tolerance—DT, and mindfulness—MI), we could propose a full factorial design in which each component would consist of a 20-min session of the skill delivered weekly. A full factorial would include eight treatment conditions (ER only, DT only, MI only, ER + DT, ER + MI, DT + MI, ER + DT + MI, all conditions off). However, in this scenario, the ER groups, on average, have 20 extra minutes of contact time with a therapist compared to those who do not receive the ER component. Therefore, if there were an effect of the ER component, we would be unable to determine whether it was due to the effect of ER, or simply due to unequal treatment times between conditions with and without ER.

To address this concern, a treatment component being “off” can be replaced with a “placebo control,” which can allow for equal total treatment time and equal amounts of each component being delivered across conditions (see Table 6). In this example, we use supportive psychotherapy as the placebo control, but another option could be utilized. In fact, each treatment component can be assigned its own placebo control to be administered when that component is “off” (e.g., for medication, it could be a placebo medication to control for the placebo effect of taking a medication daily). Assignment of a suitable control increases rigor by increasing the chances that the effect of a component can be attributed to the component rather than another variable. As can be seen in Table 6, the amount of placebo control an individual receives depends on treatment condition. If each therapy session is set to 60 min, and each active treatment component is 20 min, the amount of therapy time in each condition dedicated to placebo control ranges from 0 to 60 min. This design controls for amount of total treatment time, and allows for equal amounts of time to be dedicated to each component (e.g., amount of ER is the same in ER only and ER + IE and ER + IE + DT). It is crucial, however, that the placebo control be inert, or that its efficacy not be increased beyond the minimum amount delivered in any condition. If the efficacy produces change in increasing proportion above its minimum treatment time (in this case, 0 min), it becomes a confound. As such, there are ways of addressing the confound of time in factorial designs, but it remains a significant challenge to implementing the MOST/factorial approach and should be considered carefully.

Table 6.

Example breakdown of treatment time in a component analysis

Cell #	Treatment condition	ER	DT	MI	Placebo control time	Total time
1	ER + MI + DT	20 min	20 min	20 min	0 min	60 min
2	ER + MI	20 min	Off	20 min	20 min	60 min
3	ER + DT	20 min	20 min	Off	20 min	60 min
4	DT + MI	Off	20 min	20 min	20 min	60 min
5	ER only	20 min	Off	Off	40 min	60 min
6	DT only	Off	20 min	Off	40 min	60 min
7	MI only	Off	Off	20 min	40 min	60 min
8	All off	Off	Off	Off	60 min	60 min

ER emotion regulation, DT distress tolerance, MI mindfulness

Other potential downsides of the MOST approach

While use of the MOST approach in ED research may confer many benefits, it is worth noting some of its weaknesses. First, in Phase I, the possibilities for intervention components to be included in any one intervention span far and wide; with the proliferation of research using novel measurement (e.g., ecological momentary assessment, neuropsychological assessments, neuroimaging) to identify maintenance factors of ED behavior, there are many possible treatment targets to consider. While this “open playing field” can foster creativity and innovation, it can simultaneously be difficult to narrow the scope of treatment components to test, and there are no established guidelines for doing so. Relatedly, another downside of the MOST approach is that factorial experiments with many treatment conditions can be challenging to execute. Randomization to many treatment conditions and/or creation of several versions of a treatment manual is burdensome and mistakes can easily be made. For example, Pellegrini et al. [12] discussed above intended to conduct a fractional factorial design, but a clerical error led to the inadvertent expansion of the study to a full factorial design [38]. Although the experiment is still robust, this error illustrates the logistical complexity of implementing the approach. Researchers considering the approach must balance the benefits of what is learned scientifically from using MOST with the complexity of implementing it.

Conclusion

In summary, outcomes from treatment for EDs leave wide room for improvement in terms of efficiency, efficacy, and disseminability. The MOST approach proposes a structured, systematic methodology to treatment development prior to testing in RCTs. MOST allows for hastened identification of efficacious treatment components compared to traditional approaches, and thus conserves resources, which is especially important in the ED field because sample sizes are small. Despite its limitations, MOST has the potential to result in more effective, efficient, and disseminable interventions for EDs and is worthy of utilization in the ED field.