Advances in Clinical Trials Methodology: Intervention Optimization Approaches in Emergency Medicine

Abstract

The classical two-arm randomized clinical trial (RCT) is designed to test the efficacy or effectiveness of an intervention, which may consist of one or more components. However, this approach does not enable the investigator to obtain information that is important in intervention development, such as which individual components of the intervention are efficacious, which are not and possibly should be removed, and whether any components interact. The Multiphase Optimization Strategy (MOST) is a new framework for development, optimization, and evaluation of interventions. MOST includes the RCT for purposes of evaluation, but inserts a phase of research before the RCT aimed at intervention optimization. The optimization phase requires one or more separate trials similar in scope to an RCT, but employing a different experimental design. The design of the optimization trial is selected strategically so as to maximize the amount of scientific information gained using the available resources. One consideration in selecting this experimental design is the type of intervention to be optimized. If a fixed intervention, i.e. one in which the same intervention content and intensity is provided to all participants, is to be optimized, a factorial experiment is often appropriate. If an adaptive intervention, i.e. one in which intervention content or intensity is varied in a principled manner, is to be optimized, a sequential multiple-assignment randomized trial (SMART) is often a good choice. The objective of this article is to describe MOST and the scientific rationale for its use; describe two current applications of MOST in emergency medicine research, one using a factorial experiment and the other using a SMART; and discuss funding strategies and potential future applications in studying the care of individuals with acute illness, injury, or behavioral disorders.

1. Introduction

Suppose one is planning a randomized clinical trial (RCT) to test the efficacy of an intervention to treat emergency department (ED) patients with opioid use disorder. The intervention contains four components: (1) a brief motivational interview to elicit participants’ readiness to reduce substance use, enhance self-efficacy and skills, develop autonomy, and elicit commitment talk to change; (2) induction in the ED with buprenorphine; (3) enrollment in a smartphone SMS texting program to “push” messages to the participant to enhance skills and self-efficacy; and (4) an active referral to a drug treatment center to continue buprenorphine. The control arm receives a sheet of paper with a list of local drug treatment centers, but the participants must initiate the call themselves.

At the end of the study, the intervention arm shows a statistically significant decline in self-reported use of opioids, has a higher proportion of participants with negative urine tests, and more participants engaged in ongoing drug treatment. Satisfied with the results, the investigators prepare a manuscript for submission to a high-impact journal, and present the findings at an important national meeting. After the presentation, audience members ask these questions:

1. Which of the intervention components was most efficacious?

2. Were any not efficacious?

3. Were there important interactions between and among intervention components?

4. What, if anything, can you offer to participants who did not respond to the intervention?

5. Given these results, what would be the next study you would perform?

Realizing their inability to answer these questions, the investigators decide their next study should use methods and analytic techniques to facilitate answers to these critical questions. In reviewing the literature, and speaking with colleagues who are experienced clinical trialists, they learn of a new framework, the Multiphase Optimization Strategy (MOST).^1,2 They also learn that investigators working within this new framework use a variety of experimental designs in addition to the classical RCT, among them the factorial experiment^3,4 and the Sequential Multiple Assignment Randomized Trial (SMART).^5–7

The objective of this article is to describe MOST, the scientific and methodologic rationale for its use, how it is currently being used in emergency care research, funding strategies, and potential future applications in studying the care of individuals with acute illness, injury, or behavioral disorders.

2. The Multiphase Optimization Strategy (MOST)

Most clinical trials are designed to test the efficacy or effectiveness of a single intervention, such as a new drug or new diagnostic modality. In the classical two-arm RCT, research participants are randomized either to a control arm, to receive standard care, or an intervention arm to receive the experimental intervention. All other interventions, or accompanying maneuvers, are offered to both groups. In this article the term RCT will be reserved for this type of experiment.

However, as illustrated in the hypothetical substance use disorder treatment described earlier, many interventions contain multiple components. These interventions may show efficacy when delivered and evaluated as a package. But using the conventional approach it is not possible to disaggregate the effect of individual components of the intervention statistically, or to estimate interactions between components that may have occurred, even though this information is critical in intervention development.

MOST has been developed to address these concerns and others. As Figure 1 shows, the MOST framework distinguishes between the evaluation phase of research, in which an intervention is evaluated as a package in a classical RCT, and the optimization phase, which precedes evaluation and requires a separate optimization trial. An optimization trial is conducted using an experimental design suited to the reseach questions at hand, which typically require assessing the effects of individual intervention components and component interactions. Therefore, usually experimental designs other than the RCT, for example the factorial experiment or the SMART, are selected for the optimization phase of MOST, whereas the RCT is reserved for the evaluation phase. The optimization phase may involve a single optimization trial, or more than one if the investigators wish to identify components that perform poorly using one optimization trial, revise those components, and retest them in a subsequent trial.⁸ Decisions about the composition of the optimized intervention are made based on the results of the optimization trial(s) and, if desired, considerations of cost, which may be broadly defined to include not only money but staff time, participant burden, or any other limited resource. For example, the goal of the investigators may be to identify the combination of components that produces the best expected outcome for, say, no more than $1,000 per individual.

Figure 1.

Multiphase Optimization Strategy (MOST) design (reproduced from Ref. 1).

MOST is based on two principles adapted from industrial engineering, those of resource management and continual optimization. Resource management stipulates that resources available to conduct research should be managed strategically to maximize the amount, relevance, timeliness, and quality of information to be gained. This means the design of an optimization trial must be selected carefully, and sometimes, where appropriate, primary endpoints may be assessed relatively quickly (months, rather than a year or more) after the intervention. One implication of the resource management principle is that resources are spent on evaluation only if the results of the optimization phase suggest a sufficiently promising intervention can be constructed from the components that have been examined. Continual optimization specifies that optimization is an ongoing process, with repeated cycles making incremental improvements in the efficacy and efficiency of the intervention, using information gained from previous cycles of MOST.

As Figure 1 indicates and as mentioned above, investigators may select from a variety of experimental designs for use in an optimization trial. The selection is based on the resource management principle. One consideration is the type of intervention to be optimized. In fixed interventions, the same intervention content and intensity is provided to all participants, whereas in adaptive interventions (also termed stepped-care strategies, dynamic treatment regimens, or treatment algorithm approaches), intervention content or intensity is varied in a principled manner based on the patient’s initial clinical presentation and/or response to treatment approach. In the following sections we discuss two experimental designs frequently used for optimization trials, and present an example of each from emergency medicine. We discuss the factorial experiment, which is often a good choice for optimization of fixed interventions, and the SMART, which has been developed for optimization of time-varying adaptive interventions.

2.1. Optimization of a fixed intervention: The factorial optimization trial

One of us (SLB) is currently conducting a study to optimize a fixed tobacco treatment regimen for adult smokers seen in the ED. The study protocol has been previously reported.⁹ The study extends work previously done by our group and others demonstrating the efficacy of ED-initiated treatment for tobacco dependence,^10,11 and is supported by the National Cancer Institute (R01CA201873). In this optimization trial, four intervention components are examined: (1) a Brief Negotiated Interview (BNI), which is a shortened version of a motivational interview; (2) 6 weeks of nicotine replacement therapy (NRT, nicotine patches and gum), with the first patch applied in the ED; (3) an active referral to the state smokers’ quitline, made by the research assistant faxing a consultation form; and (4) enrollment in a 6-week texting program. Participants are contacted by telephone at 1 and 3 months. Participants who self-report abstinence from smoking at 3 months are asked to return to the ED for biochemical validation, via measurement of exhaled carbon monoxide. Study enrollment and follow-up are completed.

Table 1 illustrates the design of the optimization trial for the tobacco dependence study. In a factorial optimization trial there is a factor (independent variable) corresponding to each intervention component to be examined, and the levels of the factors correspond to options for each component (e.g. on/off; high/low). All of the factors are manipulated simultaneously in the experiment. If there are k factors each with 2 levels, the design is a 2^k. Because in this experiment there are four factors, each with 2 levels (offered/not offered), this is a 2⁴ factorial experiment, with 16 experimental conditions.

Table 1.

Experimental conditions in the trial.

Condition	BNI	NRT	QL	Text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

BNI = Brief Negotiated Interview

NRT=Nicotine Replacement Therapy

QL=Quitline referral

Text=Smoke-Free Text

Green = component is offered.

Red = component is not offered.

We note that although this design has 16 experimental conditions, it should not be considered a 16-arm RCT. A 16-arm RCT would likely require an enormous sample size, whereas an advantage of factorial designs is their efficiency in the use of experimental participants. Rather than conducting a series of two-armed trials, comparing a single intervention component to usual care (or another intervention component), all components are be assessed simultaneously, along with their interactions, in a single trial. This saves both time and resources. The chief reason is that, in a traditional RCT, effects are estimated by directly comparing the means of individual experimental conditions. By contrast, in a factorial trial main effects are estimated by comparing means aggregated from combinations of experimental conditions. All main effects are estimated based on all experimental conditions, with the conditions aggregated in different ways for different main effects. For example, the main effect of the Brief Negotiated Interview is estimated by comparing the mean of experimental conditions 1—8 in Table 1 to the mean of experimental conditions 9—16, and the main effect of Nicotine Replacement Therapy is estimated by comparing the mean of experimental conditions 1—4, 9—12 to the mean of experimental conditions 5—8, 13—16. Hence, a more modest number of study participants can be recruited, as compared to conducting four separate RCTs to examine each component.^1,12,13 This experiment is well-powered, with 66 participants per cell, for an overall sample size of 1056. Another noteworthy feature of the design, evident in Table 1, is that fully 15/16 of all study participants will receive at least one active intervention component. It is even possible to design a factorial optimization trial so that all participants receive at least some active treatment.

Based on prior work on the cost effectiveness of tobacco treatment interventions,^14,15 we prespecified that we would select components for the optimized intervention within an economic constraint of no more than $5000 per quality-adjusted life-year gained. Hence, we are collecting data on the costs of the intervention and subsequent treatment related to tobacco use.

A novel feature of our optimization trial is the addition of semi-structured qualitative interviews conducted by phone with a subsample of 65 participants, drawn from 15 of the 16 possible study conditions (participants in condition 16, in which none of the components is offered, are exempted from the qualitative work). This will allow us to assess the feasibility and acceptability of individual intervention components and combinations of intervention components. Analysis of the qualitative data is ongoing.

At the conclusion of all follow-up, we will analyze results for clinical efficacy, cost-effectiveness, and feasibility/acceptability to participants, as assessed by the phone interviews. In addition, we will examine two- and three-way interactions among study components. Components that are clinically efficacious, fit within the cost constraint, and are feasible and acceptable to participants will be retained, and packaged for evaluation in a follow-on clinical trial, which will require additional grant support.

Data from a factorial optimization trial are generally analyzed by means of factorial analysis of variance (ANOVA). When properly conducted using effect coding, this approach keeps correlations between effects to a miminum, even to zero in a perfectly balanced experiment. A generalized linear models (GLM) approach may be used to enable ready inclusion of covariates and use of different link functions, although the latter calls for care in interpretation of effect estimates, particularly interactions.

2.2. Optimization of a time-varying adaptive intervention: The sequential multiple-assignment randomized trial (SMART)

Several of this report’s co-authors (MW, PC, RC, KK) are developing an ED-based adaptive intervention to reduce risky drinking and violent behavior among adolescents. The study, called the SafERteens M-Coach study and funded by the National Institute on Alcohol Abuse and Alcoholism (R01AA024755), builds on prior work from our research group demonstrating that a selective 30-minute brief in-person therapy intervention (SafERteens intervention) for alcohol and violence delivered during emergency care reduces severe peer violence, dating violence, and alcohol consequences.^16–18 A post hoc analysis identified that response and non-response at 3 months post-intervention depended on several factors, including greater problem severity (e.g., alcohol use, violence), delinquent behaviors, mental health concerns, and negative peer influences, suggesting that although a single brief in-ED intervention session may be sufficient for many youth, a subset of youth require a more intensive and longer-term intervention.¹⁹ Pilot studies indicated that participants are amenable to extended interventions delivered via text messages and/or remote coaching.^19,20 These findings suggested that a productive direction would be development and optimization of a time-varying adaptive intervention.

Adaptive interventions are made up of decision rules that assign treatment options to participants at decision points based on tailoring variables. The treatment options may vary in approach, intensity, dose, delivery method, and/or cost (e.g., daily e-health delivered therapy messages vs. in-person weekly behavioral therapy).^5,6,21–23 These treatment options may all be available at each decision point in a patient’s course, or some may be available only if an earlier treatment option has failed or been successful. The tailoring variables are one or more individual (e.g., treatment response, adherence, side effects) or environmental (e.g., family/social characteristics) variables used to evaluate patient response on their currently assigned treatment pathway in order to tailor the treatment approach (i.e., the treatment decision). For example, tailoring variables may be used to identify early signs that a treatment is not, or is no longer, effective or sufficient for that specific patient. Finally, decision rules are a series of guidelines governing how a patient’s treatment pathway will change, depending on their tailoring variables and/or response to prior treatments. Adaptive interventions mirror “real-life” medical practice by developing a series of treatment approaches to be tried in response to how a patient responds to prior approaches. Such an approach is particularly useful for managing chronic diseases (e.g., obesity, substance use, diabetes, interpersonal violence) as they are often characterized by multiple relapses or recurrences, a series of individual and environmental factors influencing treatment outcomes that often require a specialized approach, and significant heterogeneity in treatment responses, both within the individual person and between people with similar conditions.

When the objective is to optimize a time-varying adaptive intervention, as it is in the SafERteens M-Coach study, the SMART is often a good choice. The SMART was developed by Murphy²⁴, and informed by earlier work by Lavori and Dawson.^25,26 SMARTs are multi-stage randomized trials^{5,21,22,24,27,28} in which randomization of study participants occurs sequentially, and may depend on the outcome of a prior treatment phase in the study. However, although the SMART is useful in optimization of adaptive interventions, it is not an adaptive experimental design. Unlike adaptive experimental designs, in SMARTs the treatment arms and randomization parameters do not vary during the conduct of the study.¹ SMARTs facilitate testing the efficacy of individual components of adaptive interventions (e.g., what are the best tailoring variables, key decision rules) to aid in constructing an overall efficacious adaptive intervention that can then be tested against the current standard of care or a control condition.

The SafERteens M-Coach SMART tests two novel mechanisms for increasing intervention intensity: 1) an automated text messaging program; and 2) an ongoing intensive remote health coach delivering weekly post-ED behavioral therapy sessions. This experiment addresses three research questions. The first is which is the better main first-stage approach, the single-session SafERteens intervention augmented with the automated text messaging program, or the single-session SafERteens intervention augmented with the ongoing intensive remote weekly health coach counseling? The second is, which is the more efficacious second-stage strategy for youth who are responders and those who are non-responders based on measures of binge drinking and physical aggression, assessed one month after their ED visit? Should responders continue with the same treatment or step down to a less intensive treatment? Should non-responders continue with the same treatment or step up to more intensive treatment? The third is, are there baseline or time-varying moderators of efficacy? Such moderators may make useful tailoring variables in future versions of the intervention.

Similar to the original SafERteens trial, youth (age 14-20) are enrolled in the study based on screening positive for binge drinking and physical aggression within the prior 4-months, including having a cell phone with a text messaging plan (e.g., not a messaging app only). Figure 2 illustrates the design of the SMART. After screening and consenting, youth are randomly assigned at the time of their ED visit to receive either the SafERteens brief intervention with the post-ED automated, tailored daily text messaging (two messages per day; Brief Intervention (BI) + Text Message (TM) or the SafERteens brief intervention with continued weekly post-ED remote therapy delivered by a Health Coach (HC) (one ~20-minute session per week; BI+HC). Youth involvement in drinking and violence is assessed throughout the intervention period of 4 weeks via weekly surveys. After their fourth weekly survey (4 weeks following their ED visit), responses on the four weekly surveys are used to determine whether the participant is a responder (defined as no longer binge drinking, or no moderate or severe physical aggression behaviors^29,30) or a non-responder (defined as continued binge drinking or moderate or severe physical aggression behaviors).^29,30 Specifically, a responder could have some of these behaviors in weeks 1 or 2, but none in weeks 3 and 4; in other words, those reporting any binge drinking or physical aggression behaviors in weeks 3 or 4 are considered non-responders. Importantly, those with missing data (failure to complete a weekly survey) in weeks 3 or 4 are considered non-responders for conservative purposes. As Figure 2 shows, responders are then re-randomized either to stay the course with the current treatment (where the current treatment could be either BI+TM or BI+HC, depending on initial assignment), or to step down to a resource brochure (link to study website containing resources). Non-responders are re-randomized to either stay the course with the current treatment (where again the current treatment could be either BI+TM or BI+HC, depending on initial assignment) or to step up their treatment. Non-responders who were originally assigned to the BI+TM condition (receiving automated tailored text messages) switch to receive HC (weekly remote therapy) as their treatment. Non-responders originally assigned to BI+HC receive an increased intensity of contact with the HC, including scheduled weekly sessions and in between-session personalized text messages from the HC, with option to text back and forth.

Figure 2. SafERteens M-Coach SMART trial design.

BI = Brief Intervention or the SafERteens Intervention; TM = Text Message; HC = Health Coach;

Note: Participants stepped down to Resources receive a link to the study website containing community resources, with no further TM or HC intervention. Participants stepped up to the HC+ intervention receive an increased intensity of contact with the HC, including scheduled weekly sessions and in between-session personalized text messages from the HC, with the option to text back and forth.

Participants are contacted at 4- and 8-months post-baseline to assess, via self-report, alcohol and violence behaviors. Anticipated sample size is 700; at present, the study is screening and enrolling eligible youth. At study conclusion, analysis of results, which will include the cost of the pathways, will form the basis for constructing an optimized, adaptive treatment algorithm for reducing alcohol and violent behaviors among at-risk youth seen in ED settings.

Several analytic considerations must be taken into account for SMART designs.^5,21,22,31 For certain research questions, analysis of a SMART is similar to analysis of a traditional factorial experiment. The first research question in the SafERteens M-Coach study, concerning which is the better first-stage approach, can be addressed by comparing the mean of the four leftmost experimental conditions in Figure 2 (A, B, C, D) to the mean of the four rightmost experimental conditions (E, F, G, H). The second set of research questions is answered in a similar manner, but must be conditioned on the 4-week outcome, which determines whether a youth is considered a responder or non-responder. The question of which is the better second-stage approach for responders, whether to stay the course or step the treatment down, can be addressed by comparing the mean of the two conditions for responders labeled Stay the Course (A, E) to the mean of the two conditions labeled Step Down (B, F). Non-responders are not included in this analysis. Similarly, the question of which is the better second-stage approach for non-responders, whether to stay the course or step the treatment up, can be addressed by comparing the mean of the two conditions for non-responders labeled Stay the Course (C, G) to the mean of the two conditions labeled Step Up (D, H). Responders are not included in this analysis.

Analysis of data from a SMART is usually performed using standard longitudinal analytic techniques (e.g., Generalized Linear Mixed Models – GLMM). Depending on the design, some research questions may require weighted regression models and/or participant replication to account for under-representation of all necessary participants in the data. For research questions involving moderation, investigators may consider standard moderator analyses, as well as more advanced Q-learning regression analytic techniques that allow for a determination of which sequence of decision rules produces the best expected outcome on targeted clinical outcome measures.^5,21,22,31

3. Considerations for Grant Applications

Optimization trials are not pilot studies; in fact, they are comparable to RCTs in labor- and resource-intensiveness. They are, generally, best suited for R01 or R01-equivalent mechanisms, to allow up to 5 years of funding with $500,000 in direct costs per year, although there may be circumstances under which it would be possible to conduct an optimization trial in less time or using fewer resources. The purpose of an optimization trial is to help assess which components are effective and, depending on the design, where there are interactions between components. Neither the factorial experiment nor the SMART enable evaluation of the optimized treatment package that is ultimately identified; that requires an RCT. Of note, it may not be possible to conduct both an optimization trial and a fully powered RCT of the optimized intervention within the five-year R01 time frame. This is the case with both of the studies described above. Hence, an important component of grant strategy is to indicate that a follow-on grant will be submitted, shortly after completion of the optimization trial, to evaluate the resulting optimized intervention by comparing it to the current standard-of-care via an RCT. In our experience, study sections are increasingly comfortable with this approach. However, it is a good idea not to assume reviewers are completely familiar with the factorial experiment or the SMART.

4. Future Directions

The MOST framework, which involves conducting one or more optimization trials before an RCT, is no longer new in patient-oriented research, but is still novel in emergency medicine. MOST is particularly suitable for developing and testing the efficacy of multicomponent interventions, including fixed and adaptive interventions. Methods in this area continue to evolve. One of us (SLB) has incorporated a qualitative assessment to MOST’s assessments of clinical effectiveness and cost. In addition, the growing use of “big data” approaches to collecting participant-level information, via biosensors, wearables, and smartphones has led to the development of a new experimental design for optimization of mHealth interventions, the micro-randomized trial,³² in which participants may be randomized many times in the course of a study, based on responses to assessments made, or data collected, repeatedly throughout the study.

These methods will likely further be employed by investigators in emergency care, for work conducted in the prehospital setting, the ED, aftercare settings, and post-discharge. We encourage emergency care investigators to explore the use of these innovative trial designs.