The Selection of Comparators for Randomized Controlled Trials of Health-Related Behavioral Interventions: Recommendations of an NIH Expert Panel

Abstract

Objectives:

To provide recommendations for the selection of comparators for randomized controlled trials of health-related behavioral interventions.

Study Design and Setting:

The National Institutes of Health Office of Behavioral and Social Science Research (OBSSR) convened an expert panel to critically review the literature on control or comparison groups for behavioral trials and to develop strategies for improving comparator choices and for resolving controversies and disagreements about comparators.

Results:

The panel developed a Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials. The model indicates that the optimal comparator is the one that best serves the primary purpose of the trial, but that the optimal comparator’s limitations and barriers to its use must also be taken into account.

Conclusion:

We developed best practice recommendations for the selection of comparators for health-related behavioral trials. Use of the Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials can improve the comparator selection process and help to resolve disagreements about comparator choices.

1. Introduction

Controversies and disagreements often surround the selection of comparators for randomized controlled trials (RCTs) of health-related behavioral interventions. In response to this problem, the National Institutes of Health Office of Behavioral and Social Sciences Research (OBSSR) assembled a multidisciplinary expert panel on comparator selection in health-related behavioral RCTs on April 12–13, 2017. The panel considered diverse areas of behavioral intervention research, including clinical treatment trials and community-based prevention trials. Disagreements were discussed, and votes were taken on the major issues, but the entire panel agreed on all major points. The main strength of this process is that it integrated the views of leading experts from diverse fields. Its main limitation is that public comments were not obtained. The recommendations reflect the perspective of the expert panel convened by the NIH but does not represent official policy or guidance of the NIH.

The panel’s recommendations focus on trials in which individuals are the units of randomization, but many of the principles also apply to cluster-randomized trials. The recommendations are intended primarily for researchers who are planning or proposing randomized trials of behavioral interventions and for peer reviewers of trial proposals and publications, rather than for meta-analysts. This article summarizes the panel’s findings and recommendations, and introduces the Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials. The full report is presented in Supplement 1.

2. Types and Characteristics of Comparators

The study arms to which interventions are compared are called “control” groups, arms, or conditions in some articles, and “comparison” groups, arms, or conditions in others. To minimize confusion, the generic term comparator is used whenever possible throughout this report.

Table 1 lists comparators that are often used in health-related behavioral RCTs. They are grouped under the types research questions they are typically used to address. These questions are described in Section 7.4. Table 2 defines the key attributes of comparators.

Table 1.

Comparators that are often used in health-related behavioral trials.

Name(s)	Description
Comparators Often Used to Evaluate Whether an Intervention Works at All
No treatment	No intervention is provided.
Wait list	The same intervention that is provided to the experimental group is subsequently provided to the comparator group, after the post-test evaluation has been completed.
Comparators Often Used to Determine How Well an Intervention Works Relative to a Clinically Relevant Alternative
Usual care Routine (standard) care Treatment as usual	Treatments or services that are routinely provided in the settings from which trial participants are recruited. Often differs across individuals and settings, and in some trials may be enhanced or restricted for trial participants.
Standard of care	State-of-the-art, guideline-adherent treatments or services that are routinely provided or recommended in the settings from which participants are recruited. In some circumstances, the standard of care is provided in a uniform fashion across individuals; in others, it may be individually tailored or personalized.
Optimized care Standardized care	If a “standard of care” exists but it is not being routinely practiced in the setting(s) in which the RCT is conducted, the care may be “optimized” or “standardized” by the investigators for the RCT to approximate the current standard of care.
Alternative intervention	Used when one intervention is compared to another intervention for the same problem, such as in comparative effectiveness trials.
Alternative modality	A condition that is identical to the experimental intervention except for the modality, channel, or source of delivery, e.g., an intervention that is delivered in person instead of by videoconference (i.e., a different delivery channel), or an intervention that is delivered by a human vs. a computer (i.e., a different intervention source).
Alternative content	A comparator that uses the same technology as the experimental intervention but that differs in content or in other details, e.g., a health behavior coaching smartphone app vs. a symptom monitoring smartphone app.
Supported or unsupported intervention	A comparator that uses the same technology as the experimental intervention and that provides the same content, but that adds or subtracts additional components, such as a human-delivered component (e.g., counselor guidance).
Comparators Often Used to Investigate How or Why an Intervention Works
Attention control Attention-placebo Nonspecific therapy	Umbrella terms for a variety of conditions that are usually designed to provide the same amount of contact with the intervention staff or program that will be given to the participants in the treatment arm of the trial. May be designed to control for common factors of therapy such as support, and/or other ingredients such as educational materials.
Placebo Sham	A condition that structurally resembles the experimental intervention but that lacks its putative active ingredient(s) and that is intended to alter expectancies or to stimulate a placebo response.
Component	A condition that is identical to the experimental intervention except lacking one or more of its putative active ingredients or elements.
Dosage	A condition that is identical to the experimental intervention except for one or more of its dosage parameters, e.g., frequency or duration of contacts.

Table 2.

Key characteristics of comparators used in health-related behavioral trials

Characteristic	Definition
Acceptability	Whether the comparator meets current standards for the ethical conduct of research and satisfies stakeholder requirements.
Feasibility	Whether the comparator can be successfully implemented, given the trial’s resources and environment.
Formidability	How much pre-post change the comparator induces (or is expected to induce) in the outcome.
Relevance	How closely the comparator corresponds to “real world” interventions, services, or programs.
Resemblance	How similar the comparator is to the intervention to which it is being compared.
Stringency	How well the comparator controls for threats to internal validity and helps to minimize biases.
Uniformity	How homogeneous the comparator is across participants or sites.

3. Sources of Controversy

The panel identified several reasons why comparators have been controversial in health-related behavioral RCTs. First, comparator choices affect the purpose, feasibility, fundability, results, and impact of RCTs. In many cases, disagreements about comparator choices are proxies for disagreements about the primary purpose of the trial. Such disagreements are often resolved by replacing the planned comparator with a different one. Unfortunately, the result may be a trial that cannot answer the original research question, or one whose de facto primary purpose is not the one that the investigator had intended to pursue. Second, there are often differences of opinion about unavoidable tradeoffs among comparator attributes. Third, many trials with no-treatment or wait-list comparators have been criticized as merely showing that behavioral interventions are better than nothing. This has led some researchers to conclude that behavioral trials should always control for attention or placebo effects, but others argue that it this often unnecessary or inappropriate.

Controversies also ensue when the scientific rigor of an RCT is erroneously equated with the stringency or formidability of its comparator. NIH defines scientific rigor as the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results [1]. By extension, RCTs are rigorous to the extent that they produce trustworthy, informative, and replicable findings. This requires tighter control over explanatory variables and more formidable comparators in some trials than in others.

4. Existing Approaches to Comparator Selection

4.1. Background

The panel reviewed the scientific literature on comparators and select research methodology textbooks [2–7] to identify existing guidance frameworks and informative perspectives. It found recommendations based on study purpose, research phase, research ethics, research context, empirical evidence, trial quality, and cumulative science.

4.2. Study purpose

A consistent theme in the literature on RCT methodology is that trials should be designed to serve the study’s primary purpose (i.e., its main aim, objective, hypothesis, or research question). This principle plays an especially prominent role in the International Conference on Harmonisation [8], the Purpose-Guided Trial Design (PGTD) framework [9], and the Obesity-Related Behavioral Intervention Trials (ORBIT) model [10].

4.3. Research phase

Mohr et al. [11] noted that stakeholder interests are best served in early-phase research by giving interventions a chance to demonstrate their potential value. The use of excessively formidable comparators in Phase I studies and in initial Phase II trials can decrease the chances of finding a signal, quash further work on promising interventions, and thereby deprive stakeholders of potential benefits. In contrast, later Phase II and Phase III efficacy trials are designed to inform decision-making by service providers and policy-makers. Type I errors, i.e., conclusions that interventions have benefits when in fact they do not, pose a greater risk to stakeholders in late-phase research. The use of comparators that are not very formidable in advanced Phase II and Phase III efficacy trials of behavioral interventions, including ones that have passed low-formidability tests in earlier studies, can increase the risk that ineffective interventions will be adopted. Thus, comparators with different degrees of formidability are needed at different phases of intervention research, because stakeholder interests shift.

This means that intervention developers should plan ahead during their early-phase work for the higher-formidability tests that will be conducted in later phases. An intervention may have to be refined and strengthened if early, low-formidability tests have shown promising results, to improve the chances that it will be able to withstand subsequent, higher-formidability trials.

Comparators are unnecessary in some early-phase studies. For example, the Obesity-Related Behavioral Intervention Trials (ORBIT) framework for developing behavioral interventions [10] recommends uncontrolled, small-N studies to examine an intervention’s components, dosage parameters, targets, and modes of delivery. As another example, Multiphase Optimization Strategy (MOST) trials examine the components of a complex intervention, rather than the intervention as a whole, and consequently do not require a comparator for the intervention as a whole [12]. Thus, the phase of research affects not only choices among comparators, but also whether a comparator is needed at all.

4.4. Research ethics

A variety of ethical issues have been raised about placebos and other comparators used in behavioral RCTs [13–15]. Potential risks and harms associated with comparators, and recommendations for addressing ethical concerns, have been discussed in detail elsewhere [e.g., 16]. Key considerations include inequity of care, inadequate transparency, potential risks from exposure to the comparator condition itself, and opportunity costs for participants.

4.5. Research context

Contextual factors such as current standards of practice, the setting in which a trial is conducted, or the characteristics of the population, can affect the feasibility, acceptability, and stringency of comparators [17]. For example, a no-treatment comparator may be infeasible if clinical interventions are available to participants from other health care providers [18], or if stakeholders in a community-based trial object to a no-treatment condition [19].

4.6. Empirical evidence

The growing empirical literature on comparators can help to inform comparator choices. For instance, it is often assumed that no-treatment, wait-list, and placebo comparators are interchangeable in terms of their formidability. However, recent network meta-analyses of interventions for depression [20] and social anxiety disorder [21] found greater improvement in no-treatment and placebo than in wait-list conditions.

4.7. Trial quality

Meta-analysts and clinical guideline panels often use instruments such as the Cochrane Risk of Bias tool [22] to rate their confidence in the available evidence. If a trial’s rating could be improved by choosing one kind of comparator rather than another, it would be advantageous to take this into account. However, these rating scales do not reward certain comparator choices or penalize others. Thus, they provide few clues as to which comparator to choose for an RCT.

4.8. Cumulative science

High-quality meta-analyses of RCTs are among the best sources of evidence to guide clinical and public health policies and practices [23]. Network meta-analyses can make indirect comparisons between conditions that have not been directly pitted against one another in RCTs. However, indirect meta-analyses are more vulnerable to selection biases than are meta-analyses of conditions that have been directly compared to one another in RCTs [24].

Direct, pairwise meta-analytic comparisons are possible only if enough RCTs of an intervention employ the same comparator. However, researchers have to ensure that their own trial is optimally designed to answer its primary research question, even if that means using a different comparator than has been used in previous trials. Whether the trial will meet the inclusion criteria for future meta-analyses is a less important concern for clinical trialists.

5. General Principles of Comparator Selection

5.1. Optimal comparator for the research question

The panel unanimously agreed that compatibility with the primary purpose of the trial is the single most important consideration in choosing a comparator. The optimal comparator is the one that will provide the clearest answer to the primary research question or the strongest test of the trial’s primary hypothesis. The rationale for the choice of the comparator should start with the primary purpose of the trial and not rest on less important considerations or arbitrary rules.

However, there may be barriers to the use of the optimal comparator in some circumstances. Also, the comparator that best fits the trial’s primary purpose may leave other questions unanswered or impose other limitations on the study. If the comparator’s optimality for the primary purpose of the trial would be diminished by addressing these questions or limitations, it may be better to address them in subsequent trials instead. Thus, it is necessary to consider not only the trial’s primary purpose but also barriers and limitations when choosing a comparator.

The panel also agreed that investigators should clearly explain their choice of comparator, disclose any alternatives that were considered, explain why they were rejected, and acknowledge the comparator’s limitations. Reviewers of RCT proposals and reports should judge comparator choices first and foremost in relation to the trial’s primary purpose, hypothesis, or research question, while recognizing the limitations such decisions often incur. They should also be cautious about requesting changes in comparators that would change the primary purpose of a proposed trial.

5.2. Barriers

If there is an insurmountable barrier to the use of the comparator that best fits the purpose of the trial, the investigator should consider whether it can be modified to overcome the barrier without sacrificing its goodness-of-fit, or whether a replacement comparator could overcome the barrier while still fitting the primary purpose of the trial. If the barrier cannot be overcome (e.g., if it would be unethical to randomize participants to a no-treatment arm that would deprive them of essential care, then it may not be feasible to conduct the trial in a way that would answer the primary research question or test the primary hypothesis. Faced with this dilemma, the investigator should reconsider the purpose and design of the study, and decide whether a different question, hypothesis, or design should be pursued instead. Also, when ethics board members, grant reviewers, and others who are charged with evaluating trial designs ask for a major modification or replacement of an optimal comparator, they should consider whether this will prevent the investigator from testing the primary hypothesis or answering the primary research question.

5.3. Limitations

A comparator may be the best choice for testing a trial’s primary hypothesis or answering its primary research question yet less than ideal in other respects. It may leave some secondary or exploratory questions unanswered or create opportunities for certain biases to affect the trial. Unlike ethical or resource constraints or vigorous stakeholder objections, such limitations do not create absolute barriers to the use of the comparator of choice. They may, however, affect the validity or utility of some of the conclusions that may be drawn from the results.

Thus, the justification for the choice of a comparator should address any important and foreseeable limitations and clarify any methodological compromises or tradeoffs that may have to be made to answer the primary research question or test the primary hypothesis. It should also consider whether the comparator can be modified to minimize its limitations without affecting its compatibility with the primary purpose of the trial, and whether an alternative comparator might be equally compatible but with fewer or less severe limitations.

6. The Pragmatic Model for Comparator Selection

The core principle of the Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials (Figure 1) is that the optimal comparator is the one that best serves the primary purpose of the trial. If a comparator is chosen on any other basis, it compromises the trial’s ability to answer the primary research question. This can happen if a barrier to the use of the optimal comparator is encountered and as a result, it is replaced with one that is feasible to use but that does not serve the primary purpose of the trial. It can also happen if the optimal comparator’s limitations are given greater weight than its compatibility with the trial’s primary purpose, and as a result, a suboptimal comparator is chosen instead.

Fig 1.

The Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials.

These problems can be prevented by identifying the comparator that best serves the primary purpose of the trial before addressing barriers and limitations. Some barriers may be overcome through minor modifications of the optimal comparator. Others may be insurmountable, and this may compel the investigator to revisit the feasibility or purpose of the trial. The limitations of the comparator are considered only after it has been determined that there are no insurmountable barriers to its use and any surmountable barriers have been resolved. The limitations are acknowledged, ameliorated if possible, and/or addressed in future studies if it would be both feasible and informative to do so. The output of the algorithm is the comparator that best fits the primary purpose of the trial, despite its limitations.

The Pragmatic Model for Comparator Selection encourages researchers to choose clinically relevant rather than artificial comparators that are unlikely to ever be used in practice, unless the primary purpose of the study requires an artificial comparator to isolate a mechanism of change. It acknowledges that methodological limitations and tradeoffs among comparator attributes are often unavoidable, and it asserts that tolerable limitations should not be allowed to stand in the way of informative research. It also acknowledges that resource constraints and unacceptable risks to participants or other stakeholders can pose legitimate barriers to the use of otherwise optimal comparators. The algorithm includes a path for researchers who encounter such barriers and who must therefore seek alternatives.

7. Selected Applications of the Model

7.1. Introduction

The Pragmatic Model for Comparator Selection provides a general strategy for the selection of comparators. This section applies the model to common challenges in behavioral RCT design.

7.2. Positioning and justifying the trial in an applicable research framework

By positioning and justifying the trial within an applicable translational research framework (Suppl. Figure S2, Suppl. Table S3), investigators can delineate the precursors of the current trial and the studies that may follow it. This helps to clarify the purpose of the trial and explicate the reasons why a certain comparator should be used and why others should not. For example, PRECIS-2 tool [25] can be used to position a proposed pragmatic trial along the explanatory-pragmatic continuum, and thereby help to build a case for choosing an appropriately realistic comparator and for defending its methodological limitations.

7.3. Defining the primary purpose of the trial

Positioning and justifying a trial in an appropriate research framework helps to establish its general purpose. The next step is to define the trial’s specific aims. The Population, Intervention, Comparator, Outcome, and Timing (PICOT) format is helpful for framing clear research questions [26]. It bridges the gap between the generic questions that emerge from translational research models and the specific ones that investigators have about interventions and outcomes. PICOT closely connects research questions to comparators, particularly for RCTs of relatively mature interventions.

7.4. Choosing the optimal comparator

Once the trial’s aims have been specified and justified, the next step is to choose the optimal comparator. Many research questions that can be asked about health-related behavioral interventions pertain to whether the intervention works at all, how well it works relative to clinically relevant alternatives, or how or why it works. The type of research question is a key determinant of the optimal comparator.

• Whether it works at all: Low formidability comparators such as no-treatment and wait list conditions are good candidates for trials in which this is the primary research question.

• How well it works relative to clinically relevant alternatives: Clinical or public health relevance is a key attribute of comparators in trials that evaluate how well an intervention works relative to a specific alternative. Relevant comparators include a) ones that reflect existing clinical or public health practices or services (e.g., usual care or standard of care), b) alternative interventions (e.g., a well-established, evidence-based intervention as a comparator for a newer intervention), and c) clinically-relevant variations on the experimental intervention (e.g., the same intervention except delivered via an alternative modality, such as when a face-to-face intervention is compared to the same intervention delivered via remote telehealth technology).

• How or why it works: “How” or “why” questions often require comparators that can isolate certain intervention components or underlying mechanisms of behavior change. Some trials compare the ostensibly unique or target-specific “active” ingredients of an intervention to a comparator that includes only “nonspecific” components such as attention. The resemblance of the comparator to the intervention is an especially important attribute for such trials. Comparators that are often used to address such questions include attention control conditions, nonspecific therapies, placebo or sham conditions, interventions that are identical to a multicomponent experimental intervention except with missing or modified components, and conditions that provide different dosages of the experimental intervention.

Mismatches between comparators and study purposes often occur when investigators attempt to answer more than one of these types of questions in the same trial, or when reviewers ask them to do so. Answering one question well is better than answering multiple questions poorly.

7.5. Addressing significant barriers

When a barrier to the use of the optimal comparator for the research question is encountered, the investigators should first determine if it is surmountable. If a minor modification can make the comparator acceptable or feasible while preserving the trial’s ability to answer the primary research question or test the primary hypothesis, then the modification should be made. If that is not possible, it may be necessary to choose a different comparator, if there is one that is compatible with the trial’s primary purpose. For example, if a trial is designed to ask whether an intervention works at all but a no-treatment comparator would be unacceptable for ethical reasons, the investigator could consider replacing it with a wait-list comparator or with a condition that would be ethically acceptable but inert with respect to the trial’s primary outcome. If an alternative comparator that addresses the barrier and still answers the primary research question cannot be identified, then the proposed research question may not be answerable and the research question should be reconsidered.

7.6. Addressing significant limitations

It is often impossible to design comparators with a perfect combination of attributes. Unavoidable tradeoffs among comparator attributes often impose limitations on behavioral trials. Structural limitations are built into the trial and can usually be recognized at the design or proposal stage. For example, a differential dosage of contact or attention between groups is a certainty when the experimental arm receives a behavioral intervention and the comparison group receives no treatment. Conditional limitations, in contrast, are not preordained, and they may or may not occur. For example, unanticipated differential attrition might occur in an RCT if the participants prefer one condition over the other.

The comparator’s limitations should be judged in relation to whether they would leave the primary research question unanswered. Minor limitations rarely constitute sufficient grounds to reject a comparator that is well suited to the primary research question or hypothesis. Also, many secondary questions can be addressed before an intervention is tested in a randomized controlled trial. However, trials are often conducted at times when a variety of secondary questions remain to be addressed. If the limitations of a comparator leave important secondary questions about an intervention unanswered, the model suggests that they should be addressed in subsequent trials instead of compromising the current trial’s ability to answer its primary research question.

7.7. Finalizing the choice of the comparator

The Pragmatic Model for Comparator Selection yields a comparator that addresses the primary research question, has no insurmountable barriers to its implementation, and no limitations that outweigh its compatibility with the primary research question or hypothesis. Investigators should document their decision process, so that readers understand why the comparator was selected and why other potential comparators were rejected.

8. Summary and Conclusions

Comparators are the lightning rods of health-related behavioral intervention research; they attract thunderbolts of controversy while diverting us from scrutinizing the purposes and goals of our trials. The Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials provides a way to resolve many of the disagreements and controversies that surround the comparators that are used in behavioral intervention trials. It gives greater weight to the compatibility of the comparator with the primary research question or hypothesis than it does to its limitations. It stresses the importance of carefully defining the primary research question or hypothesis, and of positioning and justifying every RCT within an applicable translational research framework. It also recognizes that there may be barriers to the use of otherwise optimal comparators, and it provides a pathway to follow when such barriers are encountered. The developers of this model hope that its adoption will help investigators, reviewers, oversight boards, and other stakeholders to address comparator-related disagreements and controversies that could impede progress in health-related behavioral intervention research if left unresolved.