The Trials and Tribulations of Selecting Comparison Groups in Randomized Trials of Nonpharmacological Complementary and Integrative Health Interventions

Abstract

Editor's Note: For over two decades, JACM Editorial Board member Karen Sherman, PhD has been among the most respected clinical trialists in complementary and integrative health research. The epidemiologist and Senior Researcher at Kaiser Permanente Washington Health Research Institute has focused on pain conditions and has led or been part of teams exploring the roles of such therapies as yoga, acupuncture, mind-body and manual therapies. In this Invited Commentary, Sherman shares wisdom gleaned from the process: trial designs, principles for selecting controls, benefits and liabilities of placebo and sham controls, attention controls, usual care controls, and more. She also discusses the effects of comparison groups on sample sizes, comparison groups for mechanistic studies, and other comparison groups' considerations. This commentary should prove a useful primer wherever research methods in complementary and integrative health are taught. We are pleased to offer it here through JACM.

—John Weeks, Editor-in-Chief, JACM (johnweeks-integrator.com)

Comparison groups play a critical role in evaluating a therapeutic intervention, because they are the benchmark by which we ascertain the therapy's usefulness for a specific health condition and at a minimum distinguish therapeutic improvement from the natural course of the disease (i.e., natural history).¹ However, serious debate about the principles of comparison group selection, including guidelines for matching such groups with the research question and explicitly describing the specific factors being controlled for, has been remarkably absent in the published literature. This is particularly true in efficacy trials of nonpharmacological therapies,^2,3 such as psychotherapy,⁴ exercise,⁵ and lifestyle interventions,⁶ where administration of a true “placebo treatment” is virtually impossible. This challenge becomes even greater for Complementary and Integrative Health (CIH) therapies, because they often arise from non-Western medical traditions,^7–9 often have multiple potential therapeutic elements with likely synergies between them,^9,10 and do not as clearly distinguish between specific and nonspecific effects.¹¹ Ideas for CIH comparison groups are often borrowed from another medical discipline without thinking clearly about what the comparison implies regarding the research question.

In this commentary, I remind readers of several key principles for selecting comparison groups. First is to ensure that the comparison group is appropriate for the research question. I will place this discussion in the context of key stakeholders for medical research and the general questions they would find most important. I will then briefly review the principal designs for full-scale clinical trials. Then, I will focus on appropriate comparison groups, their benefits and liabilities. Because most CIH nonpharmacological therapies are focused on symptom management,^12–15 my comments will be most pertinent for such conditions.

Key Stakeholders for CIH Research

For nonpharmacological CIH research, key stakeholders include the patient (and as appropriate, their caregiver and/or family), biomedical clinicians, and their professional associations, CIH clinicians or instructors and their professional associations, academic researchers, insurers, and health plans. Regulatory bodies such as the Food and Drug Administration (FDA) are usually not involved, but for some devices, such as lasers, they would be. In addition, Medicare or other governmental organizations are stakeholders for some therapies. Because of different needs for information about therapies, each stakeholder has a different perspective on the types of research they would find most valuable.

Academic researchers have been traditionally focused on questions about how the treatment works and demonstrating its effects under optimal circumstances.¹⁶ Most clinical trials have addressed these questions. However, other stakeholders have more practical needs.¹⁷ Patients and biomedical clinicians want to know whether the treatment will work for them (or their patient) and is safe so they can determine whether to try (recommend) the treatment. CIH providers will be interested in that as well (for coverage decisions) but may also be interested in investigating questions about how to optimize their treatment (e.g., dosing, most beneficial elements of the treatment).

With the rise of comparative effectiveness trials,^18,19 the creation of the Patient Centered Outcomes Research Institute (PCORI),²⁰ and implementation²¹ and dissemination²² science, there is more focus on answering those questions of greatest interest to patients, providers, health plans, insurers, and other stakeholders. Insurers and health plans want to know whether treatments that work (under ideal circumstances) will actually improve the health of those insured or treated by them.²³ In addition, they may be interested in costs and outcomes (either as cost savings, cost neutral, or cost-effective).²⁴ Government funders of health care may have similar interests to insurers, although constrained by their legislative mandate.

Principal Designs for Full-Scale Clinical Trials

In pharmacologic research, efficacy trials (Table 1) are typically required before a medication is approved by the FDA. These trials test whether the medication works under carefully controlled circumstances with a homogenous population for a short period.²⁵ Although this ensures that only “treatments that work” are added to the medical armamentarium, it does not indicate whether the treatment will work with patients typically seen by health care providers under more realistic treatment conditions.^17,26,27 For that, effectiveness trials or pragmatic trials are needed as they evaluate the usefulness of the new treatment under more real-world conditions²⁵ (Table 1). Of course, no trial is purely efficacy or effectiveness, rather they are on a continuum with each type as conceptual anchors. More recently, pragmatic trials have been featured as alternatives to efficacy trials; many of them are embedded in health care systems and feature real-world comparison treatments as controls. Other common trial designs are shown in Table 1 along with the questions they answer, information about both treatments and controls, and what actually is being controlled in the comparison groups.

Table 1.

Key Designs for Complementary and Integrative Health Clinical Trials

Type of trial	Question it answers	Treatment group	Comparison group	What is controlled	CIH examples
Efficacy	Does treatment work under ideal circumstances?	Carefully described fixed intervention	Placebo, sham, attention control (sometimes an active treatment)	Everything (natural history, placebo effects) except the “active ingredient”	Acupuncture for osteoarthritis (Berman et al.73)
Explanatory	How does the treatment work?	Carefully described fixed intervention	Placebo, sham, attention control, dismantling control (sometimes an active treatment)	Everything (natural history, placebo effects) except the “active ingredient” of interest	Acupuncture for cLBP (Cherkin et al.63)
Effectiveness	Does treatment work in the real world?	Intervention as practiced in typical care settings (may involve some constraints)	Usual care, standard of care, for comparative effectiveness, another treatment	Natural history and sometimes usual care/standard of care	Acupuncture for persistent back pain (Thomas et al.74)
Pragmatic	What are the overall benefits of routine care?	Intervention as practiced in typical care settings	Usual care, standard of care, sometimes another treatment	Natural history and sometimes usual care/standard of care	Individual versus group acupuncture for chronic pain in an underserved population (McKee et al.75)
Noninferiority	Is the new treatment roughly equivalent to the old one?	Carefully described intervention	A bonafide treatment in use in clinical practice (may be constrained or more like practice)	Natural history, placebo effects, the specific benefits of the old treatment	Yoga for cLBP (Saper et al.76)
Comparative effectiveness	How does a new treatment compare with an established one?	Intervention as practiced in typical care settings	An established treatment	Head-to-head comparison between two treatments	Acupuncture and massage for cLBP (Cherkin et al.77)
Three-arm trial	How does the treatment of interest compare with a placebo/sham/attention control (or a bonafide treatment) and with a usual care/standard-of-care control	Intervention (can be more fixed or more flexible)	Two controls: placebo/sham/attention control and usual care/standard of care	First controls for everything (natural history, placebo effects) but “active ingredient of interest (if bonafide treatment),” can serve as a “positive control”; second controls for natural history and usual care/standard of care	Yoga versus exercise versus usual care (note exercise served as a positive control) (Sherman et al.64)

CIH, Complementary and Integrative Health; cLBP, chronic low back pain.

In behavioral medicine, an analogous strategy is common. Initially, efficacy trials evaluate the benefits of a new psychotherapy.²⁸ These trials typically involve attention controls and are intended to be completed before effectiveness studies, which commonly use standard treatments as controls, are undertaken.²⁹

Comparison Groups for Efficacy Trials: Placebo and Sham Controls

For questions of efficacy of the treatment itself, the gold standard has been “placebos” that look and feel like the real treatment, if ethically feasible.³⁰ In medication trials, patients, providers, and evaluators are typically unable to determine whether the patient has received the verum or placebo pill. This allows the participants to be masked to the treatment group and effectively controls for all the reasons they may improve (i.e., natural course of the condition,³¹ regression to the mean,³¹ concurrent treatments, attention, the patient-provider relationship,^32,33 expectations about the benefits of treatment^34,35)³⁶ apart from the pharmacological actions of the medication. Placebos should minimize bias when evaluating the treatment benefits, which is especially important for subjective outcomes such as fatigue, nausea, or pain.^37,38 Providers are more likely to treat all patients the same because they do not know who is receiving a placebo treatment.

For some CIH treatments, a “sham” or “placebo” comparison (e.g., sham laser acupuncture as a control for verum laser acupuncture³⁹) could serve the same purpose as a placebo pill does. However, many sham treatments are controversial. For example, in trials investigating needle acupuncture, there is concern that “sham” controls (e.g., misplaced needling and/or shallow needling^40,41; non-insertive needling⁴²) may be active treatments.⁴³ In other trials, there is concern that the “sham control” does not adequately resemble the treatment (e.g., comparing needle acupuncture with a sham laser control⁴⁴). There are many types of “sham spinal manipulation.”⁴⁵ However, they are not true sham interventions, as they may produce a physiologic or biomechanical effect, even if small.⁴⁶ Thus, there is concern that sham treatments may actually include a treatment effect, of unknown size, in addition to controlling for natural history, regression to the mean, and contextual effects such as attention, patient-provider relationships, and expectations of treatment.

Comparison Groups for Efficacy Trials: Attention Controls

For many treatments (e.g., massage,⁴⁷ mindfulness,⁴⁸ t'ai chi, yoga) a viable “sham treatment” is not possible. Instead, various “attention controls”⁴⁹ are used. These nonidentical comparison treatments are intended to control for natural history, attention from a health care provider, expectations about the treatment's effect, group support when relevant and other contextual factors.⁵⁰ The challenge with building attention controls is that, depending on their nature, they presume that key elements of the “non-treatment reasons” patients get better can be explicitly built into these comparison groups. There is no external standard for determining that an attention control adequately controls only for the nonspecific reasons that patients may improve while undergoing treatment. Although all randomized comparison groups control for the natural history of a condition and regression to the mean, other aspects of the comparison are more difficult to precisely control.⁴⁹ Is the attention from health care practitioners of different backgrounds (e.g., surgeons vs. massage therapists) or specific providers⁵¹ equivalent in terms of how they interact with the patient? When therapies are more complex than single ingredient medications, which part of the therapies should be considered incidental and which part essential?¹¹ For example, should yoga be compared with exercise in an efficacy trial? This implies that yoga's active ingredients should only include the benefits beyond physical movement. But patients who practice yoga may receive benefits of exercise as well as breathing, relaxation, etc.¹⁰ Should massage be compared with light touch?⁴⁷ Or, should touch itself be considered part of massage? These are not questions that have easy answers.

How do you know the attention control is working as planned? I offer a conceptual example that illustrates the challenges of creating attention controls. We conducted a small trial of massage for generalized anxiety disorder.⁵² Our grant reviewers wanted us to create an attention control. Skeptical of the assumptions underlying such controls, we attempted a step-wise deconstruction of massage into three components: a control for natural history and contextual effects (by providing time to relax in a comfortable room with soft background music), an additional control for provider attention (by having the massage therapist provide “thermotherapy” using nontherapeutic doses of hot and cold provided in the same comfortable room), and, finally, the “active ingredient” of tissue manipulation (by providing massage in the same comfortable room). In the end, we found equivalent effects for all treatments, leading us to wonder whether no treatments worked beyond natural history or all were therapeutically beneficial. Qualitative participant comments revealed that they reduced their anxiety by focusing on the characteristics of their treatment. This finding suggests that our attention control group did not function as we intended. Even though our study was underpowered, this example illustrates the challenges of designing an adequate attention control.

Sometimes specific attention controls do not make sense. For example, a study of yoga for improving metabolic syndrome used a stretching control.⁵³ I was surprised to find the authors thought the “critical active ingredient” of yoga for patients with metabolic syndrome was passive relaxation (because the yoga group practiced restorative postures); the control group consisted of stretching classes.⁵⁴ Moreover, there was substantially more social interaction in the control group. The results of this efficacy study, which found both groups were equivalent, are hard to interpret.

For some symptoms, such as chronic pain, attention controls can be quite hard to develop. For example, as part of a study of t'ai chi for chronic low back pain (cLBP),⁵⁵ our grant reviewers asked us to include an “attention control” in addition to our usual care comparison. We used five principles to design this group⁵⁶:

• (1)It needs to control for attention, credibility, and time of the t'ai chi classes (i.e., 12 weeks of twice weekly hour-long classes). This would automatically control for the natural history of the condition, regression to the mean, expectations and interactions with the group and instructor.
• (2)It needs to exclude all hypothesized active ingredients for t'ai chi. These include: musculoskeletal strength, flexibility, and efficiency; breathing; concentration, attention, and mindfulness; imagery, visualization, and intention; physical touch, massage, and subtle energy; psychosocial interactions; alternative healing paradigm; rituals, icons, and environmental effects.⁹ This allows us to measure the full “specific effects” of t'ai chi.
• (3)It should not focus on alternative pathways by which pain could be alleviated or function improved. Therefore, we avoided known effective treatments (e.g., cognitive behavioral therapy, exercise, acupuncture). This is important to ensure that the full specific effects of t'ai chi are not compromised by comparison with another bonafide treatment.
• (4)It should not focus exclusively on any one common co-morbid condition that we think might be impacted by t'ai chi (e.g., depression, falls prevention, sleep). This ensures that the chronic pain is unlikely to be improved by an ancillary route (e.g., improving sleep makes the chronic pain less intense).
• (5)It should be credible to study participants. If the treatment is not credible, it cannot serve as an adequate attention control because participants would have low expectations of benefit and quite possibly, low participation.

These principles impose stringent requirements for an appropriate attention control. As a result, we thought the only suitable option was a health education comparison (HE). However, 24 h of HE strictly focusing on back pain could well have a beneficial effect on cLBP because there is evidence that educational interventions can lead to improvement in people with cLBP.^57,58 We, therefore, developed a broader curriculum focused on healthy living with cLBP for older adults that included relevant topics for this population. Although some were more related to cLBP than others, we made an explicit connection between their back pain and each topic. Topics were ordered so that those (e.g., stress, sleep, depression, osteoarthritis) that might be more likely to improve cLBP were discussed in later classes. Despite our care in developing this comparison group, including no sessions exclusively about exercise, participants noted that the need to be more active was the most common thing they learned from the classes. This illustrates the difficulty in selecting an appropriate control for pain conditions.

Comparison Groups for Pragmatic Trials: Usual Care, Standard of Care, or Another Established Treatment

For pragmatic questions asking about the value of a treatment in practice, comparisons with a standard treatment or usual care are usually the most appropriate.^17,29,50 In some studies, the intervention is considered an adjunct to usual care or standard care. Typically, such studies show greater benefit for the intervention than do studies using placebo, sham, or attention controls.^59,60 In pragmatic trials, decisions must be made about how much, if at all, to constrain the interventions permitted in the treatment and comparison groups, with fewer constraints generally preferred.

Three-Arm Trials of CIH: The Best of Both Worlds?

As mentioned earlier, efficacy trials for an intervention are typically performed before pragmatic trials of the intervention are conducted. But, for clinical trials of CIH, this staging of trial designs is not always followed because CIH interventions are already available to patients who are willing to pay for them. Thus, some researchers think it is more important to do the real-world studies.⁶¹ This can lead to some paradoxes, where efficacy may not be demonstrated but effectiveness is.⁶² One way to address arguments about which trial design is best for CIH interventions is to conduct three-arm trials with two controls groups, a more efficacy oriented control, and a more pragmatic control (Table 1). Although I have conducted several such trials^55,63–65 and appreciate their strengths, they are not perfect solutions either. Other aspects of the design (e.g., inclusion and exclusion criteria, follow-up procedures, flexibility in the treatment) will more closely reflect either efficacy or pragmatic designs.

Comparison Groups: Effects on Sample Sizes

Efficacy trials typically have smaller effect sizes than effectiveness trials.^30,60 Thus, they will require larger sample sizes to be properly powered. However, pragmatic trials where uptake of a treatment is part of the study question are likely to require much larger sample sizes, if the resulting effect sizes are small.²⁹ If the statistical power computation incorporates clinically meaningful differences for the condition studied, then the sample size will be independent of the control group.

Not all placebo or sham treatments have equivalent effects.⁶⁶ In a study comparing sham acupuncture with placebo pills for irritable bowel syndrome, for example, Kaptchuk et al.⁶⁷ found a larger effect size for sham acupuncture. However, in a study of asthma patients with placebo albuterol, sham acupuncture, and no intervention, placebo albuterol and sham acupuncture had similar effects for patient-reported improvement (45%–46%) and were superior to the no-intervention group (21%).⁶⁸

Comparison Groups: Mechanistic Studies

Mechanistic studies of CIH interventions can be challenging to conduct. For interventions with multiple components and attendant multiple mechanisms (e.g., whole systems of care such as Ayurveda, complex interventions such as t'ai chi or acupuncture with adjunctive treatments), using comparison groups to deconstruct the benefits of the intervention by looking at the benefits of specific intervention elements (e.g., posture, breathing, meditation for yoga) is conceptually attractive. However, if there are synergies among these elements, resulting findings may be unable to detect them. One way to overcome this challenge is to use the multiphase optimization strategy (MOST) design,⁶⁹ which is designed to optimize the doses of selected intervention elements.⁷⁰ However, if different patients preferentially benefit from different elements of a complex intervention, the MOST design will not detect this.

Researchers should be cautioned that many mind-body therapies (e.g., yoga, t'ai chi) may have similar or overlapping mechanisms. As such, it would not be appropriate to compare them against each other in a mechanistic study designed to measure the benefits of specific mechanisms.

Comparison Groups: Other Considerations

Additional considerations exist for selecting appropriate comparison groups. Is encouragement to practice an intervention that requires regular practice (e.g., yoga, t'ai chi) a “non-specific effect” or an essential part, even if not unique, of that intervention? I believe it is essential; it might be analogous to the idea of important “common factors”⁷¹ (e.g., therapeutic relationship, expectations about the therapy) that undergird all psychotherapies. The common factors argument suggests that such effects may be a large and critical component to the benefits of psychotherapy. I do not believe that automatically requiring the comparison group to encourage home practice, which effectively ensures that it is not part of the intervention benefits, is appropriate unless the study is explicitly looking at specific mechanisms by which the intervention works.

Ethics are another important area that merit attention when thinking about control groups.⁷² For some conditions, such as cancer, placebo controls are unethical and new medications can only be tested as add-on treatments. For procedural treatments,⁷² there is concern about ethical dilemmas that may be created for providers who must deceive the patient about the treatment they are getting.

Concluding Remarks

In this commentary, I remind researchers to think carefully about their study goals as well as the particular nuances of their therapy when designing a comparison group or groups. Hopefully, this will lead to both rigorous and appropriate study designs. It may also add to our knowledge base regarding the principles of selecting the most appropriate comparison for specific clinical trials. No doubt, additional considerations exist and will be addressed as clinical trials evolve to meet the changing needs of stakeholders.

Disclaimer

The contents are solely the responsibility of the author and do not necessarily represent the official views of the National Center for Complementary and Integrative Health (NCCIH).

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This publication was made possible by grant no R34AT009052 from the NCCIH at the National Institutes of Health.