How to design high quality acupuncture trials—a consensus informed by evidence

Abstract

An international panel including patients, clinicians, researchers, acupuncture and surgery trialists, statisticians, and experts in clinical epidemiology and methodology have developed new guidance for randomised controlled trials in acupuncture. It addresses the most prevalent and critical concerns of current acupuncture trials and will help funding agencies, trial registers, and journal editors to evaluate the relevance, importance, and quality of submitted trial proposals and completed trials

Acupuncture therapies are used in 183 countries according to a 2013 survey by the World Federation of Acupuncture-Moxibustion Societies.¹ Despite over 14 000 published randomised controlled trials in acupuncture,^{2 3} scepticism remains about the number of studies to support the effectiveness and safety of interventions, as is the case for other complementary therapies. Challenges in designing and conducting high quality acupuncture trials may partly explain the scepticism. Randomised trials provide the most rigorous support for therapeutic and health system decisions for patients, clinicians, payers, and policy makers.^{4 5} Multiple meta-epidemiological studies have identified shortcomings in acupuncture trials. Common problems include limitations in study design, applicability, and reporting.^{6 7 8 9}

In contrast to pharmacological interventions, acupuncture therapies rely on procedural expertise and are complex and multifaceted.^{8 9} These characteristics present unique methodological challenges for randomised trials, including standardisation of interventions and controls, dealing with non-specific effects of interventions, and patients’ expectations.

Summary points.

• Despite over 14 000 published randomised controlled trials in acupuncture, shortcomings persist, including research corresponding poorly to clinical practice, poor reporting, and high risk of bias

• In contrast to pharmacological interventions, acupuncture therapies rely on procedural expertise and are often complex and multifaceted

• An international panel including patients, clinicians, researchers, trialists in acupuncture and surgery, and experts in statistics, patient engagement, and clinical epidemiology and methodology developed guidance for trials

• This guidance covers methodological issues, including selection of control, blinding, and the challenges particular to acupuncture trials, including four essential design elements for acupuncture based on whether trials are primarily explanatory/mechanistic or pragmatic/practical

The original and revised Standards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA)^{10 11} focus on reporting not optimal trial conduct. Other acupuncture guidance covers type of trial (efficacy,¹² effectiveness¹³), design, and lack of representativeness (include an international panel¹⁴), but is not informed by comprehensive and systematic evidence synthesis.

To address the most prevalent design and methodological concerns in current acupuncture randomised trials, an international panel including patients, clinicians, researchers, and trialists in acupuncture, surgery, statistics, patient engagement, and clinical epidemiology developed guidance for research teams planning acupuncture trials. To inform the guidance, we conducted a systematic survey¹⁵ exploring characteristics associated with acupuncture treatment effects.

The issues covered in this guidance fall into two categories: those unique to acupuncture trials, and, based on our exhaustive review of acupuncture trials, those that are not specific but still applicable to acupuncture trials. We frame our discussion as a series of questions for readers to consider when designing trials, focusing on choices of the research question, patients, interventions, and outcomes and presenting the implications of these choices using examples from existing trials (box 1).

Box 1. Guidance to consider when designing an acupuncture trial.

Choosing the research question

Consideration 1 : Is the question important?

Trialists should establish the rationale for the research question informed by a systematic review of the relevant literature.

Enrolling patients

Consideration 2: Who should the participants be?

Trialists with a primary explanatory or mechanistic objective could include people who are most responsive to the intervention. With a primary pragmatic or practical objective, trialists could include a broad population varying in age, severity, comorbidity, and exposure to other interventions.

Consideration 3: How can trialists address possible differences in effect across patient groups?

For pragmatically oriented trials include heterogeneous populations, with prior specification of subgroup analyses to consider hypothesised effect modification.

Selecting the intervention

Consideration 4: Who should perform the intervention?

Trialists should report the expertise of the acupuncturists. A trial that aims to show whether an acupuncture treatment can work under ideal conditions will choose the most expert practitioners available. Trialists aiming to establish the effect of treatment in ordinary practice will select clinicians with typical levels of expertise.

Consideration 5: What specific acupuncture technical features should be considered if aiming to design trials for maximum treatment effect?

If aiming for maximum treatment effect, trialists should select a high frequency of acupuncture treatment sessions and penetrating type of acupuncture (manual and electroacupuncture) over lower frequency or non-penetrating (transcutaneous electrical acupoint stimulation (TEAS), laser, acupressure) acupuncture.

Choosing the comparator

Consideration 6: Are trialists interested in the specific effect or the overall (specific and non-specific) effect of acupuncture?

Trialists should blind data collectors, outcome assessors, and data analysts and carefully consider the desirability of a sham that leads to underestimation of acupuncture’s treatment effects in clinical practice.

Fine tuning the flexibility of intervention and comparator

Consideration 7: To what extent should the trialist choose a flexible intervention and comparator?

Trialists with a primary explanatory or mechanistic focus might specify that practitioners administering both the intervention and the control use a highly standardised approach. Trialists with a primary pragmatic or practical focus might include clinicians with varied techniques reflecting practice in the community and instruct them to use their usual treatment approaches.

Consideration 8: How should trialists deal with adherence?

When designing a trial that primarily takes the individual patient’s perspective, trialists might consider implementing strategies to achieve optimal adherence. When trialists take a public health perspective, there is no need to implement strategies to increase adherence.

Selecting the outcome measure and follow-up

Consideration 9: What type of outcomes should trialists choose?

Trialists with mainly mechanistic objectives can focus on surrogate outcomes (eg, blood pressure, heart rate). If trialists have a primary pragmatic objective, they should focus on patient important outcomes such as pain. Trialists should continue using symptoms and functional outcomes and increase the use of quality of life and major events (eg, cardiovascular events, hospital admission, infections) as outcomes.

Consideration 10: How should trialists decide the duration of follow-up?

When investigators have primarily explanatory objectives, short follow-up times are likely to be optimal. Acupuncture trials with primarily pragmatic objectives might choose follow-up times of six months or longer.

Engaging patients

Consideration 11: How should triaists engage patients during design and conduct of trials?

To empower patients and conduct more patient centric research, acupuncture trials should consider informing, educating, discussing, engaging, and partnering with patient organisations and with individual patients.

Development of the guidance

We established a steering committee including frontline acupuncture clinicians, acupuncture trialists, clinical trial methodologists, and a statistician (XHJ, JPL, LXL, CMW, LT, YQZ, RMJ, and GHG) with extensive experience in acupuncture and trial methodology. The steering committee recruited an international expert panel of 27 experts (from Asia, Europe, America, and Australia), including patients, frontline acupuncture clinicians, acupuncture and surgical trialists, clinical trial methodologists, and statisticians. Acupuncture trialists and methodologists were appointed to the panel on the basis of their h index and assessment of their research expertise, and frontline clinicians on the basis of their clinical experience and reputation.

Using the results of our systematic survey,¹⁵ the steering committee drafted the first version of the guidance. The purpose of the survey was to identify the most prevalent trial design and execution issues and factors associated with the magnitude of acupuncture treatment effects.¹⁵ In an iterative process, the expert panel provided feedback on the structure and content of the acupuncture trials guidance draft and on piloting of the guidance and user feedback (web appendix 1).

Use of the guidance

In the early stages of designing a randomised controlled trial of acupuncture, researchers should specify their objective. Studies may fall along a continuum from explanatory or mechanistic to pragmatic or practical (box 2).^{4 5} Trials with primary explanatory or mechanistic objectives will evaluate the specific effect of acupuncture whereas those with mainly pragmatic or practical objectives will aim to inform clinical or policy decision making. Trialists can refer to the pragmatic-explanatory continuum indicator summary (PRECIS)¹⁸ or PRECIS-2¹⁹ to assess where their trials are positioned.

Box 2. Explanatory and mechanistic frameworks for trials.

Explanatory-pragmatic

Schwartz and Lellouch’s explanatory-pragmatic framework classified trials as either aiming to achieve the maximum effects of interventions (eg, measure tumour size or cancer related mortality as outcomes) under ideal conditions or to inform decisions in clinical or health system contexts by resembling current practice settings.⁴ Witt and colleagues have provided guidance to apply this framework specific to acupuncture trials.^{13 16}

Mechanistic-practical

Karanicolas and colleagues provide an alternative framework emphasising the purpose and suggest that trial purpose depends on the decision makers’ perspectives (eg, patients and clinicians versus policy makers).⁵ They proposed a revised mechanistic-practical framework to design and interpret randomised trials. Mechanistic trials have proof-of-concept objectives and often include surrogate outcomes such as biological measures (eg, forced exhaled volume in 1 second (FEV₁) in patients with asthma¹⁷); practical trials provide comprehensive information that bears directly on healthcare decisions.

Once trialists situate their study in the explanatory-pragmatic continuum, they can use the PICOT framework²⁰ to determine critical design elements. For instance, planning a randomised trial to study the effect of acupuncture on chronic low back pain requires selection of participants (P), intervention (I) (eg, acupoints, stimulation methods, depth of the insertion), comparator (C) (eg, sham acupuncture or medication), outcomes (O), and duration of follow-up (T). If their primary objective is to evaluate the specific effect of acupuncture, trialists will likely choose sham as the comparator. If the primary objective is to explore the effectiveness and safety of acupuncture therapies and inform clinical or policy decision making, commonly used alternatives (such as non-steroidal anti-inflammatory drugs) might be suitable comparators.

In defining their objectives and consequent methods in relation to the explanatory-pragmatic continuum (box 3), trialists should consider a systematic review to establish the rationale for their research question (box 1, consideration 1).

Box 3. Acupuncture randomised trial with both pragmatic and explanatory features²¹ .

• Population—Patients with chronic low back pain

• Intervention — Clinicians gave true acupuncture treatment twice a week for five weeks (10 sessions). If patients experienced a 10-50% reduction in pain (Von Korff chronic pain grade scale), they received five additional sessions

• Comparators—Sham acupuncture (same frequency and session as true acupuncture) or German guideline based multimodal treatment programme (10 sessions of physician or physiotherapist administered physiotherapy or exercise)

• Outcomes and follow-up—The primary outcome was pain or functional status of low back pain six months after randomisation. Secondary outcomes were quality of life and patient global assessment of therapy effectiveness, medication use, and adverse events at each session after six months. Trialists defined pain status as 33% improvement on three pain related items on the Von Korff chronic pain grade scale and functional status as 12% improvement on the Hanover functional ability questionnaire

Interpretation

• Use of guideline based conventional therapy and patient important outcomes with long term follow-up indicates a pragmatic design. Having sham acupuncture as the comparator shows a more explanatory feature

Although the guidance we offer to determine the objective of pragmatic versus explanatory trials generally applies, there will be exceptions. For instance, although pragmatic objectives can generally be achieved by enrolling heterogeneous populations, if the desired focus is a narrower population, it does not preclude a primarily pragmatic objective. Once the objectives are defined trialists should consider the 11 questions below in designing their trial.

Choosing the research question

Consideration 1: Is the question important?

Published acupuncture randomised trials rarely cite previous systematic reviews in describing the rationale for conducting a trial, suggesting a high risk of wasteful use of research resources²² (web appendix 2, 2.1.1). Deciding on a trial’s objective requires an understanding of the available body of evidence on related questions, which requires the availability or conduct of systematic reviews.²² Are there already sufficient trials exploring the specific effect of acupuncture therapies? Is there already moderate or high certainty evidence establishing treatment effects in particular settings to inform decision making? If existing systematic reviews show little or no evidence on acupuncture therapies for a target condition, investigators can conclude that an optimal randomised trial will provide proof-of-concept information. If sufficient evidence establishes a therapeutic effect on a surrogate outcome or patient important outcome under narrowly defined circumstances, moving to a pragmatic trial is likely to be appropriate (box 1, consideration 1).

Enrolling patients

Consideration 2: Who should the participants include?

Trialists need to determine the population that might be the most responsive to and are interested in the intervention and what are the necessary enrolment restrictions (box 1, consideration 2). Restricting participants to those who are most responsive reflects a proof-of-concept approach. Such a trial will aim to establish the effect of acupuncture therapies in people most likely to benefit, leaving open the question of its value in routine clinical settings serving a diverse population.

Trialists with a primarily explanatory or mechanistic objective might include people who are most responsive, those at highest risk of adverse outcomes, or the most highly compliant and may exclude those with comorbidities that might affect the outcomes or likely to drop out because of severe comorbidity.

Trials with a primarily pragmatic aim would produce widely generalisable results that optimally inform clinical practice. The trial should include a broad population varying in age, severity, comorbidity, and exposure to other interventions and, depending on their target audience, either highly compliant or more general populations (box 4 ).

Box 4. Three acupuncture trials with inclusion criteria that reflect explanatory and pragmatic goals.

Example 1: Patients with knee osteoarthritis and chronic knee pain²³

• Inclusion criteria (explanatory or mechanistic)—Patients were eligible if they fulfilled all five criteria: age ≥50 years; history of knee pain for >3 months; reported knee pain on most days of the past month; average knee pain severity  ≥ 4 on an 11 point scale over the past month; and) reported morning knee stiffness of <30 minutes

• Exclusion criteria—The study excluded patients if they fulfilled any of the following 13 criteria: history of any systemic arthritic condition; knee arthroplasty on most painful knee; on waiting list for any knee surgery; any knee surgery in past six months; presence of any other condition affecting lower limb function (eg, trauma, malignancy, neurological condition); any knee injection in past six months (eg, cortisone, hyaluronic acid);) current use of oral or injectable anticoagulant medication; use of acupuncture in past 12 months; any bleeding disorder; allergy to light; referral to a pain clinic or use of morphine or pethidine within the past six months; any other medical condition precluding participation in the trial (eg, kidney or liver disease, deep vein thrombosis); knee pain subject to compensation claim; unable to give written informed consent

Example 2: Patients with chronic knee pain and knee osteoarthritis²⁴

• Inclusion criteria (pragmatic or practical)—Patients were eligible if they fulfilled all four criteria: diagnosis of knee osteoarthritis by American College of Rheumatology criteria; age ≥45 years; pain in one or both knees for three months or longer; confirmed knee osteoarthritis with radiological evidence (at least grade 1 according to the Ahlbäck classification)

Example 3: Patients with persistent non-specific low back pain using traditional acupuncture or usual care²⁵

• Inclusion criteria (pragmatic or practical)—Any patients aged 18-65 with non-specific low back pain for 4-52 weeks. Patients with spinal disease were excluded

Consideration 3: How can trialists address possible differences in effect across patient groups?

A limitation of pragmatic trials is that they include very heterogeneous and rarified populations and run the risk of producing an overall estimate of effect that is not applicable to all enrolled patients. In the extreme, they report an intermediate effect when some patients experience large benefit and some none at all. Trialists can overcome this limitation by prespecifying subgroup analyses with statistical tests of interaction to investigate hypothesised effect modification (box 1, consideration 3).

Participants’ expectation serves as an example of subgroup effect in the context of acupuncture trials. Trialists hypothesised that acupuncture would show larger effects in patients who expect a benefit from acupuncture than in those who do not.²⁶ Across therapeutic areas and health outcomes (eg, pain relief, symptom improvement, return to work), results have consistently confirmed this hypothesis: participants who have positive expectations of interventions consistently show a larger effect of acupuncture than the more pessimistic.^{27 28 29}

The repeated finding that patient expectations influence magnitude of effect suggests trialists should assess participants’ expectations at the time of randomisation. Trialists can use validated instruments such as the acupuncture expectancy scale,³⁰ expectation for treatment scale,³¹ and the expectations for complementary and alternative medicine treatments questionnaire.³²

Published acupuncture trials have rarely explored effect modification (web appendix 2, 2.1.2). Trialists can consider exploring effect modifiers when designing acupuncture trials with sufficiently large sample sizes and, if they identify possible subgroup effects, assess their credibility criteria using the rigorously constructed and user tested instrument to assess the credibility of effect modification analyses (ICEMAN).³³ Choosing a small number of hypotheses, ideally identified as promising (ie, with moderate credibility on ICEMAN) in previous trials, represents an optimal approach (box 5).³³

Box 5. Patients’ expectation as an effect modifier and credibility of effect modification assessment.

Study of acupuncture versus sham acupuncture for patients with chronic low back pain.³⁴ The study used the expectations for relief scale to measure patients’ expectations at baseline and conducted a subgroup analysis in patients with high versus low expectations. Expectations were a significant predictor of only true acupuncture response (P=0.002), such that those with greater expectations had more substantial pain relief.

We used ICEMAN³³ to assess the credibility of potentially relevant effect modification:

Was the direction of the effect modification correctly hypothesised a priori? (definitely no, probably no or unclear, probably yes, or definitely yes)

Definitely yes: subgroups (patients’ expectation) were prespecified in published protocol, and publications included an explicit statement that they had correctly predicted the direction of effect modification

Was the effect modification supported by prior evidence? (inconsistent with prior evidence, little or no support or unclear, some support, strong support)

Strong support: the main result was consistent with prior solid evidence directly applicable to the clinical scenario

Does a test for interaction suggest that chance is an unlikely explanation of the apparent effect modification (consider irrespective of the number of effect modifiers)? (chance a very likely explanation, chance a likely explanation or unclear, chance may not explain, chance an unlikely explanation)

Chance may not explain: P=0.002 (interaction P values ≤0.01 and >0.005)

Did the authors test only a small number of effect modifiers or consider the number in their statistical analysis? (definitely no, probably no or unclear, probably yes, definitely yes)

Probably yes: three effect modifiers were prespecificed but applied to other outcomes than specified in the protocol

If the effect modifier is a continuous variable, were arbitrary cut-points avoided? (definitely no, probably no or unclear, probably yes, definitely yes)

Definitely yes: The analysis was based on a full continuum—eg, assuming linear or logarithmic relationships. A repeated-measures analysis of variance (ANOVA) was used to characterise the relationship between psychiatric group and analgesia

Optional: Are there any additional consideration that may increase or decrease credibility? (yes, probably decrease credibility; yes, probably increase credibility)

Yes, probably increase credibility: the effect modification persisted after adjustment for other potential effect modifiers (interventions and psychopathology)

How would you rate the overall credibility of the proposed effect modification (very low—very likely no effect modification, use overall effect for each subgroup; low—likely no effect modification, use overall effect for each subgroup but note remaining uncertainly; moderate—likely effect modification, use separate effects for each subgroup but note remaining uncertainly; high—very likely effect modification, use separate effects for each subgroup)?

High credibility very likely: none of the response options definitely or probably reduced credibility

Interpretation

We judged the credibility of the potential effect modification as high considering a small number of clearly a priori hypotheses with a specified direction; support from previous evidence; a low P value in the test of interaction; and adjustment for other potential subgroup effects

Selecting the intervention

Consideration 4: Who should perform the intervention?

The effect of acupuncture may differ depending on various characteristics of the acupuncturists (box 1, consideration 4). These include education and training (whether they had systematic training such as undergraduate, graduate, diploma training, or short term training), length of practice, and how well the acupuncturists perform a particular complex technique (eg, fire needles and intradermal needle).

Differential expertise can create a systematic difference^{35 36 37} across interventions and between trials.³⁸ A bibliometric study investigated 7085 Chinese randomised trials of acupuncture and found that over 20% of the trials compared one acupuncture treatment with another or with an alternative treatment that required expertise (box 6).³

Box 6. Who should perform the intervention?

Consider a randomised controlled trial in which 80 participants with post-stroke pseudobulbar palsy were randomised to an acupuncture intervention—a quick needle insertion at the Aqiang point—or a comparator of routine acupuncture in combination with western medicine.³⁹ Clinicians implementing the intervention needed to perform a quick needle insertion with strong stimulation at the Aqiang point (located in the anterior midline of the neck, between the thyroid cartilage and cricoid cartilage) that may induce the patient to cough violently. In the comparator group, clinicians needed only to stimulate CV 23 (Lian quan) point with the standard acupuncture manipulation. If practitioners in the trial had a greater skill or more experience in performing the special acupuncture techniques, the results might not apply to those with less expertise in the Aqiang point approach.

This trial failed to find a difference in swallow function between the two groups, but it did not establish the expertise of those administering the more challenging intervention. The effects of expert administration therefore remain uncertain, although the results may be representative of suboptimal expertise in the community. The trial’s small sample size might also explain the failure to detect a difference.

In a patient level analysis with 9990 patients treated by 2781 physicians in Germany, neither training duration nor experience affected the extent of the acupuncture effect.⁴⁰ Unfortunately, exploration of the effect of expertise is limited because acupuncture trials often fail to report this information (web appendix 2, 2.1.3 and 2.2.1).

For studies investigating pain as an outcome, greater experience and training of practitioners (systematic acupuncture or complementary medicine education versus short term training) is associated with greater treatment effect.¹⁵

Whether a trial has explanatory or pragmatic objectives affects trialists’ decisions about the extent of expertise. A trial that primarily aims to show whether an acupuncture treatment can work under ideal conditions will choose the most expert practitioners. On the other hand, trialists primarily aiming to establish the effect of treatment in everyday practice will select clinicians with typical levels of expertise (box 7).

Box 7. Examples of acupuncture trials using highly skilled and variously skilled practitioners.

Auricular acupuncture versus sham auricular acupuncture for musculoskeletal disorders related chronic back pain⁴¹

In this explanatory mechanistic trial all participating acupuncturists had at least 10 years of experience. Having highly skilled acupuncturists perform the intervention may enhance treatment effects but raises questions for a patient consulting a practitioner who has just completed acupuncture training

Acupuncture plus advice and exercise versus advice and exercise for patients with knee osteoarthritis⁴²

In this pragmatic trial, acupuncture was delivered by 67 physiotherapists trained by the Acupuncture Association of Chartered Physiotherapists (35 hours of training), with three to over 10 years of practice. A positive result of such a trial would suggest a widely applicable intervention; a negative result would leave the question of the effect with more expert practitioners uncertain

One way to deal with the issue of expertise is by using a trial that randomises by expertise. For example, investigators interested in the relative effect of Japanese meridian therapy versus Korean constitutional acupuncture might randomise patients to clinicians who specialised in meridian therapy or those who specialised in Korean constitutional acupuncture. Such expertise based designs may enhance applicability, encourage practitioners’ participation in trials,⁴³ result in less crossover,³⁵ and provide patients the best quality care. To date,⁴⁴ no acupuncture trial has used an expertise based design, but it is an attractive option for future trials of alternative acupuncture approaches that take a practical perspective.^{45 46}

In any case, trialists should report the expertise of the acupuncturists who administered intervention and comparator treatments. More transparent reporting will enable further investigation into how expertise affects the effectiveness of acupuncture treatment.

Consideration 5: What specific acupuncture technical features should be considered if aiming to design trials for the maximum treatment effect?

The effect of acupuncture may differ depending on the choice of techniques, including underlying acupuncture theory (eg, western medical acupuncture, traditional Chinese medicine acupuncture, dry needling) and type of acupuncture (manual versus electroacupuncture).

The multivariate analysis in our systematic survey¹⁵ found that penetrating types of acupuncture (manual and electroacupuncture) showed a larger treatment effect than non-penetrating (transcutaneous electrical acupoint stimulation, laser, acupressure) acupuncture. In addition, a higher frequency of acupuncture treatment sessions was associated with larger effects than a lower frequency. Trialists can establish an expert committee to define the optimal strategy for acupuncture related details (eg, acupoints selection, stimulation technique) (box 1, consideration 5).

Choosing the comparator

Consideration 6: Are trialists interested in the specific effect or overall effect of acupuncture?

Blinding refers to those involved in randomised trials being unaware of treatment assignment after randomisation. Blinding is relevant to participants, clinicians, data collectors, outcome assessors, and data analysts. Blinding of clinicians and patients avoids non-specific effects and bias arising from co-intervention; blinding of others minimises bias in the assessment of outcome. Blinding clinicians and patients raises both theoretical and practical challenges, but to minimise bias trialists should blind data collectors, outcome assessors, and data analysts (box 1, consideration 6, web appendix 2, 2.2.2(1)).

Although legitimate scepticism exists about tests of blinding, in seven of 10 studies we identified in which authors tested for blinding (one to four times during the trial), study participants were unable to distinguish active and sham treatment, providing potentially useful reassurance of the success of blinding.¹⁵ Thus, the choice of whether to conduct tests for blinding remains, in our view, a matter of investigator judgment in which both options have merit.

In drug trials, which are closer to a explanatory or mechanistic design, trialists often use a placebo to achieve blinding. Acupuncture trials, however, face multiple challenges when using placebo or sham acupuncture for blinding.¹⁵ It is nearly impossible to blind clinicians who deliver acupuncture, although one device exists that allows practitioner blinding but limits the acupuncture to a rather superficial version (maximum 5 mm insertion).⁴⁵ Participants who have had previous acupuncture experience may also be hard to blind.^{47 48}

Whether to attempt blinding in acupuncture trials depends on the hypotheses and objectives of a trial. Here, we will refer to a treatment’s biological effects as specific effects and placebo effects as acupuncture’s non-specific effects. Trialists examining explanatory questions should include sham acupuncture control with adequately blinded participants to differentiate acupuncture’s specific and non-specific effects.⁴⁵

The results of such explanatory studies require careful interpretation, however. The clinical and basic science literature support the possibility of specific effects generated by sham acupuncture^{49 50}— that is, a failure to show a difference between real and sham acupuncture may be because the sham has effects closely related to that of the intervention. Therefore, when using sham control, trialists must consider the possibility of a specific effect generated by the sham and thus underestimating the effects of acupuncture in clinical practice compared with no intervention or other interventions such as drugs.

Previous studies of shams have focused on pain and chronic pain and conducted univariable analyses (web appendix 2, 2.2.2 (2)). We used multivariable analyses in our systematic survey, adjusting for other potential factors such as treatment frequency, flexibility of the acupuncture regimen, and sample size. Like a previous systematic review,⁵¹ we found the type of sham did not influence acupuncture’s effect.¹⁵ This is perhaps unsurprising considering the finding of meta-epidemiological studies in areas beyond acupuncture: results of such studies have proved inconsistent, with one recent thorough review showing no systematic effect of blinding.⁵²

Considering all the evidence, the effect of the type of sham remains uncertain and might vary with the type of medical condition. Over 90% of acupuncture trials focus on pain, quality of life, function, and other symptoms. These are all subjective outcomes in which blinding may be particularly important.¹⁵ Trialists should therefore carefully consider the desirability of a sham in randomised trials investigating efficacy and, if desirable, the nature of the sham in relation to their trial’s specific objectives. Trialists should also be aware that the use of non-penetrating needle shams mandates the use of the same device in the real acupuncture group. These devices may impede real acupuncture treatment effects (box 1, consideration 6; box 8).⁵⁵

Box 8. Choosing the comparator: examples of acupuncture trials investigating specific and overall (specific and non-specific) effects.

Specific and overall effects

Trial 1—Patients with knee osteoarthritis were assigned to true acupuncture plus education and attentional therapy, sham acupuncture plus education and attentional therapy, or education and attentional therapy alone. Sham acupuncture (combination of non-insertion and needle-insertion sham acupuncture) was received with two needles at sham points and tapping at nine real points.⁵³

Specific effects

Trial 2 —Patients with knee osteoarthritis were assigned to drug treatment alone, drug treatment plus true acupuncture, or drug treatment plus sham acupuncture.²⁴ Sham acupuncture was administered with the same duration and frequency and by the same specialist who performed the true acupuncture. Retractable needles were placed into small adhesive cylinders to support the needles but did not perforate the skin. The acupuncturist placed the needles at the same points as the non-sham group and used the same pairs of electrodes to simulate the electrical connection.

Trial 3 —Patients with knee osteoarthritis were randomised to intensive acupuncture or sham acupuncture.⁵⁴ Sham acupuncture (combined non-insertion and needle insertion sham acupuncture) was achieved by choosing acupoints that were away from the conventional acupoints or meridians, superficially penetrated (2-3 mm in depth), and without manipulation of the needles after insertion for “de qi” responses.

Overall (specific and non-specific) effect

Trial 4—Patients with persistent non-specific low back pain received either a short course of traditional acupuncture or usual care only. Patients in the usual care group received NHS treatment according to their general practitioner’s assessment of need.²⁵

Fine tuning the flexibility of intervention and comparator

Consideration 7: To what extent should the trialist choose a flexible intervention and comparator?

Acupuncture therapies are multifaceted interventions that incorporate patient-practitioner interactions, treatment theories, tools to deliver the stimulations, manipulation techniques, and points selection. The trial objectives should determine the flexibility of the intervention or control.

Suppose trialists aim to assess the maximum treatment effect of a particular acupuncture technique or regimen (closer to explanatory approach). In that case, they should specify that practitioners use a highly standardised approach to administer both the intervention and the control. In-person or video training for all participating practitioners regarding types of needles, the number of needles, needling details (eg, direction, depth of insertions, needle manipulation), needle retention time, and responses (eg, de qi or muscle twitches) may facilitate standardisation and may be particularly important in multicentre studies. Trialists can monitor standardisation in participating centres by visiting sites regularly or reviewing the recorded video reports. In addition, standardisation should include the number, frequency, and duration of treatment sessions (box 1, consideration 7; box 9, example 1).

Box 9. Examples of fine tuning the flexibility of intervention and comparator in acupuncture trials.

Restrictive and highly standardised intervention and comparison

Example 1: Postmenopausal women with early stage breast cancer taking an aromatase inhibitor randomised to acupuncture, sham acupuncture, or waiting list control ⁵⁶

• Acupuncture intervention—A group of acupuncturists developed a standard operating protocol based on previous acupuncture studies for aromatase inhibitor related arthralgias adhering to the STRICTA recommendation. Trialists described the frequency of treatment, acupoint prescription, whether “de-qi” occurred, depth of needle insertion, and acupoint manipulation in detail

• Comparison (sham acupuncture)—The sham acupuncture consisted of a standardised prescription of minimally invasive, shallow needle insertion using thin and short needles at the regimen of non-acupuncture points. The sham acupuncture protocol also included joint specific treatments and an auricular sham based on application of adhesives to non-acupuncture points on the ear

Varied intervention and comparator

Example 2: A pragmatic acupuncture randomised controlled for the management of chronic low back in older adults ⁵⁷

• Acupuncture intervention—An acupuncture advisory panel produced a consensus acupuncture intervention protocol. Acupuncture point sites must include both local and distal points with 6–20 insertions per session, but selection of specific acupoints was at the practitioner’s discretion. The needle retention time and length f the session ranged between 0 and 40 minutes and 45-60 minutes, respectively

Example 3: Patients visiting emergency departments with an ankle sprain, migraine, or lower back pain were randomised to acupuncture alone or combined with pharmacotherapy ⁵⁸

Comparison (pharmacotherapy)—Pharmacotherapy can be any first or second line analgesia drug.

Comments

The example of the highly standardised intervention and comparator reflects an explanatory goal for the study. The studies with flexible intervention and comparator reflect a pragmatic goal, and inferences from these studies might be highly applicable to clinical practice.

Trialists with primarily pragmatic objectives will take a different approach. They will probably recruit a wide variety of clinicians representing the variability in technique in the community and instruct practitioners to use their usual treatment approaches (box 9, examples 2 and 3).

The multivariable analysis in our systematic survey found that single centred randomised trials of acupuncture have larger treatment effects than multicentre trials.¹⁵ The greater standardisation in single centres might be responsible for the larger effects.

Reporting all technical details of the intervention and controls transparently according to the STRICTA recommendations would enhance the interpretation and applicability of acupuncture trials (web appendix 2, 2.1.4).^{10 59}

Consideration 8: How should trialists deal with issues of adherence?

Although it is often inevitable that some trial patients do not attend all treatments, how this affects inferences from the evidence may vary depending on your perspective. When patients face the decision to use acupuncture therapies, they will be interested in the effectiveness and safety of the treatment. Thus, substantial non-adherence limits the applicability of the trial from the patients’ point of view.

From a public health perspective, however, decision makers might be more interested in the treatment effect in a population with the suboptimal adherence that will actually occur in typical clinical practice. The non-adherence, if indeed typical of what happens in population implementation, will actually strengthen applicability from the public health point of view.

Thus, when designing a trial that primarily takes the individual patient’s perspective, trialists should consider implementing strategies to achieve optimal adherence. One strategy is to engage patients to help design the trial. Patients can identify and improve onerous trial design, therefore increasing participation.⁶⁰ Another strategy is to include a run-in period before randomisation and only randomise patients who are highly compliant in the run-in.

Including a run-in period is highly feasible in drug trials when patients may be offered a placebo during the run-in or when the trial is focused on morbid or fatal events and short term exposure will have no influence on the outcome. The situation is more complicated in acupuncture trials focusing on symptoms in which initial exposure to treatment during a run-in may complicate inferences about what occurs after randomisation. This is probably why acupuncture trials rarely use run-in periods to enhance adherence.

As an alternative to run-ins, acupuncture trials can assign tasks to patients before randomisation to assess their adherence (web appendix 2, 2.2.3). Trialists can select patients based on completeness and timeliness of fulfilling these tasks. Trials could also provide additional support and adequate incentives to improve adherence ( box 1, consideration 8; box 10).^{45 62}

Box 10. Example of acupuncture trial that implemented strategies to improve patient adherence.

Acupuncture versus waiting list control for menopausal patients with vasomotor symptoms⁶¹

Patients in the acupuncture group could have up to 20 acupuncture treatments by one of four acupuncturists over six months. The acupuncturist and patients decided treatment dates together, respecting the treatment frequency to increase compliance. Patients were paid $35 (£26; €31) for their baseline visit and $25 to complete each follow-up visit. Those who completed all six visits were paid an additional $50.

If trialists take a public health perspective when considering patients’ adherence, there is no need to implement strategies to increase adherence. The minimal enforcement approach closely resembles standard practice and provides a reasonable estimation of how treatment effects might be in the setting in which trials are conducted.

Selecting outcome measure and follow-up

Consideration 9: What type of outcomes should trialists choose?

Trialists with mainly mechanistic objectives can usually focus on surrogate outcomes (eg, blood pressure, heart rate) to estimate the effect on patient important outcomes. If trialists’ have a pragmatic objective primarily of guiding healthcare decision making, they should usually select patient important outcomes.⁶³ Trialists can consider taking advantage of minimal sets of agreed and standardised collection of outcomes in their clinical areas, known as core outcome set, on which investigators have agreed (https://www.comet-initiative.org/, web appendix 2, 2.2.4 (1)).^{64 65 66} Including all outcomes in a core outcome set will ensure maximum applicability of trial results to patients, the public, healthcare professionals, and other decision makers. In addition, trialists can consider outcomes recommended by research organisations and medical societies’ guidelines on conducting trials for specific diseases (web appendix 2, 2.2.4(2)). ^{64 65 67}

As well as using symptoms and functional outcomes trialists should increase the use of quality-of-life outcomes (web appendix 2, 2.1.5). Including major morbid events (eg, serious cardiovascular events, hospital admission, infections) might greatly enhance the importance and applicability of acupuncture trials when relevant but would likely require far larger sample sizes than is typical of current trials (web appendix 2, 2.2.4 (3)). Trials should also report adverse events outcomes (box 1, consideration 9; box 11).

Box 11. Selecting outcome measure and follow-up in acupuncture randomised trials.

Randomised controlled trials on acupuncture versus sham acupuncture or no acupuncture for exercise induced asthma in children¹⁷

This trial measured forced expiratory flow in first second (FEV₁), forced vital capacity (FVC), and peak expiratory flow rate (PEFR) at 0, 5, 10, 15, 20 minutes after treatment. As the trial had an explanatory function trialists used biological or surrogate outcomes

Randomised trial comparing traditional Chinese medical acupuncture, therapeutic massage, and self-care education for chronic low back pain⁶⁸

This trial used symptoms and dysfunction as the primary outcomes and disability, utilisation (provider visits, radiological procedures, operations, and hospital admissions), and cost as secondary outcomes. On a 0 to 10 scale, patients rated how “bothersome” their symptoms (back pain, leg pain, and numbness or tingling) were. Patients reported their dysfunction using the modified Roland disability scale. Blinded interviewers measured all outcomes at 4, 10, and 52 weeks after randomisation.

The aim of the study was to inform clinical or policy decision making. The follow-up selection informs the long term impact of acupuncture intervention. It also sheds light on the possible need for “booster treatments” or “maintenance treatment,” therefore helping to achieve practical objectives.

Consideration 10: How to decide the duration of follow-up

Acupuncture effects often emerge immediately or shortly after treatment sessions, although specific timing depends on the disease or conditions acupuncture treats and the outcome measures. For example, acupuncture shows a 20% reduction in chronic low back pain after 4-5 sessions,⁶⁹ and it might take up to three months to show effects on post-stroke motor functions.⁷⁰ When investigators with explanatory objectives primarily, short follow-up times are likely to be optimal. Such design choices do not answer the question of interest to patients—can I expect a longer term benefit from acupuncture? Thus, acupuncture trials with mainly pragmatic objectives are likely to choose the longer follow-up times most important for clinical or policy decision making.⁷¹

Acupuncture trials published in the past five years mainly used short to medium term follow-up times (web appendix 2, 2.1.6). Trialists should take into account the course of disease or condition for optimal follow-up times. In many instances, when designing pragmatic or practical trials, trialists need to consider measuring outcomes after six months or longer to optimally inform clinical decision making (box 1, consideration 10; box 11).

Engaging patients

Consideration 11: How should trials engage patients when designing and conducting RCTs?

Patients are among the most critical stakeholders in clinical care and health research. Involving patients when designing trials can enhance the research question’s relevance and importance, which helps to improve adherence (box 1, consideration 11).^{72 73 74} Furthermore, having patients involved as members of the trial team can facilitate the understanding of research findings and therefore facilitate dissemination and application.^{73 74}

As complex interventions, acupuncture therapies focus on patient engagement, though this is rarely included in randomised trials. Previous guidance on practical acupuncture trials also recommended engagement of relevant stakeholders (eg, practitioners, clinicians, patients, payers, researchers).¹³ Our systematic survey¹⁵ showed that over half of the acupuncture trials did not mention even the minimal extent of patient involvement—signing the informed consent form.

To empower patients and conduct more patient centric research, acupuncture trials should consider informing, educating, discussing, engaging, and partnering with patient engagement organisations such as Clinical Trials Ontario, which helps trial teams engage with patients.^{75 76}

Conclusion

We have produced guidance to consider before and during the design of acupuncture trials to deal with the unique challenges and the most prevalent concerns in current acupuncture trials (box 1). With the rapid growth of published acupuncture trials and upcoming funding opportunities worldwide in integrative medicine, failure to enhance the rigour of acupuncture trials will result in additional research waste.⁷⁷ This guidance highlights the importance of granting agencies and researchers carefully considering the need for research in the trial planning and funding processes. Our guidance will help both researchers and granting agencies consider the critical decisions linking trial design and its inferences and application to produce trials that facilitate healthcare decision making.

Web extra.

Extra material supplied by authors

Web appendix 1: Details of the methods for development of this guidance

Web appendix 2: Data and supporting literature for the guidance