Factors Associated with the Magnitude Of acUpuncture treatment effectS (FAMOUS): a meta-epidemiological study of acupuncture randomised controlled trials

Abstract

Objective

To identify factors and assess to what extent they impact the magnitude of the treatment effect of acupuncture therapies across therapeutic areas.

Data source

Medline, Embase, Cochrane Central Register of Controlled Trials, China National Knowledge Infrastructure, Wanfang Database, VIP Database, and China Biology Medicine disc, between 2015 and 2019.

Study selection

The inclusion criteria were trials with a total number of randomised patients larger than 100, at least one patient-important outcome and one of two sets of comparisons.

Data analysis

The potential independent variables were identified by reviewing relevant literature and consulting with experts. We conducted meta-regression analyses with standardised mean difference (SMD) as effect estimate for the dependent variable. The analyses included univariable meta-regression and multivariable meta-regression using a three-level robust mixed model.

Results

1304 effect estimates from 584 acupuncture randomised controlled trials (RCTs) were analysed. The multivariable analyses contained 15 independent variables. In the multivariable analysis, the following produced larger treatment effects of large magnitude (>0.4): quality of life (difference of adjusted SMDs 0.51, 95% CI 0.24 to 0.77), or pain (0.48, 95% CI 0.27 to 0.69), or function (0.41, 95% CI 0.21 to 0.61) vs major events. The following produced larger treatment effects of moderate magnitude (0.2–0.4): single-centred vs multicentred RCTs (0.38, 95% CI 0.10 to 0.66); penetration acupuncture vs non-penetration types of acupuncture (0.34, 95% CI 0.15 to 0.53); non-pain symptoms vs major events (0.32, 95% CI 0.12 to 0.52). The following produced larger treatment effects of small magnitude (<0.2): high vs low frequency treatment sessions (0.19, 95% CI 0.03 to 0.35); pain vs non-pain symptoms (0.16, 95% CI 0.04 to 0.27); unreported vs reported funding (0.12, 95% CI 0 to 0.25).

Conclusion

Patients, clinicians and policy-makers should consider penetrating over non-penetrating acupuncture and more frequent treatment sessions when feasible and acceptable. When designing future acupuncture RCTs, trialists should consider factors that impact acupuncture treatment effects.

Strengths and limitations of this study.

• This study included a comprehensive search, independent and duplicated screening and data extraction, rigorous data analysis and interpretation by multidisciplinary researchers.

• This study focused on patient-important outcomes and chose the independent variables considering literature, clinicians, and patients’ perspectives.

• This study constructed a robust three-level mixed model multivariable analysis to adjust for multiple variables to reduce the potential bias and used Cramer’s V and the weighting approach of robust regression to deal with the collinearity and substantial amount of outlier and influential values.

• The multivariable analyses excluded important independent variables such as practitioners’ experience due to poor reporting.

• Including extremely imbalanced variables (eg, country, trial registered) limits the generalisability of the study results.

Introduction

Acupuncture is one of the most used and researched interventions under the integrative medicine umbrella.^1–4 By 2014, the total number of acupuncture randomised controlled trial (RCT) has increased dramatically and accounted for 20.3% of all acupuncture studies.⁵ Since 2010, over 1000 acupuncture RCTs were published annually, with the total number exceeding 10 000 to date.⁶

Acupuncture’s treatment effect varies largely across trials.^{7 8} Efforts to determine factors associated with effect size in acupuncture RCTs have reported conflicting findings. For example, Vickers et al reported that, in studies of chronic pain, penetrating sham vs non-penetrating and non-needle sham control showed larger treatment effects.⁹ However, other studies reported that the effect of acupuncture in pain studies was unrelated to the type of sham acupuncture.^{10 11} Some found the total number of acupuncture treatments,^11–13 frequency of treatment sessions¹⁴ and acupuncture type (manual acupuncture vs electroacupuncture)¹⁴ were significant factors of the treatment effect whereas others did not.^{9 15} The reason may be related to little data variation,¹⁵ small number of included studies,^{12 14} and variation of the clinical areas and settings investigated.^{10 11 16}

To improve acupuncture RCTs’ design, and optimise acupuncture interventions’ clinical effectiveness, we conducted this meta-epidemiological study, including acupuncture RCTs published between 2015 and 2019 across therapeutic areas and outcomes, and explored the factors of acupuncture’s treatment effects. We aim to (1) identify factors regarding patient, acupuncture, comparator, outcome and methodology that impact the magnitude of the treatment effect of acupuncture therapies and (2) explore to what extent the factors impact the treatment effect across therapeutic areas.

Methods

Definitions

We define acupuncture therapies based on the WHO definition: Acupuncture literally means to puncture with a needle. However, there may also involve the application of other kinds of stimulation to certain points.¹⁷ The study addressed commonly used acupuncture modalities, including manual acupuncture, electroacupuncture (electro-acupuncture), laser acupuncture, transcutaneous electrical acupoint stimulation (TEAS), acupressure, traditional body needling, ear (auricular) acupuncture and scalp acupuncture.

We define sham acupuncture as an intervention with a minimal treatment effect designed to blind patients as they received real acupuncture.¹⁸ Often sham acupuncture includes ‘placebo’ needles with a blunt collapsing tip that does not penetrate the skin, real acupuncture but inserted at non-acupuncture points or true acupuncture points but not targeting the intended disease. Non-needle sham can be detuned lasers, deactivated transcutaneous electric nerve stimulation devices or less pressure on acupuncture points.

We define a patient-important outcome as one in which the patient would be interested, despite the risk, burden or cost, were it the only outcome to improve with an intervention.¹⁹

To differentiate from individual outcomes (eg, dysphagia), we define a construct as a category of patient-important outcomes (eg, functional status).

We define a therapeutic area as a class of related diseases or conditions based on modified International Classification of Diseases 11th Revision (ICD-11) criteria (eg, Neurology). In this study, the classification of the therapeutic areas targeted diseases or conditions for which patients seek acupuncture treatment. For example, if an acupuncture RCT investigated post-stroke depression, we would classify the RCT into ‘Mental health’ rather than ‘Neurology’.

Literature search

In collaboration with clinical and methodological experts, a medical information specialist developed a search strategy that included PubMed, Embase, the Cochrane Central Register of Controlled Trials, and 4 Chinese databases, including China National Knowledge Infrastructure (CNKI), Wanfang Database, VIP Database for Chinese Technical Periodicals (VIP) and China Biology Medicine disc (CBM). We searched acupuncture RCTs published from 2015 January to 2019 December with no language restrictions. The detailed search strategy is presented in online supplemental eAppendix 1.

Eligibility criteria

Eligible studies fulfilled the following inclusion criteria:

• RCT defined by authors.

• Reported at least one of two sets of comparisons: acupuncture vs no intervention, sham acupuncture or waiting list; or acupuncture plus other interventions vs other interventions with or without sham acupuncture. The other interventions must be conventional medical treatment and identical in both intervention and control groups.

• Reported at least one patient-important outcome.

• Randomised over 100 individuals.

• Appeared in a peer-reviewed journal publication in any language.

We excluded conference abstracts, letters, commentaries, editorials, protocols, non-human trials, cluster RCTs, n-of-1 trials, cost–utility studies, secondary analyses of RCTs, reviews and meta-analyses, RCTs in which control groups received any traditional Chinese medicine related therapies (eg, acupuncture, moxibustion, scraping, cupping, bloodletting, acupoint catgut embedding, massage, Chinese herbal medicine) and studies in which tables and text reported contradictory results on the selected outcomes.

Study selection

We exported Chinese citations to Endnote V.X9.0 and English citations to a web-based software (https://collaboratron.epistelab.com/) for eligibility screening. To conduct, independently and in duplicate, title and abstract and full-text screening, a team of 16 Chinese and 22 English reviewers worked in pairs using standardised forms with detailed instructions. To ensure screening quality, reviewers participated in a calibration exercise prior. If needed, reviewers resolved disagreements through discussion or arbitrated by a third party.

Generation and ranking of the factors that impact treatment effect

We first, through the literature review and consultation with acupuncturists, generated a list of potential factors that might be associated with the magnitude of effect resulting in 13 methodological factors and 26 clinical factors. To ensure our list was comprehensive, and to rank the importance of the factors, we conducted an online survey using Wenjuanxing (www.wjx.cn) among a global panel (n=27) composed of acupuncture trialists, acupuncturists, surgeons, trial methodologists, patients and statisticians. The survey results added seven factors, and we finally included 46 factors (online supplemental eAppendix 2) in the meta-regression analyses.

Data extraction

We classified patient-important outcomes into six constructs box 1.

Box 1. Classification of constructs.

1. Mortality.

2. Major events include morbid events (eg, incidence of myocardial infarction, fracture, stroke), recurrence (eg, the recurrence of facial spasm) or fertilisation-related events (eg, live birth rate).

3. Pain (eg, low back pain).

4. Non-pain symptoms (eg, nausea and vomiting).

5. Quality of life (eg, health-related quality of life).

6. Functional status (eg, dysphagia).

To select outcomes, we first extracted all patient-important outcomes, classified them into the six constructs (box 1), and then, within constructs, classified each outcome into therapeutic areas (we will refer to these as subconstructs). For example, for the non-pain symptoms construct, reviewers classified nausea and vomiting into ‘gastroenterology’. We retained the subconstructs, including 30 studies or more.

Within each construct/subconstruct, for each outcome, we calculated the number of studies reporting the outcome. If one study reported multiple outcomes within the same subconstruct, we extracted the more frequently reported outcome across all studies. When studies reported the same outcome measured by different instruments, we selected the most frequently reported instrument for that outcome across all studies.

If the above process excluded either the primary outcome or the first patient-important outcome in the result, in addition to the outcomes selected through that process, we also included the first patient-important or primary outcome reported in the result section.

For multiple-arm RCTs, we considered only those comparisons that met eligibility criteria. For RCTs with multiple follow-up times, we selected the outcome both at the end of treatment and at the longest follow-up time in which the loss to follow-up rate was 20% or less.

Following a calibration exercise, a team of 10 reviewers, working in pairs, independently extracted data and resolved discrepancies through discussion. If they could not reach a consensus, an arbiter resolved the conflict.

For outcome selection, three pairs of reviewers reviewed all included studies selecting outcomes. After completing the outcome selection and discussing as necessary to come to an agreement, reviewers extracted data on the preselected outcomes.

For each trial, reviewers extracted the number of randomised and analysed participants, data on all factors and recorded the selected outcomes’ effect estimates. Risk of bias was assessed using the Cochrane Collaboration tool.²⁰ For dichotomous outcomes, we collected the number of events and for continuous outcomes, point and associated variabilities, ranges and directions. To extract data from figures in which the data were unavailable in the text or tables, we used GetData Graph Digitizer V.2.25 (by Mark Mitchell) software.

Statistical analysis

Depending on the data distribution, we summarised data using means and SD, or medians and IQRs. For statistical tests, we used a threshold p value of 0.05 to indicate a statistical significance. To combine the outcomes from different measurement scales, we applied the standardised mean difference (SMD). A positive SMD indicated a beneficial effect. The variance of SMD²¹ was given by

$${\displaystyle V_d=\frac{n_1+n_2}{n_1{n}_2}+\frac{SM{D}^2}{2(n_1+n_2)}}$$

where n₁ and n₂ were the sample sizes of the acupuncture therapies group and the control group, respectively. For the dichotomous outcome, by the method of Hasselblad and Hedges,^{21 22} we converted the calculated log OR to SMD using

$${\displaystyle d=LogOddsRatio\times \frac{\sqrt{3}}{\pi }}$$

where π is the mathematical constant (approximately 3.14159). The variance of SMD was obtained by

$${\displaystyle V_d=V_{LogOddsRatio}\times \frac{3}{{\pi }^2}}$$

We initially considered 46 variables (online supplemental eAppendix 2) to investigate factors that might influence the SMD among the RCTs. However, 26 variables were excluded from the multivariate analysis because they were missing in more than 90% of the studies (online supplemental eAppendix 3). To detect possible multicollinearity, we calculated the Cramer’s V statistics^{23 24} (ranges 0–1) between every pair of the variables using a threshold of 0.70. When excessive collinearity existed, we excluded those variables from the regression analysis (online supplemental eAppendix 3).

To account for the heterogeneity between the studies and the dependency of the multiple outcomes within a study, we used a meta-regression in three-level random-effects mixed model^25–27 to simulate the sampling variation for each effect size (level one), variation over outcomes within a study (level two), and variation over studies (level 3). The dependent variable was the SMD of the acupuncture therapies. The independent variables were the study level factors treated as fixed effects.

We had three different specifications in conducting the analyses. The first specification was an empty model with no independent variables to test heterogeneity of effect sizes at the study and outcome levels. The second specification (primary analysis) was a multivariable analysis that estimated the effects of the multiple independent variables associated with the SMD. To ensure sufficient power for the estimation, we determined the number of independent variables included in the model by applying the rule of 10 observations per variable. If no enough sample would contain all independent variables, a hierarchical list of variables was used to determine the priority of entry into the model. The third specification was a univariable analysis with a single factor each time.

To limit the influence of outliers and provide the resistant (stable) results, we incorporated the robust regression approach²⁸ to the three-level random-effects mixed model for the analysis and used the difference of the least-squares means of the SMDs (or the difference of adjusted SMDs) to indicate the effect of a factor. We used 0.2 and 0.4 as the thresholds to name small, moderate and large (<0.2 as small, 0.2–0.4 as moderate,>0.4 as large) for the effect.

We conducted all the analyses in SAS, V.9.4.

Patient and public involvement

The online survey on potential factors involved empirical data and input from a global panel that included patients.

Results

The search yielded 169 406 studies, of which 6530 proved eligible. We retrieved and screened the full texts, excluded 5946 ineligible studies, and finally included 584 studies (figure 1).

Figure 1.

Study selection flow diagram.

Characteristics of included studies

The 584 eligible studies published between 2015 and 2019 reported 1304 effect estimates that met our relevance criteria. Online supplemental eTables 1.1, 1.2 and 1.3 show the basic and clinical characteristics (classification of acupuncture treatment frequency, duration and the total number of treatments provided in online supplemental eAppendix 4), and risk of bias of included studies, respectively. Over 90% of the trials (n=540, 92.5%) were conducted in China. Of the 584 studies, 444 (76%) tested traditional Chinese acupuncture, and 313 (53.6%) used manual acupuncture. Acupuncture was the add-on intervention in 564 studies (96.8%), and 542 studies (92.8%) used other interventions as control. Some variables were important but poorly reported and thus excluded from the multivariable analysis.

Included RCTs had a high risk of bias. For example, over 90% of the RCTs were labelled as inadequate or probably inadequate allocation concealment (n=536, 91.8%); close to 90% of the trials did not report any allocation concealment approaches (524, 89.7%).

The extent of the heterogeneity of the acupuncture’s treatment effect when compared with sham or no acupuncture control (unconditional model-specification 1)

We applied a robust mixed model without exploratory variables to examine the effect sizes’ variations at study and outcome levels and observed significant heterogeneity (p<0.0001). This finding provided a basis for the multivariable analysis to further explore the influencing factors of heterogeneity.

Assessment on factors influencing acupuncture treatment effect (multivariable analysis-specification 2)

Of the 46 factors, 20 met our criterion of <10% of missing (retained at least 526 studies or 1174 outcomes) factor data. The Cramer’s V assessments for multicollinearity assessment further excluded publication language, journal impact factors, trial registration, therapeutic areas and blinding of participants due to the high association with other independent variables (Cramer’s V statistic >0.7, online supplemental eAppendix 3); thus resulted in 15 variables that were eventually included in the analysis (online supplemental eAppendix 5).

The multivariable analysis, including 1133 effect estimates from 508 studies, identified 5 significant factors: type of outcome, acupuncture type, frequency of treatment sessions, number of centres and funding availability (table 1).

Table 1.

Multivariable meta-regression analysis

Factors	Significance
Acupuncture type	√
Acupuncture regimen
Frequency of treatment sessions	√
Style of acupuncture
Type of outcome	√
Type of control group
The course of disease (chronic or acute)
Random sequence generation
Allocation concealment
Blinding of outcome assessors
Sample size
Number of centres	√
Funding available	√
Country
Type of journal

√The factor is a significant predictor (p<0.05).

Blank: The factor is not a significant predictor.

Compared with major events outcomes, effects proved larger in quality of life (large magnitude, difference of adjusted SMDs 0.51, 95% CI 0.24 to 0.77; p<0.001), pain (large magnitude, 0.48, 95% CI 0.27 to 0.69; p<0.001), function (large magnitude, 0.41, 95% CI 0.21 to 0.61; p<0.001) and non-pain symptoms (moderate magnitude, 0.32, 95% CI 0.12 to 0.52; p<0.001). Compared with non-pain symptoms, effects proved larger in pain (small magnitude, 0.16, 95% CI 0.04 to 0.27; p=0.01). Single centre, compared with multicentre, was associated with moderately larger effects (0.38, 95% CI 0.10 to 0.66; p=0.01). Penetration acupuncture (ie, manual acupuncture and electroacupuncture), compared with non-penetration type of acupuncture (ie, laser acupuncture, TEAS and acupressure), was associated with moderately larger effects (0.34, 95% CI 0.15 to 0.53; p<0.001). High frequency acupuncture treatment sessions, compared with low frequency, was associated with larger effects of small magnitude (0.19, 95% CI 0.03 to 0.35; p=0.02). Compared with reported funding, effects proved larger of small magnitude in studies that did not report funding (0.12, 95% CI 0 to 0.25; p=0.03) (figure 2, online supplemental eTable 2)

Figure 2.

Forest plots of significant factors in the multivariable analysis. SMD, standardised mean difference.

Assessment on factors influencing acupuncture treatment effect (univariable analysis: specification 3)

Univariable analysis for independent variables excluded from the multivariable analysis

In univariable analysis, of 31 independent variables excluded from the multivariable analyses, 17 were statistically significant factors (table 2). However, these significances may be attributed to extremely large sample sizes and/or the absence of the other strong predictors in the model.

Table 2.

Univariable meta-regression analysis

Factors	Significance
Total number of acupuncture treatments	√
Type of acupuncture stimulation	√
Source of acupuncture regimen	√
Duration of treatment_chronic	√
Duration of treatment_acute
Education or training of practitioners	√
Acupuncturist experience
Type of comparisons	√
Therapeutic area	√
Blinding of participants	√
Longest follow-up time	√
Missing data reported	√
The proportion of missing data	√
Trial registration	√
Language of publication	√
Type of funding	√
Journal Impact factor	√
Stratification or block randomisation	√
Needle retention time(20 min)
Needling angle
Depth of insertion
Number of needles used
De qi
Patient expectation	√
Acupuncture-specific patient-practitioner interactions
Ever received acupuncture
Location of needles
The clinical specialty of practitioners
Acupuncture manipulation after needles inserted
Needling direction
Intensity of stimulation
Acupuncture type*	√
Acupuncture regimen*
Frequency of treatment sessions*	√
Style of acupuncture*	√
Type of outcome*	√
Type of control group*	√
The course of disease (chronic or acute)*	√
Random sequence generation*	√
Allocation concealment*	√
Blinding of outcome assessors*	√
Sample size*	√
Number of centres*	√
Funding available*	√
Country*	√
Type of Journal*	√

√The factor is a significant predictor (p<0.05).

Blank: The factor is not a significant predictor.

*Included in the multivariable analysis.

Online supplemental eTable 3 presents the effect sizes of significant factors impacting acupuncture’s effect in univariable analysis (excluded from multivariable analysis).

Significant factors in multivariable versus univariable analyses

Of the 15 independent variables, multivariable analysis proved five significant factors associated with the magnitude of effect; in contrast, univariable analysis proved 14 (table 2).

Discussion

Principal findings

We conducted a meta-epidemiological study including 1304 effect estimates from 584 RCTs. Our robust three-level mixed multivariable analyses identified five significant factors that impacted the magnitude of the acupuncture effect. Acupuncture produced the largest treatment effect on quality-of-life, followed by function, pain, non-pain symptoms and major events. Penetration acupuncture induced a larger effect than non-penetration acupuncture. High-frequency acupuncture sessions, single-centred acupuncture RCTs and acupuncture RCTs that did not report funding are associated with larger effects.

Strengths and limitations of the study

This study is the first three-level multivariable meta-epidemiological analysis that included the largest number of RCTs across all therapeutic areas, exploring factors associated with acupuncture’s treatment effect. Hence, the rigorous study provided robust results on critical design factors for acupuncture trialists to consider when designing future RCTs. This study provided a favourable type of acupuncture and treatment regimen for patients, clinicians and policy-makers to achieve acupuncture’s maximum treatment effect for clinical and health system decisions. Our study has several strengths. First, our study is highly patient-centred and clinically relevant. To ensure the conclusion from our study is the most pertinent for healthcare decision making, we included only patient-important outcomes. We consulted a group of international clinicians, researchers and patients when choosing the independent variables.

Second, we constructed a robust three-level mixed model multivariable analysis to adjust for multiple variables to reduce the potential bias raised from the univariable analysis. To deal with the collinearity and substantial amount of outlier and influential values in our datasets, we used Cramer’s V and the weighting approach of robust regression.

Third, our study has a high methodological rigour. We worked with an experienced medical librarian to develop a systematic and exhaustive search strategy. Teams of reviewers then screened and extracted data independently and in duplicate, with third-party adjudication of disagreement.

Our study has several limitations. First, we used a cut-off value of 0.7 in Cramer’s V statistics to identify collinearity, and when applicable, dropped the less important independent variable. Others might find a cut-off of 0.7 being too stringent and therefore left out too many independent variables from the multivariable model. Second, acupuncture RCTs poorly reported the risk of bias and acupuncture techniques related factors. Thus, we could not include some important independent variables such as practitioners’ experience in the multivariable analyses. Finally, some factors (eg, country, trial registered) distributed extremely imbalanced, limiting the results' generalisability.

Comparison with other studies

Previous studies^9–15 typically performed univariable analyses in a small number of studies (5 to 39 trials) and identified 15 significant factors, including 10 clinical, 1 methodological and 4 other factors. Although our univariable analyses confirmed all these factors, the multivariable analyses identified only five significant factors.

An individual patient data meta-analysis (IPDMA) on chronic pain trials found the total number of acupuncture treatments was a significant factor^{9 15} and more treatment sessions were associated with better effects when comparing acupuncture to no acupuncture controls. Meta-regression studies also revealed the same results.^11–13 However, due to a considerable amount of studies that didn't report the number of treatment sessions, we could not include total number of acupuncture treatment sessions in our multivariable analysis.

One study suggested treatment frequency as a significant predictor for tension-type headaches (more frequent treatment, larger effects)¹⁴ while others did not.^{9 15} In our multivariable analyses, the frequency of treatment sessions proved a significant factor. Some studies included homogeneous treatment frequency^{9 15} whereas others included varied frequency, leading to different findings.

For the type of sham acupuncture, the IPDMA^{9 15} reported that compared with non-penetrating and non-needle sham, penetrating needle sham associated with a larger effect. In contrast, a systematic review¹⁰ found no association between the type of sham and acupuncture’s treatment effect. Similarly, our multivariable analyses did not identify the type of sham as a significant factor.

Implications for practice and research

When feasible and acceptable, patients, clinicians and policy-makers should consider using penetrating over non-penetrating types of acupuncture with more frequent treatment sessions.

Identifying significant factors for acupuncture’s treatment effect in trials has important implications for future trials design and conducting secondary analyses. When trialist collaboration designs an acupuncture trial: (1) they should follow Consolidated Standards of Reporting Trials²⁹ and STandards for Reporting Interventions in Clinical Trials of Acupuncture³⁰ reporting guidelines, especially for those that might impact the treatment effect (random sequence generation and allocation concealment, acupuncture technique related information, practitioners related information, and the source of funding); (2) consider the quality of life outcome more often; (3) carefully choose the type of acupuncture, frequency of treatment sessions, choice of single or multicentre as those impact the treatment effect. When exploring factors associated with acupuncture’s treatment effect, researchers should use multivariable analyses over univariable analyses to avoid confounding variables caused biases. Researchers can further investigate factors excluded from multivariable analyses (eg, practitioners’ expertise).

Data availability statement

All data relevant to the study are included in the article or uploaded as online supplemental information.

Ethics statements

Patient consent for publication

Not applicable.