Do the effects of acupuncture vary between acupuncturists? Analysis of the Acupuncture Trialists’ Collaboration individual patient data meta-analysis

Abstract

Objectives

The degree to which the effects of acupuncture treatment vary between acupuncturists is unknown. We used a large individual patient dataset of trials of acupuncture for chronic pain to assess practitioner heterogeneity.

Methods

Individual patient data linked to identifiable acupuncturists were drawn from a dataset of 39 high-quality trials of acupuncture, where the comparators were either sham acupuncture or non-acupuncture control, such as standard care or waitlist. Heterogeneity among acupuncturists was assessed by meta-analysis.

Results

A total of 1,206 acupuncturists in 13 trials were included. Statistically significant heterogeneity was found in trials with sham-control groups (p≤0.003) and non-acupuncture control groups (p≤0.001). However, the degree of heterogeneity was very small, with the observed distribution of treatment effects virtually overlapping that expected by chance. For instance, for non-acupuncture-controlled trials, the proportion of acupuncturists with effect sizes half a standard deviation greater or less than average was expected to be 34%, but was observed to be 37%. A limitation is that the trials included a relatively limited range of acupuncturists, mainly physician acupuncturists.

Discussion

Although differences in effects between acupuncturists were greater than expected by chance, the degree of variation was small. This suggests that most chronic pain patients in clinical practice would have similar results to those reported in high-quality trials; comparably, we did not find evidence to suggest that greater standardization of acupuncture practice would improve outcomes. Further research needs to be conducted exploring variability using a sample of acupuncturists with a broader range of practice styles, training and experience.

INTRODUCTION

Some medical treatments require skill to administer, leading to the possibility that treatment effects vary between practitioners. Heterogeneity among practitioners is likely to involve a range of characteristics, including different educational and learning experiences, unique repertoires of clinical experience, and the utilisation of different interpersonal and technical skills during treatment.

Heterogeneity of outcomes among practitioners, defined as variability over and above that expected by chance, has a number of implications for research and clinical practice. It affects the interpretation of research results because, in the presence of heterogeneity, a referring clinician may be sceptical that the average results reported in the relevant literature will be obtained by a local practitioner. Heterogeneity would also suggest that changes in practice or regulation might be required in order to ensure more consistent outcomes.

Analysis of heterogeneity of outcomes among practitioners is commonly ignored in the analysis of clinical trials.(1) Methodologists are starting to explore the implications of heterogeneity in the context of surgical procedures,(2) primary care services,(3) physician-led diabetes care,(4) the teaching of behavioural interventions,(5) and psychotherapists in clinical practice.(6) In a retrospective analysis using data from four randomised controlled trial datasets, small but significant variations in outcome between health professionals were found for nurses treating leg ulcers, acupuncturists treating low back pain, physiotherapists providing pre- and post-operative care associated with total knee replacement, and homeopaths treating chronic fatigue syndrome.(7) Therapists have also been identified as a source of variability within psychotherapy outcome studies.(8)

We have recently updated an individual patient data meta-analysis of acupuncture for chronic pain.(9) We found acupuncture to lead to statistically significant improvements in pain when compared to either sham acupuncture or non-acupuncture controls. However, it is not clear to what extent acupuncture effect sizes are affected by practitioner variability. Acupuncture requires skill to administer, but it is unclear whether effects vary greatly between acupuncturists, or whether acupuncturists have approximately similar results. This is of particular interest because acupuncture is provided by a wide range of professionals, including physicians, nurses and physiotherapists with additional training in acupuncture, and specialist acupuncturists with no other healthcare professional qualifications. In addition, there is wide variation in the length of training of these practitioners in acupuncture from short courses of several days, to full Masters’ degrees. An additional reason to suspect variation in acupuncture outcomes is that there are several different styles of acupuncture based on different underlying theories.(10) Differences in the way that practitioners manipulate acupuncture needles have also been documented.(11) Here, we take the opportunity of our large individual patient dataset to further explore heterogeneity in treatment effects among different acupuncturists.

METHODS

Included Trials

Trials included in these analyses were identified through a systematic literature review that has been previously described.(12) The search included trials of acupuncture for chronic pain published prior to December 2015 and included only high-quality trials where allocation concealment was determined unambiguously to be adequate. Eligible pain types were non-specific back or neck pain, shoulder pain, chronic headache or osteoarthritis – with the additional criterion that the current episode of pain must be of at least four weeks duration for musculoskeletal disorders. This search resulted in the identification of 44 trials.

Data Acquisition

Individual patient data were obtained from 39 trials. Data on the trial-level characteristics of the acupuncture intervention were obtained directly from trialists. Twenty-six trials had a sham acupuncture control group, and twenty-five trials had a non-acupuncture control group, twelve trials being three arm trials. This study was approved by the Institutional Review Board at Memorial Sloan Kettering Cancer Center (14–113). Informed consent was not necessary as this was a secondary use of deidentified data.

Outcome

The primary outcome used for this analysis was the primary outcome defined by the study authors for each study. Where multiple criteria were considered in the primary outcome (e.g. a response defined as either a 33% reduction in pain or a 50% reduction in pain medication) or if the primary outcome was inherently categorical, we used a continuous measure of pain measured at the same time point as the original primary outcome. To make the various outcome measurements comparable between different trials, the primary endpoint of each was standardized by dividing by pooled standard deviation.

Eligible Trials

To be included in the analysis, trialists had to identify the acupuncturist who treated each patient in the acupuncture group, and in sham-controlled trials, identify the acupuncturist who treated each patient in the sham acupuncture group. Most non-acupuncture control trials identified an acupuncturist for control group patients, that is, the acupuncturist whom a patient would have seen had he or she been allocated to acupuncture. This was typically a function of geographic location, with the nearest acupuncturist identified for each patient before randomization. For two trials,(13, 14) acupuncturist information was not provided for the non-acupuncture group, so acupuncturists were randomly assigned to each of the control group patients from the same trial. This approach was deemed reasonable as there was no contact between acupuncturists and control group patients in these two trials and therefore the distribution of control group patient outcomes will be random with respect to acupuncturist. Five sham-controlled trials had practitioner data available and were included in the analysis.(15–19) The other 21 trials with sham control arms did not provide practitioner data and were excluded. (13, 20–39). The Hinman 2014 trial was excluded from the analysis of sham-controlled trials due to a sham arm that was not known to be credible.(13) Twelve trials were included in the analysis of non-acupuncture-controlled trials;(13–17, 19, 40–45) 13 trials were excluded.(20, 23, 25, 26, 33–35, 46–51)

Statistical Methods

In these analyses, we examined whether the effects of acupuncture for chronic pain varied among acupuncturists. We took two separate but complementary statistical approaches. In the first, a trial-level analysis, we examined whether effect sizes differed between acupuncturists within each trial separately and combined the results on acupuncturist heterogeneity from each trial into a meta-analysis. In the second approach, an acupuncturist-level analysis, data from acupuncturists in all trials was combined into a single analysis.

Trial-level analysis

Acupuncturists were eligible for inclusion in the trial-level analysis if they treated two or more patients in the acupuncture group. To determine whether the effects of acupuncture varied between acupuncturists within each trial separately, a mixed effects model with both a random intercept and a random slope was created for each trial to predict pain after treatment, with baseline pain and treatment group as fixed effects. Acupuncturists were included as a random effect, which assumes that the acupuncturists included in the analysis were drawn from a larger theoretical population of acupuncturists. By estimating the distribution of this theoretical population, we can investigate between-acupuncturist variability.

For these models, the coefficient for treatment is the difference in pain change from baseline between the control group and the acupuncture group, with an increase in slope representing a greater reduction in pain in the acupuncture group compared to the control group. Further modeling details are available in the supplementary material.

We meta-analyzed the random-effects coefficient from these models to test whether there was a significant random effect of acupuncturist among all trials. In some trials, the models did not converge. These models were not included in the meta-analysis, but were analyzed separately (see supplementary materials). We compared the effect of acupuncturist in both sham and non-acupuncture-controlled trials to our updated meta-analysis, to determine whether the overall acupuncturist effect was consistent with our estimate of the effect of acupuncture. Among sham-controlled trials, we also compared these results to the results from the original meta-analysis when three outlying trials(36–38) were excluded, since these three trials were also excluded from the acupuncturist-specific analysis.

Acupuncturist-level analysis

In the second approach, we then combined all data into a single analysis. A linear regression model was created for each acupuncturist that predicted outcome after treatment, and was adjusted for baseline pain, treatment group, and any variables used to stratify randomization in the original trial. For the linear regression models to converge we excluded acupuncturists who treated fewer than 3 patients in either group. The regression coefficient for treatment group and its standard error were then entered into a meta-analysis.

Since some variation among acupuncturists is expected by chance, we compared that expected variation with the variation we observed among the acupuncturists in our sample. Since it is difficult to interpret random effects coefficients and the forest plot for the meta-analysis by acupuncturist contains too many acupuncturists to be useful for visualizing the distribution of acupuncturist effects, we instead created histograms comparing the observed distribution of effect sizes to an expected distribution of effect sizes. We overlaid two histograms – a histogram of the distribution of acupuncturist effect size in our sample, and a histogram of the distribution of acupuncturist effect size that would be expected by chance, adapting the method used by Bianco et al to create a similar graph of surgical outcomes by surgeon (see Figure 1).(52) Further details of the permutation analysis are available in the supplementary materials.

Figure 1.

Difference between expected and observed complication rates between surgeons, adapted from Bianco et al.(52)

As a sensitivity analysis, we repeated these analyses after combining trials that shared acupuncturists. There were two groups of trials conducted in Germany (the ARC trials and the ART trials) in which the same acupuncturists may have treated patients with different pain types in multiple trials. The ARC trials studied the effects of acupuncture on neck pain, headache, knee and hip osteoarthritis, and lower back pain.(40–43) All ARC trials reported a clinical relevant improvement in pain for acupuncture patients compared to control patients that received only usual care. The ART trials studied acupuncture for knee osteoarthritis and lower back pain(15, 19). Both trials reported a significant improvement in pain in acupuncture patients at 8 weeks after baseline compared to non-acupuncture control; differences between acupuncture and sham were significant for osteoarthritis but not for low back pain. Since each group of trials shared acupuncturists and used the same basic principles (see supplementary materials), we performed a sensitivity analysis in which we combined the data from all four ARC trials and from both ART trials into one ARC trial dataset and one ART trial dataset, so that each acupuncturist was associated with all patients that he or she had treated, regardless of pain type.

In the German trials, including the ART and ARC trials, acupuncture was provided by physicians, whereas most other included trials involved non-physician acupuncturists. As a sensitivity analysis, we repeated the analyses excluding 8 trials where acupuncture was provided by physicians.(13, 15, 16, 19, 40–43)

All analyses were conducted using Stata 13.0 (StataCorp, College Station, TX).

RESULTS

Included trials

There were 25 trials that had eligible sham acupuncture control groups; however, only five of these trials reported acupuncturist data (Witt 2005 (19), Suarez-Almazor 2010(17) and White 2012(18) for osteoarthritis, Brinkhaus 2006 for lower back pain(15) and Diener 2006 for migraine(16)), with 213 total acupuncturists, including 9 non-physician acupuncturists from two trials (17, 18) (Table 1). The differences in the number of acupuncturists between the trial- and acupuncturist-level analysis is due to the more stringent criterion for the minimum number of patients in the latter analysis. For the sensitivity analysis, the ART(19) osteoarthritis trial was combined with the ART(15) low back pain trial. The analyses were not repeated in sham-controlled trials when excluding physician-acupuncturists as there were only two eligible trials with 9 acupuncturists.

Table 1.

Number of acupuncturists included in eligible trials, as well as included in trial-level and acupuncturist-level analysis.

Control Group	Analysis	Number of trials	Number of acupuncturists	Number in trial-level analysis	Number in acupuncturist-level analysis	Non-physician acupuncturists in trial-level analysis	Non-physician acupuncturists in acupuncturist-level analysis
Sham Acupuncture	Primary	5	213	132	67	9	9
	Combining ARC and ART trials	4	189	114	65	9	9
Non-Acupuncture	Primary	12	4,961	1,203	280	41	31
	Combining ARC and ART trials	8	3,006	1,172	438	41	31
	Excluding non-physician acupuncturists	4	43	41	31	41	31

Note that the differences in the number of acupuncturists included results from differences between analytic approaches in the minimum number of patients an acupuncturist needs to have treated to be included.

There were 25 trials with non-acupuncture control groups, and 12 of these trials reported data on acupuncturists (Thomas 2006(14), Witt 2006(43) and Brinkhaus 2006(15) for lower back pain, Witt 2005(19), Witt 2006(42), Suarez-Almazor 2010 (17) and Hinman 2014(13) for osteoarthritis, Diener 2006(16) for migraine, Vickers 2004(44) and Jena 2008(40) for headache, and MacPherson 2015(45) and Witt 2006(41) for neck pain), with 4,961 total acupuncturists, including, 43 non-physician acupuncturists from four trials(14, 17, 44, 45). For the sensitivity analysis, the German-based ARC trials of neck pain(41), osteoarthritis(42), headache(40) and low back pain(43) trials were combined. The ART osteoarthritis(19) and ART low back pain(15) trials were also combined. When combining trials for the sensitivity analysis, more acupuncturists became eligible for inclusion in the acupuncturist-level analysis, since some acupuncturists treated patients in more than one trial (Table 1). Those acupuncturists who did not treat at least three acupuncture patients and three control patients in one individual trial may have treated at least three patients in each group when including patients from multiple trials treated by the same acupuncturist. When excluding trials where acupuncture was provided by non-physicians, there were four trials with 43 acupuncturists and 943 patients included in the analysis.(14, 17, 44, 45).

Trial-level analysis

For the meta-analysis of random effect statistics from each trial calculated separately, there was a significant random effect of acupuncturist, that is, statistically significant acupuncturist heterogeneity, in both the sham-controlled trials (N = 4 trials, 126 acupuncturists and 1,136 patients, overall random effects statistic 0.17, 95% CI 0.07, 0.26, p = 0.001) and those with non-acupuncture control (N = 9 trials, 1,175 acupuncturists, and 5,662 patients, overall random effects statistic 0.08, 95% CI 0.03, 0.12, p = 0.001). The Suarez-Almazor 2010(17) osteoarthritis trial was excluded from the meta-analysis of random effects statistics in sham-controlled trials and the Thomas 2006(14) low back pain, Vickers 2004(44) headache and Hinman 2014(13) osteoarthritis trials were excluded from the meta-analysis in non-acupuncture-controlled trials since the random effects models did not converge.

We then combined trials that shared acupuncturists so that all patients treated by the same acupuncturist were analyzed together regardless of the trial in which they originally participated. There were three sham-controlled trials included in this sensitivity analysis: the Diener 2006(16) migraine trial, the White 2012(18) osteoarthritis trial and the combined ART trial (Witt 2005(19) and Brinkhaus 2006(15)). After repeating the meta-analysis for sham-controlled trials, we found similar results (N = 3 trials, 108 acupuncturists and 1,146 patients, overall random effects statistic 0.13, 95% CI 0.05, 0.22, p = 0.003). Out of the original nine trials included in the random effects meta-analysis for trials without sham control, the four ARC trials(40–43) were combined into one joint ARC trial, and the two ART trials(15, 19) were combined into one joint ART trial. The ARC trial and the ART trial were included in a meta-analysis with the Diener 2006(16) migraine trial, the MacPherson 2015 neck pain trial(45) and the Suarez-Almazor 2010(17) osteoarthritis trial. Results showed a greater variation among acupuncturists in trials without acupuncture control when the ART and ARC trials were combined, compared to kept separate (N = 5 trials, 1,148 acupuncturists and 7,733 patients, overall random effects statistic 0.12, 95% CI 0.08, 0.16, p<0.0001). This analysis was not performed when excluding physician-acupuncturists as there were too few eligible trials for which the random effects model converged.

Acupuncturist-level analysis

We first compared the effect size from this meta-analysis, which included only a subset of patients, to the effect size of acupuncture from the updated meta-analysis.(9) For the sham-controlled trials, effect sizes were slightly smaller in both analyses (N = 65 acupuncturists, effect size 0.13; combining trials that shared acupuncturists, N = 63 acupuncturists, effect size 0.12) compared to the effect sizes in the original meta-analysis comparing acupuncture to sham (between 0.16 and 0.30, depending on pain type). For trials with a non-acupuncture control, effect sizes in the full meta-analysis and in the subset of trials that provided individual acupuncturist data were similar: N = 226 acupuncturists, effect size 0.45 and N = 380 acupuncturists, effect size 0.50 when combining trials that shared acupuncturists, versus effect sizes of 0.44 to 0.63 from the updated analysis depending on pain type. As the effect size for the subset included in this analysis is slightly smaller than in the main meta-analysis, this suggests that our estimate of variability between acupuncturists may be a slight underestimate.

In both sham-controlled and non-acupuncture-controlled trials, there was significant heterogeneity among acupuncturists, meaning that there was a significant difference in effect between acupuncturists that was not due solely to chance (p < 0.0001 for both sham-controlled and non-acupuncture-controlled trials). After combining acupuncturists who participated in multiple trials, heterogeneity remained highly significant (p < 0.0001 for both sham-controlled and non-acupuncture-controlled trials).

There were 226 total acupuncturists who treated patients in the non-acupuncture-controlled trials included in the meta-analysis. As seen in Figure 2, the dispersion of effect sizes in non-acupuncture-controlled trials was statistically greater than expected; differences are small, with the observed and expected distributions mostly overlapping. When excluding physician-acupuncturists, there were 30 acupuncturists in this analysis of non-acupuncture-controlled trials. Significant heterogeneity was seen in this subset of acupuncturists (p=0.047), although the differences between acupuncturists were small and results consistent with the main analysis (Supplementary Figure 1).

Figure 2.

Distribution of observed and expected acupuncturist effect sizes in non-acupuncture-controlled trials, N=226.

Among sham-controlled trials (Figure 3), there appear to be a wider variation in values and less overlap between our observed distribution and the expected distribution. However, the number of acupuncturists treating patients in sham-controlled trials was small (N = 65). As a result, some of the bars in Figure 3, such as that at −2 SD, represent only one acupuncturist.

Figure 3.

Distribution of observed and expected acupuncturist effect sizes in sham-controlled trials, N=65.

A visual comparison with Figure 1, which represents variation in complication rates between surgeons(52), indicates that while there is significant variation among acupuncturists, the magnitude of this variation is relatively minor. For instance, in the analysis of trials with non-acupuncture controls, we expected by chance that 15% of acupuncturists would have an effect of zero or less, half a standard deviation less than average; the observed number was only very slightly higher at 18%. For an effect size of 1 or more, half a standard deviation more than average, the number of acupuncturists both expected and observed was 19%. Combining these, the proportion of acupuncturists with effect sizes half a standard deviation greater or lesser than average was expected to be 34%, but was observed to be 37%. For effect sizes different from the average by one standard deviation or more, rates of expected and observed were 7% vs. 10%. For the sham-controlled trials, the expected and observed proportion of acupuncturists with effect sizes more than half a standard deviation greater or lesser than that expected was 34% vs. 48%; for effect sizes different from the average by one standard deviation or more, rates of expected and observed were 9% vs. 15%.

DISCUSSION

We used individual patient data from a meta-analysis of high-quality trials of acupuncture for chronic pain to conduct two separate analyses addressing the question of practitioner heterogeneity of outcome. We found that differences in effect sizes among acupuncturists were greater than would be expected by chance. This effect was found for trials with sham control, as well as those with non-acupuncture control groups such as usual care. As such, differences in outcome likely include differences specific to acupuncture technique, rather than just good “bedside manner” and other non-specific aspects of clinical care.

Importantly, however, the degree of variation is relatively small. The distribution of observed acupuncturist-specific effect sizes was virtually overlapping with the distribution that would be expected by chance. This is in some distinction with, for instance, studies on variation among surgeons, where a large proportion of surgeons have outcomes very much better or worse than expected.(53, 54) Our results complement a prior study from the Acupuncture Trialists’ Collaboration that did not find important modifying effects on acupuncture outcomes for characteristics such as the number or duration of acupuncture sessions, the number of needles used, the age or sex of the practitioner, acupuncture style (Western vs. traditional Chinese) or point prescription approach (fixed, flexible or individualized).(55) The major clinical implication is that chronic pain patients should be expected to have similar results to those reported in high-quality trials. A second implication is that our findings do not provide support for greater standardization of the methods and techniques that are used within routine acupuncture provided to patients with chronic pain.

The relatively limited degree of outcome heterogeneity we found in the sham-controlled trials might be seen as predictable from consideration of the effect size of acupuncture. In the meta-analysis, we reported an effect size of acupuncture compared to sham of approximately 0.2 standard deviations.(9) Sham acupuncture involves techniques such as needles inserted to the wrong depth, and/or away from acupuncture points. If the difference between good acupuncture technique and a grossly inadequate acupuncture technique - sham acupuncture - is fairly modest, then we would not expect large differences in outcome between qualified acupuncturists based on slight variations in their technique.

The authors of the Thomas 2006 trial on acupuncture for low back pain, which was excluded from the comparison in this sub-study of acupuncture vs. non-acupuncture controls because of lack of convergence of the statistical model, have also explored outcome heterogeneity among practitioners.(56) This was assessed by comparing outcomes for the six acupuncturists in the trial, each of whom treated a minimum of 15 patients. The analysis was a nested ANCOVA, whereby acupuncturists were nested within the acupuncture group and baseline pain score was the covariate. In a result that was broadly consistent with the findings of this sub-study, it was concluded that there was no evidence of any important difference in patient outcomes between acupuncturists. Previous regression analyses of the above mentioned ARC trials that evaluated acupuncture in addition to usual care included 9990 patients treated by 2781 physicians and came to the conclusion that physician characteristics such as training and experience did not influence patients’ outcome after acupuncture.(57)

The major limitation of our study is that we included a relatively limited range of acupuncturists, and our sample is unlikely to be fully representative of those found in routine acupuncture practice. Our data-set is dominated by the trials from Germany, in which provision of acupuncture was by physicians with a minimum of 180 hours training, although most had far more training. It is not clear the extent to which our results reflect other types of acupuncturists (non-physician), with longer periods of training, from other countries (e.g. from China), practicing other styles of acupuncture (e.g. both in terms of technique and in terms of spending more time with patients, and recommending more self-care). At least theoretically, these variations in practice could be associated with more heterogeneous outcomes.

Moreover, trialists often actively selected acupuncturists based on reputation or clinical experience. For instance, in the UK National Health Service headache trial(44), acupuncturists were nominated by a professional organization. Given the high profile of the trial, it is likely that only acupuncturists who are better known within the acupuncture community would be nominated. In the UK Thomas trial, acupuncturists were chosen on the basis of experience and proximity to the centre of York, the city where the trial was conducted.(14) However, although it sounds plausible that there is less heterogeneity in outcomes between better-known or more experienced acupuncturists, as mentioned above, prior research in a routine care setting with regular acupuncturists has not found an association between experience and outcome of acupuncture in comparison to non-acupuncture control(57).

CONCLUSION

Although differences in effect sizes among acupuncturists were greater than would be expected by chance, the degree of variation was small. This suggests that most chronic pain patients in clinical practice would have similar results to those reported in high-quality trials. With respect to pain outcomes, we found no reason to suggest a need for greater standardization of acupuncture practice. However, a major limitation is that the trials included a relatively limited range of acupuncturists, which are not fully representative of those found in routine acupuncture practice. As such, further research needs to be conducted exploring practitioner variability using a wider sample of acupuncturists.