Issues of design and statistical analysis in controlled clinical acupuncture trials: An analysis of English-language reports from Western journals

Abstract

Objective

To investigate major methods of design and statistical analysis in controlled clinical acupuncture trials published in the West during the past six years (2003–2009) and, based on this analysis, to provide recommendations that address methodological issues and challenges in clinical acupuncture research.

Method

PubMed was searched for acupuncture RCTs published in Western journals in English between 2003 and 2009. The keyword used was acupuncture.

Results

One hundred and eight qualified reports of acupuncture trials that included more than 30 symptoms/conditions were identified, analyzed, and grouped into efficacy (explanatory), effectiveness (pragmatically beneficial) and other (unspecified) studies. All were randomized controlled clinical trials (RCTs). In spite of significant improvement in the quality of acupuncture RCTs in the last 30 years, these reports show that some methodological issues and shortcomings in design and analysis remain. Moreover, the quality of the efficacy studies was not superior to that of the other types of studies. Research design and reporting problems include unclear patient criteria and inadequate practitioner eligibility, inadequate randomization and blinding, deficiencies in the selection of controls, and improper outcome measurements. Problems in statistical analysis included insufficient sample sizes and power calculations, inadequate handling of missing data and multiple comparisons, and inefficient methods for dealing with repeated-measure and cluster data, baseline value adjustment, and confounding issues.

Conclusion

Despite recent advancements in acupuncture research, acupuncture RCTs can be improved, and more rigorous research methods should be carefully considered.

Because conventional medicine has limitations and side effects, increasing numbers of people turn to complementary and alternative modalities, including acupuncture, for treating their diseases and alleviating their symptoms. Since the 1970s numerous clinical trials have been conducted in the West to evaluate the effects of acupuncture, but the results remain controversial [1].

Recently, the randomized controlled clinical trial (RCT), considered the “gold standard” in conventional clinical research, has been employed extensively to study acupuncture. However, some researchers argue that the RCT might not be the most suitable method for evaluating that modality, which originated in traditional Chinese medicine (TCM), a medical system very different from that of conventional medicine in terms of diagnostic and therapeutic principles [2, 3]. While it may be true that by its nature acupuncture poses difficulties for researchers, the contradictory and inconclusive results of acupuncture RCTs may speak more to their methodological shortcomings [4, 5] than to philosophical distinctions between conventional medicine and TCM. Though the quality of acupuncture RCTs has significantly improved over the last 30 years, many researchers point out that meticulous attention to detail, more rigorous design, and better explanation of results are needed [6]. The purpose of this paper is to identify the main methodological issues of design and statistical analysis in recently published acupuncture RCTs in order to provide useful recommendations for improving the quality of future clinical trials.

Materials and Methods

Our focus was on acupuncture RCTs published in English in Western journals from August 2003 to December 2009, primarily those referenced in PubMed. Inclusive criteria were as follows: manual acupuncture (an acupuncture technique that involves manually rotating, lifting, and thrusting a needle); treatment primarily based in TCM; treatment performed for curative, palliative, or preventive purposes. We excluded observational studies and reports written in other languages. Considering that certain alternative and complementary therapies related to acupuncture may have different mechanisms that might confound the results, we excluded trials of acupressure, moxibustion, transcutaneous nerve stimulation, electro-acupuncture, and auricular acupuncture. Our analysis is not intended to be a systematic review.

Results

Accordingly, 108 identified reports [7–114] qualified for review and analysis. They encompass a wide range of conditions/symptoms, including: osteoarthritis/knee pain, migraine, low back pain, pelvic pain, hot flashes, labor pain and labor initiation, analgesic and sedative effects in surgery/ cancer pain, tension-type headache, shoulder/arm pain, embryo transfer, irritable bowel syndrome, allergic rhinitis, depression, fibromyalgia, myofascial pain, insomnia, inflammatory breast in lactating women, stroke rehabilitation, anxiety, nausea and vomiting, dry eyes, hypertension, PTSD (Post-Traumatic Stress Disorder), recurrent cystitis, psychogenic erectile dysfunction, chronic post-stroke leg spasticity, overactive bladder, refractory heartburn, rheumatoid arthritis, disabling breathlessness, impingement syndrome, cervical spondylosis, COPD (Chronic Obstructive Pulmonary Disease), diabetic bladder dysfunction, balance function after stoke. The sample size of those articles varied from sixteen to over three thousand. The reports had problems both in design and statistical analysis.

Part I. Issues in Design

For the sake of analysis and comparison, we roughly categorized the 108 studies into three groups, based on design and author’s statement of purpose. There were 45 efficacy studies, 44 effectiveness studies, and 19 that could not be defined. Clear statement of the study objective contributes to the use of appropriate research models and methods.

Efficacy trials are used to determine whether an intervention works and is safe when delivered under controlled conditions, while effectiveness trials evaluate the usefulness of an intervention’s pragmatic “real-world” applicability [115]. Efficacy studies should be well controlled and more rigorously designed than effectiveness studies, while the latter are focused on how much benefit patients gain from an intervention in actual practice. In our analysis, we found only a few efficacy studies that conformed to strict and precise designs [7, 21, 37, 46, 96].

1. Eligibility of Patients and Acupuncturists

Patient source is one of the most important issues in patient eligibility and is especially crucial to efficacy studies. Some reports did not state clearly the source of patients or used simple sampling. In total, only 26 -- 14 (29%) efficacy studies, 9 (19%) effectiveness studies, and 3 (15%) unspecified studies -- described in detail whether their patients were recruited through advertisements or other media.

Adequate and detailed inclusion and exclusion criteria can ensure a homogeneous sample. Although most of these studies clarified their selection criteria, nineteen, including four efficacy studies, did not report inclusion and exclusion criteria in detail.

Considering the uniqueness of acupuncture, in studies using non-needle-insertion sham acupuncture as control, subjects ideally should have no previous experience of acupuncture. A patient who has had true acupuncture treatment will easily recognize the sham as a fake treatment, thus confounding the blinding, so some researchers require that included patients be acupuncture naïve. Table 1 is a breakdown of how the issue of previous experience was handled in these studies.

Table 1.

Participants’ previous acupuncture experience (non-insertion sham control studies)

Acupuncture Experience	Efficacy Studies	Effectiveness Studies	Unspecified Study	Total
Acupuncture naïve	0	3 (27%)	1 (25%)	4 (15%)
No acupuncture in the past 6 or 12 months	4 (36%)	3 (27%)	1 (25%)	8 (31%)
Not reported	7 (64%)	5 (46%)	2 (50%)	14 (54%)
Total	11 (100%)	11 (100%)	4 (100%)	26 (100%)

The following categories of practitioners were reported as having performed the treatments: acupuncturists, physiotherapists, physicians, researchers, nurses, and midwives. STRICTA (Standards for Reporting Interventions in Controlled Trials of Acupuncture) recommendations by MacPherson et al. state that practitioner qualifications should be carefully established and clearly reported [116]. While point selection rationale, needling details, treatment regimen (frequency and duration of treatment) and co-interventions are important items in STRICTA, adequate training is particularly important. However, 53 reports, including 21 efficacy reports, provided no detailed information on practitioners’ acupuncture education, training, background, or experience. Some simply stated that the acupuncturists were “experienced” or “qualified” without providing specific information.

A well-trained and experienced practitioner is likely to deliver a treatment with adequate point selection, needling technique, and treatment regime. As acupuncture is a needling-skill-driven therapy, the use of qualified acupuncturists improves the quality and credibility of a study. Furthermore, adequate training of the acupuncturists in the specific treatment protocol will enhance standardization and minimize variation in their interactions with patients [117]. In addition, consultation with acupuncture experts or consensus among a group of experienced acupuncturists regarding the acupuncture treatment procedure would be a good approach for designing a clinically valid study treatment protocol.

2. Randomization and Masking

Randomization helps control for variations among patients and ensures that both known and unknown confounding factors are evenly distributed between groups. Every reviewed report mentioned using randomization to allocate patients.

Hedden et al. [110], recommends that in a clinical trial with a small sample size (n<200), where the probability of imbalance between comparison groups is high, use of modern methods of randomization, such as stratification, is best. Stratification is a procedure for achieving balance between groups in baseline characteristics, while blocking is used to achieve balance in size. Blocking can be used to ensure close numerical balance in groups at any time during a trial. To maintain blinding by keeping participants from deducing their treatment from the block size, larger or randomly varied block sizes may be used. In about half of the reviewed trials the researchers considered this issue and reported their blocking procedure. Stratified randomization provides increased confidence that groups are comparable with respect to known prognostic factors. A total of 8 (18%) of the efficacy, 5 (11%) of the effectiveness, and 2 (11%) of the unspecified studies used stratified randomization for such factors as clinic center, sex, and symptom baseline scores.

Masking patients and practitioners is done to minimize possible bias in general RCTs. Clearly masking is important. However, given the nature of acupuncture, some researchers claim that masking acupuncture practitioners is difficult or impossible and posit study designs with a blinded evaluator and blinded patients as alternatives to the standard double-blinded study used in drug trials [119,120].

In the 108 studies, two main forms of masking were used: patient-only blinding, and patient/evaluator blinding. The different methods used in masking are listed in Table 2. Three efficacy studies and two effectiveness studies attempted to blind both patients and practitioners. In the latter, the investigator used two teams of acupuncturists that were blinded to patient allocation. The first team, responsible for diagnosis and prescription, sealed the point prescriptions in envelopes given to a research assistant. Depending on patient assignment, the sealed envelopes that the acupuncturists in the second team received either contained an appropriate prescription made by the first team or a set of irrelevant acupuncture points for control. Practitioner blinding has clear limitations, since acupuncture treatment is highly skill dependent. An experienced acupuncturist can easily tell whether an acupuncture protocol suits the patient’s condition. Thus the double blinding used in standard drug trials may not be appropriate in trials of such non-pharmacologic interventions such as acupuncture and physical therapy, and it remains controversial.

Table 2.

Masking/blinding methods used

Masking/Blinding Methods	Efficacy Study	Effectiveness Study	Unspecified Study	Total
Patients only	15 (33%)	9 (20%)	7 (36%)	31 (29%)
Patients + Evaluators	14 (31%)	9 (20%)	2 (11%)	25 (23%)
Patients + Practitioners (Double Blinding)	3 (7%)	2 (5%)	0	5 (5%)
Evaluators only	4 (9%)	5 (11%)	0	9 (8%)
No Masking/Blinding	1 (2%)	2 (5%)	0	3 (3%)
Unclear	8 (18%)	17 (39%)	10 (53%)	35 (32%)
Total	45 (100%)	44 (100%)	19 (100%)	108 (100%)

Evaluation of the credibility of blinding, however, can be useful since inadequate blinding is associated with bias. Of the trials reviewed, 18 (40%) of the efficacy, 14 (32%) of the effectiveness, and 1 (5%) of the unspecified studies included such a test. The most common approach was to ask subjects which treatment they believed they had received. This approach has its limitations; there is no methodological consensus on whether such tests are appropriate [130], and some studies show that they are problematic since the guess may depend on clinical outcome if the credibility test is conducted at the end of a trial [133]. On the other hand, if this method is used at an early stage before the patient has noticed any treatment effect, it may be adequate for validating patient blinding to treatment procedure since influence from treatment result is excluded [132,133].

Better methods for evaluating the success of blinding are urgently needed [130]. To achieve better patient blinding, it is suggested that investigators make every effort to ensure that subjects from different groups do not meet in the waiting room, where they might compare treatment experiences. Additionally, point location diagrams and reading materials that discuss acupuncture, and thus potentially may reveal treatment assignment, should be removed from the clinical setting.

In addition to the blinding efforts discussed above, treatment concealment can be better achieved if the study practitioners use neutral language and avoid discussing the treatment protocol. Using a previously prepared script for acupuncturist/patient conversation might helpful to minimize the potential bias by an acupuncturist as he or she is usually not blinded to the treatment assignment. [121].

3. Control Group Choice

Control group choice is dependent on study purpose and the acupuncture modality to be tested. In these reports, control groups varied widely. Sham acupuncture was the most common control; 17 (38%) efficacy, 14 (32%) effectiveness, and 7 (37%) unspecified studies compared true acupuncture with sham acupuncture. Less frequent were reports comparing acupuncture to standard therapy or to no treatment, acupuncture plus standard treatment to standard treatment, and three-arm design comparing acupuncture, sham acupuncture, and standard or no treatment. In the effectiveness studies, the acupuncture arm was often compared to either standard care [16, 18, 23, 35, 41, 62, 67, 89, 113], or other established treatments such as physical therapy [9, 14, 38]. Although the pitfall of this type of trial is that patients cannot be blinded to treatment assignment, the advantage is that such trials reflect real-world clinical practice, and their data can help clinicians determine the choice of therapy in daily practice.

Although sham acupuncture was the most common control, there is controversy over how to choose an appropriate sham [122,123]. In the studies reviewed, four main types of sham acupuncture were used based on needling and point selection: non-insertion at non-acupuncture points, non-insertion at true points, insertion or superficial insertion at non-acupuncture points, and superficial insertion at true points. The most popular type was insertion or superficial insertion at non-acupuncture points (see Table 3). But minimal acupuncture, defined as needles inserted into non-acupuncture points and/or superficially used as placebo control, is not necessarily inert and may have both specific and non-specific effects [131]. Indeed, there is general agreement that invasive sham acupuncture is not inert. As Langevin et al. [123] stated, enhanced benefit from sham acupuncture, especially likely in the case of invasive sham needling, may result from locally induced changes in microcirculation, immune system responses, or nonspecific pain modulation mechanisms. And more invasive sham procedures, which have been shown by neuroimaging to affect pain processing regions, should be also avoided [123].

Table 3.

Different types of sham acupuncture based on needling and point selection

	Efficacy Studies	Effectiveness Studies	Other studies	Total
Non-insertion at non-acupuncture points	7 (20%)	6 (29%)	2 (15%)	15 (22%)
Non-insertion at true acupuncture points	4 (11%)	5 (24%)	2 (15%)	11 (16%)
Insertion (superficial) at non-acupuncture points	20 (58%)	10 (47%)	7 (55%)	37 (53%)
Insertion (superficial) at true acupuncture points	4 (11%)	0	2 (15%)	6 (9%)
Total	35 (100%)	21 (100%)	13 (100%)	69 (100%)

In most of the reviewed studies, non-acupuncture control points were not well defined. Some researchers use true acupuncture points unrelated to the condition being treated as non-acupuncture controls, while others used points a few centimeters away from the true points. As the size of a real acupuncture point is still not clearly defined and some schools of thought do not regard traditional acupuncture points as fixed loci, these methods of determining a non-acupuncture control point are problematical.

Different acupuncture controls in clinical trials may answer different research questions. In clinical drug trial design, the specific effect of a targeted receptor is usually clearly defined, and the placebo control pill is known to be inert. In contrast, in an acupuncture trial an investigator who chooses a non-specific sham control to determine whether acupuncture produces a non-specific effect must first clearly understand how acupuncture works and what the specific effect of acupuncture is. Because the validity of acupuncture theories, mechanisms, point indications, locations, and needling methods has not been established, we suggest that non-insertion at non-acupuncture points is generally the best sham control, as it minimizes non-specific physiological confounding effects. However, care needs to be taken even with this procedure, since some researchers maintain that touch or pressure on any point may produce unintended physiological effects [123]. Some investigators state that noninvasive sham procedures, such as tapping an empty needle guide tube on the skin surface, may trigger physiological regulatory responses, especially if done at acupuncture points, and according to some classical theories, that of certain Japanese schools for example, a non-penetrating needle is considered to have a therapeutic effect [122].

4. Outcome

Adequate choice of appropriate outcomes measurements is important. However, not every report we reviewed clearly stated outcomes. A total of 13 (29%) efficacy, 13 (30%) effectiveness and 11 (58%) unspecified studies made no mention of their outcomes in their method sections, although some reported these in their results sections.

In clinical studies, objective outcome measurement is preferred to subjective [124], and acupuncture trials using objective outcome measurements are encouraged. However, in studies on certain conditions, such as pain, only subjective measurement can be used. Because most of the trials we reviewed were on chronic pain, subjective measurements were the most common (see Table 4). The researchers often asked patients to use the visual analog scale (VAS) or numerical rating scales (NRS) at baseline and in post-intervention assessments to rate the intensity of their pain. Although these are standard and validated instruments, self-reported pain intensity can vary greatly from patient to patient, and a large sample size is needed to address this variation.

Table 4.

Different Types of Primary Outcomes Reported in Results

Primary outcomes	Efficacy studies	Effectiveness studies	Unspecified studies	Total
Subjective	27 (60%)	26 (60%)	8 (42%)	61 (57%)
Objective	9 (20%)	9 (20%)	7 (37%)	25 (23%)
Both Subjective and Objective	9 (20%)	9 (20%)	4 (21%)	22 (20%)
Total	45 (100%)	44 (100%)	19 (100%)	108 (100%)

Some of the reviewed reports used questionnaires to obtain outcome measurements, some standard and widely used, others self-designed. However, only a few authors discussed the reliability and validity of those questionnaires: 13 (29%) in efficacy studies, 14 (32%) in effectiveness studies and 2 (11%) in unspecified studies. We suggest that the reliability and validity of outcome measurement instruments should be discussed in research reports, although in the case of widely used instruments, reference citations are sufficient.

Part II. Issues of Statistical Analysis

1. Sample Size and Power Calculation

Sample size and power calculations are necessary when designing trials and planning for resource utilization. These were not well reported in the reviewed studies. A total of 20 (44%) efficacy, 12 (27%) effectiveness, and 9 (47%) unspecified studies did not mention them at all. Calculating sample size requires mean pain score, standard deviation, and effect size. In all, 59 (55%) reports gave the effect size, and 23 (21%) described the standard deviation.

Sample size calculations are approximate because the calculation equations themselves are often based on approximations to the exact statistical distributions, and some of the parameters in the equation depend on guess work [125]. Some of the studies could not achieve the original power expected because of patients who dropped out or were lost to follow-up. Only 22 (20%) trials accounted for such problems and allotted 10% to 30% more participants to compensate for possible dropouts or missing data.

2. Dropouts and Missing Data

Dropout reasons varied and included moving out of town, intolerance to adverse effects, and feeling worse after treatment. Some of these would lead to biased results if ignored by the analyst. A total of 22 (49%) efficacy, 26 (59%) effectiveness, and 12 (63%) unspecified studies gave detailed descriptions of their dropouts, including number of dropouts and time and reason for dropping out. Characteristics or baseline values between dropouts and remaining patients were compared in 5 (11%) efficacy, 2 (5%) effectiveness, and 2 (11%) unspecified studies. If a comparison test shows a significant result, then the missing data in the dropouts are not missing completely at random (MCAR). MCAR is one of the three main mechanisms for characterizing missing data, missing at random (MAR) and missing not at random (MNAR) being the other two. If comparison shows that the result is not significant, the missing data still may not be MCAR, since the samples may differ in some unmeasured characteristics, but the comparison does serve as an informal check when determining the best method for dealing with dropouts.

In reporting a trial, it is essential to disclose any missing data and state how they were handled. In the reviewed studies, the most popular methods of dealing with missing data problems were replacing missing values with baseline data and ‘last observation carried forward’ (LOCF). Six trials combined two or more methods, such as LOCF plus multiple imputation or LOCF plus replacement by mean scores obtained. Table 5 shows the methods used. Although using LOCF, mean score, or baseline score to replace a missing value is convenient and easy to handle, those methods are neither efficient nor valid. Since more powerful statistical methods for dealing with missing data, for example multiple imputations and weighted estimating equations, have been developed, we suggest using more sophisticated ways to handle this problem.

Table 5.

Methods for dealing with missing data

Methods	Efficacy Study	Effectiveness Study	Other Study	Total
Last observation carried forward	11 (65%)	13 (42%)	2 (67%)	26 (51%)
Replaced with average available data on either side of missing value	0	1 (3%)	0	1 (2%)
Replaced with mean values obtained	1 (6%)	2 (7%)	0	3 (6%)
Replaced with baseline data	1 (6%)	5 (16%)	1 (33%)	7 (13%)
Patients with missing data counted as failures	1 (6%)	1 (3%)	0	2 (4%)
Worst case method for experimental group and best case method for control group	1 (6%)	1 (3%)	0	2 (4%)
Multiple imputation	2 (11%)	8 (26%)	0	10 (20%)
Total	17 (100%)	31 (100%)	3 (100%)	51 (100%)

3. Issues in Multiple Comparison

Acupuncture studies may have multiple arms. For example, a design may compare a true acupuncture group to several different sham acupuncture groups, or true and sham acupuncture to conventional therapy. In such cases multiple comparison tests should be used to avoid the potential increase of type I error that occurs when statistical tests are used repeatedly. Table 6 showed that multiple comparison tests were not always conducted when a trial consisted of more than two groups that should have been compared to each other. Instead, some used t test to do pairwise comparisons, which may lead to faulty conclusions. Multiple comparison methods used in these studies include the Bonferroni correction test, a closed testing procedure, Scheffe’s test, and a priori-ordered two-sided null hypotheses using Student’s t test.

Table 6.

Multiple comparisons

	Efficacy study	Effectiveness study	Other study	Total
More than two groups being compared	16	10	5	31
Multiple comparison used	8 (50%)	8 (80%)	1 (25%)	17 (55%)
Multiple comparison not used	8 (50%)	2 (20%)	4 (75%)	14 (45%)
Bonferroni test used	3 (19%)	4 (40%)	0	7 (23%)

4. Controlling Baseline Measurements

The reporting of baseline demographic and clinical characteristics of each group is one of the requirements of the CONSORT (Consolidated Standards of Reporting Trials) statement [126]. Knowing baseline characteristics allows the reader to assess how closely matched the patients are and to judge the validity of the trials. In studies where important baseline factors appear well balanced, it is likely that any differences in outcome between the intervention and control groups are a real effect of treatment. Although most acupuncture clinical trials include randomization and some baseline comparisons between groups, certain unknown and important characteristics may be unbalanced in the baseline measurements. Thus Vickers [127] agrees that using co-variance (ANCOVA) to adjust for baseline scores is the preferred method of analysis in most acupuncture pain trials. He described four trials in which less powerful, unadjusted methods were used and reanalyzed them with ANCOVA. Reanalysis produced different conclusions. Therefore, in addition to comparing baseline characteristics, it is also necessary to use efficient statistical methods such as ANCOVA and regression to control potentially unbalanced baseline values between groups and to achieve more powerful conclusions. Besides ANCOVA, some studies reviewed also used multiple linear regressions, multivariate longitudinal analysis, or the linear mixed-effect model to ensure that baseline scores be well controlled and adjusted.

5. Issues in Repeated-Measure Analysis

In acupuncture trials, researchers would like to see long-term effects as well as short-term ones. This means that follow-up is done at intervals after treatment. Data collected in this way is called repeated-measures data or longitudinal data and involves measurements made on the same subject at different time points. Measurements made on the same variable for the same subject are likely to be correlated. Statistical models appropriate for longitudinal or repeated-measures data involve estimation of covariance parameters to capture correlations [128]. However, of those reports we reviewed, less than half made the proper analysis of repeated-measure data, using instead simple t or chi square tests to compare differences between groups. This neglected the aspect of time-treatment interaction and lowered the statistical efficiency.

Of the 72 trials with repeated-measure data, 19 (50%) efficacy, 8 (29%) effectiveness, and 3 (50%) unspecified studies considered repeated-measure problems and applied the corresponding analysis. Methods used to deal with repeated-measure data in these studies included repeated-measure ANOVA, multivariate repeated-measure ANOVA (MANOVA), the linear mixed model, and the GEE model. Repeated-measure ANOVA is the traditional way to manage longitudinal data. It also was the most popular in those reports we reviewed and was used in 63% of the efficacy, 63% of the effectiveness, and 67% of the unspecified studies. However, recent research suggests that the linear mixed model is a more efficient method to manage such data. This model allows users to construct designs with a broad selection of covariance structures, thus offering greater efficiency than the multivariate normal distribution model and more flexibility than models assuming sphericity. Additionally, the linear mixed model also allows subjects to have missing time points, while software for traditional repeated-measures ANOVA drops a subject from analysis if it has missing data at a single time point. Therefore we suggest that researchers consider more sophisticated and efficient methods such as the linear mixed-model to handle repeated-measure data.

6. Issues in Analysis of Cluster Data

In the acupuncture trials reviewed, patients were collected from several different clinic centers or areas, and patients in certain acupuncture groups were treated by several different acupuncturists. Patient data actually may be grouped, or sorted into clusters. The major issue in the analysis of cluster data is that observations within a cluster are not independent. The degree of similarity is typically measured by the intracluster correlation coefficient (ICC) [129]. Ignoring intracluster correlation could lead to incorrect P values, biased estimates, and very small confidence intervals, all of which could cause incorrect interpretation of associations among variables. The cluster effect is a combination of both intracluster correlation and cluster size. Small intracluster correlations coupled with large cluster size can affect the validity of conventional statistical analyses. Under such circumstances, special analysis of cluster data is essential. However, few researchers in the studies we reviewed addressed this problem. Of the 108 reports, 57 had cluster data involving either patients from multiple clinic centers, patients of a single group being treated by several physicians, or both. But cluster data analysis was presented in only 9 (16%).

Methods used to deal with cluster data are generally called multilevel analyses and are intended to explore, in greater depth, interrelationships among variables at various levels, such as clinic level, physician level, and so forth. For example, in a data set with patient and physician clusters, a comprehensive multilevel data analysis can directly assess the effects of patient- and physician-level variables on outcome and can determine whether physician-level variables moderate patient-level relationships by examining cross-level interactions between variables from both levels. In the 9 trials reporting cluster analysis, methods used included the random-effect regression model, the linear mixed model, and the GEE model.

7. Confounding Issues

In pain-related trials such as those of head pain, low back pain, and osteoarthritis, patients are usually allowed to take analgesics before, during, and after treatment and during the follow-up period. The taking of analgesics and the amount of intake are important variables which can affect treatment effect assessment greatly, and they also can be potential confounding factors for acupuncture effect analyses. For example, if patients in a true acupuncture group who seldom take analgesics have almost the same pain evaluation scores after treatment as those in a sham group taking analgesics often and in high dosages, how can investigators determine whether treatment effects are due to the acupuncture or the analgesics? Even when researchers in our sample compared analgesic usage situation between or among groups, they did not control for analgesic usage in primary outcome measures, which may invalidate their conclusions. Using advanced statistical techniques such as multivariate analysis and making analgesics usage a covariant can control for this possibly confounding effect.

Discussion

The recently published acupuncture trials illustrate the wide range of applications for which acupuncture is being used. Trials aiming to evaluate the efficacy of acupuncture are little better in quality and credibility than the effectiveness and unspecified studies, although the efficacy studies should have been rigorously designed and conducted. Good RCT design involves many aspects. Here we covered only the most common and essential of these, drawing from examples of current acupuncture RCTs, which point up the fact that much more attention needs to be given to the basics of trial design.

Acupuncture RCT design has its controversies. To avoid bias in patient selection, patient source and inclusion and exclusion criteria should be carefully considered and determined based on research purpose. Some researchers suggest that patient selection should include the complementary and alternative medicine diagnosis, not just the Western medicine diagnosis, if trials are to demonstrate more specific and direct treatment effects. Controversies regarding placebo control and effect in acupuncture RCTs are also widespread. Some researchers recommend that if a non-insertion sham is used, thorough control procedures and methods should be applied to ensure that the sham is really inert [122]. They insist that acupuncture trials should not be labeled “placebo-controlled” unless the investigator can present evidence that the sham procedure is in fact inert. Since we see an increasing use of placebo-controlled studies for acupuncture, better and verifiable controls are needed, as are more sophisticated and rigorous research models and methods for trials of complex acupuncture interventions.

In order to improve the research quality and obtain convincing and scientific results, comprehensive and rigorous methodological designs are not enough. Appropriate and well-considered methods of statistical analysis are also necessary. With the rapid developments in statistical methodology, the linear mixed model is now thought to be more efficient than ANOVA/MANOVA methods for handling repeated-measure and longitudinal data. Especially in a clinical trial in which data is unbalanced due to mistimed measurement or missing data, or which includes data with time-varying and time-invariant covariates, the linear mixed-effect model is much more flexible and parsimonious [134]. Moreover, when applying sophisticated statistical methods, we must keep scope of application in mind and be aware that not all methods are suitable in a given trial. For example, Vickers [127] mentions that ANCOVA should be used only if regression slopes are parallel and data are either normal and homoscedastic or sample sizes are large (n=25 or more per group), because ANCOVA depends on assumptions of normality of distribution and homoscedasticity, or constant variance across observations. The linear mixed-effect model is recommended for handling longitudinal data with continuous responses that are at least approximately normally distributed. However, when responses are discrete, linear models are not appropriate for relating changes in mean response to covariates. Statisticians have developed other methods based on the extended generalized linear model to handle discrete longitudinal data. These include the GEE model, the generalized mixed-effect model, and transition or response-conditioned models. Multiple imputation (MI) is considered one of the best ways to handle missing data and has been widely used. But researchers should also be aware that such factors as missing mechanism (ignorable missing or non-ignorable missing), and data type are key when choosing a suitable MI method.

We would like to emphasize that the quality of acupuncture trials has greatly improved during the last 30 years. Because of rapid developments in biostatistics, many statistical methods discussed in this paper were probably adequate at the time of these trials, and more recent advancements in statistical methods may not have been available to the researchers. The purpose of our paper is not to criticize past work, but to introduce newer and more powerful analytic methods in the hope that these will be useful for future studies.

Conclusion

As under current conditions there are no better or more suitable methods for testing the effects of acupuncture than RCTs, we must conduct rigorous RCTs that yield credible results. Although the quality of acupuncture research has greatly improved, due to the specificity and complexity of acupuncture research not all standards generally applied in the RCT are suitable for acupuncture trials, and not all researchers apply the basic standards of the RCT to their acupuncture studies. Issues such as how to choose a suitable control group and whether a valid placebo effect can be achieved are controversial. Ambiguity and uncertainty in such issues are obstacles for researchers who want to improve the quality of their research. Thus there is an urgent need to develop more valid and precise designs. However, with the development of sophisticated statistical methods, appropriate and efficient analysis can now not only assure accurate results, but also may help researchers overcome difficulties and dilemmas in design.

Hopefully, shedding light on the issues discussed here can inspire researchers to improve the design, implementation, analysis, and reporting of acupuncture RCTs. Only improvements in quality and credibility in acupuncture research can lead to better and clearer interpretations and conclusions regarding acupuncture’s effects, thus providing good studies for systematic reviews that can present “the best available evidence” to clinicians, health policymakers, and consumers on this important modality.