Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study

Abstract

Background

Clinical trials on Traditional Chinese Medicine (TCM) should be registered in a publicly accessible international trial register and report on all outcomes. We systematically assessed and evaluated TCM trials in registries with their subsequent publications.

Objective

To describe the characteristics of TCM trials, estimate bias risk and outcome-reporting bias in clinical trials.

Data sources and study selection

Fifteen trial registries were searched from their inception to July 2012 to identify randomised trials on TCM including Chinese herbs, acupuncture and/or moxibustion, cupping, tuina, qigong, etc.

Data extraction

We extracted data including TCM specialty and treated disease/conditions from the registries and searched for subsequent publications in PubMed and Chinese databases. We compared information in the registries of completed trials with any publications focusing on study design, sample size, randomisation, bias risk including reporting bias from the register protocol.

Results

1096 registered randomised trials were identified evaluating TCM, of which 505 were completed studies (46.1%). The most frequent conditions were pain (13.3%), musculoskeletal (11.7%), nervous (8.7%), digestive (7.1%), circulatory (6.5%), respiratory (6.3%), mental and behavioural disorders (6.2%) and cancer (6.0%). The trial register data identified parallel, phase II/III randomised trials with sample size estimations and blinding, but limited information about randomisation (sequence generation and allocation concealment). Comparing trial registration data of 115 completed trials (22.8%) with their subsequent 136 publications, inconsistencies were identified in one or more of the following: sample size (11%), outcome assessor blinding (37.5%), primary outcomes (29%) and safety (28%) reporting.

Conclusions

Increasing numbers of clinical trials investigating a variety of TCM interventions have been registered in international trial registries. The study design of registered TCM trials has improved in estimating sample size, use of blinding and placebos. However, selective outcome reporting is widespread and similar to conventional medicine and therefore study conclusions should be interpreted with caution.

Article summary.

Article focus

• We wished to evaluate the methodological quality of clinical trials in Traditional Chinese Medicine (TCM).

• We investigated whether the systematic identification of prior trial registration was associated with an improvement in the methodological quality of the subsequent published studies.

Key messages

• A substantial number of clinical trials in TCM, covering a broad range of therapies, are now being registered in international trial registries.

• Registration is associated with more rigorous study methodology and study design (eg, randomisation protocols, secure blinding and sample size estimates).

• Outcome-reporting bias exists when comparing the registry information and the subsequent publications, and some trials were registered after their publication.

Strengths and limitations of this study

• Systematic searches of all available international trial registries for any clinical trials of TCM.

• All interventions involving any TCM were included as was the diagnosis.

• The registered information for clinical trials is not uniform across the registries and important methodological information may be missing.

• Subsequent publications were obtained for those studies recorded as ‘completed’ in the registry. This may not represent the true situation for trials if the registry data are not updated by the researchers.

Many empirical studies have shown that the methodological quality of randomised clinical trials of Traditional Chinese Medicine (TCM) is poor with respect to risk of systematic errors (bias; generation of allocation sequence, allocation concealment, blinding, descriptions of drop-out or losses to follow-up, selective outcome reporting) and risk of random errors (play of chance).^1–6 Moreover, publications on TCM trials are uniformly positive, raising concerns that trials initiated to investigate TCM are only published if they have positive results.⁷ Poor-quality trials and risk of publication bias will reduce the strength of evidence when developing clinical practice guidelines or preparing systematic reviews. One of the ways to improve trial quality is to prospectively register clinical trial protocols in international trial registries such as clinicaltrials.gov.^{8 9} The WHO established an international clinical trial registry platform (ICTRP; http://www.who.int/ictrp/en/) in 2005, which now links 14 clinical trial registries. Furthermore, several peer-reviewed journals such as The Lancet and Trials publish trial protocols to promote transparency and improve trial quality.

In order to describe the characteristics of TCM trials, and estimate reporting bias in clinical trials, we systematically searched 15 major international trial registries to identify information about TCM trials, and compared the registered records with subsequent publications regarding outcomes and other data.

Methods

Inclusion criteria

We included randomised clinical trials for any TCM intervention singly or their combination: Chinese herbs, acupuncture, acupressure, moxibustion, cupping, dietary advice, tuina (therapeutic massage), taichi, qigong and guasha (scaping massage). We excluded non-randomised studies such as quasirandomised studies, cohort studies, phase 1 trials, retrospective clinical studies, case series or case studies. There were no limitations on study type (superiority, non-inferiority or equivalence) or study phase.

Data source

We systematically searched 15 major international trial registries (14 linked to WHO ICTRP) from their inception to July 2012 (see online supplement 01).

For trials listed as ‘completed’ in the registered records, we then searched for published protocols, as well as the full texts of subsequently published articles in PubMed and three Chinese electronic bibliographic databases including Chinese Biomedical Database (http://sinomed.imicams.ac.cn/index.jsp), China National Knowledge Infrastructure (http://www.cnki.net) and Chinese VIP Information (http://vip.hbdlib.cn/index.asp).

Data extraction

Two researchers (from MH, X-XL, Y-JM, Y-YW or G-YY) extracted data independently from each trial registry using a standardised, piloted data extraction form. The form was developed by our research group and based on general characteristics of clinical trials, methodology and the 20 minimum items required for WHO trial registration.¹⁰ The main information collected included the number of trials in each registry, year registered, trial design, methods, sample size, setting, participants and diseases/conditions, differentiation of syndrome (bian zheng lun zhi), interventions, controls, primary and secondary outcomes, inclusions and exclusion criteria, current status (eg, completed and ongoing), ethical approval, sponsors, institutions, country of origin, contact details and funding.¹¹ The extracted data were cross checked by the authors, and any discrepancy resolved by discussion with a third author (J-PL). Conditions were classified according to the WHO international classification of diseases (http://apps.who.int/classifications/icd10/browse/2010/en; last accessed 31 July 2012). We searched for publications in bibliographic databases for those trials listed as ‘completed’ in the registries, and compared the published trials with the registered records.

Data analyses

We used The Cochrane Collaboration risk of bias tool to evaluate the registered records.^4–6 This tool assesses the following domains: generation of allocation sequence; allocation concealment; blinding; incomplete outcome reporting; and selective outcome reporting (defined as change of primary outcome or new outcome added).⁶ We also evaluated the estimated sample size, explicit inclusion and exclusion criteria and the risk of funding bias.

Two authors (MH and X-XL) compared the trial design and methodology from the registered records with the resulting publications to analyse the consistency and selective outcome reporting. Selective outcome reporting was defined as “when the full paper publications reported different primary outcomes or changed it from the original pre-defined primary outcome in the registered data.” Inconsistency was defined as “when the items were not the same as described in the registry in the subsequent paper publications.” For the sample size estimation, a discrepancy of over 20% between the registered and published information was judged as inconsistent. We judged trials as positive results based on (1) authors’ conclusion showing that the intervention was superior to the control; or (2) the comparison of between groups which showed statistically significant differences (p<0.05) for primary outcome measures. We compared the dates of registration and full paper publication to assess the proportion of retrospective registrations.¹²

Results

We identified 1096 registered records of randomised clinical trials on TCM. Five hundred and five of the 1096 (46.1%) registered records indicated that the trial had been completed. Three hundred and thirty-seven of 1096 (30.7%) were from mainland China (excluding Hong Kong and Macau) (figure 1). The first trial was registered in 1999, in clinicaltrials.gov, and the number of registered trials has increased over the past decade (figure 2).

Figure 1.

Flow diagram of included trials in this study.

Figure 2.

Number of registered randomised trials on Traditional Chinese Medicine by registry.

Table 1 shows the frequency of the type of TCM intervention included in each registry. The combined therapies studied included 105 trials on integrating Chinese and Western medicine, and 38 trials combining two or more TCM therapies. Using the international disease classification (ICD-10), we identified the 10 most frequent conditions: pain (13.3%) and musculoskeletal (11.7%), nervous (8.7%), digestive (7.1%), circulatory system (6.5%), respiratory conditions (6.3%), mental and behavioural disorders (6.2%), as well as cancer (6%), endocrine, nutritional and metabolic diseases (5.7%) and pregnancy and childbirth (4.6%; table 2).

Table 1.

Numbers of registered randomised trials on Traditional Chinese Medicine by registry and country

Therapy	US registry	Chi-CTR/China	ANZCTR/Australia/New Zealand	ISRCTN/UK	IRCT/Iran	JPRN/Japan	CRIS/Korea	DRKS/Germany	REBEC/Brazil	EU-CTR/EU	NTR/Netherlands	Total
Acupuncture	244	35	34	77	19	11	8	4	4	2	2	440
Chinese herbal medicine	138	82	46	10	0	11	2	1	0	1	0	291
Combination therapy	46	76	8	4	0	0	1	0	0	0	0	135
Acupressure	34	3	2	7	21	2	0	0	0	0	0	69
Tuina	46	0	9	3	2	3	0	2	0	0	0	65
Taichi	46	1	4	0	0	0	0	0	0	0	0	51
Qigong	11	0	0	3	0	0	1	0	0	0	0	15
Moxibustion	3	8	0	0	0	0	0	0	0	0	0	11
Cupping	6	0	0	0	4	0	1	0	0	0	0	11
Gua sha	1	0	0	0	0	0	0	0	0	0	0	1
Other therapy	4	3	0	0	0	0	0	0	0	0	0	7
Total	579	208	103	104	46	28	13	7	4	3	2	1096

Chinese herbal medicine included practitioner-prescribed herbal formula, Chinese patent medicine and herbal extracts.

Combination therapy refers to two or more Chinese medicine therapies combined together, or Chinese medicine integrated with conventional therapy.

Other therapy included acupoint injection (3), acupoint herbal pasters (2), acupoint embedding therapy (1) and traditional Chinese medicine five-element music therapy (1).

Table 2.

Conditions based on ICD-10 classification treated by Traditional Chinese Medicine

Disease/conditions (ICD-10 codes)	Number of RCTs	Proportion (%)
Pain	146	13.3
M00-M99 Diseases of the musculoskeletal system and connective tissue	128	11.7
G00-G99 Diseases of the nervous system	95	8.7
K00-K93 Diseases of the digestive system	78	7.1
I00-I99 Diseases of the circulatory system	71	6.5
J00-J99 Diseases of the respiratory system	69	6.3
F00-F99 Mental and behavioural disorders	68	6.2
C00-D48 Neoplasms	66	6.0
E00-E90 Endocrine, nutritional and metabolic diseases	63	5.7
O00-O99 Pregnancy, childbirth and the puerperium	50	4.6
Z00-Z99 Factors influencing health status and contact with health services	41	3.7
Side effects of chemotherapy for cancer	34	3.1
R00-R99 Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified	32	2.9
N00-N99 Diseases of the genitourinary system	32	2.9
S00-T98 Injury, poisoning and certain other consequences of external causes	28	2.6
L00-L99 Diseases of the skin and subcutaneous tissue	23	2.1
A00-B99 Certain infectious and parasitic diseases	22	2.0
D50-D89 Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism	17	1.6
H00-H59 Diseases of the eye and adnexa	15	1.4
P00-P96 Certain conditions originating in the perinatal period	9	0.8
H60-H95 Diseases of the ear and mastoid process	8	0.7
Q00-Q99 Congenital malformations, deformations and chromosomal abnormalities	1	0.09
Total	1096	100

ICD, International Classification of Diseases; RCTs: randomised controlled trials.

There were methodological variations across TCM interventions (table 3). Our analysis of 1096 registered records showed that the majority were phase II/III (332/474, 70%) with 1024 (93.4%) using parallel groups design. One thousand and seventy-six (98.2%) included a sample size estimation, 714 (65.2%) reported that participants, personnel and/or outcome assessors were blinded, and 770 (70.3%) were two-armed (table 3). The reporting of control group varied across interventions. For example, none of the trials of tuina or qigong reported the use of a placebo/sham control, while a sham control was used in almost half of the acupuncture trials (44.8%). Other commonly used controls were western medicine (214, 19.5%), no intervention (181, 16.5%), non-pharmaceutical interventions (133, 12.1%), acupuncture (70, 6.4%) and Chinese herbal medicine (52, 4.7%). In addition, our data extraction of TCM syndrome differentiation showed that 65 of 290 (22.4%) trials on Chinese herbal medicine utilised syndrome differentiation either in the recruitment of participants (54 trials) or in the prescription of the herbal formula (11 trials). All these trials except one from Australia were registered by institutions in mainland China.

Table 3.

Methodological information of 1096 RCTs on Traditional Chinese Medicine from International Trial Registries

Items|	CHM N=290 (%)	Acupuncture/ acupressure N=509 (%)	Qigong/Taichi N=66 (%)	Tuina N=65 (%)	Other therapies N=166 (%)	Total
Design
Parallel	269 (92.8)	477 (93.7)	62 (93.9)	54 (83.1)	162 (97.6)	1024
Crossover	17 (5.9)	24 (4.7)	3 (4.6)	9 (13.9)	2 (1.2)	55
Phase II/III*	120/172 (69.8)	141/186 (75.8)	20/24 (83.3)	21/24 (87.5)	30/68 (44.1)	332
Placebo/sham-controlled	194 (66.9)	228 (44.8)	0 (0.00)	0 (0.00)	41 (24.7)	463
Sample size estimation	287 (99.0)	496 (97.5)	64 (97.0)	65 (100.0)	164 (98.8)	1076
Generation of allocation sequence	114 (39.3)	63 (12.4)	8 (12.1)	8 (12.3)	85 (51.2)	278
Allocation concealment	47 (16.2)	34 (6.7)	4 (6.1)	8 (12.3)	11 (6.6)	104
Blinding	219 (75.5)	346 (68.0)	40 (60.6)	38 (58.5)	71 (42.8)	714
TCM syndrome differentiation	65 (22.4)†

*Only 474 trials (43.2%) provided information about phase.

Blinding included participants, personnel and/or outcome assessors blinded.

†TCM syndrome differentiation (bian zheng lun zhi): data limited to CHM.

CHM: Chinese herbal medicine; RCTs, randomised clinical trials; TCM, Traditional Chinese Medicine.

The information about randomisation procedures, including the generation of allocation sequence or concealment, was under-registered and under-reported across all intervention types. The estimated sample size of the trials ranged from less than 10 (mostly pilot/feasibility trials) to over 1000 participants, with the majority (88.3%) of sample sizes between 20 and 500/trial (see online supplement 02).

Five hundred and five (46.1%) of the registered trials indicated their status as ‘completed’. Nineteen of these 505 trials (3.8%) had protocol publications. Our searches in PubMed and the Chinese databases identified full paper publications for 115 from the 505 completed trials (22.8%). The 115 trials produced 136 publications reporting study outcomes, among which 123 publications were in English and 13 in Chinese. Among the 115 trials with publications, 53.9% (62/115) trials specified primary outcomes (table 4). When comparing the 115 registered trials with their 136 publications, inconsistency was identified in sample size estimation (11%), outcome assessor blinding (37.5%), secondary outcome (34.4%) and safety reporting (28.6%; table 4). Selective outcome reporting was found in 29.2% (19/65) of the subsequent publications when comparing the stated primary outcomes in the registries. Incomplete outcome data were addressed in 34 trial publications (25%). Of these, 80.9% (110/136) of the trial publications reported positive results.

Table 4.

Comparison of methodological components between registered records and subsequent publications of RCTs

Items	Records n=115 (%)	Publication n=136 (%)	Inconsistency rate (%)
Sample size	114 (99.1%)	136 (100%)	17.6%
Generation of allocation sequence	3 (2.6%)	80 (58.8%)	66.7%
Allocation concealment	0 (0.0%)	61 (44.9%)	NA
Participant blinded	43 (37.4%)	66 (48.5%)	17.7%
Practitioner blinded	18 (15.7%)	31 (22.8%)	11.1%
Outcome assessor blinded	30 (26.1%)	54 (39.7%)	37.5%
Primary outcome	62 (53.9%)	114 (83.8%)	29.2%
Secondary outcome	52 (45.2%)	96 (70.6%)	34.4%
Safety	28 (24.4%)	58 (42.7%)	28.6%

NA: not available; RCTs, randomised clinical trials.

Among the 115 trials reporting outcomes, six trials showed other deviations from their original protocols. For example, one study was registered as prospective cohort study, but published as randomised trial; one trial was registered as parallel group, but published as a cross-over trial; one trial used different intervention from the registered record; and three trials showed inconsistent intervention arms from the registered information.

To understand the adequacy of the trial registration, we compared the date of publications with the date of registration and we found that 11 (9.6%) trials were registered later than the publication date, suggesting retrospective registration. Furthermore, in 41 trials (35.7%), the completion date for the trials was earlier than the approval date of the registration, also suggesting inappropriate registration.

Discussion

The aim of prospective registration of clinical trials is to create transparency about the research methodology, and allow access to the trial protocol before the completion of the trial. Researchers and systematic reviewers can then compare the registered information or published protocols and their subsequent publications to assess publication bias or selective outcome reporting.^{13 14}

We observed a significant increase in the number of registered TCM trials in international clinical trial registries over the past decade. Our evaluation of trial quality covers a wide variety of TCM interventions from acupuncture and Chinese herbal medicine to some very ancient interventions such as cupping and gua sha therapy. These studies mainly address chronic and long-term conditions such as pain, musculoskeletal and neurological problems. TCM trial design in terms of sample size estimation, use of placebo-control and blinding and the definition of primary and secondary outcome measures is improving according to the information held in the registers.¹⁵ However, we found that the generation of allocation sequence and concealment were under-reported in the registered trials. This might be because these two items were not mandatory for the majority of registries. Interestingly, among 291 trials on Chinese herbal medicine, less than a quarter of the trials (n=65) utilised syndrome differentiation, which is considered vital to the TCM diagnosis as the basis for classifying subtypes of participants and for the prescription of herbal formula. Almost all of these trials with syndrome differentiation were registered by institutions in China, which suggest that Chinese researchers pay more attention to the selection of optimal participants and tailored treatment based on different TCM syndromes.

We found that 29% of registered TCM trials showed selective outcome-reporting bias (discrepancies between the outcomes registered and the outcomes published). The inconsistency between the trial protocols and subsequent publications also relates to sample size, blinding, primary and secondary outcomes as well as safety. This implies that deviations from protocols might be due to the study findings which then resulted in selective outcome reporting. A previous study demonstrated that 31% of trials in Western medicine showed discrepancies between the registered protocols and the outcomes published in high-impact general medical journals,¹³ which suggest that TCM researchers are not alone in reporting outcomes selectively. Our data also suggest that there might be inappropriate registration in about half of the trials. Accordingly, there is still ample space to improve the quality of trial registration.¹⁶ Clinicians and policymakers need unbiased results from clinical trials to make informed clinical decisions.

Our study has some limitations. First, due to the lack of standardisation of the items required for registration in different registries some important information, such as randomisation methods, may be under-reported. Second, we only searched for publications for those trials indicating a ‘completed’ status. We do not know how often this information is updated in different registries. Therefore, there may be trials in the registries that are not listed as ‘completed’, but that have been completed. This may cause bias, particularly if the investigators who update the status of their trials on the registry are also more likely to adhere to the methods described in the registry when writing up their results in the subsequent publications. Third, as we undertook searches in PubMed and three Chinese databases only, we may have missed some studies that have been reported in other databases. In addition, there is a lag time between completing a study and writing for publication, submission and peer review, all of which can be considerable. Fourth, we only looked at registered trials. A very large number of TCM trials are conducted without being registered, and here we can say nothing about their risks of bias and risks of random errors. In all likelihood, these may be even worse than those we have observed.

We conclude that the study design and the quality of reporting of TCM trials have improved through prospective international trial registration compared with previous methodological studies,^1–6 although there are some inconsistencies between the registered trial protocols and subsequent publications and insufficient reporting on syndrome differentiation. Publication bias as a consequence of selective outcome reporting is still widespread and similar to conventional medicine, therefore study conclusions should be interpreted with caution. In herbal medicine trials, it would be inappropriate if a trial design does not utilise syndrome differentiation, and participants may not be properly treated.