Cochrane systematic reviews on traditional Chinese medicine: What matters–the quantity or quality of evidence?

Abstract

Background:

Systematic reviews on traditional Chinese medicine (TCM) are constantly increasing. However, if these reviews are to be of practical value, the evidence needs to be relevant, valid, and adequately reported. Cochrane Systematic Reviews (CSRs) are considered as high-quality systematic reviews that can inform health care decision making. Our aim was to provide an overview of the scope, findings, quality and impact of CSRs on the benefits and harms associated with TCM interventions for the treatment and prevention of disease to provide new information for clinical practice and future research.

Methods:

The Cochrane Database of Systematic Reviews was searched up to May 2021, and descriptive characteristics were extracted. The correspondence between the questions asked in the CSRs and the available evidence, conclusions and certainty of findings (according to GRADE assessment), methodological quality (AMSTAR 2), and impact (Altmetric Attention Score [AAS], total citations by guideline, and total citations in Web of Science [WoS]) of CSRs were extracted. Tabular and graphical summaries of these descriptive characteristics were constructed.

Results:

Of 104 CSRs on TCM identified, 70 diseases belonged to 16 disease systems and contained 1642 primary studies with 157,943 participants. Interventions included Chinese herbal medicine (n = 70), acupuncture (n = 28), TCM exercises (n = 4), and moxibustion (n = 2). Among 1642 primary studies, 662 studies included an intervention group treated with at least one TCM therapy and 980 studies included a combination of therapies. Promising outcomes from the 104 CSRs were divided into endpoint outcomes (34 diseases), doctor- or patient-reported outcomes (27 diseases), and surrogate outcomes (37 diseases). Despite the presence of promising outcomes, only 5/104 CSRs drew overall positive conclusions, 42 CSRs concluded the evidence was insufficient, and 54 failed to draw firm conclusions. GRADE assessments were reported in 41.3% of the CSRs, and the ratings were mostly low or very low. Comparing the questions asked and results obtained, there was frequently a lack of information about specific outcomes. Only 16 CSRs obtained results for all outcomes listed in the methods section. According to AMSTAR 2, 51 CSRs (49.0%) were of low quality. The total number of citations in the WoS was 2135 (mean ± SD: 20.8 ± 21.2), and 38.5% of the CSRs had been cited in guidelines 95 times.

Conclusion:

Although TCM is commonly used, evidence of its effectiveness remains largely inconclusive. Rigorous high-quality trials are needed to support the performance of high-quality reviews and to increase the evidence base. It is critical to emphasize quality over quantity in future TCM research.

Introduction

Traditional Chinese medicine (TCM), a unique diagnostic and treatment system consists of different types of interventions including; the use of Chinese herbal medicine (CHM), acupuncture, massage and therapeutic exercises. TCM interventions are described in the UNESCO Representative List of the Intangible Cultural Heritage of Humanity (UNESCO, 2020). China’s population of 1.4 billion is currently served by the parallel medical systems of TCM and Western medicine (WM). The latest statistics show that the use of TCM has spread to 183 countries and regions, and there are more than 80,000 TCM clinics outside of China (Chang et al., 2020). Research on the clinical effectiveness of TCM has increased year on year (Consentino et al., 2018). The earliest randomized controlled trial (RCT) of CHM was published in 1982 (Chen et al., 1982), and the earliest RCT of acupuncture according to PubMed was published in 1975 (Gaw et al., 1975). High-quality TCM clinical studies have been published in internationally renowned journals, such as JAMA (Cardini and Weixin, 1998; Liu et al., 2017), Annals of Internal Medicine (Liu et al., 2016; Wang et al., 2011) and The New England Journal of Medicine (Li et al., 2012; Wang et al., 2010). However, the use of TCM still faces many difficulties and opposition for a variety of reasons, including the difference in cultural approach, the philosophy underlying TCM, questions about the quality of TCM studies and the evidence base (Cyranoski, 2020).

Cochrane Systematic Reviews (CSRs), which are published in the Cochrane Database of Systematic Reviews (CDSR) in the Cochrane Library, attempt to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question (CDSR, 2021). CSRs are aimed at minimizing bias to produce more reliable findings that can be used to inform decision making. CSRs often cover critical public health issues, and their findings serve as the foundation for many national and international guidelines (WHO, 2015, 2017). A comparison between CSRs and reviews published in paper-based journals showed that CSRs appeared to be more rigorous and more frequently updated (Jadad et al., 1998). China attaches great importance and significance to the Cochrane Collaboration. At the beginning of 2021, the Cochrane China Network was officially established, with nine participating institutions, including one specializing in TCM (Cochrane China Network, 2021).

Since the publication of the first CSR (McCarney et al., 2003) in the TCM field in 1998, many TCM-related CSRs have been published. Since 2009, three overviews have summarized TCM-related CSRs (Hu et al., 2011; Ji et al., 2020; Manheimer et al., 2009), two of which were published by some of the authors of the current overview. However, two overviews analyzed only the methodological quality of the CSRs, and the third was published more than 10 years ago. Over the last 25 years, the volume of evidence has increased, and there have also been improvements in TCM research methodology. The reasons for this increase are as follows: first, in recent years, researchers have attached more importance to the application of evidence to clinical practice guidelines and principles (Professional Committee on Evidence-Based Medicine et al., 2015); and second, researchers have advocated for the use of evidence-based medicine (EBM), which they perceive as being important (Wang and Huang, 2019). Thus, it is essential to update the assessment and quality of the growing evidence, summarize the current evidence profile, and explore any existing problems and the underlying reasons for these problems. Therefore, the aim of this review was to perform a comprehensive overview of the current evidence contained in CSRs of TCM.

Methods

The methodology of the data collection and analysis in this overview is based on Chapter 22 (Overviews of reviews) in the Cochrane Handbook of Systematic Reviews of Interventions (Higgins et al., 2020).

Criteria for inclusion and exclusion

The newest versions of completed and published CSRs were included. Each CSR was required to center on a TCM intervention that was defined as an intervention based on TCM theory, including CHM (including herbal extracts, single herbs, Chinese patent medicines, or a combination of prescribed herbs), acupuncture (e.g., body acupuncture, electro-acupuncture, or acupressure therapy), moxibustion, TCM massage (Tuina), scraping (Gua Sha), cupping (Baguan), and TCM exercises (e.g., Tai Chi, Qigong, or Baduanjin). Intervention therapies such as Ayurveda, aromatherapy and yoga do not fall within the scope of TCM and were not included. There was no limitation placed on participant characteristics or outcomes.

We excluded protocols, withdrawn reviews, empty reviews (which found no studies eligible for inclusion), and reviews that had not focused on the TCM interventions defined in this review.

Search methods to identify reviews

One author (ZQD) searched the CDSR up to February 2021 using the following terms; “traditional Chinese medicine”, “Chinese herbal medicine”, “Chinese patent medicine”, “acupuncture”, “moxibustion”, “scraping”, “cupping”, “tuina”, “qigong”, “tai chi”, and “baduanjin”. An updated search was conducted on 31 May 2021. MeSH terms and free text terms related to TCM were combined to find eligible TCM-related CSRs. The search strategy can be found in Additional file 1: Table S1.

Data selection

Two reviewers (ZQD and XL) independently assessed each abstract identified by the initial search (using Endnote 9.3.3). All abstracts marked as potentially relevant by either reviewer were advanced to the full-text review stage. Two reviewers (ZQD and XL) read each selected full-text article separately to determine which articles would be included based on the pre-defined inclusion and exclusion criteria. Any discrepancies were resolved by a third reviewer (NR).

Data extraction and management

A data extraction form was developed in Microsoft Excel 2016 and used to record the pre-specified elements of each study (see the detailed extraction form in Additional file 2: Table S1). Data extraction was performed by one reviewer (ZQD), and a second reviewer (XL) checked the data for accuracy.

Our data extraction framework included 3 major items and 14 subitems as follows: a. Bibliographic data and review question: (1) publication information; (2) authors’ information; (3) participants; (4) TCM interventions; (5) outcomes; and (6) study design, b. Review results: (1) participants; (2) TCM interventions; (3) outcomes; (4) study design; (5) authors’ conclusions; and (6) results of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) assessment, if conducted, c. Methodological quality and impact of the CSRs: (1) quality evaluation according to the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR 2) and (2) impact according to the Altmetric Attention Score (AAS), number of citations by guidelines, and total number of citations in the Web of Science (WoS).

Analysis and synthesis of the reviews

The analysis was conducted in three parts. First, we extracted the general characteristics of included studies according to the “PIOSC” (Participant, Intervention, Outcome, Study design, and Authors’ conclusion)” information found in the methods section of each included CSR. Second, we analyzed the review results in each CSR based on the PIOSC information and the GRADE assessment provided by each CSR. Third, the methodological quality was analyzed by evaluating using the AMSTAR 2 and the impact of each CSR assessed.

The results, describing the review characteristics, are presented as tables and graphically. Descriptive statistics, including the frequency and percentage, median and interquartile range (IQR), and mean and standard deviation (SD), were used to describe the basic characteristics.

The length of time since publication of the CSRs was calculated in years between the original publication year and 2021. Time since the last update was calculated in years between the date reported in the history section of each CSR and May 31 2021.

A bar chart was utilized to present the data on disease systems, classified according to the International Classification of Diseases-10 (ICD-10). A bubble chart was used to present the data on the relationships among TCM therapies, disease systems, the number of CSRs, and the number of primary studies included in each CSR. The bubble chart X-axis represents different disease systems, and the Y-axis represents the number of CSRs. The circle size corresponds to the total number of primary studies included in each CSR for each disease system.

The authors’ conclusion was classified based on the abstract conclusion statement in each CSR according to one of the following categories: (i) the conclusion indicated that the results supported the use of TCM therapy; (ii) the conclusion indicated a suggestion of a benefit, but the evidence was insufficient due to low methodological quality or a small number of included studies; (iii) the conclusion indicated that the currently available data did not allow any conclusions to be drawn; and (iv) the conclusion indicated that the results did not support the use of TCM therapy. One author (ZQD) categorized the conclusions, and this categorization was verified by a second author (XL).

For the GRADE assessment, we summarized whether the GRADE assessment was carried out in each CSR, and we extracted the GRADE information from each finding’s summary.

In addition, we analyzed the differences in PIOSC components between the information included in the methods and results sections.

Quality assessment and impact of the reviews

We assessed and analyzed the methodological quality of the CSRs using AMSTAR 2. AMSTAR 2 contains 16 items with a total of 4 options, namely, high, moderate, low and critically low (Shea et al., 2017). Seven of the 16 items are by default labeled as “Critical domains” as they can critically affect the validity of a review and its conclusions. The “Critical domains” 2, 4, 7, 9, 11, 13, and 15 are shown in the Table. Data summarizing the impact of CSRs from three perspectives was collected.

First, the AAS (2020), which is presented on each page of CSRs in the CDSR, represents the immediate availability of information on the reach and influence of an article and confers the ability to track how the attention paid to a CSR has changed over time (Elmore, 2018). The AASs were extracted (a higher score corresponds to increased online presence, with a score ≥20 generally representing an article that performs well compared to its contemporaries (Barakat et al., 2019)).

Second, citation counts were retrieved by searching the WoS for the included CSRs. If a CSR had been updated, all the citations were summed from the published versions up to the total number of citations.

Third, the number of citations of each CSR by guidelines, which is also presented on each page of CSRs in the CDSR, were collected and analyzed.

For these three aspects of the impact of CSRs, a radar chart was used to indicate the total impact. Different colors represented different interventions (orange for CHM, purple for acupuncture, yellow for moxibustion, and green for TCM exercises). The mean values of the total scores for these four types of intervention were used. Each vertex represents the size of the three impact aspects. A larger area enclosed by the vertices indicates a higher impact.

Results

A total of 296 CSRs were obtained by the search (on 10 February 2021), and 4 additional CSRs were found in the updated search (on 31 May 2021). Finally, 168 CSRs relevant to TCM were identified. Of these 168 CSRs, 47 included both TCM and non-TCM therapies as interventions, and 17 CSRs were empty reviews, leaving 104 CSRs eligible for inclusion in this overview (the search results are presented in Fig. 1; the references are given in Additional file 1: Table S2; the full extracted information is provided in Additional file 2: Table S2; and the general characteristics of the 104 included CSRs are given in Table 1). The 104 CSRs were published between 2000 and 2020 and originated from 9 countries. The included systematic reviews investigated the following therapies: acupuncture (n = 28), herbal medicine (n = 70), moxibustion (n = 2) and TCM exercises (n = 4).

Fig. 1.

PRISMA flow diagram.

Table 1.

General characteristics of the 104 included CSRs.

	Number (%)
Publication year
2000–2002	4 (3.8%)
2003–2005	6 (5.8%)
2006–2008	14 (13.5%)
2009–2011	12 (11.5%)
2012–2014	36 (34.6%)
2015–2017	18 (17.3%)
2018–2020	14 (13.5%)
Network and Cochrane Review Group
Cochrane Circulation and Breathing
Stroke Group	12 (11.5%)
Heart Group	9 (8.7%)
Airways Group	2 (1.9%)
Vascular Group	1 (1.0%)
Hypertension Group	1 (1.0%)
Cochrane Abdomen and Endocrine
Hepato-Biliary Group	9 (8.7%)
Kidney and Transplant Group	8 (7.7%)
Metabolic and Endocrine Disorders Group	6 (5.8%)
Gut Group *a	5 (4.8%)
Cochrane Children and Families
Gynecology and fertility group	9 (8.7%)
Pregnancy and Childbirth Group	4 (3.8%)
Incontinence Group	1 (1.0%)
Cochrane Musculoskeletal, Oral, Skin and Sensory
Musculoskeletal Group	7 (6.7%)
Eyes and Vision Group	4 (3.8%)
Pain, Palliative and Supportive Care Group	2 (1.9%)
Skin Group	1 (1.0%)
Back and Neck Group	1 (1.0%)
Cochrane Acute and Emergency Care
Acute Respiratory Infections Group	4 (3.8%)
Injuries Group	2 (1.9%)
Bone, Joint and Muscle Trauma Group	1 (1.0%)
Cochrane Mental Health and Neuroscience
Schizophrenia Group	3 (2.9%)
Dementia and Cognitive Improvement Group	2 (1.9%)
Movement Disorders Group	1 (1.0%)
Epilepsy Group	1 (1.0%)
Developmental, Psychosocial and Learning Problems Group	1 (1.0%)
Common Mental Disorders Group	1 (1.0%)
Cochrane Cancer
Colorectal Group *b	3 (2.9%)
Gynecological, neuro-oncology and orphan cancer group	2 (1.9%)
Breast Cancer Group	1 (1.0%)
Country of First Author
China	74 (71.2%)
Australia	12 (11.5%)
UK	7 (6.7%)
USA	3 (2.9%)
Korea	3 (2.9%)
Canada	2 (1.9%)
Germany	1 (1.0%)
Netherlands	1 (1.0%)
Singapore	1 (1.0%)
Classification of disease (based on the ICD-10)
Diseases of the circulatory system	23 (22.1%)
Diseases of the genitourinary system	16 (15.4%)
Endocrine, nutritional and metabolic diseases	9 (8.7%)
Diseases of the digestive system	8 (7.7%)
Certain infectious and parasitic diseases	7 (6.7%)
Mental and behavioral disorders	7 (6.7%)
Diseases of the musculoskeletal system and connective tissue	6 (5.8%)
Neoplasms	6 (5.8%)
Diseases of the respiratory system	5 (4.8%)
Diseases of the nervous system	5 (4.8%)
Pregnancy, childbirth and the puerperium	4 (3.8%)
Diseases of the eye and adnexa	3 (2.9%)
Injury, poisoning and certain other consequences of external causes	2 (1.9%)
Diseases of the skin and subcutaneous tissue	1 (1.0%)
Codes for special purposes	1 (1.0%)
Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified	1 (1.0%)
Interventions evaluated in CSRs
Chinese herbal medicine	70 (67.3%)
Acupuncture	28 (26.9%)
Tai Chi	3 (2.9%)
Moxibustion	2 (1.9%)
Qigong	1 (1.0%)
Interventions evaluated in primary studies *c
TCM therapy alone	662 (40.3%)
TCM and non-TCM therapy combination	955 (58.2%)
TCM and TCM therapy combination	25 (1.5%)
Authors’ conclusion
No conclusion could be drawn	54 (51.9%)
Benefit with insufficient evidence	42 (40.4%)
Supported	5 (4.8%)
Not supported	3 (2.9%)
Study design
Only RCT	94 (90.4%)
Both RCT and CCT *d	10 (9.6%)
GRADE assessment was done (Y/N)
Y	43 (41.3%)
N	61 (58.7%)

^*aUpper GI and Pancreatic Diseases Group and IBD Group are now combined into one group named the Gut Group.

^*bColorectal Cancer Group no longer exists in the Cochrane Library; therefore, this group was incorporated into the Colorectal Group.

^*cWe counted TCM interventions in the primary study included in each CSR, including TCM interventions alone, the combined use of different TCM interventions, and combined the use of TCM and non-TCM therapies.

^*dBoth RCTs and controlled clinical trials (CCTs) were included in 10 CSRs, among which 9 included RCTs and quasi-RCTs.

General information

Time since original publication of CSRs ranged from 1 to 22 years with a mean of 10.8 years (SD= 4.8). Among them, 33/104 (31.7%) had been updated at least once. Time since these CSRs were last assessed as up-to-date published between 2000 and 2020, ranged from 1 to 20 years with a mean of 8.7 years (SD= 4.5). Ninety-eight (94.2%) CSRs had not been updated in more than 2 years; Eight-four (80.8%) CSRs had not been updated in more than 5 years; and thirty-two (30.8%) CSRs had not been updated in more than 10 years.

Thirty of the Cochrane Collaboration’s 53 Review Groups hosted at least one TCM-related CSR, with the number within these 30 Review Groups ranging from 1 to 12. The groups publishing the most were the Stroke Group (n = 12), the Gynaecology and Fertility Group (n = 9), the Heart Group (n = 9), and the Hepato-Biliary Group (n = 9).

The number of authors participating in the 104 CSRs ranged from 1 to 14, and the median (IQR) was 5 (4–7). The first authors were mainly from China (n = 74 [71.2%]); Australia (n = 12); the UK (n = 7); the USA (n = 3); Korea (n = 3); Canada (n = 2); and Germany, the Netherlands and Singapore, which each had one first author of a CSR. Sixty-seven (64.4%) CSRs had at least one author with an affiliation relevant to EBM, and thirty-four (32.7%) CSRs had a corresponding author with a relevant affiliation. There were seven different EBM-related affiliations of corresponding authors, of which the most common were the Chinese Cochrane Center, Chinese Evidence-Based Medicine Center, West China Hospital Sichuan University (n = 19), and the Beijing University of Chinese Medicine Center for Evidence-Based Chinese Medicine (n = 12). Fifty-seven (54.8%) CSRs had at least one author with an affiliation relevant to TCM or complementary and alternative medicine (CAM), and thirty-seven (65%) of these CSRs had a corresponding author with a relevant affiliation. The most common relevant affiliations were the Beijing University of Chinese Medicine (n = 12), the Shanghai University of Traditional Chinese Medicine (n = 4), and the Chinese University of Hong Kong (n = 3).

Analysis of the ‘PIOSC’ components in the CSR results section

Participants

The 104 included CSRs contained 1617 RCTs and 157,943 participants (median 879.5, IQR 397.3–1810). In total, 70 diseases and symptoms were investigated in the included CSRs. According to the ICD-10 classification system, 23 (22.1%) CSRs addressed diseases of the circulatory system (12 CSRs in stroke); 16 investigated diseases of the genitourinary system; and 9 reported endocrine, nutritional and metabolic diseases.

TCM interventions

A total of 104 CSRs involved four TCM interventions. Among them, −70 (67.3%) CSRs reported CHM, of which 23 were concerned with 16 individual herbs or herbal preparations, 8 involved 6 Chinese patent medicines, 2 were concerned with TCM decoctions, and the remainder (n = 37) addressed multiple CHM formulae. CHM was used most frequently to treat diseases of the circulatory system (n = 17), followed by diseases of the genitourinary system (n = 12), and endocrine, nutritional and metabolic diseases (n = 8). Twenty-eight (26.9%) CSRs had acupuncture in their title, of which 4 CSRs included both acupuncture and moxibustion. Acupuncture was used mostly to treat diseases of the nervous system (n = 5), followed by diseases of the genitourinary system (n = 4), and diseases of the circulatory system (n = 4). Of the remaining CSRs, 2 CSRs reported moxibustion alone; 4 CSRs were concerned with TCM exercises, of which 3 CSRs investigated Tai Chi and 1 CSR assessed Qigong. (See Fig. 2).

Fig. 2.

Evidence map of the CSRs, describing the number of CSRs and number of included studies in each CSR on different disease systems. The X-axis represents different disease systems, and the Y-axis represents the number of CSRs. The circle size corresponds to the total number of studies included in the CSRs for each disease system.

An analysis of the interventions in the experimental groups in 1642 primary studies in 104 CSRs showed that there were 662 studies involving an experimental group treated with TCM therapy alone and 980 studies involving an experimental group treated with combination of therapies, among which there were 955 studies involving non-TCM combination therapy and 25 involving TCM combination therapy.

Outcomes

The promising outcomes (i.e., a statistically significant benefit was found in the relevant CSRs) for 70 different diseases investigated in 104 CSRs were summarized and analyzed (see Fig. 3 and Additional file 1: Table S3). Among these 70 diseases, no promising outcomes were reported for 5 diseases, and no summary of outcomes was reported for 6 diseases. For the remaining 59 diseases, we divided the promising outcomes into three categories, namely, endpoint outcomes (n = 34 diseases), surrogate outcomes (n = 37 diseases) and doctor- or patient-reported outcomes (D/PROs) (n = 27 diseases). All three types of outcomes were reported for 8 diseases (13.6% of 59 diseases), and only endpoint outcomes were reported for 14 diseases (23.7% of 59 diseases).

Fig. 3.

Graphical presentation of promising outcomes for each disease system involved in the included CSRs.

Study design

A total of 1642 trials were included in the 104 CSRs, with a median (IQR) of 10 (5–18.5) trials per CSR. In addition, among the 104 CSRs, 94 (90.4%) included only RCTs, while ten CSRs also included quasi-RCTs or controlled clinical trials (CCTs).

Authors’ conclusions

Of the 104 CSRs, 5 (4.8%) CSRs reported a positive conclusion, of which 3 were about acupuncture and 2 were about CHM. Forty-two (40.4%) CSRs reported that the evidence was insufficient, 54 (51.9%) did not draw a firm conclusion, and 3 CSRs drew unsupported conclusions.

Evidence quality evaluation—GRADE

In terms of the GRADE assessments, 43 (41.3%) of the 104 CSRs, among which the earliest was published in 2011, reported the evidence quality or certainty rated according to the GRADE system. The remaining CSRs, which were published between 2000 and 2016, did not report the evidence quality or certainty using the GRADE system. The GRADE assessments of these 43 CSRs were further evaluated (see Additional file 3). The results showed that these 43 CSRs involved 32 diseases; covered 4 types of TCM interventions, namely, CHM, acupuncture, moxibustion and Tai Chi; and reported 145 kinds of outcomes. Thus, a total of 353 outcomes were evaluated by the GRADE system, of which 6 were found to be of high quality, 45 were found to be of moderate quality, 172 were found to be of low quality, and 130 were found to be of very low quality. The outcomes with high-quality evidence were all from a CSR on the use of Xiongshao capsules and WM in the treatment of coronary heart disease. The outcomes with moderate-quality evidence involved 16 conditions, the most common being vascular dementia (n = 9), fibromyalgia (n = 9), and pain management during labor (n = 4). In terms of the intervention, seven of the outcomes were for CHM, eight for acupuncture, and one for Tai Chi. The detailed results can be found in Additional file 3, and the overall evaluation can be seen in Fig. 4

Fig. 4.

General quality of evidence as determined by GRADE in the included CSRs.

PIOS components described in the methods section vs those described in the results section

We compared the PIOS components specified in the methods section of each CSR with those in the results section of the completed review.

For the participant component of the PIOS, the description in the methods section was essentially the same as the description in the results section.

For the intervention component of the PIOS, the interventions in 70 CSRs were CHM, among which 33 CSRs analyzed specific Chinese patent medicines or herbal extracts. The remaining 36 stated "including all CHM interventions" in the methods section, in which some individual herbs, combinations of herbs or Chinese proprietary medicines were further mentioned. There were 28 acupuncture-related CSRs, of which 16 had detailed descriptions and specific examples of the acupuncture performed in the methods section. Out of these 16 acupuncture CSRs, only 5 described all the interventions listed in the methods section in the results section. There was no discrepancy between the methods section and the results section in the CSRs of Qigong, Tai Chi and moxibustion.

For the outcome component of the PIOS, out of the 104 CSRs, only 16 CSRs completed reported all the outcomes listed in the methods section. And 16 CSRs reported none of the primary outcomes. Fifty CSRs finally reported all the primary outcomes and some of the secondary outcomes proposed in the methods section.

For the study design component of the PIOS, among the 104 CSRs, the study design in the methods section included RCTs (parallel RCTs [n = 104], crossover RCTs [n = 9], cluster RCTs [n = 3]), quasi-RCTs (n = 22), and CCTs (n = 3). No information about cluster RCTs was found in the final results for any of the CSRs. Twenty-two CSRs had fewer types of studies included in the final analysis than expected.

Methodological quality and impact of CSRs

Evaluation of quality based on AMSTAR 2

Methodological quality was evaluated using the AMSTAR 2. Among the 104 CSRs, 8 CSRs (7.7%) were assessed as high quality, 17 CSRs (16.3%) were assessed as medium quality, 51 CSRs (49.0%) were of low quality, and 28 CSRs (26.9%) were of critical low quality. The high-quality CSRs were related to endometriosis, chronic hepatitis B, diabetic peripheral neuropathy, symptomatic gastroparesis, chronic neck pain, atopic eczema, age-related macular degeneration, and impaired glucose tolerance or impaired fasting blood glucose. Analysing Fig. 5 by column, eight items all had “Yes”, involving item 1, 2, 5, 6, 7,12, 14, and 16, and the evaluation results of the six items of the AMSTAR 2 were not all “Yes”, including item 3, 4, 8, 9, 10, and 15.

Fig. 5.

The summary of AMSTAR 2: each item for each included study by year.

Impact

We analyzed the impact of the CSRs after publication in four areas: CHM, acupuncture, moxibustion, and TCM exercises (see Fig. 6).

Fig. 6.

Radar chart comparing the value of four TCM interventions.

The first analysis was based on the AAS. The total AASs (means) for the four types of interventions were 625 (8.9) for CHM, 1350 (48.2) for acupuncture, 110 (55) for moxibustion, and 227 (56.8) for TCM exercises.

In terms of citations in the WoS, with the exclusion of one CSR that could not be found in the WoS, the number of citations was 2135 (mean ± SD: 20.8 ± 21.2). The number of citations (means) in the WoS for CHM, acupuncture, moxibustion, and TCM exercises were 1141 (16.3), 901 (32.2), 41 (20.5), and 61 (15.3), respectively.

Subsequently, according to the information provided by the Cochrane Library, we analyzed the numbers of citations of each CSR in clinical guidelines. There were 40 (38.5%) CSRs that were cited in guidelines 95 times, and the number of citations ranged from 1 to 8.

In this, we compared the impact of four TCM Interventions from the included CSRs with the axes varying from 0 to 60. Different variables are the AAS, citation in WoS, and citation by guideline. The mean scores were calculated. Four TCM Interventions were compared on the basis of these variables. Different colors represent different interventions (orange for CHM, purple for acupuncture, yellow for moxibustion, green for TCM exercises). Each vertex represents the size of the three impact aspects. Larger areas enclosed by the vertices indicate greater impacts.

Discussion

An international team was organized to perform an overview of the topics, results, quality, and impact of TCM CSRs. Our goal was to not only update previous work in this area but also comprehensively profile the high-quality evidence of TCM in the CDSR. Substantial advances have been made, and changes in clinical research have accelerated with the advent of affordable and easily accessible information about clinical epidemiology since the introduction of EBM to China at the beginning of the 20th century. In the early period, TCM clinical studies were largely informal, and in many cases, there was no awareness of scientific study design among researchers and no formal quality control standards in place that could serve as a reference to those who wanted to learn. This has changed dramatically over time, and there is now a center of evidence-based TCM (CACMS, 2019) that oversees research quality and access nationwide. This study analyzed 104 TCM CSRs published between 2000 and 2020 that involved 70 diseases. Data were verified and then summarized by various methods to obtain relatively objective results.

Participants and intervention

The current analysis demonstrates that the evaluated CSRs covered 70 different diseases/conditions belonged to 16 different disease systems as defined by the ICD-10 classification system. These diseases/conditions ranged from chronic diseases, such as diabetes (Liu et al., 2002) and cancer (Zhang et al., 2018), to acute conditions, such as influenza (Jiang et al., 2013) and acute myocardial infarction (Wu et al., 2008). TCM interventions, especially herbal medicines and acupuncture, emphasize individualized treatment based on syndrome differentiation, which is the pragmatic nature of TCM. For example, herbal formulas and the selection of acupoints are regularly modified according to the cluster of symptoms (syndrome) experienced by each patient. The individualized diagnosis and treatment method that is integral to TCM has become an important factor limiting the reliability of related clinical research conclusions. In addition to the CSR of one specific Chinese patent medicine, in other CSRs, researchers usually combine the addition and subtraction formulas for analysis, which affects the reliability of the conclusion to a certain extent. For example, in the study of the treatment of schizophrenia with Wendan decoction (WDD) (Deng and Xu, 2017), which was actually a study in which the individual medicines were checked by an experienced TCM practitioner to decide whether they were truly WDD. Although all studies were on WDD, the composition and dosage of the decoction was not completely consistent due to additions and subtractions, which affected the reliability of the conclusions.

Acupuncture is also similar in that acupoints are added and subtracted, different acupuncture techniques are used, and the method of operation by clinicians is different, which affects the reliability of the results.

Therefore, given the increasing number of TCM RCTs in recent years, the most promising TCM interventions should be analyzed separately to ensure that the conclusions are more targeted and that clinicians and decision makers can apply the evidence more quickly.

The forms of herbal medicines used vary from traditional raw herbs made into decoctions, pills, liniments, plasters and ointments to concentrated powders and liquid extracts prepared by modern pharmaceutical methods. The routes of administration also differ, as they can be oral, topical, intravenous or injections into specific acupuncture points (Zhu et al., 2016). Evidence regarding CHM has been obtained for the treatment of 15 disease systems, including diseases of the circulatory system; diseases of the genitourinary system; and endocrine, nutritional and metabolic diseases. It is worth noting that for certain infectious and parasitic diseases, only CSRs on herbal medicines are available. However, since Shang Han Lun laid the foundation for the use of TCM for the treatment of infectious diseases 1800 years ago (Yuan, 2020), there have been more than 300 epidemics in China’s history (Hu and Zhang, 2020) during which herbal medicines have played important roles. Moreover, the discovery of artemisinin has contributed to solving the problem of resistance to antimalarial treatments (Ma et al., 2020). Therefore, herbal medicine has great potential for the prevention and treatment of infectious diseases.

There are many categories of acupuncture, such as traditional acupuncture, electro-acupuncture, laser acupuncture, acupoint injection, scalp acupuncture, and auricular acupuncture (Shen et al., 2014). Evidence for the treatment of diseases affecting 11 systems, including the genitourinary system, the nervous system, and the circulatory system has been provided. Notably, among the CSRs evaluated, only acupuncture appeared in those involving diseases of the nervous system. Many studies in animals and humans have demonstrated that acupuncture can produce multiple biological responses (Wang et al., 2001) that can occur both locally and close to the site of application (Jansen et al., 1989) and that are mediated mainly by sensory neurons connected to many structures within the central nervous system (Magnusson et al., 1994). The result is the activation of pathways affecting various physiological systems in the brain and in the periphery (Liu et al., 2004; Middlekauff et al., 2004; Sun and Li, 2001), which is probably why only acupuncture appeared in the CSRs involving diseases of the nervous system.

For Tai Chi and Qigong (TCM exercises), evidence for the treatment of diseases of the circulatory system, respiratory system, musculoskeletal system and connective tissue was provided. This evidence was also verified in another study. Although the number of CSRs involving Tai Chi and Qigong has been limited, the results are worth exploring. Tai Chi is a healing art combining martial art movements with Qi and Qigong (Jahnke et al., 2010; Wang et al., 2017). Tai Chi and Qigong are designed to increase muscle strength, stretch the body, improve balance, improve the flow of blood and other fluids throughout the body, improve common cardiovascular risk factors, and improve aerobic capacity (Field, 2011; Hempel et al., 2014; Lan et al., 2013). Thus, Tai Chi and Qigong might provide new options for alternative treatments of these diseases.

Outcomes

In the 104 CSRs, the endpoint outcomes (such as mortality and recurrence rate), D/PROs (such as pain, the Mini-Mental State Examination score and quality of life) and surrogate outcomes (such as creatinine clearance, serum HBsAg and blood pressure) were used to evaluate the efficacy of TCM. The analysis of outcomes is very broad, but it cannot fully reflect the advantages of TCM for which type of outcome. The next step should be to further clarify for which types of outcomes there is clear evidence and for which there is currently insufficient evidence, as this might provide guidance regarding the areas in which additional good-quality research would be most likely to provide a firm answer.

Furthermore, there is an outcome called the “cure rate”. The “cure rate” has been promoted by the guideline (Zheng, 2002) in the field of TCM research for a long time in the past. However, “cure rate” is a kind of a composite outcome which is not recognized in WM, although it is used in most TCM primary studies (Zhang et al., 2020). The “cure rate” cannot reflect the true effect of TCM (Armstrong and Westerhout, 2017). Fortunately, given the guidance provided by each Cochrane group, only 5 CSRs used the cure rate as an outcome. Most of the CSRs selected standardized, quantifiable outcomes with clinical meaningfulness.

With regard to the gap between the review question and results, only 16 CSRs completely reported all of the outcomes, and fewer than half of the CSRs reported all of the primary outcomes. We analyzed three main reasons for this mismatch. First, due to the time lag in the publication of the primary study and the CSR, the outcomes that the primary studies had reported were not those the CSR researchers were most interested in at that moment, or even not the interesting issue for relevant research fields at present. Therefore, when those researchers pooled the results, they did not obtain enough information, which may have lead to some discrepancies between methods section and results sections. Second, different researchers have different concerns about a disease, which may lead them to have specific preferences. Third, there is no consensus on the outcomes for the same disease. For instance, while some TCM practitioners have confirmed the efficacy of acupuncture with regard to neurological recovery in patients with stroke, others believe that oral CHM is superior with regard to improving patients’ quality of life. It is promising that an increasing number of TCM researchers are actively carrying out research on Core Outcome Measures in Effectiveness Trials (COMETs) (COMET, 2020). Thus, we believe that in the next 5–10 years, clinical research on TCM will improve as the selection of outcomes will be more disease specific and internationally recognized.

Author’s conclusions

Our study found that only 5 CSRs provided promising positive conclusions. Most of the CSRs drew a similar conclusion: “some of the evidence supports positive effectiveness; however, the data are limited due to bias or low methodological quality. Thus, in the future, more high-quality evidence is needed”. This is usually due to the poor quality of the methodology used in the included studies or the insufficient sample size (Tang, 2006; Tang et al., 1999). It is suggested that many TCM primary studies with low quality and small sample size have been carried out, which leads to the waste of research to a certain extent. This issue has been previously mentioned: ‘our medical research investment should be protected from the avoidable waste of inadequately producing and reporting research’ (Chalmers and Glasziou, 2009). As they suggested, we should examine the causes and degree of waste occurring at four successive stages: the selection of research questions, the quality of research design and methodology, the adequacy of publication practices, and the quality of reports of research. Therefore, we suggest that the most important goals at the present moment are to improve the quality of primary studies, especially RCTs, which are the gold standard for ascertaining the efficacy and safety of an intervention, instead of constantly performing secondary studies, and to embrace the concept of “less is more” (Grady and Redberg, 2010; Soong et al., 2021).

Quality–GRADE and AMSTAR2

The GRADE assessment system can be used to rate the quality of evidence in systematic reviews and guidelines (Guyatt et al., 2008). The GRADE system was created by the GRADE Working Group in 2000 and officially launched in 2004 (Atkins et al., 2004). In handbook version 5.0.0, which was published in 2008, Cochrane pointed out for the first time that GRADE should be used to assess the certainty (or quality) of a body of evidence (Higgins et al., 2008). However, in our study, all 104 CSRs were either published or updated after 2009; among them, only 43 (41.3%) of the CSRs were evaluated using the GRADE approach. It is obvious that the rate of the utilization of the GRADE system is not very high despite the recommendations of the Cochrane Handbook.

Second, in these evaluated CSRs, the results of TCM intervention were better than the control, but the quality of evidence was mostly low or very low, which affects the reliability of the results. Beside improving the quality of TCM primary study at the source, from the perspective of CSR, we suggest that the inclusion and exclusion criteria should be stricter and more limited. For example, primary studies that are single-page trials, single-author trials, and trials published in ‘predatory’ or pseudo-scientific journals should not be considered.

AMSTAR 2, which was revised and published in 2017, is a critical appraisal tool for systematic reviews (Shea et al., 2017). According to our results, the methodological quality of included CSRs was not high. The reason is that there are problems in the implementation of some items in these CSRs. Among the 16 items, the critical domains item 15 (If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?), that is, publication bias, has a significant impact on the quality of CSRs. Most CSRs mentioned that because too few studies were identified in one analysis for a funnel plot construction. Thus, we evaluated this item as “No”, which led to the final score of the evaluation being considered low or critical low. The findings indicate that in CSRs, researchers may not pay enough attention to the analysis of publication bias. In addition to using the funnel plot, other statistical methods can be used to analyze publication bias, also a very important part of methodological quality of CSR. Moreover, it is worth noting that we found that the methodological quality of CSRs has improved over time, especially in terms of literature search strategy (item 4), detailed description of included studies (item 8), and assessment of risk of bias (Item 9).

Impact and translation

The AAS (AAS, 2020; Barakat et al., 2019) is an emerging tool that measures the online presence of published articles by automatically calculating the number of mentions of an article on social media, thereby weighting the attention a research output has received. The total score and average score of AASs of CSRs on acupuncture, moxibustion, and TCM exercises were found to be greater than those of CSRs on CHM, especially acupuncture. The AAS is a weighted count that depends on mentions in news, blogs, policy documents, patents, and Wikipedia articles, among others (AAS, 2021). Thus, a higher AAS likely indicates that more attention is paid to these interventions on social media than that received by the other interventions. However, due to the small number of CSRs on moxibustion and TCM exercises, a statistical analysis of various interventions could not be carried out. In addition, we did not observe a clear correlation with the timing of publication. In future studies in which sufficient numbers of CSRs are included, we can limit the type of disease, intervention and time of publication, enabling us to conduct a rigorous statistical analysis.

The results of CSRs can be used to support the development of evidence-based guidelines (Higgins et al., 2020). According to our findings, the results 40 CSRs of the 104 CSRs have been used in the guidelines. The remaining 64 CSRs were not used in guidelines. Possible reasons may be: first, many of these CSRs were published relatively early and were not updated, which means that the evidence is not current. Second, the majority of the CSRs concentrated on CHM, and these CHM interventions spanned a broad range, and cannot provide sufficient and clear clinical guidance, therefore guideline developers may overlook them, hence the relative lack of citations.

The application of CSR results is of great value. As the World Health Organization (WHO) guidelines state (Peters; et al., 2013), evidence should also advocate utilization and transformation. It is a major waste of the resources, time, and effort involved in performing a CSR if scant attention is given to how the results of these reviews can be used and how their recommendations can be implemented. In the future, continuous, dynamic and comprehensive research should be conducted on the entire process of the implementation of results from TCM-related CSRs. (1) Before they are implemented, we should systematically and comprehensively evaluate the international and domestic situation to study the evidence provided by the existing guidelines to prioritize diseases with a heavy burden and no effective clinical treatments; (2) during the implementation of these findings, quality should be strictly controlled to ensure the generation of high-quality reports that are more suitable for the evaluation of TCM interventions; and (3) after the implementation of these findings, the applicability of the results and recommendations of the CSR should be evaluated, as should their ability to support clinical decision-making. Timely updates of reviews should be performed to ensure a continuous, dynamic research cycle.

Other issues

Our analysis found that there was a problem in the provision of timely updates of CSRs, as 94.2% of the included CSRs had not been updated in more than 2 years, according to the biennial update principle for each CSR. And 30.8% had not been updated more than 10 years. Although the new updated Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (Higgins et al., 2020) does not limit the exact time within which a CSR should be updated, and there is no consensus on when to update a review. However, the decision to update a CSR should be based on the continuing importance of the review question to decision makers and the availability of new data or new methods that would have a meaningful impact on the review findings. The CSRs that had not been updated in two years covered various types of TCM therapies and different types of diseases. After a rapid search of PubMed, we found that more than 3000 TCM-related RCTs were published in the past ten years. Reasons for failing to update CSRs vary. It is the responsibility of the review group to give their reasons for failing to update the review and state when they will update it. Clinical practice guideline development is based on systematic reviews. If systematic reviews are not be updated in a timely manner, the guidelines cannot be efficiently developed. If CSRs are not available, guideline developers need to conduct a new systematic review on the topic under consideration. Therefore, it is necessary to update CSRs in a timely fashion.

Second, regarding the composition of authors, 63.8% of the included TCM-related CSRs had at least one author whose affiliation was relevant to TCM or CAM. In a high-quality CSR of TCM, the professional and clinical advice and understanding of TCM experts play vital roles. In addition, at least one author in 64.4% of the CSR was engaged in EBM. Despite the assistance of the Cochrane collaboration, the presence of EBM experts in the team was also critical to the completion of a high-quality CSR. Therefore, we suggest that future CSRs should be composed of multidisciplinary team, involving authors with TCM expertise, methodological experts, statisticians and WM experts.

Limitations

Several limitations must be considered when interpreting the findings of this study. First, only CSRs that evaluated TCM-related therapies as the main intervention were analyzed. Reviews that evaluated both TCM-related therapies and other therapies as the major interventions and empty reviews were not analyzed because they provided either mixed evidence or no evidence, which is a potential bias of this study. Second, our results can only represent the condition of the published TCM-related CSRs in the past, not the quality of TCM-related systematic reviews published in other journals (Junhua et al., 2009; Manheimer et al., 2008), let alone the quality of other TCM studies and in the future. Actually, high quality clinical studies have been published in most recent years (Liu et al., 2021; Siu et al., 2021; Sun et al., 2021), which have not been incorporated or analyzed in the CSRs. Third, this review was based on the Cochrane Library and provides a general picture of the evidence regarding TCM from the highest level in the evidence pyramid, but data extraction and analysis were conducted at the review level, rather than the individual trial data. This meant that the risk of bias of the individual trials could not be accounted for, but only the quality of the systematic reviews that synthesised them. With regard to the specific application of these results, this review can serve as a guide, and it is necessary to analyze the specific problems according to the real-world situation. Fourth, we analyzed only the citations by guidelines provided by the Cochrane Library and did not retrieve the guidelines ourselves, which may have also led to a certain level of bias. Fifth, this review might have overlapping primary studies across the reviews, which could overstate sample size and number of events, falsely leading to greater precision in the analysis (Pieper et al., 2014).

Conclusions

This study provides deeper insight into the current status of TCM CSRs. Although TCM-related CSRs have covered a large number of diseases, the current quality of evidence is insufficient to support the efficacy of most TCM interventions. It is not worthwhile to spend time and resources performing a CSR when the included trials have poor methodological quality. TCM-related CSRs should aim to fill the gap between research and clinical practice and ensure that evidence-based effective health care interventions are introduced into clinical practice. A fundamental problem is highlighted in this study: the quality of the primary trials determines the quality of the systematic review. The China Cochrane Network should address this problem in the future.

Abbreviations:

AAS

Altmetric attention score

AMSTAR-2

Assessing the methodological quality of systematic reviews 2

CAM

Complementary and alternative medicine

CCT

Controlled clinical trials

CDSR

Cochrane database of systematic reviews

CHM

Chinese herbal medicine

CSR

Cochrane systematic review

COMET

Core outcome measures in effectiveness trial

D/PROs

Doctor- or patient-reported outcomes

EBM

Evidence-based medicine

GRADE

Grading of recommendations assessment, development, and evaluation

ICD-10

International classification of diseases-10

IQR

Interquartile range

RCT

Randomized controlled trial

SD

Standard deviation

TCM

Traditional Chinese medicine

WDD

Wendan decoction

WM

Western medicine

WoS

Web of science