Assessment of the Reporting Quality of Placebo-controlled Randomized Trials on the Treatment of Type 2 Diabetes With Traditional Chinese Medicine in Mainland China

Abstract

Placebo-controlled randomized trials are often used to evaluate the absolute effect of new treatments and are considered gold standard for clinical trials. No studies, however, have yet been conducted evaluating the reporting quality of placebo-controlled randomized trials. The current study aims to assess the reporting quality of placebo-controlled randomized trials on treatment of diabetes with Traditional Chinese Medicine (TCM) in Mainland China and to provide recommendations for improvements.

China National Knowledge Infrastructure database, Wanfang database, China Biology Medicine database, and VIP database were searched for placebo-controlled randomized trials on treatment of diabetes with TCM. Review, animal experiment, and randomized controlled trials without placebo control were excluded. According to Consolidated Standards of Reporting Trials (CONSORT) 2010 checklists items, each item was given a yes or no depending on whether it was reported or not.

A total of 68 articles were included. The reporting percentage in each article ranged from 24.3% to 73%, and 30.9% articles reported more than 50% of the items. Seven of the 37 items were reported more than 90% of the items, whereas 7 items were not mentioned at all. The average reporting for “title and abstract,” “introduction,” “methods,” “results,” “discussion,” and “other information” was 43.4%, 78.7%, 40.1%, 49.9%, 71.1%, and 17.2%, respectively. The percentage of each section had increased after 2010. In addition, the reporting of multiple study centers, funding, placebo species, informed consent forms, and ethical approvals were 14.7%, 50%, 36.85%, 33.8%, and 4.4%, respectively.

Although a scoring system was created according to the CONSORT 2010 checklist, it was not designed as an assessment tool. According to CONSORT 2010, the reporting quality of placebo-controlled randomized trials on the treatment of diabetes with TCM improved after 2010. Future improvements, however, are still needed, particularly in methods sections.

INTRODUCTION

The number of people with diabetes is growing rapidly worldwide.¹ China has the highest number of people with diabetes.¹ The overall prevalence of diabetes in the adult population of China was 0.67% in 1980 ² and had increased to 11.6% by 2010.³ With the increasing acceptance of Traditional Chinese Medicine (TCM) worldwide and the heightened interest in the clinical efficacy of TCM,⁴ more and more randomized clinical trials (RCTs) have been conducted examining the use TCM for treating of diabetes.^5–7

Randomized clinical trials are considered to be the gold standard for clinical trials. Randomized clinical trials effectively reduce bias and provide evidence for use in clinical practice. The Consolidated Standards of Reporting Trials (CONSORT) statement, first published in 1996⁸ and updated in 2001⁹ and 2010,¹⁰ is a guideline for RCTs. The aim of the CONSORT statement is to improve the quality of RCTs. Placebo-controlled randomized trials, as a classic type of RCT, are often used to evaluate the absolute effect of a new treatment by reducing all factors except the treatment. It is most commonly designed with blinding, in which participants do not know what treatment they have received: real or placebo. Without placebo groups to compare against the treated groups in clinical trials, it is impossible to know whether a new treatment itself had any effect.

To the best of our knowledge, no studies have yet been conducted evaluating the reporting quality of placebo-controlled randomized trials examining the treatment of diabetes with TCM that were conducted over the last 30 years in Mainland China. The objective of this study was to assess the reporting quality of placebo-controlled randomized trials in Mainland China that evaluated the treatment of diabetes with TCM and to provide recommendations for improvements.

METHODS

Search Strategy

The China National Knowledge Infrastructure electronic database, Wanfang database, China Biology Medicine database, and VIP database were searched for placebo-controlled randomized trials up to November 10, 2015. The search topic terms were “terms related to RCTs,” “terms related to diabetes,” and “limited to placebo.”

Eligibility Criteria

Eligibility criteria were as follows: subjects diagnosed with type 2 diabetes or prediabetes; TCM intervention, including Chinese herbs, Chinese patent drugs, and acupuncture; a prospective RCT with a placebo control group; study published in a Chinese Journal; and original research. Criteria for exclusion included diabetes complications, diabetes concomitant disease, meta-analysis, animal experiment, review, articles from the same study, RCTs without placebo control, abstract, or case report.

Assessment of Quality of Reporting

In the current study, CONSORT 2010 was selected to assess the reporting quality of placebo-controlled randomized trials. The CONSORT 2010 checklist contains 6 sections: title and abstract, introduction, methods, results, discussion, and other information, and a total of 25 items (refined to 37 items, 25 primary, and 12 secondary items). In addition, study center, funding, placebo species, informed consent forms, and ethical approvals were also were recorded and analyzed. All of the data were analyzed before and post 2010 for comparison.

Data Extraction and Analysis

Two trained investigators extracted the information and independently evaluated each article. The results from both investigators were checked jointly and differences of opinion were resolved through third party consultation.

To evaluate the reporting quality of articles to determine if they were reliable and valid, we created a scoring system according to the CONSORT 2010 checklist in the analysis, following methods detailed in previous studies.^11,12 In the CONSORT 2010 checklist, for each of the 37 items, a yes (score of 1) is allocated if the author reported the item, whereas a no (score of 0) is given if the author did not report the item. The sum of reported items was calculated for each article. The number and percentage of items reported in each article was analyzed. Descriptive statistics were performed. Microsoft Excel 2010 (Microsoft, USA) and SPSS software version 19.0 (IBM, USA) were used to analyze data.

RESULTS

A total of 4873 studies published in Mainland China were searched. A total of 68 articles^13–80 were included for analysis (Table 1).

TABLE 1.

The Reporting Number and Percentage for Each Article of the Consolidated Standards of Reporting Trials 2010 Checklist in 68 Included Articles

According to the 37 items in CONSORT 2010, the reporting percentage in each of the 68 articles ranged from 24.3% to 73%. A total of 21 (30.9%) articles reported more than 50% of the items (Table 1).

Seven of the 37 items (4a, 5, 6a, 13a, 16, 17a, and 22) were reported more than 90% of the items, whereas 7 items (3b, 6b, 7b, 12b, 14b, 18, and 24) were not mentioned at all (Table 2). Only 4 (5.9%) of the articles were identified as randomized trials in the titles, 2 (2.9%) reported determination of sample size, 8 (11.8%) reported the mechanism for allocation concealment, and 8 (11.8%) mentioned implementation of randomization. Twenty-four (92.3%) items had increased after 2010.

TABLE 2.

The Reported Number and Percentage of Each Item on the Consolidated Standards of Reporting Trials 2010 Checklist

The average reported number and percentage of each section, according to CONSORT 2010 checklist, are shown in Table 3. The average reporting for “title and abstract,” “introduction,” “methods,” “results,” “discussion,” “other information” was 43.4%, 78.7%, 40.1%, 49.9%, 71.1%, and 17.2%, respectively and all had increased after 2010.

TABLE 3.

The Average Reported Number and Percentage of Each Section of Consolidated Standards of Reporting Trials 2010 Checklist

In addition, the general characteristics that were not included in CONSORT 2010 checklist was analyzed (Table 4). There were 10 (14.7%) studies that had multiple study centers, 34 (50%) that acknowledged funding, 25 (36.85%) that reported placebo species, 23 (33.8%) that mentioned informed consent forms, and 3 (4.4%) that reported ethical approval.

TABLE 4.

The Quantity of Reported General Characteristics and Average Reported Number and Percentage of Each Topic According to Consolidated Standards of Reporting Trials 2010 Checklist

DISCUSSION

Chinese herbal medicine has a long history and played a dominant role in China before the spread of Western medicine.^81,82 With the development of evidence-based medicine, more and more RCTs have been used to evaluate the efficacy and safety of TCM. Adequate reporting of RCTs allows for easy determination of the RCT quality, which is important because RCTs of poor quality may exaggerate the effects of treatment.⁸³ Placebo-controlled randomized trials account for small proportion of the RCTs performed; however, they are required to be stricter in design and conduct.

To the best of our knowledge, this is the first study assessing the reporting quality of randomized placebo-controlled trials on the treatment of diabetes with TCM in Mainland China, according to the revised CONSORT 2010 guidelines. Results of the current study indicated that the quality of placebo-controlled randomized trials on TCM needs improvement, especially in the methods section.

In the current study, only 4 (5.9%) of titles indicated that the studies were randomized controlled trials. This, however, was a higher percentage than that reported in recent reviews of RCTs studied in certain Chinese Journals.^11,12,84 There was only 1 (2.6%) title that reported “randomized” before 2010 and 3 (10%) titles after 2010. The total score of the 4 articles reporting “randomized” in the titles was ranked in the top 10. This illuminated the fact that articles with “randomized controlled trial” in the title commonly had a high reporting quality in our study. Thus, authors should use “randomized” in the title to indicate the trial design and to allow readers to easily identify the type of study. A structured abstract contains trial design, methods, results, and conclusions. In the current study, 80.9% articles had structured abstracts. Eleven articles before 2010, however, did not have structured abstracts, whereas 2 articles did not have structured abstracts after 2010. The results indicate that the update of the CONSORT statement promoted the reporting of structured abstracts in Mainland China. In addition, only 19 (27.9%) articles reported trial design in the abstract according to CONSORT for abstracts.

Scientific background and explanation of the rational as well as specific objectives or hypotheses should be included. Biomedical research involving people should be based on a thorough knowledge of the scientific literature, according to the Declaration of Helsinki.⁸⁵ In the current study, the average reporting percentage for the “Introduction” section was 63.2% before 2010, and increased to 98.3% post 2010. In spite of this, the background in many articles was inadequate to explain the rationale and no hypothesis was reported in all of included articles.

The percentage of reported trial design was 57.9% before 2010 and decrease to 46.7% post 2010. This result, however, is similar to a previous study in which 42.9% of RCTs reported trial design, which were published in the Chinese Journal of Integrated Traditional and Western Medicine, and were on the treatment of coronary heart disease with TCM.¹² Trial design should be described in both the abstract and text. In this study, 4 articles described the trial design in the abstract, but not in the text.

A total of 50% of the articles before 2010 and 90% of articles post 2010 reported the settings and locations where the data were collected. Because the trials were conducted in China and diabetes was noncommunicable disease, nearly all of the articles only reported the hospital, whether outpatient or inpatient, and the community, and not including the cultural environment and the climate. Only 5 articles distinguished the primary and secondary outcomes in our study. Most of the articles, however, listed unordered test ratings. The primary outcome should be set before conducting a trial and should be used to calculate the sample size.

Although only 2 (2.9%) of articles reported how the sample size was determined, this result was better than several recently published studies on Chinese Journals.^11,12,84,86 Hu et al¹⁶ reported adequate information to calculate sample size in our study. A study with too small sample size may conclude that there is no statistical difference with a particular treatment or intervention,⁸⁷ whereas a study with an unnecessarily large sample size may require a huge amount of funding. Clinicians who lack statistical knowledge can turn to statisticians for help, but the sample size calculation process should still be reported in articles.

Randomization can minimize bias between groups. In our study, the methods used to generate the random allocation sequence were reported as 15.8% and increased to 30% after 2010, and type of randomization was reported as 21.1% and increased to 36.7%. The percentage, however, is lower, compared with articles published in Science Citation Index.⁸⁸ The random number table was found to be the most commonly used method for generating random allocation sequences according to the results of our study. Reporting that patients were randomized into 2 groups is inadequate. The process used to generate randomization can help readers to identify whether or not any bias was caused by the randomization method. Therefore, authors should report the details of the randomization process instead of only reporting that the RCT was randomized.

Items 9 and 10 emphasize the allocation concealment and implementation of randomization. In the current study, 8 (11.8%) articles reported allocation concealment and implementation of randomization. There was no significant change after 2010. Among the 8 articles, 6 articles reported funding indicating that funding promotes the reporting quality to a certain extent. Reporting the method of concealment is important for determining whether or not the randomization has been subverted. The method used for implementation of randomization can also produce bias. If possible, researchers involved in the sequence generation or allocation concealment steps should not be involved in the implementation step.¹⁰

Most placebo-controlled randomized trials use blinding to reduce bias. In the current study, 37 (54.4%) of the articles reported the implementation of blinding including single blind and double blind, and 30 (44.1%) articles described interventions similar to blinding. A total of 7 articles were single blind, all of which were published before 2010. The remaining article may have used blinding in the conduct of studies, but that it was not specifically reported.

The statistical method used in most of the articles was the t test. This relates to the fact that the primary and secondary outcomes of diabetes are continuously variable. Only 2 articles reported adequate information for analyses of the primary and secondary outcomes in our study.

There was no participant flow reported in any of the included articles. All articles, however, reported 13a by sentence descriptions instead of a flow chart of participants. The proportion of articles reporting losses and exclusions was 18.4% before 2010 and 40% post 2010. This result roughly concurs with a recent review of RCTs conducted to evaluate TCM.^11,12 None of the articles included in the study ended or stopped. For most of the articles, the percentage of reported dates for the periods of recruitment increased from 39.5% to 76.7% after 2010, but no articles reported the minimum, maximum, and median durations of follow-up after treatment.

Baseline data are especially valuable for outcomes. Item 15 emphasizes baseline data. Findings in the current study showed that articles reporting baseline demographic and clinical characteristics for each group in table format increased from 15.8% before 2010 to 46.7% after 2010. Among the remaining articles, 87.5%, however, presented the information in text. Consolidated Standards of Reporting Trials 2010 states that it is most efficient to present such information in a table.

Sixty-eight (95.6%) articles reported number analyses shown in tables and/or sentences. The intention-to-treat analysis was found in 2 articles, more than in other studies.¹¹ It is necessary to divide people into different groups for different analyses and intentions. The safety evaluation often uses intention-to-treat analysis, whereas the effectiveness evaluation often uses per protocol analysis. The outcomes and estimation were reported in most articles, though only 1 article showed 95% confidence intervals and 4 articles reported accurate P values. Similar results have been reported in other studies.^11,84

The safety of the intervention is indicated by harm; therefore, it is necessary to report whether or not a specific intervention or treatment has harms or adverse effects. In this study, 29 (42.6%) articles reported harms, which was better than that found in previous studies of TCM.^11,89,90 For each group in a study, the number of participants who withdraw because of adverse events and the appropriate metrics for recurrent events should be reported.⁹¹

The discussion section was adequate to some extent. The generalizability, interpretations, and limitations were presented in 88.2%, 95.6%, and 29.4% of the articles, respectively. This result is similar to those of other studies in China.^11,12 The limitations section is very important, particularly according to CONSORT 2010, and reporting of limitations needs improvement in future studies. Limitations should contain weaknesses in a study, imprecision in the results, and the status of the sample size.

There was only 1 article that reported the registration number and no articles provided protocols. Chen et al⁶⁴ published effects of TCM combined with general lifestyle on 210 patients with Impaired Glucose Tolerance combined MS in China Journal of Traditional Chinese Medicine and Pharmacy in 2012. They, however, did not reported where the full trial protocol can be accessed.

In addition, the study center, funding, placebo species, informed consent forms, and ethical approvals were examined in this study to evaluate the reporting quality. The results indicate that the average total score of reported items of an article is influenced by multiple study centers, funding support, especially funding from national organizations, informed consent forms from participants, and ethical approvals. In our study, only 3 articles reported ethical approval. Internationally, any study related to human or animals should be approved by an ethical organization.

There were some limitations in the current study. First, although the reporting quality of articles was assessed, only articles from Chinese Journals were selected. Articles from Chinese authors that were published in English were not included. Most studies of TCM, however, have been conducted in China, particularly during the earlier years we examined. Secondly, though we created a scoring system according to the CONSORT 2010 checklist in the study to evaluating the reporting quality of articles, the CONSORT checklist focuses on items related to the internal and external validity of trials, rather than being designed as an assessment tool. Third, the authors were not contacted to see if they performed items in the study when the items in CONSORT 2010 checklist were not reported.

CONCLUSIONS

Although a scoring system was created according to the CONSORT 2010 checklist, it was not designed as an assessment tool. In general, according to CONSORT 2010, the reporting quality of placebo-controlled randomized trials on the treatment of diabetes with TCM, had improved after 2010. More improvement, however, needs to be made in the future, especially in the methods sections. To further improve the quality of placebo-controlled randomized trials, both journals and authors in China need to follow the CONSORT 2010 guidelines.