Abstract
Objectives
High-quality reporting of treatment details can aid replication of study results in real-world clinical practice. The Standards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) is a reporting guideline for key elements of acupuncture interventions in clinical trials. This study used STRICTA to investigate whether Cochrane reviews of acupuncture adequately report important treatment details.
Design
Systematic review
Methods
Cochrane reviews of acupuncture were identified from The Cochrane Library (issue 7, 2012). Randomised controlled trials (RCTs) included in the reviews and published after 2005 were obtained. Using STRICTA, we extracted acupuncture-related information from the Cochrane reviews and the RCTs. The characteristics of the included studies’ table were the major source of intervention information from Cochrane reviews. Reporting quality of acupuncture interventions in Cochrane reviews was assessed and compared with the respective RCTs.
Results
25 Cochrane reviews of acupuncture and 92 RCTs met the selection criteria. Cochrane reviews were 16% less likely to report the acupuncture-related items of STRICTA than RCTs (risk ratio 0.84, 95% CI 0.79 to 0.88, I2=8%). Information was significantly better reported for 10 of the 15 treatment-group items of STRICTA in RCTs than in Cochrane reviews (p<0.05), while four items did so without statistical significance. One item related to practitioner background was significantly better reported in Cochrane reviews.
Conclusions
Reporting quality of treatment details in Cochrane reviews of acupuncture was insufficient with regard to STRICTA, even though such information was readily reported in RCTs. The overall quality of reporting of the RCTs, while better than the reviews, was also often suboptimal. Use of STRICTA guideline during the review process is recommended to adequately report the key treatment components in Cochrane reviews of acupuncture. The potential impact of STRICTA to the replicability and utilisation of reviews in future research and practice needs to be investigated.
Keywords: Complementary Medicine, Statistics & Research Methods, Medical Education & Training
Strengths and limitations of this study.
This study is the first investigation that revealed the avoidable incompleteness of reporting quality with regard to important treatment components of acupuncture in the Cochrane reviews, even though information was readily reported in relevant primary randomised controlled trials.
Our findings suggest that there is a loss of treatment-related information during data abstraction for Cochrane reviews which may influence the replicability of trial interventions.
Whether loss of treatment-related information actually leads to the altered replicability of reported interventions in Cochrane reviews was not investigated.
Background
Cochrane systematic reviews have summaries of study characteristics, which aim to include details of study interventions to be replicated in practice.1 Developing methods to improve the descriptions of complex interventions is considered as an essential task to complement existing systematic review methodology.2 Therefore, adherence to international standards for reporting interventions might be helpful for detailed description of study interventions in Characteristics summaries in Cochrane systematic reviews. The Standards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) is an internationally recommended standard for reporting details of acupuncture interventions in clinical trials of acupuncture. It can be used to provide sufficient information for researchers or clinicians to understand acupuncture interventions used in the study and replicate them in other researches or practices.3 Consolidated Standards of Reporting Trials (CONSORT) for non-pharmacological treatments has also suggested to provide detailed information for study interventions, thus to improve the uptake of study results in studies of complex interventions, including surgery, psychological interventions and acupuncture.4 However, many current trials and reviews have been found to often omit crucial details of non-pharmacological treatments.5 In case of Cochrane reviews, clear description on the intervention of interests has been emphasised to improve applicability of review into real clinical practice.6 To the best of our knowledge, however, there is no study assessing whether Cochrane reviews provide satisfactory description on any particular intervention by adopting standards for reporting interventions. Although previous researches focused on managing the reporting quality of Cochrane review in terms of methodological process,7 the reporting quality for interventions in Cochrane review had been largely ignored. Hence, this study aimed to assess the reporting quality of acupuncture treatment-related information in the Cochrane reviews of acupuncture, as well as to investigate whether utilisation of the reporting guideline for treatment intervention can improve the quality of reporting in Cochrane reviews of acupuncture.
Methods
Search strategy and selection criteria for Cochrane reviews of acupuncture
Cochrane Database of Systematic Reviews (CDSR) of issue 7 March 2012 was searched to identify Cochrane reviews of acupuncture using the term ‘acupuncture’. Cochrane reviews that investigated acupuncture as a primary treatment intervention or as one of various treatment or control interventions with at least one primary component study that assessed the effects of acupuncture were included. Reviews without any included study (ie, empty reviews) or reviews that did not include acupuncture-related studies were excluded.
Search strategy and selection criteria for component studies in eligible Cochrane reviews of acupuncture
Component study was defined as a study included in the Cochrane review regardless of the contribution to the qualitative or quantitative analyses. These were searched using the reference citation information provided in the relevant Cochrane reviews. Among studies in eligible Cochrane systematic reviews, only trials published since 2005 were considered to be analysed in this study. This is because of the possible necessary time-span for uptake of STRICTA guideline. This approach is similar to the work of Prady et al8 which analysed acupuncture studies published after 3–4 years following the publication of STRICTA. Component studies which involved the assessment of acupuncture as a treatment or control intervention were eligible in our study. The term ‘acupuncture’ was defined as interventions involving penetration of certain points on the skin by needling regardless of manual or electrical stimulation, since needling is believed to be the most representative feature of acupuncture and STRICTA items were originally developed for the reporting of needle acupuncture in clinical studies.3 Thus, component studies using other types of non-penetrating stimulation on acupuncture points, such as laser acupuncture, acupressure, device-involved acupuncture point stimulation (ie, wrist band application) and injections on acupuncture points were excluded. Studies employing stimulation on non-classical acupuncture points, such as trigger points, were eligible only if they clearly mentioned in the review that the intervention was ‘acupuncture’.
Data extraction in Cochrane reviews of acupuncture
Intervention details for acupuncture treatments in tables for characteristics of included component studies were extracted in each Cochrane review. Characteristics of included component studies were chosen for the main data source of our analysis because they intend to provide sufficient information on various study components including details of interventions to enable readers to understand the study better or replicate interventions in their own contexts.1 Methods and results of the included Cochrane reviews were also examined to identify further acupuncture-related information which was not reported in the tables. Two independent authors underwent these processes.
General characteristics of each component study, including publication languages (English or non-English), type of control groups in primary component studies, publication or last-updated years and the number of included studies in the Cochrane reviews were extracted. Whether clinical heterogeneity related to the acupuncture treatments were planned to be investigated by subgroup, sensitivity or other analysis regardless of the availability of those analyses was also identified. One review author conducted these processes.
Data extraction in component studies
For included studies to be analysed, full-texts were obtained to check whether there is any difference of reported information for acupuncture treatments in terms of STRICTA items between the primary component studies and the Cochrane review. This was carried out to identify any selective omission of treatment-related information by the Cochrane review author. Published protocols or supplements of included studies for more detailed information were further sought when the primary component studies had relevant information for tracking such data (ie, citation information in the reference or web-appendix provided by the component study).
Selection of STRICTA items for data extraction
Acupuncture-related items of STRICTA were used to assess the quality of reporting for acupuncture treatment-related information in Cochrane reviews of acupuncture. Items with regard to the description of the comparison interventions were not used, because the main interest of this review was to assess the completeness of description of the acupuncture treatment itself. The revised STRICTA checklist published in 2010 served as a primary source of extraction form of acupuncture-related information.3
Rating methods of items in STRICTA
Each item of STRICTA was rated with a dichotomous scale (ie, ‘reported’ or ‘not reported’). The rating of ‘reported’ was given when relevant information is at least partially reported in the Cochrane review or primary component studies. The rating of ‘not reported’ was given when relevant information is completely lacked in the Cochrane reviews or primary component studies. When there is written evidence in the Cochrane review that the review authors attempted to report and sought the relevant information but could not find it in the component study, the item was rated as successfully reported one in the Cochrane review. Our approach is in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statements that stressed the importance of reporting all variables for which data were sought, regardless of its availability in the component studies.7 STRICTA recommends reporting of actually performed acupuncture intervention, rather than reporting of treatment protocols without any evidence of implementation. However, we assumed the reporting of Cochrane reviews or of component studies as performed, because distinctions were not clear in most cases.
Inquiries to the review authors and review groups
One author (KHK) visited websites of relevant Cochrane review groups (CRGs) to see whether any reporting guidelines for acupuncture-related intervention were provided for review authors (accessed on 12 July 2012). Email queries were sent to the contact point of CRGs. Two questions were used for the survey as follows: ‘Do your CRG have a policy or specific recommendations about the reporting details of acupuncture interventions for review authors? If any, please specify’ and ‘Have you ever recommended any specific guideline for reporting details of acupuncture interventions for the review to review authors? If any, please specify’, respectively. Similar email queries with four questions were sent to the correspondence of the included Cochrane reviews (table 1). This was carried out to check whether CRGs or review authors had been aware of the existence of reporting guidelines on acupuncture (ie, STRICTA), had or had been recommended to use reporting guidelines for detailed description of interventions regarding acupuncture.
Table 1.
Response of review authors for the reporting of acupuncture details in the Cochrane review
| N=25 | Q1 (%) | Q2 (%) | Q3 (%) | Q4 (%) |
|---|---|---|---|---|
| Yes | 11 (44) | 4 (16) | 0 (0) | 0 (0) |
| No | 2 (8) | 5 (20) | 10 (40) | 8 (32) |
| No answer | 12 (48) | 16 (64) | 15 (60) | 17 (68) |
Q1. Are you aware of any guideline or recommendation for reporting details of acupuncture treatment in journal publication?
Q2. Have you ever referred or used any reporting guidelines or recommendations when you reported the details of acupuncture treatments in your Cochrane review?
Q3. Have you ever received any guideline or recommendation for reporting details of acupuncture treatment from the Cochrane review group, when writing or updating your review?
Q4. Have you ever received any constraints or restrictions when reporting the details of acupuncture treatments in your Cochrane review?
Training of trial assessors
Two reviewers (KHK and JWK), experienced acupuncture researchers and systematic reviewers, received training on STRICTA assessment checklists. The training course aimed to minimise inconsistency in data extraction and scoring. Ten trials in the Cochrane reviews that were published before 2005, thus excluded in our analyses, were randomly selected for initial scoring by both reviewers. Two reviewers independently assessed and scored the quality of reporting in terms of acupuncture treatment-related information of 10 component trials in the Cochrane reviews. Following this, we attempted to develop and reassure the standardised scoring instruction for the main analysis.
Statistical analysis
The number of items rated as ‘reported’ in the Cochrane review was compared with those in the relevant primary component studies. Results were presented as risk ratios (RRs) with 95% CIs. Random effects model was used, and heterogeneity was analysed using χ2 test on N-1 degrees of freedom, with an α of 0.05 used for statistical significance and with the I2 statistic. The number and proportion of component studies in the Cochrane reviews and the original randomised controlled trials (RCTs) that reported information for each STRICTA item were compared using the McNemar test as percentage reported with binomial 95% CIs. STATA V.10.0 (Stata-Corp, College Station, Texas, USA) and RevMan V.5.2 (The Nordic Cochrane Centre, Copenhagen, Denmark) were used for statistical analyses and the forest plot, respectively.
Results
A total of 74 Cochrane reviews were identified following the search of CDSR, of which 26 reviews with 96 component studies were deemed to be eligible. Among them, one review and four component studies were excluded from the analysis, because of the wrong citation (n=3) and no accessibility (n=1) of component studies that resulted in the analysis of 25 reviews and 92 component studies (figure 1).
Figure 1.
Flowchart of Cochrane reviews and component study selections. CDSR; Cochrane Database of Systematic Reviews.
Study characteristics
Of the 25 included Cochrane reviews of acupuncture, 17 reviews (68%) evaluated acupuncture as a primary intervention of interest. The median value of publication year or year that last update was performed was 2010. In total, 12 reviews (48%) attempted to investigate the clinical heterogeneity related to acupuncture treatment by the means of subgroup, sensitivity or other type of analysis (ie, acupuncture adequacy test).
Of the 92 component studies, 58 studies (63%) were published in English. The median value of publication year of component studies published after 2005 was 2007. Type of comparison to acupuncture was non-acupuncture interventions (n=68, 74%), sham acupuncture (n=34, 37%) and active acupuncture treatments (n=14, 15%). Details of characteristics for Cochrane acupuncture reviews and component studies are provided in table 2.
Table 2.
General characteristics of Cochrane reviews related to acupuncture
| Included Cochrane reviews N=25 | |
|---|---|
| Number of Cochrane reviews, N (%) | |
| Acupuncture as primary | 17 |
| Acupuncture as secondary | 8 |
| Number of component study | |
| Total | 409 |
| Mean | 16.36 (11.77) |
| Median (Q1–Q3) | 13 (3–47) |
| Number of component study published after 2005 | |
| Total | 92 |
| Mean | 3.7 (3.85) |
| Median (Q1–Q3) | 2 (1–5) |
| Publication language of component studies after 2005, N (%) | |
| Total | 92 |
| English | 58 (63) |
| Languages other than English | 34 (37) |
| Number of component studies published in | |
| Acupuncture-related journal | 47 |
| Western medicine journal | 43 |
| Publication year | |
| Cochrane review (Q1–Q3) | 2010 (2008–2011) |
| Component studies (Q1–Q3) | 2007 (2006–2008) |
| Types of control*, N (%) | |
| Sham acupuncture | 34 (37) |
| Active acupuncture | 14 (15) |
| Non-acupuncture | 68 (74) |
| Clinical heterogeneity related to acupuncture | |
| Investigated | 12 (48%) |
| in subgroup analysis | 5 (20%) |
| in sensitivity analysis | 6 (24%) |
| in other analysis | 8 (32%) |
| Not investigated | 13 (52%) |
*Sum of the number of each control group may exceed the total number of included trials, since there are trials that have more than two control groups.
Q1–Q3: IQR 1–3.
Review policy or recommendations for the reporting acupuncture intervention
No CRG policy or recommendations given to the review author for reporting of acupuncture treatment were identified by an email survey to the CRG correspondence or review authors. Five of 16 CRGs replied that the Cochrane Handbook should serve as a general standard guideline for reporting in the review, although no detailed instructions or any recommendations for reporting the details or at least core components of complex interventions such as acupuncture were provided to authors. One CRG of the Cochrane review that did not evaluate acupuncture as primary intervention reported that they sought the feedback from an acupuncture specialist during editorial process. Less than half of the reviewers (44%) reported that they were aware of STRICTA and only 16% of reviewers had utilised STRICTA during the review process. The reviewers did not receive either recommendation or restriction in terms of reporting acupuncture treatment during their review process (table 1).
STRICTA reporting in the Cochrane reviews and in the component studies
RR of the reporting rate of STRICTA items was 0.84 (95% CI 0.79 to 0.88, I2=8%), indicating that Cochrane reviews were 16% less likely to report the acupuncture-related items of STRICTA than component RCTs (figure 2). Information was significantly better reported for 10 of the 15 acupuncture treatment-related items of STRICTA in RCTs than in Cochrane reviews (p<0.05), while four items did so without statistical significance (table 3). The most significant difference of reporting rate was found for the item A2 (reasoning of treatment) and B7 (needle type) with better reporting in 39.1% and 45.7% of RCTs, respectively. One item (E1) related to practitioner background was significantly better reported in Cochrane reviews. Four items including A2 (reasoning of treatment), B3 (depth of insertion), D1 (details of other treatments) and D2 (setting and context) were reported in less than half of component studies in the Cochrane reviews and of RCTs.
Figure 2.
The number of reported items of STRICTA in Cochrane reviews of acupuncture and related RCTs. RCT, randomised controlled trial.
Table 3.
The number of component studies in Cochrane reviews and of original RCTs with reporting of selected STRICTA items
| Items | Component studies in Cochrane reviews |
Original RCTs |
Difference (95% CI)* | ||
|---|---|---|---|---|---|
| n/N | (%) | n/N | (%) | ||
| A. Acupuncture rationale | |||||
| (1) Style of acupuncture treatment | 59/92 | (64.1) | 66/92 | (71.7) | 7.6 (1.1 to 14.1) |
| (2) Reasoning of treatment | 6/92 | (6.5) | 42/92 | (45.6) | 39.1 (28.1 to 50.2) |
| (3) Extent to which treatment was varied | 76/92 | (82.6) | 87/92 | (94.6) | 12.0 (4.2 to 19.5) |
| Total section A | |||||
| B. Needling details | |||||
| (1) Number of needles | 76/92 | (82.6) | 84/92 | (91.3) | 8.7 (1.9 to 15.5) |
| (2) Names of points | 74/92 | (80.4) | 84/92 | (91.3) | 10.9 (2.7 to 19.0) |
| (3) Depths of insertion | 36/92 | (39.1) | 41/92 | (44.6) | 5.4 (−7.1 to 17.9) |
| (4) Response to needle | 53/92 | (57.6) | 63/92 | (68.5) | 10.9 (0.0 to 21.7) |
| (5) Needle stimulation | 42/92 | (45.7) | 64/92 | (69.6) | 23.9 (13.6 to 34.2) |
| (6) Retention time | 69/92 | (75.0) | 81/92 | (88.0) | 13.0 (4.4 to 21.6) |
| (7) Needle type | 18/92 | (19.6) | 60/92 | (65.2) | 45.7 (34.4 to 56.9) |
| Total section B | |||||
| C. Treatment regimen | |||||
| (1) Number of sessions | 85/92 | (92.4) | 90/92 | (97.8) | 5.4 (−0.3 to 11.2) |
| (2) Frequency/duration | 85/92 | (92.4) | 88/92 | (95.7) | 3.3 (−2.5 to 9.1) |
| Total section C | |||||
| D. Treatment context | |||||
| (1) Details of other treatments | 15/92 | (16.3) | 31/92 | (33.7) | 17.4 (8.6 to 26.2) |
| (2) Setting and context | 2/92 | (2.2) | 19/92 | (20.7) | 18.5 (9.5 to 27.5) |
| Total section D | |||||
| E. Practitioner background | |||||
| (1) Description of acupuncturists | 56/92 | (60.9) | 51/92 | (55.4) | −5.4 (−15.2 to −4.4) |
*Higher number means favourable results to RCTs.
RCTs, randomised controlled trials.
Discussion
This study is the first systematic investigation of selective reporting in terms of details of interventions in the Cochrane reviews of acupuncture. We found some items which have been regarded as core components of acupuncture treatments were not reported in the Cochrane review, even though that information was already reported in the component RCTs. Details of interventions were significantly better reported in the RCTs in most items of STRICTA. This was achieved without further contact of authors of RCTs, which might imply the improvement of the reporting quality in terms of intervention details could be achieved directly by adopting STRICTA as a guideline for data abstraction by reviewers, editorials and peer-reviewers. However, no specific recommendation or guidelines for review authors were provided by CRG, and only a minor portion of review authors or members of CRGs were identified to be aware of the existing CONSORT extension version of acupuncture. Overall, there is an insufficient reporting of acupuncture details which could have been improved if the existing reporting guidelines were well utilised in the review process.
In our review, less reporting of treatment-related information in the Cochrane reviews was observed in most of the selected STRICTA items. To some extent, missing details of complex intervention might be inevitable during abstraction of information due to the complexity in practice.5 However, some items showed almost compatible or even better reporting quality in the Cochrane reviews than in the RCTs. For example, information regarding the practitioner qualification was relatively well reported (over 60% of component studies). Some of the mechanical aspects of acupuncture treatments were also relatively well reported in Cochrane reviews and primary component studies, although some showed modest reporting quality in Cochrane reviews and RCTs. Item B7 (needle type) showed the most difference (45.7% difference) of reporting rates between Cochrane reviews and primary component studies. This is in line with the previous research that common missing element of non-pharmacological interventions in RCTs were materials necessary to provide the intervention (47% complete) published in six major general journals.9 On the contrary, poor reporting of contextual factors (D1, details of other treatments; and D2, setting and context) was obvious in Cochrane reviews and primary component studies. Although treatment context can have a significant influence on treatment effects,10 reporting information related to study contexts and patient–practitioner interactions seems to have been largely ignored by researchers of complex intervention.11 Since acupuncture is a practitioner-dependent, non-pharmacological complex intervention,12 sufficient details of intervention delivered and study contexts might be of particular importance to enable readers to consider the application and reproduction of review results in different treatment settings.11 Items that explain theoretical background of acupuncture intervention were also largely under-reported in Cochrane reviews of acupuncture, although theoretically-derived therapeutic actions or strategies in complex interventions such as acupuncture may also have an important role in constituting overall therapeutic effects.12 It is still unclear for acupuncture which component exerts therapeutic effects which combination of potential therapeutic components works best, or whether combination of several components during the acupuncture treatment is additive or synergistic.13 Therefore, it would be a reasonable suggestion for reviewers to report component factors of acupuncture that seems potentially contributable to the effectiveness which are well-represented in STRICTA, until future researches elucidate the therapeutic role of each component or group of components in the acupuncture treatment.
One possible explanation of inconsistent and suboptimal reporting of acupuncture treatment-related information would be that the Cochrane reviewers might not perceive some items of STRICTA as relevant to their review and selectively depicted some items of STRICTA. In a small survey of Prady and Macpherson,1,4 some trialists and Cochrane review authors did not perceive some items of the previous version of STRICTA as relevant to acupuncture treatments. Another reason might be a lack of standard reporting guideline for interventions in the Cochrane review. Either Cochrane review authors or CRG correspondence reported that they did not know the existence of reporting guidelines for acupuncture. To date, however, there is no study investigating reasons for the selective reporting intervention-related information (including acupuncture) in Cochrane reviews. Since information related to the treatment intervention is essential to enhance the external validity, applicability and implementation of results of systematic review,5 barriers to the optimal quality of reporting for treatment intervention should be explored and tackled in future studies.
Removal of STRICTA-specific information in the reporting due to the journal's space constraints or suggestions of editors as well as peer-reviewers was addressed by some trial authors.14 This may be the case for the systematic review published in non-Cochrane medical journals which have limited word counts.14 In our research, the possibility of potential influence of space constraints on suboptimal reporting seems unlikely because Cochrane review allows review authors to describe the main characteristics of each included study without any space constraints. However, lack of comparison of the reporting quality of acupuncture intervention in Cochrane reviews with those in non-Cochrane reviews published in journals with space constraints do not provide evidence to support this argument. Future research incorporating Cochrane and non-Cochrane reviews may reveal whether there is any systematic differences of reporting quality in terms of the details of treatment intervention between Cochrane and non-Cochrane reviews, thus space constraints could be an influencing factor to the reporting quality of intervention-related information or not.
Evaluation of internal validity has been put on great emphasis in the Cochrane systematic reviews.9 15 However, evaluating applicability of review results may also be of similar importance, since uptake and reproduction of review results may be able to change the routine clinical practice towards evidence-based practice.5 Lack of consideration of external validity is a major criticism of systematic reviews and RCTs, and is a potential threat of poor uptake of evidence into routine clinical practice.16–19 Intervention details and treatment contexts which STRICTA aims to address are suggested as the determinants of external validity in trial, systematic review and guideline publications,20–22 although little is known whether detailed description of acupuncture interventions in systematic reviews can affect uptake of evidence in practice. Whether selective reporting of acupuncture interventions observed in our study would influence the uptake of evidence may deserve further research.
Strengths and weakness
STRICTA is the best representative recommendation agreed by international acupuncture experts, constituting core components of acupuncture treatment that may be inter-related.3 However, STRICTA had not been developed to measure the quality of reporting acupuncture interventions, although we summed the scores of each items to score the overall quality of reporting acupuncture interventions. Since this method is not based on the original purpose of STRICTA and there is no weighting criteria for each item, interpretations of our results should be cautious.8 The scores should not be understood as the exact quantitative estimates across the studies, but as possible indicators of incompleteness of reporting details of acupuncture.
Results of rating might be prone to subjective assessment, although extensive training of authors using CONSORT elaboration document3 was preceded. Two assessors were unblinded to the study characteristics, such as publication dates or allocation of acupuncture as a primary or control intervention. Since both assessors are acupuncture researchers as well as practitioners, they may be familiar with included acupuncture studies; we did not attempt to blind the assessors. Prady et al8 found that unblinded assessor gave higher scores to some of the acupuncture studies in terms of items of STRICTA, although this tendency was not consistent with the rest of the studies to be assessed. Nevertheless, unblinded assessors with prior knowledge of study characteristics in our review may have introduced bias.
The first STRCITA was published in 2001, and as a revised version in 2010. We adopted the revised version of STRICTA for studies which had been published before and after it. Thus, component studies published before the publication of revised STRICTA may not address some revised items assessed in this review. This may have yielded a systematic bias when assessing the component studies by disadvantaging articles following a previous version of the guideline.8 Nevertheless, the utilisation of the latest version of STRICTA was justified in our study with three reasons: first, the CONSORT initiatives recommended the use of most recently released version of reporting standards when reporting and analysing RCTs.20 Second, the ultimate aim of using STRICTA in this study was not to score component studies per each item, but to reflect whether and how Cochrane reviews report or omit essential component of acupuncture treatments which were deemed the best representation of consensus across international acupuncture experts. Third, we identified that selected components for this review in the original STRICTA and in the revised one is almost consistent enough to justify the use of the latest version of STRICTA. In this sense, the latest version of STRICTA was preferred.
Uptake of reporting guidelines by individual researchers and journal editors may take a longer period than expected. A survey of author instructions conducted in 2007 revealed that only 38% of 165 high-impact journals endorsed the CONSORT statement, which was initially published in 1996.21 The median values of the publication years of the primary component studies and Cochrane reviews used in this study were 2007 and 2010, respectively. The first STRICTA statement was published in 2001, and insufficient time may have elapsed to justify our research. This should be recognised as a weakness of our study, and future follow-up studies may overcome this issue.
We did not contact authors of the primary component studies to gain additional information with regard to the under-reported items of STRICTA. This was because our primary interest is to identify selective reporting/omissions of acupuncture-related information during the review process in the Cochrane reviews. Given the authors of primary component studies are often contacted to gain additional information when conducting systematic review, however, our study might not reflect common information-seeking procedure during review process. Author's contact would have brought to what extent Cochrane reviews could have comprehensively described the treatment-related information of acupuncture based on the standard author query process recommended in the Cochrane Handbook.22 Future follow-up study may address such limitation.
Implication for future research
Factors associated with how and why review authors selectively summarise the treatment intervention during the review process should be investigated to identify the potential barrier of optimal quality of reporting in Cochrane reviews of acupuncture. Recent empirical evidence supports that the utilisation of reporting guideline (ie, CONSORT, STROBE, TREND and STARD) with regard to general methodological issues during peer review processes improves the quality of publication in biomedical journals.23 Whether and how reporting guidelines for the description of treatment intervention such as STRICTA can be efficiently utilised during editorial and peer-review process and provided as author-support resources in CRG websites should be explored to improve the reporting quality of treatment intervention. Whether and how reporting items of STRICTA explaining theoretical background, mechanical aspects and contextual factors of acupuncture could contribute to the investigation of clinical heterogeneity of acupuncture treatments and their potential impacts on the direction and significance of effect estimates should also be explored. Whether loss of treatment-related information actually leads to an altered replicability of reported interventions in Cochrane reviews was not investigated in this study, and should also be investigated in future research.
Conclusions
The reporting quality of treatment details in Cochrane reviews of acupuncture was insufficient with regard to STRICTA recommendation, even though information was readily reported in primary component studies. STRICTA was rarely utilised by CRGs and review authors. Use of STRICTA guideline for the reporting treatment details in Cochrane reviews and peer-review process should be considered to improve the replicability and utilisation of review results in future research and clinical practice.
Supplementary Material
Acknowledgments
The authors would like to thank Dr Hugh MacPherson, Steve McDonald and Professor Tae-Hun Kim for their helpful comments on the drafts of the manuscript. They also thank Cochrane review authors and Cochrane Review Groups who participated in the survey for this work.
Footnotes
Contributors: KHK conceived and designed the study, searched the literature from electrical databases, and wrote the first draft of this manuscript. KHK and JWK involved in analysing the data. MSL and JDL provided critical comments and contributed to the interpretation of analysed results. All authors read and revised the draft.
Funding: This work was supported by the Korea Institute of Oriental Medicine (K13400).
Competing interests: None.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: Authors adhere to the data sharing statement of BMJ Open. Please send an email to the first author (KHK) to obtain additional unpublished data.
References
- 1.Schunemman HJ, Oxman AD, Higgins JPT, et al. Chapter 11: presenting results and ‘Summary of findings’ table. In: Higgins JPT, Green S. eds Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, 2008:336–7 [Google Scholar]
- 2.Shepperd S, Lewin S, Straus S, et al. Can we systematically review studies that evaluate complex interventions? PLoS Med 2009;6:e1000086. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.MacPherson H, Altman DG, Hammerschlag R, et al. Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA): extending the CONSORT statement. PLoS Med 2010;7:e1000261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Boutron I, Moher D, Altman DG, et al. Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann Intern Med 2008;148:295–309 [DOI] [PubMed] [Google Scholar]
- 5.Glasziou P, Meats E, Heneghan C, et al. What is missing from descriptions of treatment in trials and reviews? BMJ 2008;336:1472–4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Schunemman HJ, Oxman AD, Vist GE, et al. Chapter 12: interpreting results and drawing conclusions. In: Higgins JPT, Green S. eds Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, 2008:367–9 [Google Scholar]
- 7.Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Prady SL, Richmond SJ, Morton VM, et al. A systematic evaluation of the impact of STRICTA and CONSORT recommendations on quality of reporting for acupuncture trials. PLoS One 2008; 3:e1577. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Hoffmann TC, Erueti C, Glasziou PP. Poor description of non-pharmacological interventions: analysis of consecutive sample of randomised trials. BMJ 2013;347:f3755. [DOI] [PMC free article] [PubMed]
- 10.Kaptchuk TJ, Kelley JM, Conboy LA, et al. Components of placebo effect: randomised controlled trial in patients with irritable bowel syndrome. BMJ 2008;336:999–1003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wells M, Williams B, Treweek S, et al. Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials 2012;13:95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Paterson C, Dieppe P. Characteristic and incidental (placebo) effects in complex interventions such as acupuncture. BMJ 2005;330:1202–5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Macpherson H, Tilbrook H, Bland MJ, et al. Acupuncture for irritable bowel syndrome: primary care based pragmatic randomised controlled trial. BMC Gastroenterol 2012;12:150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Prady SL, Macpherson H. Assessing the utility of the standards for reporting trials of acupuncture (STRICTA): a survey of authors. J Altern Complement Med 2007;13:939–43 [DOI] [PubMed] [Google Scholar]
- 15.Higgins JPT, Altman DG. Chapter 8: assessing risk of bias in included studies. In: Higgins JPT, Green S. eds Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, 2008:189–241 [Google Scholar]
- 16.Eldridge S, Ashby D, Bennett C, et al. Internal and external validity of cluster randomised trials: systematic review of recent trials. BMJ 2008;336:876–80 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Ethgen M, Boutron L, Steg PG, et al. Quality of reporting internal and external validity data from randomized controlled trials evaluating stents for percutaneous coronary intervention. BMC Med Res Methodol 2009;9:24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Rothwell PM. Assessment of the external validity of randomised controlled trials. In: Rothwell PM. ed Treating individuals: From randomised trials to personalised medicine. Spain: Elsevier, 2007:61–82 [Google Scholar]
- 19.Purepong N, Jitvimonrat A, Sitthipornvorakul E, et al. External validity in randomised controlled trials of acupuncture for osteoarthritis knee pain. Acupunct Med 2012;30:187–94 [DOI] [PubMed] [Google Scholar]
- 20.http://www.consort-statement.org/ (assessed 9 Sep 2013).
- 21.Hopewell S, Altman DG, Moher D, et al. Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal ‘Instructions to Authors’. Trials 2008;9:20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Higgins JPT, Deeks JJ. Chapter 7: selecting studies and collecting data. In: Higgins JPT, Green S. eds Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, 2008:151–85 [Google Scholar]
- 23.Cobo E, Cortés J, Ribera JM, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ 2011;343:d6783. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.