Application of the CONSORT statement to randomized controlled trials comparing endoscopic and open carpal tunnel release

Abstract

BACKGROUND:

The CONSORT (Consolidated Standards of Reporting Trials) statement was developed by a group of clinical trialists, biostatisticians, epidemiologists and biomedical editors as a means to improve the quality of reports of randomized controlled trials (RCTs). The purpose of the present study is to assess the reporting quality of published RCTs that compare endoscopic carpal tunnel release (ECTR) with open carpal tunnel release (OCTR) using the CONSORT statement.

METHODS:

A computerized literature search was conducted to identify all RCTs published from January 1989 to November 2004 that compared ECTR with OCTR. Foreign language studies were also included, and translated versions of these studies were obtained. Two investigators independently reviewed each eligible article and determined whether the authors reported on each of the 22 items of the CONSORT statement. Disagreements were resolved by consensus. The mean scores for studies published before the introduction of the CONSORT statement and those published afterward were compared. Similarly, a comparison was made between foreign language studies and those published in English.

RESULTS:

Eighteen RCTs comparing ECTR with OCTR met the inclusion criteria. The total scores on the CONSORT checklist ranged from 3 to 20, with a mean score of 9.83±3.79 (the maximum possible score was 22). The six studies published in foreign language journals had a statistically significantly lower mean score than the studies published in English language journals (7.00±2.76 versus 11.25±3.49, respectively; P<0.05). The mean score was higher for studies published after 1996 than for those published in 1996 or earlier (12.14±3.80 versus 8.36±3.11, respectively; P<0.05).

CONCLUSIONS:

The quality of reporting improved over time, but no study met all 22 criteria of the CONSORT statement. The CONSORT scores were higher for studies published after 1996 and for studies published in English language journals. Despite the improvement after 1996, most of these RCTs only reported one-half of the items listed on the CONSORT statement. Future investigators of surgical RCTs should make an effort to comply with the CONSORT checklist.

The randomized controlled trial (RCT) is generally regarded as the most scientifically rigorous study design to evaluate the effect of a surgical intervention because it offers the maximum protection against selection bias (1,2). Health research methodologists in general recognize the following hierarchy of strength of evidence for surgical treatment decisions: ‘mega’ RCT (ie, one with a large sample size: thousands of patients versus hundreds of patients); systematic review of RCTs or meta-analysis; single RCT; systematic review of observational studies addressing important patient outcomes; single observational study addressing important patient outcomes; physiological studies such as rodent experiments on nerve regeneration or bone healing; and unsystematic clinical observations.

The popular beliefs that only the RCT produces trustworthy results and that all observational studies are misleading do disservice to patient care, clinical investigation and the education of health care professionals (3). In investigating harm causation, an RCT is not feasible, necessary or appropriate. Nevertheless, a large RCT is the ideal study design to answer questions related to the effectiveness of surgical interventions, when the effect size is small to moderate (4).

Historically, surgical RCTs are carried out less frequently than RCTs that compare medical interventions, partly because using an RCT to evaluate a surgical procedure is difficult and demands special consideration of issues such as blinding, the effects of the surgeon factor, the learning curve, and the differences in pre- and postprocedural care (5). However, RCTs are becoming more common in surgical literature as more surgeons become acquainted with the research methodology required to conduct them (5). The mere reporting of a trial as randomized does not allow surgeons to infer its validity. When reading a report of an RCT in surgical literature, it is important to know the quality of the methodology used. The RCT report should convey to the reader why the study was undertaken, how it was conducted and how data were analyzed. All of this information has to be reported in a clear and transparent fashion. There is evidence suggesting that inadequately reported randomization in RCTs has been associated with bias in estimating the effectiveness of interventions (2,6).

In the mid-1990s, two independent initiatives were undertaken by a group of international epidemiologists, clinical trialists, biostatisticians and biomedical editors to improve the reporting of RCTs. These led to the publication of the CONSORT statement (Consolidated Standards of Reporting Trials) (7), which is composed of a checklist and a flow diagram to be used by investigators when reporting an RCT (8).

An increasing number of medical journals require authors who submit manuscripts reporting the results of an RCT to include the CONSORT flow diagram (Figure 1), which shows the progress of patients throughout the trial in the manuscript (8). In addition, the CONSORT checklist (Table 1, columns 1 and 2) must be completed and submitted with the manuscript (8). The purpose of the present review is to assess the compliance of published RCTs that compared endoscopic carpal tunnel release (ECTR) with open carpal tunnel release (OCTR) to the CONSORT statement. In the preparation of a recent meta-analysis of ECTR versus OCTR (9), we identified several methodological weaknesses and decided to apply the CONSORT statement to all published RCTs comparing ECTR with OCTR.

Figure 1).

The Consolidated Standards of Reporting Trials (CONSORT) flow diagram that demonstrates progress through the phases of a randomized controlled trial. Reproduced with permission from reference 8

TABLE 1.

Number of studies reporting each item on the Consolidated Standards of Reporting Trials (CONSORT) statement checklist (n=18)

Item	Descriptor	Number of studies, n (%)
Title and abstract	How participants were allocated to interventions (eg, ‘random allocation’, ‘randomized’or ‘randomly assigned’)	18 (100)
Introduction
Background	Scientific background and explanation of rationale	17 (94)
Methods
Participants	Eligibility criteria for participants, and the settings and locations where the data were collected	7 (39)
Interventions	Precise details of the interventions intended for each group and how and when they were actually administered	17 (94)
Objectives	Specific objectives and hypotheses	12 (67)
Outcomes	Clearly defined primary and secondary outcome measures and, when applicable, any methods used toenhance the quality of measurements (eg, multiple observations, training of assessors, etc)	4 (22)
Sample size	How sample size was determined and, when applicable, explanation of any interim analyses and stopping rules	1 (5.6)
Randomization
Sequence generation	Method used to generate the random allocation sequence, including details of any restriction (eg, blocking, stratification)	6 (33)
Allocation concealment	Method used to implement the random allocation sequence (eg, numbered containers or central telephone), clarifying whether the sequence was concealed until interventions were assigned	6 (33)
Implementation	Who generated the allocation sequence, who enrolled participants and who assigned participants to their groups	3 (17)
Blinding (masking)	Whether participants, those administering the interventions and those assessing the outcomes wereaware of group assignment. If not aware, this includes how the success of masking was assessed	5 (28)
Statistical methods	Statistical methods used to compare groups for primary outcome(s) and methods for additional analyses, such as subgroup analyses and adjusted analyses	13 (72)
Results
Participant flow	Flow of participants through each stage (a diagram is strongly recommended). Specifically, for each groupreport the numbers of participants randomly assigned, receiving intended treatment, completing the studyprotocol and analyzed for the primary outcome. Describe protocol deviations from study as planned, together with reasons	1 (5.6)
Recruitment	Dates defining the periods of recruitment and follow-up	9 (50)
Baseline data	Baseline demographic and clinical characteristics of each group	10 (56)
Numbers analyzed	Number of participants (denominator) in each group included in each analysis and whether the analysis wasby ‘intention to treat’. State the results in absolute numbers when feasible (eg, ‘10 of 20’, not ‘50%’)	2 (11)
Outcomes and estimation	For each primary and secondary outcome, a summary of results for each group, and the estimated effect sizeand its precision (eg, 95% CI)	5 (28)
Ancillary analyses	Address multiplicity by reporting any other analyses performed, including subgroup analyses and adjustedanalyses, indicating those prespecified and those exploratory	3 (17)
Adverse events	All important adverse events or side effects in each intervention group	17 (94)
Discussion
Interpretation	Interpretation of the results, taking into account study hypotheses, sources of potential bias or imprecisionand the dangers associated with multiplicity of analyses and outcomes	3 (17)
Generalizability	Generalizability (external validity) of the trial findings	1 (5.6)
Overall evidence	General interpretation of the results in the context of current evidence	17 (94)

Reproduced with permission from reference 8

METHODS

A computerized literature search of the Cochrane, MEDLINE, EMBASE, CINAHL and HealthSTAR databases was conducted for the period from January 1989 to November 2004. The search was limited to this time period because the first ECTR was described in 1989 (10). MeSH or the key words ‘endoscopic’, ‘open’, ‘versus’ or ‘vs’, and ‘carpal tunnel release’ were used. The result of this search was then combined with articles found using the MeSH or key word of either ‘randomized’ or ‘randomization’. Personal files of patients with carpal tunnel syndrome were also hand-searched. The inclusion criterion for the present study was that the study was designed as an RCT that compared OCTR with ECTR. Foreign language studies were also included, and translated versions of these studies were obtained.

Two investigators (RTC and KV) independently reviewed each article and determined whether the RCT reported on each of the 22 items of the revised CONSORT statement (11). Each reviewer independently gave a ‘yes’ or ‘no’ answer to each question on the CONSORT checklist. Disagreements between the reviewers were resolved by consensus, if possible, and a third investigator (AT) acted as an arbitrator for items where consensus could not be reached. The kappa statistic for chance-corrected agreement was calculated. A score out of 22 was given for each study, with one point assigned to each item of the CONSORT statement satisfied. The items were not weighted because the CONSORT statement is not a validated instrument. The mean scores for studies published before the introduction of the CONSORT statement (7) and those published afterward were compared using the Mann-Whitney U test. Similarly, a comparison was made between foreign language studies and studies published in English. P<0.05 was considered statistically significant.

RESULTS

Eighteen RCTs comparing ECTR with OCTR were identified during the study period (12–29). Ten of these studies were published in English, four in German, two in Dutch, one in French and one in Portuguese. The complete publications for all six foreign language studies were translated to English by a professional medical translator. The two reviewers (RTC, KV) disagreed on 20 items in nine of the studies with a kappa value of 0.90 and a third investigator (AT) was needed to resolve the differences on all disagreed items.

The majority of the studies did not report on most of the items on the CONSORT statement, and none of the studies reported all 22 items. The total scores on the CONSORT checklist ranged from 3 to 20, with a mean score of 9.83±3.79 (the maximum possible score was 22). The findings are summarized in Table 1; notably, only one study (5.6%) reported how sample size was determined a priori or clearly reported the flow of participants through each stage of the study. While all of these studies were reported as RCTs, only 33% (n=6) provided information on sequence generation, 33% (n=6) mentioned concealment of the allocation sequence and 17% (n=3) described the implementation of the randomization. Furthermore, only two studies (11%) reported on all three items relating to randomization, two studies (11%) reported on two items and five studies (28%) reported only one item, with the remaining nine studies (50%) not reporting any information regarding randomization.

Seven studies (39%) reported adequate information regarding the eligibility criteria for study participants, and only two studies (11%) reported whether the analysis was by ‘intention to treat’. The eight studies published in foreign language journals had a significantly lower mean score than the studies published in English journals (7.00±2.76 versus 11.25±3.49, respectively; P<0.05). Eleven studies were published before or in 1996 (Figure 2). The mean total scores on the CONSORT checklist by year of publication are shown in Table 2. The mean score for studies published after 1996 was significantly higher than for those published in 1996 or earlier (12.14±3.80 versus 8.36±3.11, respectively; P<0.05).

Figure 2).

Number of studies published by year of publication

TABLE 2.

The mean Consolidated Standards of Reporting Trials (CONSORT) statement score of studies by publication year

Publication year	Mean score ± SD
1992	7.00±3.00
1993	15.00±3.00
1994	6.00±3.00
1995	10.00±3.00
1996	7.60±0.81
1997	11.00±3.00
2000	8.00±3.00
2002	11.50±0.50
2003	14.33±2.96

DISCUSSION

The overall reporting quality of published RCTs that compared ECTR with OCTR was extremely poor. Incomplete reporting of methodology in studies limits the reader’s ability to assess the validity of results and may be associated with bias of these studies’ findings. We did detect a statistically significant improvement in the quality of reporting in the RCTs after the publication of the CONSORT statement in 1996; however, most of the improved studies still satisfied only one-half of the items in the CONSORT statement.

The study that had the highest score (20 of the 22 items) failed to report on the implementation of randomization and sample size calculation (28). Given that their results found no significant differences in symptom and functional activity scores, grip strength or pillar pain in the first three months between the OTCR and ETCR groups, sample size becomes an important issue because negative findings could be related to inadequate statistical power to detect a difference (28).

A well-designed and well-reported RCT in surgery should meet all of the criteria of the CONSORT statement (4). With adequate reporting, readers will understand what was actually done, rather than assume what was done. The CONSORT items do not actually assess the quality of the methodology of an RCT, but rather assess the reporting of key items that are crucial in determining the validity and quality of the RCT. The CONSORT checklist was developed as a guideline, not as an actual scale for assessing methodology of an RCT. In the present study, we may have underestimated the quality of the RCTs; the authors may have used the correct methodology, but may not have explicitly reported all of the methodology used. The clarity of the items on the CONSORT checklist is evidenced by the high kappa scores between the two independent reviewers.

The quality of RCTs can be assessed using validated scales, such as the Detsky scale (30) or the Jadad scale (31). However, if the RCTs are not adequately reported, it is not possible to assess their quality. The studies we evaluated in the present report generally reported poorly on many items deemed important on the Detsky scale (ie, randomization, outcome measures, participant criteria, sample size and precision of measurement), making it difficult for any reader to determine the quality and validity of results without needing to make various assumptions. Investigators should therefore use the CONSORT statement to ensure that their study methodology is clearly reported. The readers of the literature may then use the Detsky scale or the Jadad scale to assess the quality of the research.

Some journals require that manuscripts reporting the results of RCTs include the CONSORT flow diagram (Figure 1) showing the progress of patients throughout the trial, and that the CONSORT checklist (Table 1) also be completed and submitted with the manuscript. The Journal of the American Medical Association (JAMA), The Lancet and Annals of Internal Medicine all endorse the CONSORT statement. Even in those journals that require CONSORT compliance reporting on submitted trials, the published RCTs are not always 100% CONSORT-compliant (32). The RCTs included in the present review were published in multiple journals. Of the journals published in English, only two, The Journal of Bone and Joint Surgery and The Journal of Hand Surgery (British and European Volume), require the CONSORT checklist and flow diagram to accompany any reports of RCTs.

It is interesting to find that the foreign language studies had a significantly lower mean score than the studies published in English literature. Because the CONSORT statement was developed in English, it may explain why the non-English journals had less stringent requirements for publication. The CONSORT statement has recently been translated into multiple languages, including Chinese, Dutch, French, German, Italian, Japanese, Korean, Portuguese, Russian and Spanish.

The findings of the present study suggest that many surgeons are not familiar with the CONSORT statement, despite it being published a decade ago, in 1996. The findings also suggest that surgical investigators are carrying out RCTs that compare techniques of carpal tunnel release (or any other area of surgery for that matter), while lacking the methodological skills to execute such studies. What does this mean? It means that the basic lack of understanding of how to properly design and execute an RCT only adds bias, or ‘noise’, to the prevailing controversy of which technique is superior. Some may consider it unethical to inadequately report an RCT when readers are not clear on the methodology used or whether the study was adequately powered (33). Because these studies may not adequately test the underlying hypotheses, they may be of limited clinical value, and therefore be unethical, in their exposure of patients to the risks and burdens of clinical research (33). Poorly designed studies are also unethical because they waste scarce resources, and if an incorrect result is published and this result becomes accepted as general practice, harm may be caused.

The alternative to poorly designed RCTs is surgeons becoming aware of the methodology of the CONSORT statement. It is entirely possible that the studies themselves were methodologically sound, but were incompletely described. There is no evidence that the failure to mention methodological details equates to the lack of methodological knowledge or skills. It is possible, however, that peer reviewers and editors are at fault for not insisting on complete descriptions of the studies as dictated by the CONSORT statement. The responsibility for reporting lies not only with the authors.

CONCLUSIONS

If the surgical community wishes to answer the question of which of the carpal tunnel release techniques is superior, a properly designed RCT needs to be undertaken that has an adequate sample size and power to answer the question once and for all (4). Failure to do so leads to a situation in which we remain uncertain for decades as to which technique is superior. This leads to a waste of scarce health care resources because we continue to use a technique that may be inferior. Our study focused on carpal tunnel literature merely as an example of the problem that undoubtedly exists in other areas of surgical research. Future investigators engaging in RCTs in surgery should ensure that the quality of reporting and execution satisfies the current expected standards through the use of the CONSORT statement checklist.