A systematic review of the quality of publications reporting coronary artery bypass grafting trials

Abstract

Objective

Several studies have shown that the quality of reports of randomized controlled trials (RCTs) in medicine is variable and often poor, whereas the quality of those in surgery is unknown. We aimed to assess the quality of reports of RCTs in coronary artery bypass grafting (CABG) surgery when comparing off-and on-pump techniques.

Methods

From electronic searches of MEDLINE, the Cochrane Library, CINAHL, HealthSTAR and EMBASE, we identified RCTs published between 2000 and 2005 comparing off-and on-pump CABG. We assessed the report quality, using 35 items from the Consolidated Standards for Reporting Trials (CONSORT) statement and 54 additional indicators relevant to CABG surgery. Some of the indicators comprised several small parts, making the maximum possible total score 105. Two authors independently reviewed and assessed the reporting quality of each RCT. The level of agreement was assessed with kappa statistics, and disagreements were resolved by consensus. We expressed descriptive analyses as median and interquartile range; we used a generalized estimating equation (GEE) for data analysis.

Results

We included 50 trials, for a total of 5134 patients. The kappa value was greater than 0.6 for 73 of 105 (70%) indicators. The overall report quality score varied from 35 to 93 of 105. The CONSORT score reporting quality varied from 16 to 39 of 42. The quality of reporting was poor and insufficient for the methods (particularly, the sample size, allocation and blinding subsections), results and discussion sections. With GEE modelling, the reporting quality had a strong association with trial size, publication year, trial location and funding source, but not with the results and type of primary outcome.

Conclusion

The quality of the publications' reporting methods, results and discussion sections was suboptimal. It is critical that, in reporting surgical trials, authors follow the CONSORT guidelines as well as consider the surgical factors.

Coronary artery bypass grafting (CABG) plays a central role in the management of coronary artery disease. For more than 3 decades, surgeons have used cardiopulmonary bypass (CPB) to provide a still and bloodless field in which to accomplish optimal revascularization. The recent resurgence of operating on a beating heart, or off-pump, as an alternative procedure for coronary artery revascularization is intended to decrease the adverse events typically associated with CPB (on-pump CABG). Despite the growing body of literature, the results of off-pump CABG are controversial regarding its benefits or adverse effects, compared with on-pump CABG. Most of the available information on off-pump surgery is from observational studies. Currently, randomized controlled trials (RCTs) are applied to assess the superiority of off-pump CABG as a new technique against conventional on-pump CABG, but the results of these RCTs can only be understood if they are reported with high quality. The results of meta-analyses^1–3 performed on the available RCTs show no reduction in mortality but a significant reduction in several indicators of morbidities. Studies in other branches of medicine suggest that the quality of publications describing RCTs vary considerably in terms of their reporting quality,⁴ but no study has explicitly reviewed the quality of reports on RCTs in the surgical field.^5–13 The poor quality of published information could be misleading, and it might introduce difficulties in decision making and during peer review, systematic review or meta-analysis. Hence, we aimed to system-atically review the quality of published reports of RCTs comparing off-and on-pump CABG surgery. We discuss the results of this review, highlighting areas where improvements are needed and providing some direction on additional resources that can be used.

Methods

This systematic review was performed in accordance with a protocol that prescribed eligibility criteria, search strategy, outcomes and statistical analyses.

We performed an Ovid literature search to select potential reports according to the strategy for MEDLINE, in the Cochrane Reviewers' Handbook.¹⁴ We used similar strategies to search other databases, such as CINAHL, HealthSTAR and EMBASE. We also performed manual searches to identify articles missed in the computer-assisted searches. We limited the data searches to randomized controlled trials published in English between Jan. 1, 2000 and Dec. 31, 2005; subjects were aged 18 years and over. To refine the articles, we used the keywords “coronary artery bypass,” “CABG,” “off-pump,” “beating heart,” “OPCAB,” “on-pump,” “cardiopulmonary bypass,” “conventional CABG,” “ONCAB,” “valve,” “repair,” “balloon,” “stent,” “PCI,” “PTCA,” “angioplasty” and “drug.”

Eligibility criteria for study selection

Two authors independently identified trials for inclusion and extracted information on interventions and outcomes. A study was considered for inclusion if it was an RCT designed to compare off-pump CABG and on-pump CABG in adult populations, on the basis of at least 1 relevant clinical outcome. All studies were published in English and used human subjects. We included all RCTs with the primary objective of assessing a categorical or continuous variable as an outcome. For this reason, the number of trials in our study is different from the recently published meta-analysis.³ We excluded duplicate publications of previously published trials on the same patients. We included substudies of larger trials with a predefined protocol, eligibility criteria and different sample size. The time period for the inclusion of published studies was Jan. 1, 2000 to Dec. 31, 2005. We excluded 5 RCTs before the year 2000 (1 from 1995, 1 from 1998 and 3 from 1999). We chose this time period to better assess the reporting quality of CABG publications, because the medical journals adopted the Consolidated Standards for Reporting Trials (CONSORT) statement by 1996, which was developed for clinical trials. The CONSORT statement would have had a very marginal effect in CABG surgical trials until the emergence of the revised CONSORT statement.^4,15 We excluded studies that involved a combination of CABG surgery with another procedure, drug or device trials and subgroup analyses of an original study and those that only explained the rationale and design of the study.

Data extraction and trial assessment

The revised CONSORT statement¹⁵ has been used by many impact journals to criticize the quality of articles. We used a modified CONSORT statement⁴ and added the factors relevant to surgical trials and CABG surgery (Appendix 1). We created a data extraction form before data collection, and we used 89 indicators to assess the selected trials (35 indicators identified by the modified CONSORT statement and 54 indicators relevant to surgical trials or CABG surgery). Using a standardized data abstraction form, 2 authors independently reviewed the articles to assess the reporting quality of each RCT. The form was previously pilot tested and revised. Disagreements were resolved by consensus. One credit was awarded to each appropriately reported indicator, and no credit was assigned otherwise. Some of the indicators comprised several small parts. Therefore, the maximum possible score was greater than the number of indicators and varied from 0 to 42 for the CONSORT score and from 0 to 105 for the total score. We collected each reviewer's data set separately, using Microsoft Access (Microsoft Corporation, Redmond, Wash., 2000).

Definition of variables

When we found significant differences comparing off-and on-pump CABG, we defined the result of the primary outcomes as positive; otherwise, results were defined as negative. The trial size was decided based on the number of patients enrolled in each trial. The journal of publication refers to the journal in which the trial was published; the trial location refers to the location where the trial was conducted or, in the case of multicentre trials, to the location of the corresponding author. Type of primary outcome was defined as a categorical or continuous variable.

Data analysis

We performed Cohen's kappa analysis¹⁶ to measure the level of agreement between the reviewers on all items on the data abstraction form. A kappa of > 0.8 was considered good, 0.6–0.8 was substantial, 0.4–0.6 was moderate, 0.2–0.4 was fair and < 0.2 was poor. We performed 2 sets of analysis on extracted data, one on all 105 indicators and the other on the 42 CONSORT statement indicators as a sensitivity analysis. We reported categorical data as frequencies, percentages and 95% confidence intervals (CIs), using the Wilson Score method. Continuous data were reported as mean and standard deviation (SD), median, interquartile range (IQR) and minimum (min) and maximum (max) score. Tests of associations between total reporting score and sectional reporting scores were performed using partial correlation coefficient when the trial size, publication year, primary outcome results, journal of publication, type of primary outcome, funding source and trial location were held constant. We performed univariate analysis and multivariable analysis, using a generalized estimating equation (GEE)¹⁷ to model the total quality score as a function of final sample size, publication year, funding source, trial location, and type and result of primary outcome. GEE was chosen to incorporate the possible correlation between quality scores of publications in the same journal. We expressed the GEE results as a coefficient, corresponding standard error (SE), 95% CI and associated p value. We included variables, in the multivariable analysis, with a p value of < 0.1 on univariate analysis. We assessed goodness of fit, using the likelihood ratio statistic.¹⁸ All analyses were conducted with SPSS 15.0 and Stata 8.0.

Results

The MEDLINE database search identified 129 RCT articles published between Jan. 1, 2000 and Dec. 31, 2005. The combined search for additional databases yielded 114 trials for the same time period. After reviewing both databases, we identified 133 articles for systematic review. Another 4 articles were found by reviewing the references of the articles or metaanalyses. Of 137 articles, 50 trials fulfilled the eligibility criteria for this study (Fig. 1).^19–68 Seven of 9 duplicates or nonoriginal publications were published in 2005. A total of 5134 patients were studied, and the trial enrolment varied from 20 to 401 patients. Table 1 summarizes the characteristics of these trials. Thirty-two (64%) of 50 trials were published in the surgical journals, 31 (62%) were completely or partially funded by governmental agencies or foundations and 36 (72%) were conducted in Europe. The interreviewer level of agreement was good (κ > 0.8) for 47 (45%) and substantial (κ = 0.6–0.8) for 26 (25%) of 105 indicators; it was moderate (κ = 0.4–0.6) for 17 (16%) and fair (κ = 0.2–0.4) for 15 (14%) indicators.

FIG. 1. Identification of eligible randomized controlled trials between Jan. 1, 2000 and Dec. 31, 2005.^19–68

Table 1

Table 2 summarizes the total scores, the CONSORT scores and the breakdown of scores for each section. The total report quality varied from 35 to 93 out of a possible score of 105. The report quality assessed by CONSORT criteria varied from 16 to 39 out of a possible score of 42. The report quality varied from 7 to 39 out of a possible score of 47 for methods sections and from 13 to 35 of a possible 39 for results sections. Within method subsections, sample size (min = 0, max = 11 of a possible score of 17), allocation or blinding (min = 0, max = 7 of a possible 7) and protocol (min = 1, max = 10) of a possible 10) had the least report quality score, respectively. There was a significant positive correlation between the total report quality score and report quality scores of methods (correlation coefficient [r] = 0.85, p < 0.001), results (r = 0.38, p = 0.011), and discussion (r = 0.36, p = 0.018) sections of the trials while controlling for trial size, publication year, primary outcome results, trial location, funding source and type of primary outcome. The report quality scores of components of methods sections were also significantly and positively correlated with the total score. The correlation coefficient was 0.49 (p = 0.001) for protocol, 0.49 (p = 0.001) for allocation and blinding, 0.54 for sample size calculation (p < 0.001) and 0.76 (p < 0.001) for statistical methods, respectively.

Table 2

Table 3 presents the percentages of the trials that reported the CONSORT and surgical indicators adequately. The abstract or title described the study as an RCT, the introduction, description of study participants, surgical procedure, statistical methods of analysis for primary outcome, baseline data, results of primary outcomes and ancillary analyses were reported adequately in more than 90% of the trials. Indicators that were reported less frequently included the number of surgeons (46%), the learning curve for off-pump technique (20%), sample size (28%), method of generating allocation sequence (30%), person or group that generated allocation sequence (10%), blinding (34%), intention-to-treat analysis (30%), the participants' flow chart for recruitment and follow-up (16%), the study period (66%) and the strengths and weaknesses of the findings (51%).

Table 3

The detailed report quality for each item is presented in Appendix 1. An overall explanation of the study design was described adequately in only 24 trials (48%). In the protocol sections, 30% reported whether patients were consecutive, 60% reported the number of centres and 34% defined secondary outcomes. Although a sample size calculation was provided, there was little information on the desired significance level (28%), power (26%), the statement of smallest difference between groups (28%), the type of test used for primary outcomes (8%) or the method of analysis for the sample size calculation (4%). Other poorly presented indicators included handling missing information (40%), stating the method used in adjusted (40%) or subgroup (8%) analyses, and stating the software used for analysis (58%). Research Ethics Board approval and patient's informed consent were reported in more than 80% of the trials.

GEE modelling results

Univariate GEE analysis showed that the trial size, publication year, location of trial, funding source and type of primary outcome were significantly associated with the total quality of reporting score when we adjusted for the journal of publication (Table 4). The results of primary outcome (negative or positive) were not a significant factor in relation to the quality of reporting CABG trials. When we adjusted for the journal of publication, multivariable analysis of GEE revealed that the sample size of the trial was significantly and positively associated with the quality of reporting (Table 5). With respect to the publication year, the report quality of publications for CABG surgery was significantly and positively improved in 2002, 2004 and 2005, compared with 2000, and this improvement was greatest in 2005. The report quality in 2001 and 2003 was similar to that in 2000. With respect to the trial location, the quality of reporting was similar between North America and the United Kingdom, but that of CABG trials was significantly lower for Scandinavia and other countries. The funding source and type of primary outcome (continuous v. categorical) were not significantly associated with the quality of reporting. Most trials with a continuous variable as the primary outcome had positive results (66%), compared with those with categorical data as the primary outcome (34%).

Table 4

Table 5

Discussion

We comprehensively searched the databases to identify RCT articles comparing any type of outcome between off-and on-pump CABG surgery. Only 50 original RCT articles published between Jan. 1, 2000 and Dec. 31, 2005 met our inclusion criteria. The total reporting quality of trials comparing off-and on-pump CABG surgery in this systematic review varied substantially between publications (35–93 of a possible score of 105). The results of this systematic review showed that there is room for improvement. The reporting quality of the methods was poor, with insufficient and inadequate information. Information on the methods and study design, allocation sequence (30%), blinding mechanism (34%), sample size calculation (28%), intention-to-treat approach (30%), number of surgeons in the study (46%), learning curve and surgeon's experience with the off-pump procedure (20%) was minimal in most of the published trials. Although all of the trials reported the number of patients (sample size) in the final reports, the details on sample size calculation were lacking from most of the studies. Because the present study only reviewed trials published in peer-reviewed journals, we expected that the quality of reports on CABG trials (when including nonpeer-reviewed reports) would be even lower. The publication year and the size and location of the trials were significantly associated with the reporting quality of the CABG trial findings. The reasons for improving the reporting quality of CABG trials in recent years are as follows: 1) authors of surgical trials are gaining more experience and skill at conducting RCTs and writing manuscripts and 2) referees and journals are demanding better quality in reports of RCTs, since medical journals adopted the CONSORT statement. However, these improvements are not as evident in reporting indicators relevant in surgical trials.

In the hierarchy of evidence, RCTs represent the highest quality design for evaluating clinical practice. They are most valuable when conducted with precision and when the results are adequately reported. The main reason for undertaking RCTs is to inform and alter medical practice. Without adequate information on allocation, randomization or blinding, the validity of a trial becomes questionable. Until recently, trials of surgical methods have been uncommon, and the decision making on surgical methods has been modelled on the experience gained from observational studies on a small number of patients. With the increasing popularity of RCTs in surgery, several pitfalls related to their conduct may introduce significant biases.

There are different types of RCTs that are applicable in surgery patients.⁶⁹ Type I studies the superiority of a new surgical procedure versus the conventional one. Type II compares a surgical technique to a laparoscopic method or to medical treatment. Type III does not study surgical methods but, instead, relates to such medical treatments as pain relief and prophylaxis (i.e., infection, thrombosis, bleeding) in surgical patients.⁷⁰ Although most of the difficulties apply to all 3 types, the first 2 require a more challenging design process. This systematic review focuses on type I studies.

A surgical intervention can be distinguished from a drug intervention by the skill required to administer the treatment. Drug trials do not risk any differential skills in administering an active medication versus placebo to patients; however, surgery is a skilled, complex and multistep process, which makes the design of RCTs problematic in surgical studies for several reasons.^70–72 Training experience is required to develop expertise in a surgical technique. There is a learning process in every new surgical technique, and the quality of performance improves with frequent repetition over time.⁷⁰ This learning process most likely varies from surgeon to surgeon. To avoid serious bias, the learning curve for the new procedure needs to be acknowledged and controlled at the design stage or evaluated at the analysis stage.⁷⁰ In a surgical trial, if a participating surgeon is performing the new technique as well as the conventional technique and has restricted expertise with the former, the results will be biased in favour of the conventional technique. Devereaux and colleagues⁷³ refer to this as differential bias. Surgeon's skill variation is another issue that poses a problem in conducting surgical trials. There is an inherent variation in different surgeons performing the same procedure.⁷⁰ The effect of trainees, fellows and other surgical teams in the operating room (OR) makes the problem even more profound. The recruitment time is a challenging issue in surgical trials, especially in rare conditions. It is difficult to recruit when patient accrual time is long or when the numbers needed to achieve an effect is large.⁶⁹ Another concern is that patients who have been recruited might become ineligible by the time they have surgery, because of long waiting lists. The classic method preferred to avoid outcome bias in clinical trials is double blinding, wherein neither the patients nor the investigators know which treatment the patient receives. In surgical RCTs, surgeons cannot be blinded to the type of surgery they are performing; patients usually know the type of surgery they are having because it is unethical and difficult for the surgeon not to tell them.⁶⁹ The blinded approach would be impossible when a surgical technique is being compared with a nonsurgical one. The best approach is to conduct a single-blind study in which the people who assessthe outcome and the data analysts are unaware of the patient's treatment. This is also difficult to guarantee because the assessor is usually the operating surgeon. It becomes difficult to get reliable results if a long time was required to observe the end points. It is often difficult to track a group of patients for a long time or to keep them interested in clinical visits if no treatment or minimal treatment is given after surgical intervention.

Within the aspects of study design, the lack of reporting adequate information on the timing of the trial, the definition of study population, the learning curve for the new procedure, the technical skill and ability of the surgeon(s), a trainee's assistance in the OR, variation in surgeons' experience with the surgical technique, allocation sequence and concealment, blinding mechanism, intention-to-treat analysis and recruitment time period all encounter certain barriers that undermine the validity, applicability and ethical integrity of RCTs in surgery. Surgical factors that are not incorporated in the CONSORT statement are very important and should be regarded with significance in surgical trials. Researchers and surgeons are encouraged to consider these factors when designing their trials and when reporting them in their publications. In the case of impracticality or lack of feasibility, the limitations inherent in their approach should be discussed in detail to allow readers to interpret the use of the findings. While our review did not investigate the accuracy of reporting statistical aspects, there were some instances where common errors, such as lack of reporting goodness of fit and assessment of model assumptions, were apparent. As part of the user's guide to the surgical literature series, Thoma and colleagues⁷⁴ critically reviewed a surgical RCT in general surgery and reached a similar conclusion.

Strengths and limitations

Due to the rigor of this systematic review, we are confident that it is a complete summary of the available evidence. There are, however, several limitations to our study. First, our study evaluated only the reporting quality of the trials and not the quality of the trials themselves. Soares and colleagues⁸ evaluated the methodology of the RCTs by the Radiation Therapy Oncology Group and concluded that “poor reporting of RCTs may not indicate poor quality of the trials themselves.” Because only published reports are available to readers, researchers should be encouraged to publish manuscripts that contain more details of experimental methods and to discuss the strengths and weaknesses of their protocol in greater detail. Several indicators of our instrument are subjective, for example, “Is the study design well explained?” Because indicators are recommended by the CONSORT statement and are necessary to report in clinical trials, we kept them and examined their accuracy by interreviewer analysis. Criteria for both the CONSORT statement and extended variables relevant to surgical RCTs or CABG surgery were not weighted for scoring purposes, based on the assumption that all of the indicators have equal importance. We avoided weighting because it would have been arbitrary and subjective and thus subject to criticism.⁴ This means that the CONSORT and surgical indicators, such as the learning curve for the off-pump technique, number of participating surgeons, sample size calculation, allocation and blinding, and title identifying the study as an RCT, are treated equally. To the best of our knowledge, we included all known baseline risk factors in the protocol; these depend on the patient population under study, which may vary between trials. To avoid underscoring bias, we included the risk factors in the inclusion and exclusion criteria. It is, however, possible that the baseline data section in some trials was underscored. These trials might have included a healthier patient population and did not need to report some of the risk factors. Finally, the modified CONSORT statement introduced in this manuscript and its scoring system are not validated, nor is there any validated tool to evaluate the reporting quality of the RCTs reported in the field of surgery.

Conclusion

Our findings are consistent with other studies on the quality of RCT publications in clinical trials,^4,75–77 although no studies have focused on surgical trials. The published CABG RCTs lack good reporting quality, especially with respect to the methods and discussion sections. The results of this review should strongly encourage journal editors to change the instructions to authors to ensure that the issues that affect the understanding of a manuscript, whether by referees or readers, and how the study was undertaken are adequately described. The implementation of CONSORT statements and surgical indicators by referees in reviewing surgical trials would facilitate peer review and enhance the scientific quality of the data retrieved from the publications. The CONSORT statements provide trialists, referees and editors with guidelines for improving the quality of RCT reports, but they do not cover the issues related to the surgical trials. Therefore, authors, editors and referees are encouraged to consider the CONSORT statements and factors relevant to the surgical trials described in this review. These criteria are likely generalizable to other fields of surgery, when an investigator is conducting an RCT while adjusting for baseline risk factors. To embrace this tool and its scoring system as a practical tool in the evaluation of surgical RCTs, it needs to be established and validated in the different fields of surgery.

Appendix 1.

Contributors: Drs. Farrokhyar and Thabane designed the study. Drs. Farrokhyar and Whitlock and Ms. Chu aquired the data, which Drs. Farrokhyar and Thabane and Ms. Chu analyzed. Dr. Farrokhyar wrote the article, and all authors reviewed it. All authors gave final approval for the article to be published.

Competing interests: None declared.

Accepted for publication Sept. 25, 2006

Correspondence to: Dr. Lehana Thabane, Centre for Evaluation of Medicines, Father Sean O'Sullivan Research Centre, 105 Main St. E., Level P1, Hamilton ON L8N 1G6; fax 905 528-7386; thabanl@mcmaster.ca