Abstract
Objectives Randomized controlled trials and systematic reviews of such trials are the gold standard for assessing the effectiveness of interventions. There have been concerns about the anecdotal evidence underpinning many of the interventions used and introduced into surgical care. The American College of Surgeons has prioritized the need for more trials and systematic reviews of trials.
To investigate the assertion that the methodological quality of studies conducted in surgery is in general poor and to assess the possible impact of new policy developments in the US, we sought to compare the number and proportion of published randomized controlled trials and systematic reviews in the leading two US and UK general surgical journals. Two reviewers systematically and independently hand searched all issues of these journals over a 12-month period to identify randomised controlled trials and systematic reviews.
Design Systematic searching and independent abstraction of data from all volumes of the top two general surgical journals published in the USA and the UK in 2004.
Setting 519 original reports in UK journals and 616 original reports in USA journals.
Main outcome measures Number and proportion of randomized controlled trials and systematic reviews.
Results Overall, the proportion of randomized controlled trials in all four journals was 5.6% (95% confidence interval [CI] 4.4-7.0) and 5.2% (95% CI 4.1-6.7) for systematic reviews. For journals published in the UK 29/519 (5.6%) of the publications were reports of randomized controlled trials, and for the USA journals this figure was 34/616 (5.5%); odds ratio [OR]=0.99, 95%CI 0.6-1.6; P=0.96. Systematic reviews were significantly more commonly reported in the UK journals: UK 37/519 (7.1%) versus USA 22/616 (3.6%); OR=0.48, 95%CI 0.3-0.8; P<0.01.
Conclusions The concerns expressed almost a decade ago remain valid: there are still very few reports of randomized controlled trials and systematic reviews published in leading USA and UK surgical journals, with relatively little difference between these countries in the proportion of reported studies employing these designs. The American College of Surgeons initiative has yet to make an impact.
INTRODUCTION
Randomized controlled trials and systematic reviews of homogeneous randomized controlled trials constitute the most robust form of clinical evidence.1-3 Concern has, however, existed for a number of years that the majority of surgical research is based on case reports and case-series raising questions about the strength of evidence underpinning many of the routinely performed interventions.4 Horton demonstrated, back in 1996, that only 7% of studies published in leading surgical journals presented data derived from randomized controlled trials.5
Most studies of operations have historically been retrospective case series, with randomized controlled trials accounting for less than 10% of the total.6-8 In cases where trials have been performed, these have often been small and poorly designed, this also leading to concerns about interpreting findings as their design affords them unwarranted credibility.9 Based on these and other data it is estimated that treatments in general surgery are half as likely to be based on rigorous evidence as treatments used in internal medicine.8,10
The American College of Surgeons has since 2000 been undergoing major restructuring, this includes the establishment of an Office of Evidence Based Surgery (now renamed to Continuous Quality Improvement) designed to facilitate, by providing the administrative and infrastructure support needed, the conducting of rigorous studies including randomized controlled trials and systematic reviews.3 Given this major boost, we hypothesized that there would exist greater momentum to undertake and publish studies employing these designs in the USA when compared to the UK. To investigate this, we compared the proportions of randomized controlled trials and systematic reviews published in leading general surgical journals in the UK and the USA.
METHODS
Using the ISI Web of Knowledge database [http://wok.mimas.ac.uk], we sourced the top two general, nonspecialty-based surgical journals published in the USA (Annals of Surgery and Archives of Surgery) and the UK (British Journal of Surgery and Annals of the Royal College of Surgeons). The rank of journals was judged by its impact factor. Our research period of interest was January-December 2004 and for this 12-month period, two researchers (SSP and RT) independently hand-searched all issues of these four journals.
Data were independently extracted and appraised the data onto a pre-piloted sheet. Our aim was to describe the designs employed and each original research study was categorized as employing one of the following approaches:
randomized controlled trial
systematic review
analytical study (case control, cohort, modelling, audit, survey, cross sectional)
descriptive/qualitative study (individual case study/series, surgical technique, focus group, in-depth interview study and ethnographic study).
Any disagreements were resolved through discussion, with referral to a third member of the team (AS) to arbitrate if necessary.
We excluded all other publications including editorials, non-systematic reviews, panel discussions, short communication letters with no original hypothesis/data, book reviews, errata, commentaries, critique, obituaries, presidential addresses, abstracts and Minerva.
Descriptive statistics were used to determine the proportion of the various types of studies reported and the χ2 test was used to compare difference in the proportions of these studies between USA and UK journals.
Previous studies have shown that randomized controlled trials in UK journals comprise approximately 5% of the general surgical research literature.7,8 In the USA, we estimated this figure might be about 20%.11 In order to have 80% power, at the 5% significance level, of detecting a difference of this magnitude, we calculated that we would need to identify approximately 55 reports of randomized controlled trials in each country. A calendar year's analysis of two journals from each country was deemed sufficient to yield these numbers of trials.
RESULTS
The overall proportion of the various studies for all four journals is shown in Table 1.
Table 1.
Number and proportion of reported studies employing different methodological designs
| Study type | UK (n=519) | USA (n=616) | Overall (n=1135) for UK and USA (with 95% confidence interval (CI)) |
|---|---|---|---|
| Randomized controlled trials [n (%)] | 29 (5.6) | 34 (5.6) | 63 (5.6, 95% CI 4.4-7.0) |
| Systematic reviews & meta-analyses [n (%)] | 37 (7.1) | 22 (3.8) | 59 (5.2, 95% CI 4.1-6.7) |
| Analytical [n (%)] | 178 (34.3) | 294 (48.8) | 474 (41.5, 95% CI 38.7-44.4) |
| Descriptive/qualitative [n (%)] | 116 (22.4) | 79 (12.8) | 195 (17.2, 95% CI 9.3-13.0) |
| All others [n (%)] | 159 (30.6) | 187 (30.4) | 346 (30.5, 95% CI 27.9-33.2) |
In the UK journals, 29/519 (5.6%) of publications were randomized controlled trials, this being very comparable to the picture in USA journals: 34/616 (5.5%); OR=0.99, 95% CI 0.59-1.64; χ2=1.01, P=0.96.
Turning to systematic reviews/meta-analysis, 37/519 (7.1%) of publications in UK journals employed this design, which is significantly higher than that in the USA publications: 22/616 (3.6%); OR=0.48, 95% CI 0.2-0.8; χ2=15.4, P50.01.
In the UK journals, 178/519 (34.3%) of studies were classified as analytical, compared with 294/616 (47.7%) of the same in the USA journals (OR=1.75, 95% CI 1.4-2.2, χ2=22.8, P50.001). In contrast, descriptive/qualitative studies were more common in UK journals: UK 116/519 (22.3%) versus USA 43/616 (12.8%); OR=0.51, 95% CI 0.4-0.7; χ2=25.4, P50.001.
DISCUSSION
Main findings
We have found that it is still the case that relatively few surgical papers employ rigorous designs. Our results are in keeping with those previously shown and demonstrate the lack of changes in the types of study conducted in a surgical arena.7,8 This picture is very comparable in both countries for randomized controlled trials, although systematic reviews are somewhat more common in the UK than the USA.
Strengths and limitations of the work
The main strength of this work are the explicit criteria for selecting journals for inclusion, the comprehensiveness of the searches of journals over the study period and the independent assessment of studies to ensure validity and reliability.
The limitations need to be appreciated. Our study has provided a 1-year generalized snapshot view of the state of surgical research by analysing original research reports in only four surgical journals published in the UK and USA. It provides some insight into trends over time;7,8 but we did not assess the trends in study size or the quality of the studies by using measurement scales such as those employed by the Cochrane Collaboration, as these were not reported for the original studies thus making direct comparison difficult.12 The different impact factors of the various journals incorporated into the study may also influence the outcome as different journals have different criteria on the particular types of studies that they will accept for publication. Surgical studies are also published in a range of other surgical journals and also in some generalist medical journals. We have in this study been unable to assess whether the picture uncovered in the present analysis is representative of the broader state of the general surgical literature. Furthermore, there are instances in which UKled studies are published in USA journals and vice versa. However, given that the impact factors for the USA journals were higher than those from the UK, any systematic misclassification error would have been likely to bias against the null hypothesis in question. It may also reasonably be argued that given the time it takes to secure funding and obtain ethics approval, execute, analyse and publish trial data, it was always unlikely that at this relatively early stage of its existence the Office of Evidence Based Surgery would have had the time to make much of an impact: that said, our work does provide a baseline from which to assess future progress of this key new initiative.
Considering the findings in relation to previous work
Several reasons have been cited for the poor proportion of randomized controlled trials in surgical journals. Unlike in medicine, where clinical trials are relatively easy to carry out using pharmacological interventions, surgical trials are difficult to conduct. Most surgeons would rather avoid the uncertainties, paperwork and hassle. Patients too may be reluctant to participate in these studies as they may have a preference for a particular technique, e.g. laparoscopic surgery. In addition, there are financial challenges—pharmaceutical companies do not provide funding. Also, in the USA the government imposes less stringent regulations on new operations and technologies than it does on new drugs. One can, therefore, appreciate surgeons' reluctance to undertake the large and administratively complex trials that are often needed in order to provide a secure evidence-base for procedures.2
Implications for research
Although the medical community accepts conventional surgical randomized controlled trials as ethical, some surgeons may have ethical problems with enrolling patients in a trial when they know they may have to do a procedure with which they feel inexperienced. This problem does not arise to the same extent in expertise-based randomized controlled trials because surgeons perform only the procedures in which they have established expertise.13 This may, therefore, represent a more acceptable strategy than employed hitherto to weaning academic surgeons off their unhealthy addiction to comic operas.5
CONCLUSIONS
There remains a need for funders, researchers and editors to prioritize the undertaking and reporting of studies employing robust designs in order to improve the body of evidence underpinning surgical practice.
Competing interests None declared.
Acknowledgments AS conceived this study, formulated the study protocol, and led the writing of the manuscript. SSP and RT extracted data. SSP was responsible for data analysis and write-up. TA helped to interpret the results. AS is guarantor.
References
- 1.Sheikh A, Smeeth L, Ashcroft R. Randomised controlled trials in primary care: scope and application. Br J Gen Pract 2002;482: 746-51 [PMC free article] [PubMed] [Google Scholar]
- 2.Jones RS, Richards K. Office of Evidence-based surgery charts course for improved system of care. Bull Am Coll Surg 2003; April 11-21 [PubMed]
- 3.Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based Medicine: How to Practice and Teach EBM, 2nd edn. London: Churchill Livingstone, 2000: 105-9
- 4.Solomon MJ, Laxamana A, Devore L, McLeod RS. Randomised controlled trials in surgery. Surgery 1995;117: 355-6 [PubMed] [Google Scholar]
- 5.Horton R. Surgical research or comic opera: questions, but few answers. Lancet 1996;347: 984. [DOI] [PubMed] [Google Scholar]
- 6.Pollock AV. The rise and fall of the random controlled trial in surgery. Theoret Surg 1989;4: 163-70 [Google Scholar]
- 7.Pollock AV. Surgical evaluation at the crossroads. Br J Surg 1993;80: 964-6 [DOI] [PubMed] [Google Scholar]
- 8.Ellis J, Mulligan I, Rowe J, Sackett DL. Inpatient general medicine is evidence based. Lancet 1995;364: 407-10 [PubMed] [Google Scholar]
- 9.McCulloch P, Taylor I, Sasako M, Lovett B, Griffin D. BMJ 2002;324: 1448-51 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Howes N, Chagla L, Thorpe M, McCulloch P. Surgical practice is evidence based. Br J Surg 1997;84: 1220-3 [PubMed] [Google Scholar]
- 11.Thomas T, Fahey T, Somerset M. The content and methodology of research papers published in three United Kingdom primary care journals. Br J Gen Pract 1998;48: 1229-32 [PMC free article] [PubMed] [Google Scholar]
- 12.Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. The Cochrane Library, Issue 3, 2005. Chichester: John Wiley
- 13.Devereaux PJ, Bhandari M, Clarke M, et al. Need for expertise based randomised controlled trials. BMJ 2005;330: 88-91 [DOI] [PMC free article] [PubMed] [Google Scholar]