Abstract
Aims
Guidance has been published on how best to report randomised controlled trials (Consolidated Standards of Reporting Trials - CONSORT) and systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-analysis - PRISMA). In 2011, we reported a low rate of enforcement by surgery journals for submitted manuscripts to conform to these guidelines. The aim of this follow-up study is to establish whether there has been any improvement.
Methods
We studied the 134 surgery journals indexed in the Journal Citation Report. The ‘Instructions to Authors’ were scrutinised for inclusion of the following guidance: CONSORT, PRISMA, clinical trial registration and systematic review registration.
Results
Compared to 2011, there has been an improvement in the endorsement of reporting guidance in journals' ‘Instructions to Authors’ in 2014, as follows: trial registration (42% vs 33%), CONSORT (42% vs 30%) and PRISMA (19% vs 10%, all p < 0.001). As in 2011, journals with a higher impact were more likely to adopt trial registration (p < 0.001), CONSORT (p < 0.001) and PRISMA (p = 0.002). Journals with editorial offices in the UK were more likely to endorse guidance compared to those outside the UK (p < 0.05). Only one journal mentioned registration for systematic reviews.
Conclusions
Surgery journals are presently more likely to require submitted manuscripts to follow published reporting guidance compared to three years ago. However, overall concordance rates are still low, and an improvement is required to help enhance the quality of reporting – and ultimately the conduct – of randomised control trials and systematic reviews in surgery.
Keywords: Editorial policies, Clinical trials, Conflict of interests, Journalism, Guidelines
Highlights
-
•
There has been an improvement in the endorsement of reporting guidance in surgery journals’ instructions to authors.
-
•
Higher impact journals are more likely to require trial registration and adherence to CONSORT and PRISMA.
-
•
There is still scope to improve the quality of reporting in surgery journals.
1. Introduction
Reporting guidelines exist to encourage transparency and improve the standards of reporting in evidence-based medicine. Universal implementation of such guidelines would go some way towards addressing the suboptimal reporting of randomised controlled trials (RCTs), systemic reviews and meta-analyses in surgery.
Randomised controlled trials (RCTs) are an indispensable tool for high quality assessment of causality and, aside from systematic reviews, remains the gold standard for assessing treatment effects. Trial registries exist to ensure clinical transparency, prevent the duplication of research and to enable regulation. The failure to register RCTs can be regarded as a breach of scientific and ethical conduct [1]. The registration of clinical trials is now essential in some countries, it is strongly recommended by most official bodies and should be included in published reports [1]. In 2010, to address problems of inadequacy in the objective evaluation of RCTs, the Consolidated Standards of Reporting Trials (CONSORT) statement was revised [2–4]. This 25-item checklist and flow diagram facilitates good reporting and manuscript development in interventional studies [3].
In 1999, the Quality of Reporting of Meta-analyses (QUOROM) statement was published. This included a 27-item checklist to encourage good reporting of systematic reviews [5]. QUOROM later gained support from many surgery and medicine journal editors including the Cochrane Collaboration [6]. A decade later, the QUOROM Statement was revised to incorporate recent methodological advances, and renamed PRISMA (Preferred Reporting Items of Systematic reviews and Meta-Analyses) [7]. In 2011, a prospective registry of systematic reviews in health and social care – PROSPERO – was created by the Centre for Reviews and Dissemination (CRD) [8]. This is an international database of prospectively-registered systematic reviews to help address problems with transparency, reporting bias and the duplication of reports. It is hoped that the complementary use of PRISMA and PROSPERO in the conduct and publication of systematic reviews and meta-analyses will ensure good reporting in the medical literature.
It is believed that enforcement of these guidelines will lead to an improvement in the quality of reporting, and thus the conduct, of RCTs and systematic reviews in medical research [9]. In 2011, we looked at the “Instructions to Authors” of a group of journals dedicated to surgical research, to see how frequently trial registration and adherence to guidelines was required [9]. We found that most journals did not mention, let alone insist upon, this guidance. The purpose of the current study is to ascertain whether surgery journals more frequently require manuscript submissions to adhere to these guidelines, compared to three years ago.
2. Methods
In our previous report, we studied all journals listed under “Surgery” in the science edition of the 2009 Journal Citation Reports (Thomson Reuters, New York, US). We used a similar approach in this report, (http://apps.webofknowledge.com, last accessed 14th July, 2014). We only included journals which actively publish original research. In 2011, this resulted in a list of 136 surgery journals. However, of these, two no longer exist (one discontinued, the other incorporated into another publication), so 134 journals were available for follow-up analysis (Fig. 1, below).
An internet-based investigation was performed by accessing the online instructions for authors for each surgery journal. Information regarding a journal's enforcement of PRISMA, CONSORT and trial registration was entered into a custom-designed spreadsheet (Excel 2010, Microsoft, Redmond) in March 2011, and then repeated in July 2014. PROSPERO did not exist when the original study was conducted, so a paired analysis could not be undertaken in this case. The journal's impact factor was recorded. Each journal was assigned a geographical zone based upon the location of their editorial office, and this was categorised into four distinct groups: UK, Europe (excluding the UK), North America (US and Canada) and other (the remaining countries).
For each of the guidelines mentioned above, and for each journal, the level of enforcement was documented as compulsory (an essential criteria for manuscript acceptance), recommended (usage encouraged, but not mandatory) or unclear (guidelines were mentioned but the necessity of their inclusion in the manuscript remained ambiguous). Data were collected independently by two authors (T.S., P.K.), with Yes/No discrepancies noted in 0.7% of items. This equated to a good intra-observer reliability (Cohen's kappa, κ = 0.992). All inconsistencies were reviewed by a third author (S.S.) and resolved after subsequent discussion with all authors.
Impact factor (IF) was analysed as a continuous variable, and compared to reporting frequency using logistic regression [10]. The associations between geographical region and frequency of guideline recommendations by journals were calculated using Pearson's chi-squared test (with three degrees of freedom), with subgroup analyses calculated using post hoc z-tests for proportions. To look for evidence of an improvement between the years 2011 and 2014 of paired frequencies, McNemar's test was used. All statistical analyses were performed using SPSS (v18, IBM, Chicago).
3. Results
As previously mentioned, 136 surgery journals were included in our original study in 2011 [9]. In our most recent search, two journals were found to no longer be in existence, therefore we were unable to provide follow-up data and they were excluded from our analysis. This left 134 journals for analysis with a range of impact factor from 0.1 to 6.3.
The overall distribution of data with regards to impact factor and geographical location is represented in Tables 1 and 2. Impact factors were similar in the 2011 and 2014 analysis showing that 65% and 62% of journals had an impact factor of ≤2, respectively. As noted in our previous report, the majority (55%) of surgery journals' editorial offices were located in Northern America.
Table 1.
Impact factor |
p value |
||||
---|---|---|---|---|---|
0–1 |
1–2 |
2–3 |
>3 |
||
(n = 33, 25%) | (n = 50, 37%) | (n = 29, 22%) | (n = 22, 16%) | ||
CONSORT | 7(21%) | 18(36%) | 15(52%) | 16(73%) | P < 0.001 |
PRISMA | 3(9%) | 8(16%) | 7(24%) | 9(41%) | p = 0.002 |
Trial registration | 8(24%) | 17(34%) | 12(41%) | 19(86%) | p < 0.001 |
Review registration | 0(0%) | 0(0%) | 1(3%) | 0(0%) | p = 0.748 |
P values highlighted in bold are statistically significant.
Table 2.
Geographical location |
p value | ||||
---|---|---|---|---|---|
North America |
Europe (excl. UK) |
UK |
Other |
||
(n = 74, 55%) | (n = 36, 27%) | (n = 13, 10%) | (n = 11, 8%) | ||
CONSORT | 30(41%) | 8(22%) | 12(92%) | 6(55%) | p < 0.001 |
PRISMA | 17(23%) | 3(8%) | 6(46%) | 1(9%) | p = 0.002 |
Trial registration | 31(42%) | 10(28%) | 8(62%) | 7(64%) | p = 0.067 |
Review registration | 1(1%) | 0(0%) | 0(0%) | 0(0%) | p = 0.845 |
P values highlighted in bold are statistically significant.
Currently, guidelines for CONSORT are endorsed by 56 (42%) of surgery journals. Of these, this was an absolute requirement of 44 (33%) journals (Fig. 2, below), and most journals requiring CONSORT provided a web reference (91%). Journals with a higher impact factor are more likely to endorse the use of CONSORT (p < 0.001). A significant difference was found between geographical location and endorsement of CONSORT (p < 0.001), with the UK-based journals being the most likely to enforce this guidance (92%, p < 0.05). On the other hand, only 22% of European journals (excluding the UK) endorsed the use of CONSORT. Over the last three years, overall endorsement of CONSORT has improved by 40% (p < 0.001, Table 3).
Table 3.
2011 |
2014 |
p value | |
---|---|---|---|
(n = 134) | (n = 134) | ||
CONSORT | 40(30%) | 56(42%) | p < 0.001 |
PRISMA | 13(10%) | 26(19%) | p < 0.001 |
Trial registration | 44(33%) | 56(42%) | p < 0.001 |
P values highlighted in bold are statistically significant.
Clinical trial registration was mentioned by 56 (42%) journals; of these it was an absolute requirement in 91%. Web links to trial registration databases were provided by most of these journals (77%). Journals with a higher impact factor were more likely to enforce trial registration (p < 0.001). No association between a journal's geographical location and enforcement of trial registration was identified (p = 0.067). In the last three years, there has been a 27% increase in the rate of enforcement of trial registration (p < 0.001).
We observed that PRISMA was enforced by 26 (19%) surgery journals, with 21 (81%) of these providing a web reference. Surprisingly, only one journal stated PRISMA as an absolute requirement for manuscript submission. Journals with a higher impact factor are more likely to encourage use of PRISMA (p = 0.002). Again, a significant difference in geographical location and endorsement of PRISMA has been observed (p = 0.002), with the best concordance found in UK-based journals (p < 0.05). Compared to our initial analysis in 2011, the overall endorsement of PRISMA has increased by 100% (p < 0.001). Only one journal (<1%) mentioned systematic review registration (by PROSPERO) in their instructions to authors; furthermore, this was a recommendation not a requirement. No correlation between impact factor or geographical location and endorsement of review registration was identified.
4. Discussion
In this study, we analysed the “Instructions for Authors” of indexed surgery journals to determine the current rate of endorsement of published guidelines for systematic reviews (PRIMSA, review registration) and RCTs (CONSORT, trial registration) in the surgery literature. In addition, we compared these results to our previous study performed in 2011 to see if there has been an improvement [9]. Overall, the endorsement of these guidelines has increased significantly since 2011, although rates of endorsement are still far below ideal. We found that journals with a higher impact factor were more likely to enforce reporting guidelines. Of the guidelines analysed, CONSORT and trial registration were most likely to be endorsed. During the study period, a significant increase in the endorsement of CONSORT, PRISMA and trial registration in journals' instructions to authors was noted (p < 0.001). Systematic review registration was the least widely-used guideline; this may in part be because it is the most recent addition of the guidelines studied.
There is growing body of evidence suggesting that manuscripts published in surgery fail to report RCTs according to the CONSORT statement [11–14]. In particular, studies show that articles comply on average with 49–55% of the CONSORT items [11–13]. Notably, these studies revealed substandard reporting for methods of randomisation, blinding, trial registration and funding. The specific inclusion of trial registration identifiers in published RCTs has also been scrutinised in the surgery literature. Lee et al. identified that only 10% of articles specified a trial registry number in published RCTs pertaining to trauma surgery [12]. In addition, this study also identified that journals with a higher impact factor are more likely to publish reports who follow CONSORT guidelines [12]. Furthermore, a study of high impact orthopaedic journals reporting systematic reviews has shown that on average only 68% of PRISMA statement items are included in their articles [15]. Together, these reports suggest that the application of guidelines dedicated to improving the quality of systematic reviews and randomised controlled trials in surgery is poor. Only with the support of journals can we hope for this suboptimal reporting to improve.
The issue of suboptimal endorsement in journals' editorial policies has been found in non-surgical specialities. For example, similar studies in paediatric and psychiatric journals found that endorsement of CONSORT occurred in 20% and 23% of cases, respectively [16,17], although this level was greater (37%) in higher-impact medical journals [18]. In a study of general medical journals, clinical trial registration was recommended by 51% (vs. 42% in this study) [19]. In line with our results, endorsement is more frequent in journals with a higher impact factor [16].
There are limitations to this study. Firstly, not all surgery journals are indexed in the Journal Citation Report and thus there is a risk of selection bias. Moreover, many high quality surgery articles are published in medical journals and these would not be included in our analysis. The 2014 Journal Citation Report now identifies 199 surgery journals in publication. However, these additional journals were not appraised because we had no previous data to compare them with. We do not know the impact of this exclusion from our study because it is not known if the age of a journal impacts its likeliness to adhere to reporting guidelines. There may be a feeling that generic reporting guidelines may not be entirely appropriate for use in surgical trials (for example, due to difficulties in blinding), and hence the lack of a journal's requirement for following guidelines could be an intended omission. However, this should not preclude a report conforming to the other appropriate items in the guidelines. Furthermore, “Instructions to Authors” guidelines may not accurately reflect the editorial policies of the journal, and these were not investigated. However, for purposes of transparency, we would recommend journals report consistent guidelines in their editorial policies and instructions for authors. Finally, despite its known weaknesses, our assessment of a journal's profile was made according to impact factor [20].
We report that guidelines intended to standardise and enhance the clarity of reporting in systematic reviews and randomised controlled trials are not widely enforced by surgery journals, although there has been some improvement over the last three years. It is vital that good publication practice is supported in the medicine and surgery literature to allow sound decision-making in healthcare. Overriding responsibility remains within the remit of the investigators to conduct high-quality research. However, this should be encouraged by journals and editors through their editorial and peer-review policies, and made explicit in their instructions to authors.
Conflict of interests
None.
Financial support
None.
Ethical approval
None.
Consent
None.
Author contribution
Tanya Smith: data collection, analysis, interpretation, drafting manuscript.
Priyantha Kulatilake: data collection, analysis, interpretation, drafting manuscript.
Lucy Brown: data collection, interpretation, drafting manuscript.
James Wigley: study concept and design, interpretation, critical revision of manuscript.
Waseem Hameed, study design, interpretation, critical revision of manuscript.
Saran Shantikumar, study concept and design, data collection, analysis, interpretation, critical revision of manuscript.
Guarantor
Saran Shantikumar.
Contributor Information
Tanya A. Smith, Email: ts9829@my.bristol.ac.uk.
Priyantha Kulatilake, Email: p.kulatilake@bristol.ac.uk.
Lucy J. Brown, Email: lucy.brown@warwick.ac.uk.
James Wigley, Email: james.wigley@gmail.com.
Waseem Hameed, Email: hameedwaseem@hotmail.com.
Saran Shantikumar, Email: saran.shantikumar@gmail.com.
References
- 1.De Angelis C., Drazen J.M., Frizelle F.A., Haug C., Hoey J., Horton R. Clinical trial registration: a statement from the international committee of medical journal editors. Ann Intern Med. 2004;141(6):477–478. doi: 10.7326/0003-4819-141-6-200409210-00109. [DOI] [PubMed] [Google Scholar]
- 2.Schulz K.F., Altman D.G., Moher D., the CONSOR Group CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340 doi: 10.4103/0976-500X.72352. c332. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Begg C., Cho M., Eastwood S., Horton R., Moher D., Olkin I. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA J Am Med Assoc. 1996;276(8):637–639. doi: 10.1001/jama.276.8.637. [DOI] [PubMed] [Google Scholar]
- 4.Plint A.C., Moher D., Morrison A., Schulz K., Altman D.G., Hill C. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006;185(5):263–267. doi: 10.5694/j.1326-5377.2006.tb00557.x. [DOI] [PubMed] [Google Scholar]
- 5.Moher D., Cook D.J., Eastwood S., Olkin I., Rennie D., Stroup D.F. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet. 1999;354(9193):1896–1900. doi: 10.1016/s0140-6736(99)04149-5. [DOI] [PubMed] [Google Scholar]
- 6.Biondi-Zoccai G.G., Lotrionte M., Abbate A., Testa L., Remigi E., Burzotta F. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. Bmj. 2006;332(7535):202–209. doi: 10.1136/bmj.38693.516782.7C. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Moher D., Liberati A., Tetzlaff J., Altman D.G., PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi: 10.1371/journal.pmed.1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Booth A., Clarke M., Dooley G., Ghersi D., Moher D., Petticrew M. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev. 2012;1:2. doi: 10.1186/2046-4053-1-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Shantikumar S., Wigley J., Hameed W., Handa A. A survey of instructions to authors in surgical journals on reporting by CONSORT and PRISMA. Ann R Coll Surg Engl. 2012;94:468–471. doi: 10.1016/j.amsu.2014.12.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Garfield E. The history and meaning of the journal impact factor. JAMA J Am Med Assoc. 2006;295(1):90–93. doi: 10.1001/jama.295.1.90. [DOI] [PubMed] [Google Scholar]
- 11.Adie S., Harris I.A., Naylor J.M., Mittal R. CONSORT compliance in surgical randomized trials: are we there yet? A systematic review. Ann Surg. 2013;258(6):872–878. doi: 10.1097/SLA.0b013e31829664b9. [DOI] [PubMed] [Google Scholar]
- 12.Lee S.Y., Teoh P.J., Camm C.F., Agha R.A. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–572. doi: 10.1097/TA.0b013e3182a5399e. [DOI] [PubMed] [Google Scholar]
- 13.Gray R., Sullivan M., Altman D.G., Gordon-Weeks A.N. Adherence of trials of operative intervention to the CONSORT statement extension for non-pharmacological treatments: a comparative before and after study. Ann R Coll Surg Engl. 2012;94(6):388–394. doi: 10.1308/003588412X13171221592339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Knobloch K., Gohritz A., Vogt P.M. CONSORT and QUOROM statements revisited: standards of reporting of randomized controlled trials in general surgery. Ann Surg. 2008;248(6) doi: 10.1097/SLA.0b013e31818f9c81. 1106-0037; discussion 07–08. [DOI] [PubMed] [Google Scholar]
- 15.Gagnier J.J., Kellam P.J. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Jt Surg Am. 2013;95(11):e771–e777. doi: 10.2106/JBJS.L.00597. [DOI] [PubMed] [Google Scholar]
- 16.Knuppel H., Metz C., Meerpohl J.J., Strech D. How psychiatry journals support the unbiased translation of clinical research. A cross-sectional study of editorial policies. PloS One. 2013;8(10):e75995. doi: 10.1371/journal.pone.0075995. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Meerpohl J.J., Wolff R.F., Niemeyer C.M., Antes G., von Elm E. Editorial policies of pediatric journals: survey of instructions for authors. Arch Pediatr Adolesc Med. 2010;164(3):268–272. doi: 10.1001/archpediatrics.2009.287. [DOI] [PubMed] [Google Scholar]
- 18.Hopewell S., Altman D.G., Moher D., Schulz K.F. Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal 'Instructions to Authors. Trials. 2008;9:20. doi: 10.1186/1745-6215-9-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hooft L., Korevaar D.A., Molenaar N., Bossuyt P.M., Scholten R.J. Endorsement of ICMJE's clinical trial registration policy: a survey among journal editors. Neth J Med. 2014;72(7):349–355. [PubMed] [Google Scholar]
- 20.Hernan M.A. Impact factor: a call to reason. Epidemiology. 2009;20(3):317–318. doi: 10.1097/EDE.0b013e31819ed4a6. discussion 19–20. [DOI] [PubMed] [Google Scholar]