Abstract
Open science practices are research tools used to improve research quality and transparency. These practices have been used by researchers in various medical fields, though the usage of these practices in the surgical research ecosystem has not been quantified. In this work, we studied the use of open science practices in general surgery journals. Eight of the highest-ranked general surgery journals by SJR2 were chosen and their author guidelines were reviewed. From each journal, 30 articles published between January 1, 2019 and August 11, 2021 were randomly chosen and analyzed. Five open science practices were measured (preprint publication prior to peer-reviewed publication, use of Equator guidelines, study protocol preregistration prior to peer-reviewed publication, published peer review, and public accessibility of data, methods, and/or code). Across all 240 articles, 82 (34%) used one or more open science practices. Articles in the International Journal of Surgery showed greatest use of open science practices, with a mean of 1.6 open science practices compared to 0.36 across the other journals (p<.001). Adoption of open science practices in surgical research remains low, and further work is needed to increase utilization of these tools.
Introduction:
Reproducibility and transparency have long been considered important considerations in medical research.1 Recent retractions of studies in several medical journals have underscored the relevance of these issues.2 Many tools now exist to promote research quality and transparency, including protocol preregistration websites, Equator Network reporting guidelines for common study types, and preprint servers.3 However, the extent to which surgical research has adopted these tools is unknown. The purpose of this study is to describe the use of these quality-promoting practices in surgical research.
Methods:
The use of five open science practices were measured (preprint publication prior to peer-reviewed publication, use of Equator guidelines, study protocol preregistration prior to peer-reviewed publication, published peer review, and public accessibility of data, methods, and/or code) in surgical journals and manuscripts. The top eight general surgery journals by Scimago Journal Rank (SJR2) were included. A random sample of 240 original research articles published from January 1, 2019 to August 11, 2021 from these journals (30 from each journal) comprised the study cohort. The number of journals and studies that explicitly endorsed or used these practices was measured. The study was reported using STROBE guidelines. It was preregistered on osf.io.
Results:
In their author guidelines, seven (88%) journals recommended the use of Equator guidelines prior to journal submission. Five (63%) journals explicitly stated that they allowed for submissions to have been previously released as preprints. Seven (88%) of journal recommended that authors preregister protocols for clinical trials, though only two (25%) recommended that authors preregister their protocols for observational studies. None (0%) published peer reviewer comments. Four (50%) explicitly recommended that their authors make their methods publicly available. Across all 240 articles, 82 (34%) used one or more open science practices. 63 (26%) explicitly complied with the appropriate Equator guideline. Only 12 (5%) prospective studies preregistered their study protocols. None of the articles were posted on a preprint server prior to journal publication. Only 22 (9%) studies fully disclosed their methods in the form of making their code public or publishing a separate protocol (Table 1). Research in the International Journal of Surgery exhibited the highest utilization of open science practices. Studies in this journal utilized a mean of 1.6 open science practices compared to 0.36 across the other journals (p<.001). Journals that recommended or required usage of Equator guidelines saw higher levels of open science practice utilization in their research than journals that did not mention the practice (30% vs. 0% utilization, p value <.001). There was no relationship between journal rank and utilization of open science practices in each journal’s published studies.
Table 1.
Adoption of various open science tools currently used to promote research quality, transparency, and reproducibility.
| Open Science Tool | Journals (n=8)1 | Original Research Articles (n=240) |
|---|---|---|
| Equator Network guidelines | 7 (88%) | 63 (26%) |
| Preprints2 | 5 (63%) | 0 (0%) |
| Preregistration of clinical trials | 7 (88%) | 12 (5%) |
| Preregistration of observational studies | 2 (25%) | 26 (11%) |
| Published peer review | 0 (0%) | 0 (0%) |
| Open source methods | 4 (50%) | 22 (9%) |
The eight journals included in our analysis were: JAMA Surgery, Annals of Surgery, British Journal of Surgery, Journal of the American College of Surgeons, Surgery, International Journal of Surgery, World Journal of Surgery, and American Journal of Surgery.
Adoption of preprint servers was counted among journals if their author guidelines explicitly allowed for submission of preprint papers, and use of preprint servers was counted among original research articles if it had been posted on a preprint server prior to peer-reviewed publication.
Discussion:
Surgical research is adapting slowly to open science practices in academia, leaving the field potentially vulnerable to poor research quality. We found that surgical journal guidelines vary in their recommendations of open science practices, and use of these practices remains low. Drivers of this lack of open science usage may include limited incentives to follow open science practices, as well as added time required to do so.4 To increase usage, researchers, journals, and policymakers can collaborate to provide incentives for open science use, such as publicly available badges, and can advocate for funding structures that require open science usage.5,6
This study has limitations, as we only reviewed a sample of published research and excluded subspecialty journals, which may have led us to underestimate open science usage. Despite this, this study demonstrates a lack of open science usage in general surgery journals. The responsibility falls on both researchers and journals to adopt these new tools to promote high quality research generation and dissemination in surgery.
Supplementary Material
Support:
JSM is supported by a grant from the National Library of Medicine/National Institutes of Health (T15LM007092-30) and the Biomedical Informatics and Data Science Research Training (BIRT) Program of Harvard University.
Footnotes
Conflicts of Interest: The authors declare no competing interests.
References
- 1.Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Ledford H, Van Noorden R. High-profile coronavirus retractions raise concerns about data oversight. Nature. 2020;582(7811):160–160. doi: 10.1038/d41586-020-01695-w [DOI] [PubMed] [Google Scholar]
- 3.Open Science. PLOS. Accessed January 7, 2023. Open Science. PLOS. Published December 18, 2019. Accessed January 26, 2022. https://plos.org/open-science/ [Google Scholar]
- 4.Allen C, Mehler DMA. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 2019;17(5):e3000246. doi: 10.1371/journal.pbio.3000246 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Gardener AD, Hick EJ, Jacklin C, et al. Open science and conflict of interest policies of medical and health sciences journals before and during the COVID-19 pandemic: A repeat cross-sectional study. JRSM Open. 2022;13(11):205427042211321. doi: 10.1177/20542704221132139 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kadakia KT, Beckman AL, Ross JS, Krumholz HM. Leveraging Open Science to Accelerate Research. New England Journal of Medicine. 2021;384(17):e61. doi: 10.1056/NEJMp2034518 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.