Abstract
Introduction
Spin is defined as an inaccurate interpretation of results, intentionally or not, leading to equivocal conclusions and misdirecting readers to look at the data in an overly optimistic way. Previous studies have shown a high prevalence of spin in scientific papers and this systematic review aims to investigate the nature and prevalence of spin in the neurosurgical trauma literature. Any associated factors will be identified to guide future research practice recommendations.
Methods and analysis
The Preferred Reporting Item for Systematic Reviews and Meta-Analyses recommendations will be followed. Randomised clinical trials (RCTs) that enrolled only patients with traumatic brain injury and investigated any type of intervention (surgical or non-surgical) will be eligible for inclusion. The MEDLINE/PubMed database will be searched for articles in English published in 15 top-ranked journals. Spin will be defined as (1) a focus on statistically significant results not based on the primary outcome; (2) interpreting statistically non-significant results for a superiority analysis of the primary outcome; (3) claiming or emphasising the beneficial effect of the treatment despite statistically non-significant results; (4) conclusion focused in the per-protocol or as-treated analysis instead of the intention-to-treat results; (5) incorrect statistical analysis; (6) republication of a significant secondary analysis without proper acknowledgement of the primary outcome analysis result. Traditional descriptive statistics will be used to present RCT characteristics. Standardised differences between the groups with or without spin will be calculated. The variables with a standardised difference equal or above 0.2 and 0.5 will be considered weakly and strongly associated with spin, respectively.
Ethics and dissemination
This study will not involve primary data collection and patients will not be involved.
Trial registration number
10.17605/OSF.IO/H3FGY.
Keywords: statistics & research methods, neurosurgery, neurology
Strengths and limitations of this study.
This is the first systematic review to evaluate the prevalence of spin in the neurosurgical trauma literature.
Inclusion of studies with different interventions will allow identification of good and poor examples of randomised clinical trials.
The inclusion of only randomised clinical trials may strengthen our results.
Exclusion of papers not published in specific journals may mean that important additional findings are missed.
Introduction
Spin is defined as an inaccurate interpretation of results, intentionally or not, leading to equivocal conclusions and misdirecting readers to look at the data in an overly optimistic way.1 Spin is considered by many as a consequence of the highly competitive and commercial academic system all scientists work within, since it puts negative findings in a more palatable way to editors, journals, patients, funders and readers.2
Spin may be present in the results, discussion or conclusion sections. Boutron et al demonstrated in a series of randomised clinical trials (RCT) that spin was present in 29.2%, 43.1% and 50.0% of the studies at the results, discussion and conclusion sections, respectively.3 Series of observational studies exhibited more than 80% of spin prevalence.4 Such findings represent a serious problem for the scientific community.
There are three main different spin strategies. First, to focus on statistically significant results not based on the primary outcome, as a within-group comparisons, secondary outcomes or subgroup analyses. Second, interpreting statistically non-significant results for a superiority analysis of the primary outcome as treatment equivalence or comparable effectiveness. Finally, claiming or emphasising the beneficial effect of the treatment despite statistically non-significant results.3
Spin comes in different shapes and sizes, and medical fields with a less robust, fragile scientific output may be more biased by such malpractice. Considering that the quantity and quality of the neurosurgical literature remains suboptimal, it would be thought-provoking to acknowledge how often spins are encountered, how they are presented and its potential to the specialty practice.5–8
Therefore, this systematic review aims to investigate the nature and prevalence of spin in the neurosurgical trauma literature and identify any associated factors which could guide future good research practice recommendations.
Methods and analysis
Protocol and registration
This systematic review and meta-analysis will be reported following the Preferred Reporting Item for Systematic Reviews and Meta-Analyses guidelines.
Eligibility criteria
Published RCT that enrolled only patients with traumatic brain injury (TBI) and investigated any type of intervention (surgical or non-surgical) will be eligible for inclusion. Studies without a clear primary outcome will be excluded. The following journals will be selected for screening based in the impact factor and importance to the neurosurgical trauma literature: New England Journal of Medicine (NEJM), Lancet, Lancet Neurology, Journal of the American Medical Association (JAMA), JAMA Neurology, Journal of Neurology, Neurosurgery and Psychiatry (JNNP), Neurosurgery, Journal of Neurosurgery, Neurosurgical Focus, World Neurosurgery, Acta Neurochirurgica, Journal of Neurotrauma, Intensive Care Medicine, Critical Care, Neurocritical Care, Journal of Trauma and Acute Care Surgery or Critical Care Medicine.
Information sources and search strategy
The MEDLINE/PubMed database will be searched for articles in English published from January 1960 to July 2020. The descriptors (((((((((((Traumatic brain injury[Title/Abstract]) OR (TBI[Title/Abstract])) OR (Brain trauma[Title/Abstract])) OR (Brain concussion[Title/Abstract])) OR (Brain contusion[Title/Abstract])) OR (Head trauma[Title/Abstract])) OR (Head injury[Title/Abstract])) OR (Brain injury[Title/Abstract])) OR (Traumatic subarachnoid hemorrhage[Title/Abstract])) OR (subdural hematoma[Title/Abstract])) OR (epidural hematoma[Title/Abstract])) OR (traumatic intraparenchymal hemorrhage[Title/Abstract]) will be used. The filters ‘Clinical trial’ and ‘Randomized clinical trials’ will be applied.
Study selection
The search strategy aimed to achieve a sample of RCTs published in the neurosurgical trauma literature. Spin in sections other than the conclusion (manuscript or abstract), such as the results and discussion sections, will not be considered due to a perceived higher subjectivity in their definition. After all, a reasonable presentation of results not based on the primary outcome, and their discussion is not necessarily a bad practice—and indeed expected. All articles’ titles and abstracts will be screened by two authors (A and J) for eligibility. The selected articles will be adjudicated by a third author (DJFS) and disagreements will be resolved by consensus. Additional studies identified in the reference section of the selected articles could be included if the eligibility criteria will be fulfilled.
Data collection and analysis
Data will be abstracted and recorded on a standardised form regarding: journal, year of publication, study country (high income or low-and-middle income), first and last authors affiliations (neurosurgery or other), the presence of a statistician among the authors, single centre or multicentre trial, type of trial design (superiority, non-inferiority or equivalence), the presence of a priori sample size and power calculation, type of TBI (mild, moderate, severe), setting (pre-hospital or intra-hospital), intervention (surgical, drug, other), allocation concealment and blinding, number of patients, follow-up period, the event rates in the two treatment arms, the presence of a post hoc power calculation, the discussion of lack of power as a limitation, funding (industry, independent or none) and conflict of interest (when explicitly stated). The full text of the included articles will be systematically reviewed by two authors (A and J). The interobserver agreement and the κ statistic will be calculated. Disagreements will be resolved by a third author (DJFS). The data analysis will be blinded to the authors and institutions of the study. GRADE (Grading of Recommendations Assessment, Development and Evaluations) framework will be used to assess the level of evidence of each paper, varying from very low up to high.
Spin will be defined as1 a focus on statistically significant results not based on the primary outcome, as a within-group comparisons, secondary outcomes or subgroup analyses2; interpreting statistically non-significant results for a superiority analysis of the primary outcome as treatment equivalence or comparable effectiveness3; claiming or emphasising the beneficial effect of the treatment despite statistically non-significant results4; conclusion focused in the per-protocol or as-treated analysis instead of the intention-to-treat results5; incorrect statistical analysis6; republication of a significant secondary analysis without proper acknowledgement of the primary outcome analysis result.
Traditional descriptive statistics will be used to present the included RCT characteristics. Standardised differences between the groups with or without spin will be calculated as proposed by Yang and Dalton. They are indexes which measure the effect size between two groups.9 The variables with a standardised difference equal or above 0.2 and 0.5 will be considered weakly and strongly associated with spin, respectively. All analyses will be conducted with the SPSS software (IBM, SPSS Statistics for Windows, V.24.0).
Risk of bias in individual studies
The entire text of each included paper will be evaluated in a structured fashion for prespecified attributes relating to the assessment and ultimate claim of equivalency. Cochrane’s tool to assess the risk of bias in randomised trials was used.10 This way, the investigators were able to access power and sample size calculation, allocation concealment and many other criteria regarding the methodological quality of each RCT.
Patient and public involvement
No patient involved.
Ethics and dissemination
This study will not involve primary data collection and patients will not be involved. Therefore, formal ethical approval will not be required. The final systematic review will be published in a peer-reviewed journal and presented at appropriate conferences. This protocol may be adapted for the analysis of other innovative surgical and invasive procedures.
Supplementary Material
Footnotes
Twitter: @davisolla
Contributors: DJFS conceived the research idea. JVMPO, ALFOJ and DJFS designed and drafted the study protocol. AGK and WSP all made critical review of the study protocol and edited the final manuscript.
Funding: DJFS, AGK and WSP are supported by the NIHR Global Health Research Group on Neurotrauma, which was commissioned by the National Institute for Health Research (NIHR) using UK aid from the UK Government (project 16/137/105). The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. AGK is supported by a Clinical Lectureship, School of Clinical Medicine, University of Cambridge and the Royal College of Surgeons of England.
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review: Not commissioned; externally peer reviewed.
Ethics statements
Patient consent for publication
Not applicable.
References
- 1.Chiu K, Grundy Q, Bero L. 'Spin' in published biomedical literature: a methodological systematic review. PLoS Biol 2017;15:e2002173–16. 10.1371/journal.pbio.2002173 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Harvey LA. Spin kills science. Spinal Cord 2015;53:417. 10.1038/sc.2015.89 [DOI] [PubMed] [Google Scholar]
- 3.Boutron I, Dutton S, Ravaud P, et al. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 2010;303:2058–64. 10.1001/jama.2010.651 [DOI] [PubMed] [Google Scholar]
- 4.Lazarus C, Haneef R, Ravaud P, et al. Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention. BMC Med Res Methodol 2015;15:1–8. 10.1186/s12874-015-0079-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kiehna EN, Starke RM, Pouratian N, et al. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg 2011;114:280–5. 10.3171/2010.8.JNS091770 [DOI] [PubMed] [Google Scholar]
- 6.Vranos G, Tatsioni A, Polyzoidis K, et al. Randomized trials of neurosurgical interventions: a systematic appraisal. Neurosurgery 2004;55:18–26. 10.1227/01.NEU.0000126873.00845.A7 [DOI] [PubMed] [Google Scholar]
- 7.Azad TD, Veeravagu A, Mittal V, et al. Neurosurgical randomized controlled trials-distance travelled. Neurosurgery 2018;82:604–12. 10.1093/neuros/nyx319 [DOI] [PubMed] [Google Scholar]
- 8.Mansouri A, Shin S, Cooper B, et al. Randomized controlled trials and neuro-oncology: should alternative designs be considered? J Neurooncol 2015;124:345–56. 10.1007/s11060-015-1870-6 [DOI] [PubMed] [Google Scholar]
- 9.Yang D, Dalton J. A unified approach to measuring the effect size between two groups using SAS. SAS Global forum 2012;335 https://www.semanticscholar.org/paper/A-unified-approach-to-measuring-the-effect-size-two-Yang-Dalton/6cf4bd36ca4c90006a5d6563f646a391c255581b [Google Scholar]
- 10.Sterne JAC, Savović J, Page MJ, et al. Rob 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898–8. 10.1136/bmj.l4898 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.