ABSTRACT
Objective:
This study aims to assess the quality of articles published in the leading orthopedic surgery journals, by measuring the relation between the impact factor and the number studies with a high level of evidence.
Methods:
A literature review was performed of articles published in four previously selected journals. A score of journal evidence (RER - Relation between Randomized clinical trials and Systematic reviews) was calculated, considering the number of RCTs and SR published and the total number of full-text articles.
Results:
The selected journals were JBJS-Am, ASMJ, BJJ-Br and Arthroscopy, with Impact factors of 5.280, 4.362, 3.309 and 3.206 respectively in 2015. In the study, the RER Scores, in the same order, were 9.408, 6.153, 7.456 and 7.779.
Conclusion:
The journal JBJS-Am is the best available source of information on orthopedic surgery from this point of view. It has the highest Impact Factor and clearly the highest RER Score. On the other hand, we could conclude that the number of published RCT and good quality SR is very low, with less than 10% of all the articles. Level of evidence III, Analyses based on limited alternatives and costs, and poor estimates.
Keywords: Review Literature as Topic, Impact Factor, Evidence-Based Medicine, Orthopedics, Study Characteristics
RESUMO
Objetivo:
Este estudo tem como objetivo avaliar a qualidade dos artigos publicados nos principais periódicos de cirurgia ortopédica, medindo a relação entre o fator de impacto e o número de estudos com alto nível de evidência.
Métodos:
Realizou-se a revisão de literatura com artigos publicados em quatro periódicos previamente selecionados. Um escore de evidência de periódicos (RER - Relação entre Ensaios Clínicos Randomizados e Revisões Sistemáticas) foi calculado, considerando-se o número de ECR e RS publicados e número total de artigos com textos completos.
Resultados:
Os periódicos selecionados tiveram o fator de impacto de 5.280, 4.362, 3.309 e 3.206 respectivamente para JBJS-Am, ASMJ, BJJ-Br e Arthroscopy no ano de 2015. No estudo, os escores RER foram, na mesma ordem, 9.408, 6.153, 7.456 e 7.779.
Conclusão:
A revista JBJS-Am é a melhor fonte disponível de informações sobre cirurgia ortopédica deste ponto de vista. Tem o maior fator de impacto e claramente o maior escore RER. Por outro lado, pudemos concluir que o número de ECR e RS publicados de boa qualidade é muito baixo, com menos de 10% do total de artigos. Nível de Evidência III, Análises baseadas em alternativas e custos limitados, e estimativas ruins.
Descritores: Literatura de revisão como assunto, Fator de Impacto, Medicina baseada em evidências, Ortopedia, Características dos estudos
INTRODUCTION
An enormous number of articles are published annually by each orthopedic journal, which leads to a progressive increase of new information, new surgical techniques, updates of diseases and case reports that can be easily accessed.
With too much data available in the literature, effective and judicious analysis of these data should be done in order to guide and complement surgeon's decision-making process, defining therefore, the central point of practicing Evidence-Based Medicine (EBM). It should involve integration of clinical expertise, patients’ perceptions and values, and the best available research evidence. However, most of knowledge of orthopedic surgeons is based only on clinical experience of some experts. 1
Aiming for the best quality of information, students, researchers, clinical practitioners and surgeons may choose an article based on the journal according to its relevance, which can be measured by Impact Factor (IF). The IF was created as a measurement of the number of citations of scientific articles published in a determined period. 2 The IF shows indirectly the relevance and quality of the publications by a journal, and it is calculated by a relation between published articles and articles that are mentioned, quoted or cited_in the biennium prior to the year in which the IF is calculated.
Another method to assess the quality of the study is the analysis its design, considering the level of evidence, which is categorized in six levels: 3 (Figure 1)
Figure 1. The Hierarchy of evidence – Rockwood and Green's Fracture in Adults, 7Ed.
OBJECTIVE
This study aims to assess the quality of published articles in the most important orthopedic surgery journals, by measuring the relation between the impact factor and the number of high-evidence level studies.
METHODS
Inclusion criteria used was the four orthopedic surgery journals with the highest impact factor (IF) included in the list of Journal of Citation Report of Web of Knowledge. 2
A systematic assessment of published articles of these four journals was performed. Full articles published from January 1st 2013 to December 31st 2014 were selected. A score of journal evidence (RRS - Relation of Randomized clinical trials and Systematic Revision) was calculated considering the number of RCTs and SR published and total of full articles (Figure 2).
Figure 2. RRS score calculation formula.
Editorials, letters and communications were excluded from the calculation. Also, narrative reviews were not considered high-evidence articles and were not included in the sum of RCTs and SRs. From that point, we could estimate the RRS score of each journal selected and compare with IF.
RESULTS
The selected journals in orthopedic surgery field with higher IF selected were:
Journal of Bone and Joint Surgery - American Volume (JBJS-Am);
American Journal of Sports Medicine (AJSM);
The Bone and Joint Journal - British Volume (BJJ-Br) and
Arthroscopy - The Journal of Arthroscopic and Related Surgery. A total of 210 level 1 and 2 (high-evidence level) articles were identified in the four selected journals. Of these 26 were systematic reviews / meta-analysis and 184 RCTs (Tabela 1).
Table 1. Demonstrates relation of total published articles to number to high-evidence articles, RRS Score and Impact factor.
| Total published articles 2013 – 2014 | RCTs/SRs | RRS Score | IF | |
|---|---|---|---|---|
| JBJS-Am | 659 | 62 | 9.408 | 5.280 |
| AJSM | 650 | 40 | 6.153 | 4.362 |
| BJJ-Br | 523 | 39 | 7.456 | 3.309 |
| Arthroscopy | 887 | 69 | 7.779 | 3.206 |
RCT: Randomized Controlled Trial; SR: Systematic Review; RRS***; IF: Impact Factor (JCR-Web of Knowledge); JBJS-AM: Journal of Bone and Joint Surgery - American Volume; AJSM: American Journal of Sports Medicine; BJJ-Br: The Bone and Joint Journal - British Volume; Arthroscopy - The Journal of Arthroscopic and Related Surgery.
DISCUSSION
Since 2003, the main orthopedic journal - The Journal of Bone and Joint Surgery - American Volume (JBJS-Am), have been using a scale based on the pyramid of evidence, ranking their published articles according to level of evidence. 4 Actions like this promote readers and authors to practice EBM, improve the quality of studies to be published and demonstrate the increasing influence of this practice in science of orthopedic surgery. Some journals publish a large number of letters, communications and editorials. In order to not compromise results, these publications were excluded from the analysis in this study.
The proposed Score assesses the proportion of high-evidence articles published by a journal each year. However, the highest obtained RRS score in this study of 9.408 of JBJS-Am means that less than 10% of all articles published in the journal of highest impact factor represents high-evidence quality. Other studies also demonstrate even lower rates of quality articles published. Rodrigues have found 0.84% in plastic surgery journals, Moraes found 2.4% in orthopedic literature and Rosales found from zero to 8.3% in hand surgery journals. 5 6 7
The impact factor of a scientific journal reflects its importance and relevance in the literature. 8 Despite the fact that there are some criticisms about the validity of IF, once many authors self-citation and policies practiced by journal editors to increase its IF rating. Limitation of this study is lack of assessment of quality of published RCTs. Tools like the one described by Jadad demonstrate fails and risks of biases in these studies. 9 This further and deeper evaluation may be postponed and be the next step when a larger amount of high-quality evidence studies is published.
Another limitation is that prospective cohort studies for prognosis and accuracy studies for diagnostics were not included as high quality evidence articles even though they represent the highest evidence for each study design.
CONCLUSION
The conclusion of this study was that the JBJS-Am is the best available source of information in orthopedic surgery. It has the highest Impact Factor and clearly the highest RRS Score. On the other hand, we could conclude that the number of published RCT and good quality SR is very low, with less than 10% of all the articles.
REFERÊNCIAS
- 1.Moraes VY, Belloti JC, Moraes FY, Galbiatti JA, Palácio EP, Santos JBG. Hierarchy of evidence relating to hand surgery in Brazilian orthopedic journals. Sao Paulo Med J. 2011;129(2):94–98. doi: 10.1590/S1516-31802011000200007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Garfield E. Citation analysis as a tool in journal evaluation. Science. 1972;178(4060):471–479. doi: 10.1126/science.178.4060.471. [DOI] [PubMed] [Google Scholar]
- 3.Bucholz RW, Heckman JD, Court-Brown CM, Tornetta P. Outcome Studies in Trauma. In: Rockwood CA, Green DP, editors. Fracture in Adults. 7th ed. Philadelphia: Lippincott Williams & Wilkins; 2010. pp. 410–430. [Google Scholar]
- 4.Wright JG, Swiontkowski MF, Heckman JD, James D. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85(1):1–3. [PubMed] [Google Scholar]
- 5.Rodrigues MA, Tedesco AC, Nahas FX, Ferreira LM. Journal impact factor versus the evidence level of articles published in plastic surgery journals. Plast Reconstr Surg. 2014;133(6):1502–1507. doi: 10.1097/PRS.0000000000000214. [DOI] [PubMed] [Google Scholar]
- 6.Moraes VY, Moreira CD, Tamaoki MJC, Faloppa F, Belloti JC. Randomized clinical trials in orthopedics and traumatology: Systematic assessment of the national evidence. Rev Bras Ortop. 2010;45(6):601–605. doi: 10.1016/S2255-4971(15)30310-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Rosales RS, Reboso-Morales L, Martin-Hidalgo Y, Diez de la Lastra-Bosch I. Level of evidence in hand surgery. BMC Res Notes. 2012;5:665–665. doi: 10.1186/1756-0500-5-665. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Garfield E. The evolution of the Science Citation Index. Int Microbiology. 2007;10(1):65–69. [PubMed] [Google Scholar]
- 9.Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12. doi: 10.1016/0197-2456(95)00134-4. [DOI] [PubMed] [Google Scholar]