Abstract
Objective
Profile analysis of articles from scientific journals is rare in our country. The aim of this study was to perform an analysis of publications of the Revista Brasileira de Ortopedia (RBO), to specify the designs of the studies and their level of evidence.
Methods
All articles published in RBO from January 2006 to December 2010 were classified according to the design of the study. The clinical studies were further stratified according to the level of evidence, in agreement with the norm of the journal. The studies classified as randomized and controlled clinical trials (RCTs) had their quality assessed by the system proposed by Jadad.
Results
In this period, there were 376 articles published in RBO. Clinical studies represented most of the papers, with 60.64% of the total. Case series represented 61.4% of the clinical studies. Thirteen RCTs were published, accounting for 3.46% of the total, and 5.7% of the clinical studies. The analysis of the quality of the RCTs showed that 5 (38.46%) were considered high quality, while 8 (61.54%) were of low quality. Among the studies in which the level of evidence does not apply (non-clinical), non-systematic reviews (46 articles) and basic research (40 articles) have prevailed, representing 12.23% and 10.64% respectively of the total.
Conclusions
Case series were the most prevalent (37.23%) studies published in RBO between 2006 and 2010, while RCTs accounted for 3.46% of the articles. The majority of RCTs (61.54%) were considered low quality, and only 1.32% of the clinical studies were classified as level I evidence.
Keywords: Methodology, Bibliometrics, Periodicals, Orthopedics
1. Introduction
Evidence-based medicine has gained prominence in current medical literature. This clinical practice is aimed at combining clinical experience with statistical results.1 Using evidence- based medicine, physicians can better understand treatment outcomes and use the information to better treat patients and reduce costs.2
Randomized controlled clinical trials (RCTs) provide the most evidence of all clinical research models. However, some medical areas have characteristics that make it difficult to conduct RCTs with maximum accuracy.3 Orthopedics is a surgical discipline that presents its own obstacles for carrying out high-quality clinical trials.4, 5 Therefore, the majority of research in this field is observational and produces data that may be inconclusive and bring into doubt its credibility.6, 7
Assessment papers published in scientific journals are rare in our field, but they are useful tools to measure scientific productivity in Brazil.
This study aimed to perform a quantitative analysis of the publications in the Revista Brasileira de Ortopedia (RBO) from 2006 to 2010 to evaluate the types of study designs and levels of scientific evidence produced by the studies.
2. Material and Methods
All scientific papers published in RBO from January 2006 to December 2010 were evaluated. Papers were classified into 14 categories based on the type of study. Clinical papers were divided into the following seven categories: technique description, case report, case series, cohort, RCT and systematic review. Non-clinical papers were divided into seven other categories: update papers or non-systematic reviews, accuracy analysis of diagnostic methods, epidemiological studies, basic research, biomechanical testing, anatomic studies and others.
Clinical studies were further stratified according to the level of clinical evidence produced in the study; studies were classified from I to V as per the magazine's editorial rule.8
Three participants in this study classified papers independently. If there were any disagreements regarding papers, it was re-reviewed as a group and the evaluation given by at least two reviewers was considered to be the final classification.
The quality of the studies classified as RCTs was evaluated according to the system developed by Jadad et al.9 Studies are assessed based on whether they are randomized, whether they are double-blind, whether they describe reasons for patient dropout, and whether the randomization and blinding are appropriate. Good quality papers are those that have positive responses for three or more of the above criteria. When the paper did not specify the items of the Jadad system in its methodology or when these items were not implicit in the text, they were considered to be missing.
2.1.1. Statistical Analysis
A descriptive analysis (absolute numbers and percentages) of the results was performed using Excel® (Microsoft).
3. Results
During the period studied, 376 papers were published in RBO. The distribution according to the type of s tudy is shown in Table 1.
Table 1.
Distribution of types of studies.
| Type of Study | Definition | n | % |
|---|---|---|---|
| Case series | Description of intervention in a series of subjects without bias control | 140 | 37.23 |
| Non-systematic review | Non-systematic review of a series of articles | 46 | 12.23 |
| Basic research | Analysis of animal, tissue, cellular or molecular models | 40 | 10.64 |
| Case report | Description of a rare condition or unexpected intervention result | 33 | 8.78 |
| Cohort | Outcome evaluation regarding exposure | 31 | 8.24 |
| Anatomic study | Description of anatomic characteristics | 20 | 5.32 |
| Biomechanical study | Evaluation of the resistance of tissues | 16 | 4.26 |
| Randomized and controlled clinical study | Randomized comparative experimental study | 13 | 3.46 |
| Accuracy of the diagnostic method | Evaluation of diagnostic techniques and their reproducibility | 12 | 3.19 |
| Epidemiological study | Description of the characteristics of a population | 12 | 3.19 |
| Case-control | Search for causal factors for certain outcomes | 5 | 1.33 |
| Technique description | New technique description | 3 | 0.8 |
| Systematic review | Systematic review of a series of articles | 3 | 0.8 |
| Others | Do not fit into the previous categories | 2 | 0.53 |
| Total | 376 | 100,00 |
Clinical studies accounted for 60.64% of all reports. Case series were also prevalent and represented 37.23% of all publications and 61.40% of published clinical research. Thirteen RCTs were published, comprising 3.46% of all papers and 5.7% of clinical studies that were published. The distribution of the different types of clinical studies and their evidence levels can be observed in Table 2, Table 3, respectively.
Table 2.
Distribution of types of clinical studies.
| Clinical studies | n | % |
|---|---|---|
| Systematic review | 3 | 1.32 |
| Randomized and controlled clinical trial | 13 | 5.7 |
| Case-control | 5 | 2.19 |
| Cohort | 31 | 13.6 |
| Case series | 140 | 61.4 |
| Case report | 33 | 14.47 |
| Technique description | 3 | 1.32 |
| Total | 228 | 100 |
Table 3.
Distribution of clinical studies regarding the level of evidence.
| Level of evidence | n | % |
|---|---|---|
| I | 3 | 1.32% |
| II | 12 | 5.26% |
| III | 37 | 16.23% |
| IV | 140 | 61.40% |
| V | 36 | 15.79% |
| Total | 228 | 100.00% |
Quality analysis of RCTs showed that five trials (38.46%) were high quality, while eight (61.54%) were low quality. Based on Jadad's classification, the most-commonly omitted criteria were appropriate blinding and the description of the study as double-blind. Both were present in only 23.08% of the published studies (Table 4). Furthermore, only three RCTs were considered to be studies with level I evidence.
Table 4.
Evaluation of the randomized and controlled clinical trial according to the Jadad classification.
| n | % | |
|---|---|---|
| Randomized | 13 | 100.00% |
| Double-blind | 3 | 23.08% |
| Description of patient dropout | 7 | 53.85% |
| Appropriate randomization | 6 | 46.15% |
| Appropriate blinding | 3 | 23.08% |
| High-quality papers | 5 | 38.46% |
| Low-quality papers | 8 | 61.54% |
| Total of papers | 13 | 100.00% |
Among the studies in which the level of clinical, non-systematic review papers (46) and basic research papers (40) were prevalent, which represent 12.23% and 10.64% of the total number of publications, respectively. Table 5 shows the detailed distribution of non-clinical papers.
Table 5.
Distribution of non-clinical studies.
| Non-clinical studies | n | % |
|---|---|---|
| Non-systematic review | 46 | 31.08% |
| Basic research | 40 | 27.03% |
| Anatomic study | 20 | 13.51% |
| Biomechanical study | 16 | 10.81% |
| Accuracy of the diagnostic method | 12 | 8.11% |
| Epidemiological study | 12 | 8.11% |
| Others | 2 | 1.35% |
| Total | 148 | 100.00% |
4. Discussion
The percentage of studies in the RBO from January 2006 to December 2010 classified as having level I clinical evidence is lower than that found in international literature. Hanzlik et al.10, who analyzed papers published in the Journal of Bone and Joint Surgery (American Volume) in 2005, found that 21% of all studies had level I clinical evidence. Obremskey et al.11 evaluated publications from eight scientific magazines in the first semester of 2003 and found that 11.3% were level I studies. Cashin et al.12, who studied publications from the Journal of Pediatric Orthopaedics, found that level I studies made up 3% of all papers. For the period assessed, we observed that 1.32% of published studies had this level of evidence.Furthermore, 5.26% of the studies were classified as having level II evidence, a rate comparable to that found by Cashin et al.12 (5%) but lower than that found by Hanzlik et al.10 and Obremskey et al.11 (15% and 20.7%, respectively). The percentage of papers with level III evidence (16.23%) was higher than that found by Obremskey et al.11 (9.9%), similar to that found by Hanzlik et al.10 (16%), and lower than that observed by Cashin et al.12 (24.1%). If we sum the level-II and level-III studies, the results from the RBO (21.49%) are lower than those found in the literature, where they are approximately 30%.10, 11, 12
The two studies used here to compare the levels of scientific evidence used slightly different classifications, in which the case reports were classified together with series cases as level IV.11, 12
Summing these two types of studies, we observed that the percentage found in RBO (77.19%) is higher than that found by Cashin et al.12 (58%) and Obremskey et al.11 (58.1%). Hanzlik et al.,10 who, like us, graded case reports as having level V evidence, did not find occurrences of this type of study in the Journal of Bone and Joint Surgery (American Volume) in 2005. The percentage of case reports in RBO is 15.79%. When appropriately elaborated, case series play a significant role in surgical specialties and are an important to generating hypotheses for high-level studies,13 but they must be carefully considered when used as a guide for decision-making. Case reports, on the other hand, are important in describing rare diseases or unusual procedural complications, but they present the lowest level of clinical evidence among studies.
We hypothesize that the lower rates of studies with better levels of clinical evidence in the RBO can be at least partially explained by the fact that Brazilian authors search for journals with higher impact factors. The two most important national orthopedic magazines, RBO and Acta Ortopédica Brasileira (AOB), are not indexed by Pubmed and the articles published in these two magazines reach considerably fewer readers. This produces a bias, as Brazilian authors try to publish their best papers in international magazines.
Regarding the quality of RCTs, most fail to appropriately describe their methodology. According to the Jadad scale, only 38.46% were considered to be high quality. Appropriate blinding and a description of the study as double-blind were present in only 23.08% of the studies. Similar findings have also been reported in international literature. Dulai et al.14, who assessed studies by means of the Detsky Scale, noted that only 19% of RCTs satisfied the threshold for a satisfactory level of quality. Failures in blinding were also described by Poolman et al.15, who found that the description of the appropriate blinding of patients and treatment providers were only present in 19% and 6% of RCTs, respectively. Blinding is important to minimize bias and obtain a high-quality study and presents additional difficulties in trials involving surgical procedures, as in the case of orthopedics.4 Depending on the type of study, the appropriate blinding of the surgeon or the patient is not possible.4
During the evaluation of the studies, when raters did not agree upon their classifications, they performed a group review. Even after a thorough analysis, some papers were difficult to classify. Description and methodology failures were common, possibly due to the author's lack of knowledge regarding some methodological principles. The type of study and its level of evidence are needed when submitting an article to RBO, although this information is not published. We believe that the publication of this information, similar to international publications, would be interesting for readers. However, as many orthopedists are not familiar with the classification system and also because of the risk of evaluation bias (the author may have a tendency to overvalue the scientific level of his/her study), this information should be reviewed by the editorial board of the magazine.
De Moraes et al.16 assessed 22 RCTs published in RBO and AOB from 2000 to 2009. The authors did not find significant differences in quality or types of publications when comparing the first five-year period with the second one. Hanzlik et al.10, who assessed the percentage of studies with level I evidence in the Journal of Bone and Joint Surgery (American Volume), showed a significant evolution in the types of orthopedics publications over time. In 1975, orthopedics papers comprised only 4% of all publications in the journal, but by 2005, this number had increased to 21% of all publications in the journal. We believe that the lack of changes found by De Moraes et al.16 is because a shorter period was assessed and that the analysis was performed in two periods of five years (2000-2004 and 2005-2009). In our study, we did not analyze historical trends because we considered five years to be too short to do so.
The limitations of this work are that only one magazine (RBO) was analyzed over a shorter period compared to other studies.10, 16 However, unlike researchers who only assessed clinical studies,10, 11, 12, 16 we provided information about the distribution of different types of articles published in the magazine, In addition, we provided data and information about all Jadad classifications of RCTs, which is unlike the study by de Moraes et al.,16 who only provided mean values of the scoring system.
Critical analyses of the publications in journals are essential to obtain a better understanding of the national scientific scenario. Furthermore, it these analyses are useful to promote changes in editorial policies and can improve the quality of the studies.
5. Conclusion
Case series were the most prevalent (37.23% of the total number of publications) studies, while RCTs accounted for 3.46% of the total number of publications. 61.54% of RCTs were considered to be low quality, and only 1.32% of the clinical studies were classified as having level I evidence.
Conflicts of Interest
The authors have no conflicts of interest to declare associated with this paper.
Footnotes
Work performed at the Hospital das Clínicas, School of Medicine at USP, São Paulo, SP, Brazil.
References
- 1.Sackett D., Rosenberg W., Gray J., Haynes R., Richardson W. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(13):71–72. doi: 10.1136/bmj.312.7023.71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Carter M.J. Evidence-based medicine: an overview of key concepts. Ostomy Wound Management. 2010;56(4):68–85. [PubMed] [Google Scholar]
- 3.Carr A. Evidence-based orthopaedic surgery: what type of research will best improve clinical practice? J Bone Joint Surg Br. 2005;87(12):1593–1594. doi: 10.1302/0301-620X.87B12.17085. [DOI] [PubMed] [Google Scholar]
- 4.Malavolta E.A., Demange M.K., Gobbi R.G., Imamura M., Fregni F. Ensaios clínicos controlados e randomizados na Ortopedia: dificuldades e limitações. Rev Bras Ortop. 2011;46(4):452–459. doi: 10.1016/S2255-4971(15)30261-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Chaudhry H., Mundi R., Singh I., Einhorn T.A., Bhandari M. How good is the orthopaedic literature? Indian J Orthop. 2008;42(2):144–149. doi: 10.4103/0019-5413.40250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hartz A., Marsh J.L. Methodologic issues in observational studies. Clin Orthop Relat Res. 2003;413:33–42. doi: 10.1097/01.blo.0000079325.41006.95. [DOI] [PubMed] [Google Scholar]
- 7.Parsons N.R., Hiskens R., Price C.L., Achten J., Costa M.L. A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br. 2011;93(9):1154–1159. doi: 10.1302/0301-620X.93B9.27193. [DOI] [PubMed] [Google Scholar]
- 8.Camanho GL. Nivel de evidência. [Editorial]. Rev Bras Ortop. 2009; 44.(6).
- 9.Jadad A.R., Moore R.A., Carroll D., Jenkinson C., Reynolds D.J., Gavaghan D.J. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled clinical trials. 1996;17(1):1–12. doi: 10.1016/0197-2456(95)00134-4. [DOI] [PubMed] [Google Scholar]
- 10.Hanzlik S., Mahabir R.C., Baynosa R.C., Khiabani K.T. Levels of evidence in research published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91(2):425–428. doi: 10.2106/JBJS.H.00108. [DOI] [PubMed] [Google Scholar]
- 11.Obremskey W.T., Pappas N., Attallah-Wasif E., Tornetta P., Bhandari M. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87(12):2632–2638. doi: 10.2106/JBJS.E.00370. [DOI] [PubMed] [Google Scholar]
- 12.Cashin M.S., Kelley S.P., Douziech J.R., Varghese R.A., Hamilton Q.P., Mulpuri K. The levels of evidence in pediatric orthopaedic journals: where are we now? J Pediatr Orthop. 2011;31(6):721–725. doi: 10.1097/BPO.0b013e31822aa11a. [DOI] [PubMed] [Google Scholar]
- 13.Brighton B., Bhandari M., Tornetta P., Felson D.T. Hierarchy of evidence: from case reports to randomized controlled trials. Clin Orthop Relat Res. 2003;(413):19–24. doi: 10.1097/01.blo.0000079323.41006.12. [DOI] [PubMed] [Google Scholar]
- 14.Dulai S.K., Epi M., Slobogean ÞBLT, Beauchamp ÞRD, Mulpuri K., Ortho M.S. A Quality Assessment of Randomized Clinical Trials in Pediatric Orthopaedics. Clinical Trials. 2007;27(5):573–581. doi: 10.1097/bpo.0b013e3180621f3e. [DOI] [PubMed] [Google Scholar]
- 15.Poolman R.W., Struijs P.A., Krips R., Sierevelt I.N., Lutz K.H., Bhandari M. Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC medical research methodology. BMC Med Res Methodol. 2006;6:44. doi: 10.1186/1471-2288-6-44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.De Moraes V.Y., Moreira C.D., Jun M., Tamaoki S. Ensaios clínicos randomizados na ortopedia e traumatologia: avaliação sistemática da evidência nacional. Trauma. 2010;45(6):601–605. [Google Scholar]