RANDOMIZED CONTROLLED TRIALS IN ORTHOPEDICS AND TRAUMATOLOGY: SYSTEMATIC ANALYSIS ON THE NATIONAL EVIDENCE

Abstract

Objective: To assess whether there has been any improvement in the quality and quantity of randomized controlled trials (RCTs) in nationally published journals through the application of standardized and validated scores. Methods: We electronically selected all RCTs published in the two indexed Brazilian journals that focus on orthopedics, over the period 2000-2009: Acta Ortopédica Brasileira (AOB) and Revista Brasileira de Ortopedia (RBO). These RCTs were identified and scored by two independent researchers in accordance with the Jadad scale and the Cochrane Bone, Joint and Muscle Trauma Group score. The studies selected were grouped as follows: 1) publication period (2000-2004 or 2004-2009); 2) journal of publication (AOB or RBO). Results: Twenty-two papers were selected: 10 from AOB and 12 from RBO. No statistically significant differences were found between the proportions (nRCT/nTotal of published papers) of RCTs published in the two journals (p = 0.458), or in the Jadad score (p = 0.722) and Cochrane score (p = 0.630). Conclusion: The relative quality and quantity of RCTs in the journals analyzed were similar. There was a trend towards improvement of quality, but there was no increase in the number of RCTs between the two periods analyzed.

INTRODUCTION

Critical assessment of the orthopedic literature has become a necessary tool for those who seek up-to-date knowledge1, 2, 3. Within this context, acquisition of information in a systematic, evidence-based manner, and consequent categorization into levels of evidence⁽⁴⁾ becomes mandatory, in view of the great number of published papers and the frequent methodological traps that may lead readers to faulty conclusions that are potentially dangerous for medical practice5, 6, 7, 8.

Efforts towards defining adequate and uniform methodological criteria for publishing good-quality studies have led researchers to rethink the planning and publication of research, so that greater scientific rigor and greater external validity are achieved9, 10, 11. Among these efforts, with regard to treatment, randomized controlled trials (RCTs) are the type of study that presents the best level of evidence12, 13, given that their methodological design makes it possible to reach comparative randomized conclusions regarding the best treatment option for each clinical question, using validated tools for measuring the significant outcomes from the clinical condition under examination. Today, RCTs attract greater prestige and attention within the literature.

Despite the attribution of levels of evidence, assessments on studies in an individualized manner cannot be neglected, given that inconsistencies in such attributions occur frequently, as made clear by some authors5, 14. A previous study demonstrated that there was equivalence between the quality of Brazilian national and foreign production between 1988 and 2002⁽¹⁵⁾. Since then, it was recently demonstrated regarding the American literature that there had been an improvement in the quality of published papers, despite a considerable number of potentially correctable methodological faults9, 16.

It would be hoped that, in the Brazilian literature, quantitative and qualitative improvements in the level of evidence of published studies would be seen, in line with the worldwide trend⁽¹⁷⁾.

The aims of the present study were:

• 1.To investigate whether there was any increase in the numbers of RCTs published, comparing the periods 2000-2004 and 2004-2009 (quantitative evaluation);
• 2.To evaluate these studies in accordance with their methodological quality (Jadad score and Cochrane Collaboration score) (qualitative evaluation).

MATERIALS AND METHOD

The investigation was conducted by two independent researchers (V.Y.M, C.D.M), who used an electronic search to select all the editions, and perform manual extraction of all the studies, that were described as RCTs in the two Brazilian indexed journals that focus mainly on publishing research relating to orthopedics and traumatology: Acta Ortopédica Brasileira (AOB) and Revista Brasileira de Ortopedia (RBO).

We defined RCTs as follows: I. Trials that have a design that is planned before data gathering; II. Trials involving treatment; and III. Trials in which a given patient can be included in any of the allocation groups with the same chance⁽¹³⁾.

The two investigators independently evaluated all the titles and structured abstracts of these journals and set aside for detailed evaluation the studies that were presented as RCTs that, a priori, were published between 2000 and 2009. Studies that presented imprecise or inconclusive abstracts were also set aside and doubts regarding them were resolved by reading the full text and/or contacting the principal investigator of the study. These were included or excluded from the study as soon as the missing information was obtained and/or after reaching a consensus between the researchers.

The studies included were scored in accordance with the score of Jadad et al⁽¹⁸⁾ and the score of the Cochrane Bone, Joint and Muscle Trauma Group⁽¹⁹⁾. After reading the studies in full, the scores were considered separately (S1 and S2), and as the mean between them (S1/S2), in order to analyze the groups. These groups were defined as follows: Group I: studies published before 2004; Group II: studies published between 2004 and 2009. Together, an evaluation according to the journal of publication was performed (AOB and RBO), for the same scores defined above.

This study was approved by the research ethics committee of our institution (no. 0016/10).

Inclusion criteria

RCTs published in indexed Brazilian journals focusing on orthopedics and traumatology (AOB and RBO) between 2000 and 2009 were included.

Exclusion criteria

Studies that resulted in methodological doubts that remained even after a consensus meeting between the investigators, and for which no contact with the principal investigator through traditional means could be achieved, were excluded.

Statistical analysis

The premise of normality of distribution of the sample was assessed using the Kolmogorov-Smirnov test. The Mann-Whitney test was used for comparisons between the means for the scores in the different groups. To evaluate the reliability, with the aim of assessing the internal consistency of the inter-observer Jadad scores, the Cohen kappa test was used, with paired samples⁽²⁰⁾. The method proposed by Landis and Koch⁽²¹⁾ for interpreting the degree of concordance was used: I. < 0 — poor; II. 0 to 0.20 — slight; III. 0.21 to 0.40 — fair; IV. 0.41 to 0.60 — moderate; V. 0.61 to 0.80 — excellent, substantial; VI. 0.81 to 1.00 — almost perfect. The chisquare test was used for proportional evaluation between the different periods (proportion of RCTs in period 1 and proportion of RCTs in period 2).

RESULTS

Twenty-two studies were included, of which 10 came from AOB22, 23, 24, 25, 26, 27, 28, 29, 30, 31 and 12 from RBO32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43 (Table 1). Evaluation of the inter-observer reliability (S1 and S2) for the Jadad score resulted in a kappa value of 0.611 (substantial, excellent).

Table 1.

Number and percentage of RCTs and non-RCTs in the journals.

	AOB	RBO	P value
RCT	10 (2.4%)	12 (1.7%)
Non-RCT	411	679	0.458

RCT - randomized controlled trial.

Non-RCT -non-randomized controlled trial.

Comparison of the methodological quality between the studies, according to the Jadad and Cochrane collaboration scores, did not demonstrate any statistical differences when the studies were grouped according to publication period (Tables 2 and 3). When grouped according to publication period, there were no statistically significant differences (Tables 4 and 5), except for examiner 2, for the Jadad score, in which the methodological quality in the period from 2004 to 2009 was favored (Mann-Whitney test; p = 0.02). The ratio between all the studies published and the RCTs did not show any difference between the two journals (p = 0.867).

Table 2.

Mean Cochrane scores for the RCTs according to the journal.

	Journal
AOB		RBO
Mean	SD	Mean	SD	P value
Cochrane score	12.75	4.63	14.33	3.93	0.722

Table 3.

Mean Jadad score for the RCTs according to the journal.

		JADAD S1		JADAD S2		Mean (S1/S2)
Mean	SD	Mean	SD	Mean	SD
Journal	AOB	2.1	1.29	2.1	1.45	2.1	1.31
	RBO	2.5	1.38	2.17	1.19	2.33	1.23
P value			0.497		0.923		0.674

Table 4.

Mean Jadad score for the RCTs in the two periods.

		JADAD S1		JADAD S2		Mean (S1/S2)
Mean	SD	Mean	SD	Mean	SD
Period	2000-2004 (n = 7)	1.86	0.69	1.29	0.49	1.57	0.53
2004-2009 (n = 15)	2.53	1.51	2.53	1.36	2.53	1.37
P value			0.447		0.021*		0.09

Table 5.

Mean Cochrane score for the RCTs in the two periods.

	Period
2000-2004		2004-2009
Mean	SD	Mean	SD	P value
Cochrane score	14.07	3.62	13.4	4.6	0.63

DISCUSSION

The results from this study demonstrate a trend towards improvement in the methodological quality of the RCTs published within our setting, even though neither of the investigators found statistical significance. This last observation goes against the efforts coming from foreign and national initiatives1, 2, 4, 10, 44, 45. Another factor to which attention is drawn is the equivalence of the methodological quality of the two national journals, at least with regard to this single analysis.

The absolute number of RCTs (close to 2%) demonstrates two difficulties in real situations: the first is the difficulty of carrying out RCTs with blinding in specialties of a surgical nature; and the second is the characteristic of placing value on case series, which reflects a list of patients who are treated uniformly without methodological criteria. In a certain way, this is congruent with the worldwide literature, which presents similar percentages of RCTs, with improved methodological rigor now demonstrated46, 47. Despite the merit of case series, therapeutics cannot be guided by studies of level III or IV, at least for diseases that are known to occur with high frequency. It is within this panorama that protocols with adequate methodological quality involving several research centers can be created. This suggests that, in this respect, orthopedics and traumatology societies have important activities⁽⁴⁸⁾.

Some bias should be taken into consideration in this investigation: the difficulty involved in using the evaluation scales for RCTs (Jadad and Cochrane) and, because of the low frequency of trials encountered, there is the possibility that there may have been a beta error. The first of these is due to the inherent difficulties of classification systems, and the second is due to aspects of Brazilian realities.

CONCLUSION

The relative quality and quantity of RCTs in the journals analyzed was similar.

There was a trend towards improved quality, but there was no increase in the quantity of RCTs over the two periods analyzed.

Future studies on clinical treatment questions should focus on stimulating the elaboration of research protocols with methodological refinement (RCTs), with the aim of furnishing the best level of evidence and establishing inter-institution cooperation.