Abstract
Purpose
To assess the quality of randomized controlled urological trials conducted by Korean medical institutions.
Materials and Methods
Quality assessment was conducted by using the Jadad scale; in addition, the van Tulder scale and the Cochrane Collaboration risk of bias tool were used as individual indices. All assessments were performed by two reviewers. If the outcomes differed, the two reviewers and a third reviewer adjusted the discrepancy in the results through discussion. Starting from 1986, a quality analysis of randomized controlled trials (RCTs) was conducted in 1-year and 5-year units. The quality assessment was conducted by subject, type of intervention, presence of double blinding, presence of funding, and review by an Institutional Review Board (IRB).
Results
Whereas the number of RCTs published has gradually increased, there was no significant difference in the quality of the RCTs according to publication year. Drug studies, double-blind studies, studies with funding, and studies reviewed by IRBs had higher quality scores and a higher percentage of high-quality RCTs than did other studies. Thirty-six RCTs were published in journals included in the Science Citation Index and 20 RCTs were published in journals included in the Science Citation Index Expanded. The largest number of RCTs (32.32%) were published by the Korean Journal of Urology.
Conclusions
A quantitative increase was observed in RCTs over time, but no qualitative improvement in the RCTs was observed. It seems necessary to put effort into the quality improvement of RCTs at the design stage.
Keywords: Journal article, Randomized controlled trial
INTRODUCTION
Evidence-based medicine (EBM) was introduced by the EBM Working Group in 1992 and is defined as the conscientious, judicious, and explicit use of the best evidence in making decisions for the care of individual patients [1]. Randomized controlled trials (RCTs) have been acknowledged as important standards for EBM. The RCT is a study design that can minimize bias and produce the most dependable data among study methods [2]. However, even the RCT cannot eliminate all bias, which can occur at the designing, conducting, or reporting phase and can yield misleading results [3]. Peer review is an essential method for preventing such results from being used as the basis of clinical applications [4]. Moreover, objective methodological quality assessment of published studies is essential because these studies can influence the quality of medical care received by patients [5]. The methodological quality of an article can represent its overall quality and should therefore be assessed at the design, conduct, and analysis steps [6,7]. In addition, any erroneous data can be identified throughout the assessment process, thereby eliminating the erroneous data from clinical care and saving medical expenses [8]. When meta-analyses or systematic reviews are conducted with the use of data from low-quality RCTs, the reliability of the conclusions drawn can be impaired [3].
Several methods can be used to assess the methodological quality of clinical trials, such as scales, individual markers, and checklists. The scales method is easy to apply to interstudy comparisons and is more applicable to performing quantitative assessments on the quality of a clinical trial than are the other methods. Randomization, blinding, and dropout are the factors of scale that directly relate to reducing bias. The Jadad quality assessment scale (Jadad scale) is a representative quality assessment tool consisting of these three items [9]. The Jadad scale has been widely used because of its simple questionnaire and capacity to make assessment easy, but it does not include an assessment item for concealment of allocation. However, there is an individual marker method that assesses allocation concealment, which is a way to randomize the allocation sequence to avoid any selection bias in the allocation of patients for treatment [10]. The van Tulder scale and the Cochrane Collaboration risk of bias tool (CCRBT) include an assessment item for concealment of allocation.
Some recent studies have examined issues in the quality assessment of RCTs. For example, Lee et al. [11] analyzed all RCTs published in the Korean Journal of Urology (Korean J Urol). The present study assessed the quality of urological RCTs that were conducted in Korea by using the Jadad scale, the van Tulder scale, and the CCRBT. The results of the present study can be used to assess the past and present status of urology research in Korea. Moreover, the results of this study can suggest guidelines for future studies and thus improve the medical practice of urology in Korea.
MATERIALS AND METHODS
1. Cohort study
Original articles that described urological studies conducted in Korean institutions were manually searched.
2. Selection of RCTs
Two reviewers independently searched all reports of RCTs conducted in Korean institutions by using the PubMed Medline database and KoreaMed. The reviewers used search limits and searched for terms such as "random," "randomized," and "randomly." All articles searched were selected as RCTs by the Methods section of the articles. The third reviewer made a final selection by adjusting the data collected.
3. Assessment of the quality of RCTs
Quality assessment was conducted by using the Jadad scale; in addition, the van Tulder scale and the CCRBT were used as individual indices. All assessments were performed by two reviewers. If there were different outcomes, the two reviewers and a third reviewer adjusted the discrepancy in the results through discussion. Starting from 1986, a quality analysis of RCTs was conducted in 1-year and 5-year units. The quality assessment was conducted by subject, type of intervention, presence of double blinding, presence of funding, and review by an Institutional Review Board (IRB).
1) Jadad scale
The Jadad scale is also known as the Oxford quality scoring system and assesses RCT-related literature. It is composed of five points in total: two in relation to randomization, two in relation to blinding, and one in relation to the dropout rate [9]. When the report includes only general comments with no detailed description of randomization and blinding, one point in each category is given. One point is added when there is a detailed description of the appropriate method. However, when the description method is inappropriate, one point is deducted. When the specified number and reasons for dropouts by each subject group are provided, one point is given. Even if there are no dropouts, this should be stated. If the total is ≥3 points, the study is considered to be high quality, but if the total is ≤2 points, the study is considered to be low quality. However, if it was not possible for the design of the study to be double-blinded, the study considered to be high quality if the total score is ≥2 points.
2) van Tulder scale
The van Tulder scale is designed to make assessments of 11 components including randomization, allocation concealment, baseline characteristics, patient blinding, caregiver blinding, observer blinding, cointervention, compliance, dropout rate, end point assessment time point, and intention-to-treat analysis [12]. The assessment method is the selection of "yes," "no," or "don't know" for each item. If ≥5 items are satisfied (≥5 points), the quality of the report is deemed to be high.
3) CCRBT
The CCRBT assesses the quality of RCTs by using six classifications: sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential threats to validity. The assessment indicates "yes," "no," or "unclear" for each domain, designating a low, high, and unclear risk of bias, respectively. In cases in which the first three questions are answered with "yes" and when no important concerns related to the last three domains are identified, the study is classified as having a low risk of bias [13]. If ≤2 domains are answered with "unclear" or "no," the study is classified as having a moderate risk of bias. If ≥3 domains are answered with "unclear" or "no," the study is classified as having a high risk of bias.
4. Statistical analysis
The one-way analysis of variance test was used to compare and analyze the respective scores obtained by each assessment tool, and a chi-square test was used to compare and analyze the ratio of high-quality articles and the quality assessment outcomes from the CCRBT. The quality assessment of RCTs according to intervention type, double blinding, funding, and IRB review was analyzed by Student's t-test. IBM SPSS ver. 18.0 (IBM Co., Armonk, NY, USA) was used for all statistical analyses, and a p-value of <0.05 was considered statistically significant.
RESULTS
1. Quantitative variation of RCTs over time
Ninety-nine urological RCTs were published between 1986 and 2011 (Fig. 1). Among these, 2 articles (2.02%) were published from 1986 to 1990, 6 articles (6.06%) from 1991 to 1995, 12 articles (12.12%) from 1996 to 1997, 14 articles (14.14%) from 2001 to 2005, 52 articles (52.53%) from 2006 to 2010, and 13 articles (13.13%) in 2011 (Table 1).
FIG. 1.
Flow sheet.
TABLE 1.
Characteristics of RCTs according to publication year
Values are presented as number (%).
RCT, randomized controlled trial; IRB, Institutional Review Board.
a:Chi-square test.
2. Qualitative variation in RCTs over time
1) Jadad scale
The results of the quality assessment are presented in 5-year units starting in 1986. The mean Jadad scale for RCTs from 1986 to 1990 was 1.50±0.71, and the scores increased to 2.38±0.96 in 2011 (p=0.045) (Table 1). However, no statistical difference was seen in the quality assessment of RCTs according to publication year with the use of 1-year units (p=0.192) (Fig. 2). There were no high-quality articles published from 1986 to 1990, but this number increased to 7 of 13 RCTs (53.85%) in 2011 (p=0.344) (Table 2).
FIG. 2.
Quality assessment of randomized controlled trials according to publication year by use of 1-year units. One-way analysis of variance. a:Chi-square test.
TABLE 2.
Quality assessment of RCTs according to publication year
Values are presented as mean±standard deviation or number (%).
RCT, randomized controlled trial.
One-way analysis of variance.
a:Chi-square test.
2) van Tulder assessment scale
There was no statistical difference in the quality assessment of RCTs according to publication year (5-year units, p=0.974; 1-year units, p=0.811). The percentage of high-quality articles showed no statistical difference according to publication year (p=0.904) (Table 2).
3) CCRBT
There were no low risk of bias articles among the RCTs published up to 2011 by the CCRBT assessment. There was no statistical difference according to publication year by the CCRBT (5-year units, p=0.252; 1-year units, p=0.632).
3. Analysis of RCT quality by medical subject
Among the RCT articles presented during the 26 years (1986 to 2011) included in this study, 20 articles were about erectile dysfunction, 15 were about benign prostatic hyperplasia, 12 were about prostate cancer, 8 were about stone disease, 8 were about overactive bladder, 7 were about premature ejaculation, 6 were about chronic pelvic pain syndrome, 4 were about stress urinary incontinence, 3 were about bladder cancer, and 16 were about other subjects. There were no statistically significant differences observed among groups by the CCRBT assessment. However, both in the Jadad scale assessment and in the van Tulder scale assessment, the quality of RCTs and the percentage of high-quality RCTs differed significantly among subjects (Table 3).
TABLE 3.
Characteristics of RCTs according to subjects
Values are presented as mean±standard deviation or number (%).
RCT, randomized controlled trial; CPPS, chronic pelvic pain syndrome.
One-way analysis of variance.
a:Chi-square test.
4. Analysis of factors related to the quality of the articles
Drug studies, double-blind studies, studies with funding, and studies reviewed by IRBs had higher quality scores and a higher percentage of high-quality RCTs than did other studies (Table 4). The differences were statistically significant.
TABLE 4.
Factors associated with quality of RCTs
Values are presented as mean±standard deviation or number (%).
RCT, randomized controlled trial; IRB, Institutional Review Board.
Student's t-test.
a:Chi-square test.
5. Publication journal and institution of study
Thirty-six RCTs were published in journals included in the Science Citation Index and 20 RCTs were published in journals included in the Science Citation Index Expanded. The largest number of RCTs (32.32%) was published by Korean J Urol (Fig. 3). Sixteen RCTs were conducted at Seoul National University (Fig. 4).
FIG. 3.
Journals in which articles were published. SCI, science citation idex; SCIE, Science Citation Index Expanded.
FIG. 4.
Medical institution of the corresponding author.
DISCUSSION
In the present study, the quality of published urological RCTs conducted in Korea from 1986 to 2011 was assessed. Whereas the number of RCTs published has gradually increased, there was no significant difference in the quality of RCTs according to publication year. The small number of methodological descriptions of concealment of allocation and the lack of double-blind studies were the greatest factors in preventing studies from receiving high-quality assessments.
There have been few qualitative analyses of RCTs. Uetani et al. [2] analyzed medical RCTs conducted in Japan by using consolidated standards for reporting of trials (CONSORT) statements. The CONSORT statement was announced in 1996 with an aim to decrease the number of poorly conducted RCTs by detailing appropriate standards. Since the release of the CONSORT statement, various journals and organizations have played a leading role in enhancing the quality of RCTs [14]. Uetani et al. [2] showed that of 98 RCTs conducted in Japan from January to March 2004 that had been published, only 11 were in accordance with the CONSORT statement. However, because the CONSORT statement is not a quality assessment tool, it cannot be used to quantify the qualitative analysis.
Various types of methodological quality assessment tools for RCTs are available, including the Campell, Chalmers, CCRBT, Jadad, Moher, Newell's, and van Tulder methods. However, assessment of the quality of trials remains controversial, and no consensus exists on which tools are highly accurate and valid [15]. The present study was able to overcome such limitations through the use of three different tools. These three tools are representative assessment tools and are commonly used both nationwide and worldwide. The Jadad scale is a simple and easy method of performance assessment, but does not include assessment items for concealment of allocation. Therefore, additional analyses were performed by using the van Tulder scale and the CCRBT to supplement the Jadad scale. Patient, caregiver, and observer blinding as well as allocation concealment are considered in the van Tulder scale. Also, the term for intention-to-treat analysis is included in van Tulder scale. Although the CCRBT has a limitation in that it cannot quantify the quality of RCTs, it can be used to perform more objective analysis of RCTs because of its more detailed interpretation of each classification.
Kim et al. [16] analyzed the quality of RTCs published in five academic journals (the Korean Journal of Internal Medicine, the Journal of the Korean Surgical Society, the Korean Journal of Obstetrics and Gynecology, the Korean Journal of Pediatrics, and the Korean Journal of Family Medicine) by using the Jadad scale. They found that the number of RCTs with a Jadad score of more than 2 points increased in the 1990s compared with the 1980s. In addition, Chung et al. [5] analyzed RCTs published in the Korean Journal of Family Medicine from 1980 to 2005 and reported that the number of RCTs increased according to publication year. This increase in the number of RCTs may have been a result of the growing influence of EBM [17]. In the present study, as in these previous reports, the number of RCTs was found to have increased over time. However, the quality of the RCTs did not significantly change according to publication year.
Lee et al. [11] previously analyzed the quality of RCTs published in the Korean J Urol. According to their study, there were 28 RCTs out of 3,156 original articles presented from 1991 to 2010. The mean Jadad scale score was 1.75 and 8 RCT articles were determined to be high-quality articles. They found that the number of RCT articles and their quality improved over time. Moreover, they suggested that the descriptions of allocation concealment in the RCTs published in the Korean J Urol needed improvement. In the present study, among the urological RCTs conducted in Korea, only three articles had proper descriptions of allocation concealment. Hewitt et al. [18] showed that 46% of RCTs published in four different medical journals (the British Medical Journal, the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine) had inappropriate descriptions of allocation concealment. According to Schulz and Grimes [19], not incorporating allocation concealment into a study can impair the effects of randomization and blinding. The omission of allocation concealment could distort the results of the intervention by more than 40%. The quality of Korean urology reports would be improved if clinical studies included adequate blinding and allocation concealment.
Double blinding can be applied to drug studies more easily than to nondrug studies, such as those investigating surgery. Therefore, studies investigating drugs were scored as higher quality in comparison with studies not investigating drugs. In the present study, double-blind studies and drug studies were of better quality than other studies.
The present study confirmed that studies that were under IRB review were of better quality than RCTs that were not. To the best of our knowledge, no previously published study has analyzed the association of IRB review and the quality of articles. IRB review partially serves to assess the feasibility of the study design and the performance of the study protocol. The system by which RCT study protocols are validated and approved by IRBs has played a major role in raising the quality of articles, because IRB approval is considered an international quality standard. Clifford et al. [20] reported that there was no association between the funding source and the quality of the article. However, in the present study, funded studies were of better quality than were those that were not funded. This result may indicate that receiving financial support helps investigators to design well-organized studies and perform orderly research, resulting in higher quality articles. Moreover, the results of the present study indicated that most RCTs were published by selected universities and most RCTs were published in Korean J Urol.
This study had at least one limitation. It is possible that some RCTs were not considered in the present study. The extraction of RCTs and their quality assessments were based on the subjective judgment of the researchers. Therefore, two medical doctors independently extracted the RCTs. The assessments were performed independently by two reviewers and the outcomes were adjusted to ensure objectivity and reliability.
CONCLUSIONS
For the first time, the quality of urological RCTs conducted in Korea was assessed. The number of RCTs conducted by Korean medical institutions has increased over time. However, no qualitative improvement of RCTs was observed over time. In present study, areas for improvement were identified, and the findings of this study may contribute significantly to the overall quality of urological medical research in Korea.
Footnotes
The authors have nothing to disclose.
References
- 1.Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312:71–72. doi: 10.1136/bmj.312.7023.71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Uetani K, Nakayama T, Ikai H, Yonemoto N, Moher D. Quality of reports on randomized controlled trials conducted in Japan: evaluation of adherence to the CONSORT statement. Intern Med. 2009;48:307–313. doi: 10.2169/internalmedicine.48.1358. [DOI] [PubMed] [Google Scholar]
- 3.Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276:637–639. doi: 10.1001/jama.276.8.637. [DOI] [PubMed] [Google Scholar]
- 4.Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The validity of peer review in a general medicine journal. PLoS One. 2011;6:e22475. doi: 10.1371/journal.pone.0022475. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Chung W, Lee KW, Hwang IH, Lee DH, Kim SY. Quality assessment of randomized controlled trials in the Journal of the Korean Academy of Family Medicine. Korean J Fam Med. 2009;30:626–631. [Google Scholar]
- 6.Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol. 1986;4:942–951. doi: 10.1200/JCO.1986.4.6.942. [DOI] [PubMed] [Google Scholar]
- 7.Autorino R, Borges C, White MA, Altunrende F, Perdona S, Haber GP, et al. Randomized clinical trials presented at the World Congress of Endourology: how is the quality of reporting? J Endourol. 2010;24:2067–2073. doi: 10.1089/end.2009.0541. [DOI] [PubMed] [Google Scholar]
- 8.Lim SM, Shin ES, Lee SH, Seo KH, Jung YM, Jang JE. Tools for assessing quality and risk of bias by levels of evidence. J Korean Med Assoc. 2011;54:419–429. [Google Scholar]
- 9.Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1–12. doi: 10.1016/0197-2456(95)00134-4. [DOI] [PubMed] [Google Scholar]
- 10.Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess. 1999;3:i–iv. 1–98. [PubMed] [Google Scholar]
- 11.Lee JY, Chung JH, Kang DH, Lee JW, Moon HS, Yoo TK, et al. Quality assessment of randomized controlled trials published in the Korean Journal of Urology over the past 20 years. Korean J Urol. 2011;52:642–646. doi: 10.4111/kju.2011.52.9.642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.van Tulder M, Furlan A, Bombardier C, Bouter L Editorial Board of the Cochrane Collaboration Back Review Group. Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976) 2003;28:1290–1299. doi: 10.1097/01.BRS.0000065484.95996.AF. [DOI] [PubMed] [Google Scholar]
- 13.Higgins JP, Green S, editors. Cochrane handbook for systematic reviews of interventions. Ver. 5.1.0 [Internet] The Cochrane Collaboration; 2011. [cited 2013 Jan 7]. Available from: http://www.cochrane-handbook.org. [Google Scholar]
- 14.Moher D, Schulz KF, Altman D CONSORT Group (Consolidated Standards of Reporting Trials) The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285:1987–1991. doi: 10.1001/jama.285.15.1987. [DOI] [PubMed] [Google Scholar]
- 15.Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG. Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract. 2012;18:12–18. doi: 10.1111/j.1365-2753.2010.01516.x. [DOI] [PubMed] [Google Scholar]
- 16.Kim SW, Choi YS, Ahn HS, Lee HY, Ahn DS, Lee YM. Quantity and quality assessment of randomized controlled trials published in five Korean medical journals, from 1980 to 2000. J Korean Acad Fam Med. 2004;25:118–125. [Google Scholar]
- 17.Keech AC, Pike R, Granger RE, Gebski VJ. Interpreting the results of a clinical trial. Med J Aust. 2007;186:318–319. doi: 10.5694/j.1326-5377.2007.tb00911.x. [DOI] [PubMed] [Google Scholar]
- 18.Hewitt C, Hahn S, Torgerson DJ, Watson J, Bland JM. Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ. 2005;330:1057–1058. doi: 10.1136/bmj.38413.576713.AE. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359:614–618. doi: 10.1016/S0140-6736(02)07750-4. [DOI] [PubMed] [Google Scholar]
- 20.Clifford TJ, Barrowman NJ, Moher D. Funding source, trial outcome and reporting quality: are they related? Results of a pilot study. BMC Health Serv Res. 2002;2:18. doi: 10.1186/1472-6963-2-18. [DOI] [PMC free article] [PubMed] [Google Scholar]