A Quality Analysis of Randomized Controlled Trials about Erectile Dysfunction

Abstract

Purpose

A low quality clinical trial could produce errors, and these errors could, in turn, distort the results of the clinical trial. To avoid applying distorted results of trials clinically, a quality analysis of clinical trials is needed.

Materials and Methods

We selected randomized controlled trials (RCTs) about erectile dysfunction (ED) conducted in Korea using Medline and KoreaMed. Quality assessment of selected RCTs was performed using three assessment tools (Jadad scales, van Tulder scale, Cochrane Collaboration Risk of Bias Tool [CCRBT]).

Results

The first RCT about ED conducted in Korea was published in 2002. Since 2002, a total of 20 RCTs have been published in medical journals. Among the 20 articles, only 1 article was found to have a low risk of bias according to the CCRBT. On the Jadad scale, there were 17 high quality articles, while 19 articles were assessed as high quality by the VTS. Only 2 RCTs described the randomization method adequately. Only 1 RCT presented allocation concealment.

Conclusions

A low quality clinical trial could produce errors, and these errors could, in turn, distort the results of the clinical trial. To avoid applying distorted results of trials clinically, a quality analysis of clinical trials is needed. The quality of RCTs was found to be high because almost all of the selected RCTs were double blinded studies. However, the quality of RCTs was inadequate with regard to the lack of randomization and absence of allocation concealment. Therefore, performing adequate randomization and adding a description of the appropriate concealment of allocation may improve the quality of RCTs.

INTRODUCTION

Among the many designs for clinical trials, the randomized controlled trial (RCT) shows the highest level of evidence, which can reduce the bias most effectively. The results of RCTs are typically used as the basis for clinical application.1 Moreover, according to the increasing importance of evidence-based medicine, the importance of RCTs has drawn even more attention.2 However, even RCTs cannot eliminate all bias. Bias can occur at any of the steps in conducting trials and can lead to incorrect results.3 These incorrect results can harm patients and cause unnecessary medical expenses.4 There are some methods for preventing the use of incorrect information in clinical applications. Representatively, peer review is one of the methods by which articles are assessed before publication.5 Another method involves the methodological quality assessment of articles, in which the design, conduct, and analysis of clinical trials is evaluated.6 Assessment of the methodological quality of articles represents the overall quality of clinical trials.7 Methodological quality assessment of clinical trials can include the use of scales, individual markers, and checklists.2 Scales address randomization, double blinding, and dropout rates. The Jadad Quality Assessment Scale (Jadad scale) is the assessment tool that can assess the scales.8 Allocation concealment, as an individual marker, is an index that can assess the randomized allocation sequence. Allocation concealment is essential to avoid selection bias.9 The Jadad scale has been widely used because of its simplicity. However, the Jadad scale cannot evaluate individual markers. On the other hand, the van Tulder scale (VTS) and Cochrane Collaboration Risk of Bias Tool (CCRBT) include indices of individual markers.

Erectile dysfunction (ED) is an inability to achieve an erection sufficient to enable sexual intercourse. Many factors are associated with ED, including vascular, psychological, hormonal, and neurological factors.10 ED affects 150 million males worldwide, and the prevalence of ED is expected to more than double within 10 years due to the increasing lifespan.11 In Korea, some clinical trials about ED have been performed. In the present study, to suggest a direction for clinical trials about ED, we performed a methodological quality assessment of RCTs about ED.

MATERIALS AND METHODS

1. Data collection

Data collection was done by using a Medline and KoreaMed search. Medical Subject Headings (MeSH) phrases related to ED were used for searching. There were no limitations on the language. In total, 20 RCTs were identified in the search.

2. Assessment of the quality of RCTs

The Jadad scale, the VTS, and the CCRBT were used for quality assessment. The quality assessment of RCTs was conducted according to the publication year, while the quality assessment was conducted by type of intervention, presence of funding, and whether it was reviewed by an institutional review board (IRB).

1) Jadad scale

The Jadad scale is composed of five points: two in relation to randomization, two in relation to blinding, and one in relation to the drop-out rate.8 When the RCT includes only general comments without a detailed description of randomization and blinding, one point for each index is given. One point is added when there is a detailed and appropriate description. However, one point is deducted when the description is inappropriate. When the specified number and reasons for drop-outs by each subject group are provided, one point is given. Even if there are no drop-outs, this should be stated specifically. When the total is ≥3 points, it is assessed as high quality but when it is ≤2 points, it is assessed as low quality. However, if it was not possible for the design of the study to be double blinded, it is assessed as high quality when the total score is ≥2 points.

2) van Tulder scale

The VTS is designed to assess 11 components (randomization, allocation concealment, baseline characteristics, patient blinding, caregiver blinding, observer blinding, co-intervention, compliance, drop-out rate, end-point assessment time point, and intention-to-treat analysis).12 Its assessment method is to select 'yes', 'no', or 'don't know' for each item, and when ≥5 items are satisfied (≥5 points), the quality of the report is deemed high.

3) Cochrane collaboration risk of bias tool

The CCRBT assesses the quality of RCTs in six classifications (sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential threats to validity). The assessment indicates 'yes', 'no', or 'unclear' for each domain. In cases where the first three questions are answered with 'yes' and when no important concerns related to the last three domains are identified, it is classified as having a low risk of bias, while cases where it is assessed in ≤2 domains with 'unclear' or 'no', it is classified as having a moderate risk of bias. The cases assessed in ≥3 domains with 'unclear' or 'no' are classified as having a high risk of bias.

3. Statistical analysis

The Kruskal-Wallis test was used to compare and analyze the respective scores obtained by each assessment tool, while a chi-squared test was used to compare and analyze the ratio of the high quality articles and the quality assessment outcomes from the CCRBT. The quality assessment of RCTs according to the publication year was analyzed by one-way ANOVA analysis. PASW Statistics 18.0 (IBM Co., Armonk, NY, USA) was used for all statistical analyses, and a p value of <0.05 was considered statistically significant.

RESULTS

A total of 20 RCTs were selected in an electronic search. All of the selected RCTs were conducted to evaluate drugs. Among the 20 RCTs, 19 RCTs were double-blind studies, and there were only 2 RCTs which described the randomization method adequately. Only 1 RCT presented allocation concealment. Eighteen RCTs mentioned review by an IRB and 15 RCTs were funded studies. Sixteen RCTs had been published in Science Citation Index (SCI) or Science Citation Index Expanded (SCIE) journals (Table 1).

Table 1.

Characteristics of RCTs

Values are presented as number (%).

RCTs: randomized controlled trials, IRB: institutional review board, SCI: science citation index, SCIE: science citation index expanded.

The largest number of articles had been conducted to evaluate mirodenafil (n=6), and 5 articles evaluated udenafil (Table 2).

Table 2.

Drugs evaluated in the included studies

Among the 20 articles, only 1 article was assessed as having a low risk of bias, 3 articles were assessed as having a moderate risk of bias, and 16 articles were evaluated as having a high risk of bias by the CCRBT. In the Jadad scale, there were 17 high quality articles, while 19 articles were assessed as high quality articles by the VTS. In the Jadad scale and VTS, there was no statistical improvement according to the publication year. However, in the CCRBT, there was a statistical difference in quality according to the publication year (p=0.036) (Table 3).

Table 3.

Quality assessment of RCTs according to publication year

Values are presented as number (%) or mean±standard deviation.

RCT: randomized controlled trial.

Kruskal-Wallis test, ^†chi-squared test.

Sixteen RCTs had been conducted at Seoul National University. Chonbuk National University and Yonsei University had each published three RCTs. Kyung Hee University and Pusan National University had conducted two RCTs each (Fig. 1).

Fig. 1.

Medical institution of corresponding author.

Three RCTs had been published in The Korean Journal of Urology, in Asian Journal of Andrology and in British Journal of Urology International. And, 2 RCTs had published in Journal of Urology.

DISCUSSION

In the present study, the quality of published RCTs about ED conducted in Korean medical institutions was assessed. In the Medline database, about 900 RCTs that dealt with ED were found. However, there were only 20 RCTs about ED that had been conducted in Korea. Only one RCT was assessed as having a low risk of bias according to the CCRBT. On the Jadad scale and the VTS, there was no significant difference in the quality of RCTs according to the publication year.

ED is defined as the inability to attain or maintain a penile erection sufficient for successful vaginal intercourse.13 In a landmark population-based study, the crude incidence of ED was 26 per 1,000 man-years.14 Many factors such as vascular, psychological, neurological and hormonal abnormalities are associated with ED.15 Many therapeutic strategies have been introduced, such as psychological therapy, lifestyle modification, phosphodiesterase type 5 inhibitors (PDE5-Is), testosterone, and penile prostheses.16 Notably, PDE5-Is have been used as a first line of treatment for ED since sildenafil was introduced in 1998.17,18 Even though many treatment options have been proposed for ED, there are many limitations in the management of ED.16 Because of the high incidence of ED and therapeutic limitations, many studies have been conducted about ED. However, none have assessed the quality of RCTs about ED.

Some previous analyses have been performed on the quality of urological RCTs. Lee et al19 analyzed the quality of RCTs that had been published in the Korean Journal of Urology by the Jadad scale. Twenty-eight RCTs have appeared since 1991 in the Korean Journal of Urology. The mean Jadad scale score of those RCTs was 1.75. In addition, 8 RCTs were assessed as high quality. Moreover, Lee et al19 showed that only one article had adequate allocation concealment. In the present study, compared to Lee et al,19 the mean Jadad scale score (2.95±0.76) was higher. However, in the present study, only 1 RCT had adequate allocation concealment. According to Schulz and Grimes,20 the absence of concealment of allocation can damage the randomization in the process of the study, even if the randomization is well conducted. This may distort the results by up to 40%. Hewitt et al21 reported that about 46% of RCTs published in four journals (British Medical Journal, Journal of the American Medical Association, Lancet, and New England Journal of Medicine) described inappropriate allocation concealment or failed to describe it clearly. Appropriate study design and presentation of allocation concealment should improve the quality of RCTs.

Chung et al2 assessed the quality of RCTs published in the Journal of Korean Medical Science. They reported that the Jadad scale score was 2.30±1.39 and the VTS score was 4.98±2.18. According to the CCRBT, 68.18% of the RCTs were assessed as having a high risk of bias, 20.45% of the RCTs were assessed as having a moderate risk of bias, and 11.36% of the RCTs were assessed as having a low risk of bias. Compared to the previous two studies, the quality of RCTs in the present study seemed to be higher according to the Jadad scale and VTS. This may be because almost all of the selected RCTs in the present study were drug studies. In studies that use a drug as the intervention, double blinding is easier than in non-drug studies because of to the possibility of using a placebo.

The present study showed that there were differences in the quality analysis of RCTs according to the three different tools. There is no consensus on what are considered highly accurate and valid quality assessment tools.2 The Jadad scale has simple quality assessment questions, but it does not include assessment of individual markers. Therefore, additional quality analyses were performed using the VTS and CCRBT to supplement in this regard.

The review from an IRB could serve as an acknowledgement of the feasibility of a study. IRB review should be an important factor in the improvement of the quality of RCTs.2 Funding of clinical trials should enable the establishment of a well-designed study and performance of well-organized research, resulting in many high quality articles.2,6,19 In the present study, almost all of the studies were funded study and had been reviewed by an IRB. This may be an one of the factors that improved the quality of the RCTs. The relatively small numbers of selected RCTs enabled the comparison of quality between funded studies and non-funded studies. It also enabled the comparison of quality according to whether the IRB reviewed the study or not.

The limitation of the present study is the small number of RCTs included. In addition, the assessment of quality was conducted by one researcher. This could have introduced bias because the selection of RCTs and quality assessment depends on the subjective judgment of the researcher.

CONCLUSIONS

The assessed number of RCTs about ED that have been conducted in Korea is small. Even though the quality of RCTs in the present study was superior to that of other previous studies, considering that almost all of the RCTs included in the present study were double blinded, we cannot conclude that the RCTs in the present study are superior to others. In any case, our results emphasize that urological researchers should focus on performing high quality studies that have been reviewed by an IRB, performing adequate randomization, and describing the appropriate concealment of allocation.