Abstract
Background: Iranian Registry of Clinical Trials (IRCT) started as a primary registry in 2008. We examined the characteristics and scientometric measures of prospectively registered clinical trials in IRCT over time and compared them with that of ClinicalTrials.gov.
Methods: We selected eligible trial records between 2008 and 2016 from the IRCT database. We assessed their characteristics and the journal metrics of ensuing outputs over the study period and compared our findings with the corresponding information from ClinicalTrials.gov reported by Magdalena Zwierzyna et al. and a random sample of trials registered with this registry. We used the chi-square test for comparison of proportions and Mann-Whitney U test for comparison of medians. P-value <0.05 was considered statistically significant. Statistical analyses were performed using IBM SPSS Statistics V.22.
Results: 1751 prospectively registered clinical trials were eligible for analysis, of which 1526 (87%) had parallel-group design, 1541 (88%) reported to be randomized, 753 (43%) used double-blinding design, 485 (%27.7) had sample size more than 100, 1313 (75%) completed within a year, 1539 (87.9%) were single centered and 1529 (87.3%) exclusively used public money. Comparison with ClinicalTrials.gov showed that they are less likely to have multiple centers, funded by private sectors, continue beyond one year; and more likely to be randomized, double-blind and get published as a paper. The sample sizes were similar. Journal scientometric measures remained constant over the study period for both databases but were higher in ClinicalTrials.gov (median SJR=1.67, IQR=1.1-3.23) compared with IRCT (median SJR=0.58, IQR=0.34-0.91).
Conclusion: Our findings suggest that clinical trials registered in IRCT are predominantly investigator-initiated studies with acceptable methodological features and high publication rate albeit in journals with substantially lower scientometric measures compared with that of ClinicalTrials.gov. Journal metric indices remained constant despite an increase in the number of registrations in IRCT.
Keywords: Scientometric measures, Clinical trials, IRCT, Characteristics
↑ What is “already known” in this topic:
Journal’s scientometric indices represent the quality of papers it publishes. The number of registered trial studies in IRCT has risen steadily, however, it is not known if their quality is comparable to the trials conducted within the rest of the world.
→ What this article adds:
Our findings suggest that clinical trials registered in IRCT are predominantly investigator initiated studies with acceptable methodological features and high publication rate albeit in journals with substantially lower scientometric measures compared with that of ClinicalTrials.gov. Journal metric indices remained constant despite increase in the number of registrations in IRCT.
Introduction
Randomized clinical trials are the gold standard to assess preventive, therapeutic, diagnostic, or rehabilitative interventions and constitute a foundation for evidence-based medicine (1). They play a critical role in evidence-based policy decision making, provided that they are high quality, transparent and discoverable and their results have been fully disclosed (2, 3). The design, conduct, and reporting of clinical trials have been under focus worldwide and some initiatives such as protocol registration have tried to increase transparency and availability of clinical trials’ results (4).
The objective of clinical trial registration is to make key information of all clinical trials accessible to the public before they start recruiting, although retrospective registration is still allowed in most WHO primary registries (5, 6). It increases transparency in clinical trial conduct and reduces the chance of publication bias and selective reporting (7, 8). Clinical trial registration is now widely considered as an ethical and scientific responsibility (9) and has been included in the World Medical Association’s 3rd Helsinki declaration since 2008 (10).
Iranian Registry of Clinical Trials (IRCT) was established as a primary registry in 2008. Its number of annual registrations has increased steadily since then to more than 3500 per year in 2018 and currently contains over 20000 trial records. However, it is not certain whether or not the quality of conducted trials has reciprocated this rapid quantitative growth, and the characteristics of registered trial protocols are comparable to that of global standards. Or, has this considerable increase in quantity come at the price of decline in quality? To examine this, we assessed the characteristics of the registered records and the journal metrics of resulted outputs over the study period and compared our findings with the corresponding information from ClinicalTrials.gov reported by Magdalena Zwierzyna et al. and a random sample of trials registered with this registry.
Methods
Data source
We used trial protocols registered in the Iranian Registry of Clinical Trials and ClinicalTrials.gov. IRCT is a primary registry in the WHO registry network developed and maintained by the Ministry of Health and Medical Education of Iran. It registers all trial protocols according to International Clinical Trial Registry Platform (ICTRP) guidelines. ClinicalTrials.gov is a Web-based resource maintained by the National Library of Medicine (NLM) at the National Institutes of Health (NIH) currently holding registrations from 320,080 research studies from 209 countries all over the world.
Search strategy
We selected prospectively registered trial protocols (date of registration approval on or before the date of recruitment) between 29th of October 2008 and 31st of December 2016 (Diagram. 1) conducted in Iran, which had valid and complete registration dates. We extracted the characteristics of clinical trial protocols including registration date, expected recruitment start and end dates, gender of participants, purpose, funding source, health conditions studied, use of randomization or blinding methods, use of placebo, type of intervention, study design, study phase, number of recruitment centers and sample size. We categorized the condition studied using the ICD-10 coding system. If several ICD-10 codes were listed in this part, we only considered the first code.
Fig. 1.
Trends of three year moving average of median CiteScore, SNIP and SJR over the period of 2009 to 2015 in journals publishing papers resulted from the trial protocols registered in IRCT and ClinicalTrials.gov
Diagram 1.
Clinical trial inclusion flow chart
We systematically assessed the publication status of the records by comprehensive searching of electronic databases, including PubMed, EMBASE, Google Scholar, Cochrane, Scopus and general Google search, up until 15th of February 2018. Our search terms included the IRCT registration number, the name of the corresponding author for scientific inquiries and the scientific title of the study. The corresponding authors for scientific inquiries of the study were also contacted via email to determine if the clinical trial has already been published. An independent search of electronic databases by a second investigator was conducted on a 10% random sample of the trial protocols to check if all resultant published papers have been retrieved.
We used the published results of the study conducted by Magdalena Zwierzyna et al. (11) that covered the period between 2005 and 2017 to compare the characteristics of registered clinical trials in IRCT with those of ClinicalTrials.gov. We reclassified funding data in Zwierzyna’s paper into three major groups: industry (small and big pharma), public (NIH) and others. Furthermore, we selected and downloaded a random sample of 250 trial protocols registered in ClinicalTrials.gov. Our criteria for inclusion in the selection pool were completed interventional studies with cited results and a start date later than October 2008 and a completion date before 2017.
Scientometric Indicators
Journal metrics for the resultant published papers from both IRCT and ClinicalTrials.gov samples, including CiteScore, Source Normalized Impact per Paper (SNIP) and Scimago Journal Rank (SJR), were extracted over the study period from 2017 published Scopus journal metrics. CiteScore measures average citations received per document published in the serial in the past three years. SNIP and SJR measure weighted citations on subject field and prestige of the citing serial, respectively.
Data Analysis
We used descriptive statistics to summarize the general characteristics of prospectively registered clinical trials in IRCT. Categorical variables were reported as frequencies and percentages; continuous variables as medians and interquartile range (IQR). Three-year moving average was used to depict the trends of scientometric measures over the study period. We used the chi-square test for comparison of proportions and Mann-Whitney U test for comparison of medians. P-value <0.05 was considered statistically significant. Statistical analyses were performed using IBM SPSS Statistics V.22.
Results
A total number of 1751 prospectively registered clinical trials were eligible for analysis (Diagram. 1), of which 87% had parallel-group design, 88% reported to be randomized, 43% used double-blinding design, %27.7 had sample size more than 100, 75% completed within a year, 87.9% were single centered and 87.3% exclusively used public money (Tables 1 and 2).
Table 1. Comparison of characteristics of registered protocols in ClinicalTrials.gov a and IRCT .
| Characteristics | ClinicalTrials.gov ( N=119,840) | IRCT ( N=1751) | P-value | |
| Randomization | <0.001 b | |||
| Non-randomized | 41266 (34.4%) | 44 (2.5%) | ||
| Randomized | 74313 (62.0%) | 1541 (88%) | ||
| N/A | 0 | 166 (9.5%) | ||
| Allocation missing | 4261 (3.6%) | 0 | ||
| Blinding | <0.001 b | |||
| Open label | 64222 (53.6%) | 578 (33%) | ||
| Single blind | 6383 (5.3%) | 322 (18.4%) | ||
| Double blind | 45094 (37.6%) | 753 (43%) | ||
| Triple blind | 0 | 98 (5.6%) | ||
| Masking missing | 4141 (3.5%) | 0 | ||
| Study centers | <0.001 b | |||
| Multi-center | 44775 (37.4%) | 212 (12.1%) | ||
| Single-center | 75065 (62.6%) | 1539 (87.9%) | ||
| Median sample size (IQR) f | 60 (26-157) | 66 (45-100) | <0.001 b | |
| <100 | 74780 (62.4%) | 1247 (71.2%) | ||
| ≥100 | 45059 (37.6%) | 485 (27.7%) | ||
| Study duration (days) Median (IQR) f | 701.0 (335-1,218) | 177.5 (90-364) | - | |
| Funding: f | <0.001 b | |||
| Industry | 62556 (52.2%) | 37 (2.1%) | ||
| Public | 13541 (11.3%) | 1529 (87.3%) | ||
| Others | 43741 (36.5%) | 185 (10.6%) | ||
| Purpose | <0.001 b | |||
| Treatment | 94074 (78.5%) | 1022 (58.4%) | ||
| Prevention | 10306 (8.6%) | 360 (20.6%) | ||
| Number of study groups | <0.001 b | |||
| Single group | 42902 (35.8%) | 146 (8.3%) | ||
| Controlled group | 76937 (64.2%) | 1605 (91.7%) | ||
| Phase II-IV trials published in scientific journal | 8338 (29.9%) d | 160 (47.7%) e | - | |
| Scientometric measures | ||||
| CiteScore | 3.01 (2.28-4.46) | 1.44 (0.88-2.22) | <0.001 c | |
| SNIP | 1.32 (0.98-2) | 0.83 (0.59-1.01) | <0.001c | |
| SJR | 1.67 (1.1-3.23) | 0.58 (0.34-0.91) | <0.001c |
aData is reported by Magdalena Zwierzyna et alb P-values are calculated using chi squared test; cP-values are calculated using Mann-Whitney U test; d Of 27835 phase II-IV registered protocols in ClinicalTrials.gov; e Of 335 phase II-IV of registered protocols in IRCT; fmissing values are less than 1.5% in IRCT
Table 2. Additional characteristics of prospectively registered clinical trials in IRCT between 2008 and 2017 .
| Characteristics | All trials (N=1751) |
| N (%) | |
| Purpose | |
| Treatment | 1022 (58.4%) |
| Prevention | 360 (20.6%) |
| Supportive | 229 (13.1%) |
| Basic sciences | 13 (0.7%) |
| Diagnostic | 24 (1.4%) |
| Health service research | 49 (2.8%) |
| Screening | 8 (0.5%) |
| Other | 46 (2.6%) |
| Phase | |
| Phase 1 or 0 | 32 (1.8%) |
| Phase 2 or 1-2 | 280 (16%) |
| Phase 3 or 2-3 | 472 (27%) |
| Phase 4 | 15 (0.9%) |
| N/A | 951 (54.3%) |
| Bioequivalence | 1 (0.1%) |
| Placebo | |
| Not used | 1061 (60.6%) |
| Used | 690 (39.4%) |
| Trial design | |
| Single group | 146 (8.3%) |
| Parallel group | 1526 (87.2%) |
| Crossover | 53 (3.0%) |
| Factorial | 11 (0.6%) |
| Other | 15 (0.9%) |
| Characteristics | All trials (N=1751) |
| N (%) | |
| Number of study groups | |
| 1.00 | 146 (8.3%) |
| ≥ 2.00 | 1605 (91.7%) |
| Type of intervention | |
| Treatment - Drugs | 1550 (41.5%) |
| Treatment - Other | 255 (6.8%) |
| Treatment - Devices | 46 (1.2%) |
| Treatment - Surgery | 75 (2.0%) |
| Prevention | 410 (11.0%) |
| Rehabilitation | 145 (3.9%) |
| Placebo | 418 (11.2%) |
| Lifestyle | 114 (3.1%) |
| Early detection | 16 (0.4%) |
| Diagnosis | 34 (0.9%) |
| Behavior | 128 (3.4%) |
| N/A | 146 (3.9%) |
| Missing | 8 (0.2 %) |
| Other | 386 (10.4%) |
| Gender of the study population | |
| Female | 440 (25.1%) |
| Male | 86 (4.9%) |
| Both | 1224 (69.9%) |
| Missing | 1 (0.1%) |
Of the 119840 registered clinical trials in ClinicalTrials.gov, 62% was reported to be randomized, 37.6% used double-blinding design, %37.6 had sample size more than 100, 62.6% were single centered and 52.2% exclusively used industry funders. Comparison with ClinicalTrials.gov showed that trials registered in IRCT are less likely to have multiple centers, funded by private sectors, and take more than one year to complete, and more likely to be randomized, double-blind and get published as a paper. Median sample sizes were similar (Table 1).
The more frequently studied medical conditions in IRCT were mental and behavioral disorders with 12.3% trials, followed by diseases of the genitourinary system and endocrine, nutritional and metabolic diseases, whereas oncology was the most represented discipline in ClinicalTrials.gov with 29.1% of the trials, followed by infectious and cardiovascular diseases.
Of the phase II-IV clinical trials registered in IRCT, %47.7 were published in scientific journals while this figure for ClinicalTrials.gov has been reported to be about %30. The median CiteScore for IRCT was 1.44, IQR=0.88-2.22 that was half of the corresponding scientometric measure in ClinicalTrials.gov of 3.01, IQR=2.28-4.46. This was also true for both SNIP (median SNIP =0.83, IQR=0.59-1.01 versus 1.32, IQR=0.98-2) and SJR (median SJR=0.58, IQR=0.34-0.91 versus 1.67, IQR=1.1-3.23) in IRCT and ClinicalTrials.gov respectively (Table 1).
Trends of the three years moving average of median CiteScore, SNIP and SJR over the period of 2009 to 2015 in the published papers resulted from the registered trial protocols in IRCT and ClinicalTrials.gov showed that the Journal scientometric measures remained constant over the study period for both databases (Fig. 1).
Discussion
We found that prospectively registered clinical trials in IRCT within the period of 2008-2017 are usually single centered and investigator-initiated studies which are mostly completed in less than a year and sponsored by publicly funded universities. In contrast, trials registered with ClinicalTrials.gov are more likely to have multiple centres, funded by private sectors and take more than a year to complete. The journal scientometric measures for the published papers in IRCT were generally much lower than the figures for ClinicalTrials.gov with CiteScore value being half and SJR third the corresponding figure in ClinicalTrials.gov. In both data sources, however, they remained constant over the study period.
The observed differences in the characteristics of registered trials in IRCT and ClinicalTrials.gov could be partly explained by their source of funding (11). Following the 1985 integration of medical education into the National Health Service of Iran, there was a rapid increase in the number of public universities and postgraduate students over the decade after, resulting in a surge in the number of investigator-initiated trials (12). The number of international pharmaceutical industry-sponsored trials, however, has been limited due to sanctions and the domestic pharmaceutical companies have only recently started contributing substantially to this area. Therefore, the bulk of trials registered in IRCT are investigator-initiated studies funded by the public sector.
On the other hand, the median sample size is similar in both IRCT and ClinicalTrials.gov registered trials, and the publication rate is even higher for IRCT registered trial protocols (47% versus 30%). We believe, therefore, that the scale of the study alone is unlikely to explain the low journal metrics.
Our data showed that the IRCT registered trials are more likely to be randomized (88% vs. 62%) and double-blind (43% vs. 37.6%) compared with ClinicalTrials.gov. This suggests, on the face of it, that IRCT registered trials are conducted by researchers that are familiar with trial methodologies. However, low scientometric measures in IRCT registered trials (Table 1) indicate that this hasn’t been enough for high ranking journals to publish their results. Given that the novelty of the subject besides sound methodology plays an important role in the acceptance of a paper in a high ranking journal, we believe this to be an important contributing factor for low journal metrics of IRCT registered trials.
The authors believe that the reason behind selecting subject matters with a lower degree of novelty by academic staff at publicly funded universities of Iran is the policy of excessive pressure to increase research paper production. For example, asking master’s degree postgraduate students to publish a paper as a condition for graduation (13), where the resources are inadequate, could negatively impact the quality of the research conducted. Review of the content of the registered trial protocols showed that, in some extreme cases, healthcare and clinical audit studies might be transformed into trial lookalikes by choosing a control group and comparing the outcomes in the two groups. Audit studies by themselves are valuable tools for improving the quality of clinical and healthcare services; however, they could be misused to create studies with questionable scientific value and ethical justifications and lack of equipoise.
We found that journal metric indices did not change over the study period. This could be interpreted in both ways. On one hand, this shows that no effective intervention has been implemented over the study period to improve the quality of the trials. On the other hand, this could be a positive sign showing that despite a substantial increase in the number of registrations in IRCT (14) the increase has not come at the price of further losing quality.
Our study had some limitations. CiteScore, the journal metrics index, of the resultant publication was used as an indication for the overall quality of the study. This might not always be true. Furthermore, this may vary across disciplines and citations are not of equal value everywhere. We used the Source Normalized Impact per Paper (SNIP) and Scimago Journal Ranking (SJR) to account for this. Some trials conducted in Iran might not be registered in IRCT; however, we believe the mainstream trials conducted in Iran are sufficiently represented particularly in the later study period where the registration coverage has greatly improved because of the complete integration IRCT registration in the country’s medical research management system.
The number of missing values in IRCT was very low (less than 1.5%) for each of the variables used in the analysis partly because most of the collected variables are designated as “required” at the time of data entry. Furthermore, the entries are reviewed by IRCT officers upon submission and missing data are normally detected and affected records are returned to the registrant for completion.
Our findings suggest that the quality of the trials registered in IRCT is not acceptable compared with the global standard and therefore is in need of urgent attention. We believe clinical trials with no novelty in their subject matters should be discouraged, and in extreme cases, trial lookalikes should be detected and stopped. In the process of approving a trial subject, enough attention should be paid to the scientific values of the study. Thorough literature review and if necessary, systematic reviews should be conducted first before choosing a subject for a trial study. Institutional review boards and ethics committees should make sure that clinical equipoise has been demonstrated before allowing the research team to embark on the study. It is also necessary to review some of the policies that may contribute to the current situation. Implementing global standards (ICH GCP) (15) in design (SPIRIT) (16), conduct and reporting (CONSORT) (17) of trials could be another important step in improving the quality. However, this is not a task only for those governing researches such as research councils or ethics committees, but it is the responsibility of all members of the research community to increase the awareness about the problem and work towards improving the quality of conducted trials.
Conclusion
Our findings suggest that clinical trials registered in IRCT are predominantly investigator-initiated studies with acceptable methodological features and high publication rate albeit in journals with substantially lower scientometric measures compared with that of ClinicalTrials.gov. Journal metric indices remained constant despite an increase in the number of registrations in IRCT.
Acknowledgment
This work was supported by grant number 971193 from the National Institute for Medical Research Development of Iran.
Conflict of Interests
The authors declare that they have no competing interests.
Cite this article as: Lotfi S, Janani L, Ghalichi L, Tanha K, Solaymani-Dodaran M. Scientometric measures of prospectively registered clinical trials over time: A comparison of IRCT and ClinicalTrials.gov. Med J Islam Repub Iran. 2020 (8 Sep);34:116. https://doi.org/10.34171/mjiri.34.116
Footnotes
Conflicts of Interest: None declared
Funding: This work was supported by grant number 971193 from the National Institute for Medical Research Development of Iran.
References
- 1.Amstutz A, Schandelmaier S, Frei R, Surina J, Agarwal A, Olu KK. et al. Discontinuation and non-publication of randomised clinical trials supported by the main public funding body in Switzerland: a retrospective cohort study. BMJ Open. 2017;7(7):e016216. doi: 10.1136/bmjopen-2017-016216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.van Valkenhoef G, Tervonen T, de Brock B, Hillege H. Deficiencies in the transfer and availability of clinical trials evidence: a review of existing systems and standards. BMC Med Inform Decis Mak. 2012;12(1):95. doi: 10.1186/1472-6947-12-95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hill KD, Henderson HT, Hornik CP, Li JS. Paediatric cardiovascular clinical trials: an analysis of ClinicalTrials gov and the Food and Drug Administration Pediatric Drug Labeling Database. Cardiol Young. 2015;25(S2):172–80. doi: 10.1017/S1047951115000980. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Shamliyan TA, Kane RL. Availability of results from clinical research: failing policy efforts. J Epidemiol Glob Health. 2014;4(1):1–12. doi: 10.1016/j.jegh.2013.08.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Viergever RF, Li K. Trends in global clinical trial registration: an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open. 2015;5(9):e008932. doi: 10.1136/bmjopen-2015-008932. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. WHO. International Clinical Trials Registry Platform (ICTRP): World Health Orginization. Available from: https://www.who.int/ictrp/network/primary/en/.
- 7.Tang W, Fukuzawa M, Ishikawa H, Tsutani K, Kiuchi T. Review of the registration of clinical trials in UMIN-CTR from 2 June 2005 to 1 June 2010-focus on Japan domestic, academic clinical trials. Trials. 2013;14(1):333. doi: 10.1186/1745-6215-14-333. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Loder E, Loder S, Cook S. Characteristics and publication fate of unregistered and retrospectively registered clinical trials submitted to <em>The BMJ</em> over 4 years. BMJ Open. 2018;8(2):e020037. doi: 10.1136/bmjopen-2017-020037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Viergever RF, Karam G, Reis A, Ghersi D. The Quality of Registration of Clinical Trials: Still a Problem. PloS One. 2014;9(1):e84727. doi: 10.1371/journal.pone.0084727. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.World MAI. Declaration of Helsinki Ethical principles for medical research involving human subjects. J Indian Medical Assoc. 2009;107(6):403. [PubMed] [Google Scholar]
- 11.Zwierzyna M, Davies M, Hingorani AD, Hunter J. Clinical trial design and dissemination: comprehensive analysis of ClinicalTrialsgov and PubMed data since 2005. BMJ (Clinical research ed) 2018;361:k2130. doi: 10.1136/bmj.k2130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Solaymani‐Dodaran M, Ostovar A, Khalili D, Vasei M. Iranian Registry of Clinical Trials: path and challenges from conception to a World Health Organization primary register. J Evid Based Med. 2009;2(1):32–5. doi: 10.1111/j.1756-5391.2009.01002.x. [DOI] [PubMed] [Google Scholar]
- 13.Malakoti S K, Naseripour M, Nemati-Anaraki L. A Critique or Violation of the Past Research Country. Rahyaft. 2018;28(70):21–36. [Google Scholar]
- 14.Solaymani-Dodaran M, Vasei M, Ghanei M. Iranian Registry of Clinical Trials: A four-year steady progress. Arch Iran Med. 2013;16(11):671–4. [PubMed] [Google Scholar]
- 15. ICH GCP. International Conference on Harmonisation of technical requirements for registration of pharmaceuticals for human use 2015. Available from: https://ichgcp.net/en/. [DOI] [PMC free article] [PubMed]
- 16.Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K. et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7. doi: 10.7326/0003-4819-158-3-201302050-00583. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Schulz KF, Altman DG, Moher D, Group tC. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8(1):18. doi: 10.1186/1741-7015-8-18. [DOI] [PMC free article] [PubMed] [Google Scholar]