Abstract
Objective
To assess the time from completion to publication of randomized controlled trials (RCTs) on connective tissue diseases (CTDs), investigate the factors associated with, and explore the influence of significance of study results on time to publication (time‐lag publication bias).
Methods
We included interventional, phase 2/3, 3, or 4 RCTs on CTDs registered in Clinicaltrials.gov from 2000 to 2016, whose results had been published in a peer‐review journal less than 5 years after their completion. Main trial features, including the significance of primary outcome results, were collected. Time to publication was the time from study completion to the earliest publication date. Multivariable linear regression was used to identify factors associated with time to publication.
Results
We included 62 studies, mostly phase 3 (61%) trials on pharmacologic treatments (94%); we recruited patients with systemic lupus (55%) or systemic sclerosis (23%) and planned to enroll a median of 131 (IQR [interquartile range]: 61‐288) patients. Twenty‐two (35%) reported at least a statistically significant primary outcome. Median time to publication was 28 months (IQR: 17‐36). In a multivariable analysis, time to publication progressively improved over time (faster publication in recent years, with the average time to publication decreasing by 1.3 [95% CI: 0.3‐2.3] months per year) and was not influenced by the significance of primary outcome results, funder, impact factor of the journal, number of recruiting countries, and comparator.
Conclusion
A high proportion of CTDs‐RCTs is published beyond 2 years from completion. We did not find evidence of time‐lag publication bias, and time to publication improved over time.
INTRODUCTION
Nonpublication or there being a long time from completion to publication of randomized controlled trials (RCTs) results can harm patients by allowing inefficacious or dangerous intervention being prescribed, or useful treatment implementation being delayed. Moreover, delaying the publication of study findings can incentivize other research groups to address the same questions, with further waste of time and energy (1), and can slow the development of management guidelines and financial policy. The World Health Organization (WHO) has underscored the need to submit RCT findings for publication in a peer‐reviewed journal within 12 months of study completion, with further 12 months allowed from first submission to publication (2).
Preparation and submission of the manuscript, peer‐review process, multiple resubmissions, quality of the reporting, and study methodology all contribute to the time needed for a paper to be published (3). Additionally, it has been shown that publication and time from study completion to publication is often associated with the tendency from investigators, study sponsors, or journal editors to favor a more rapid dissemination of results that are statistically significant (4, 5) or have a relatively large treatment effect (time‐lag publication bias) (6). This practice undermines the validity of the overall knowledge on a given topic skewing evidence toward positive findings and is therefore as deleterious as nonpublication of study results.
Connective tissue diseases (CTDs) are systemic autoimmune diseases, some of which rare, characterized by a dysregulated immune response, heterogeneous clinical picture and are burdened by high disability, poor quality of life, and impaired survival. We have recently shown that one third of CTDs‐RCTs registered after 2000 never reached publication (7), but we did not estimate the extent of time‐lag publication bias, nor analyzed factors associated with delayed publication. Because most of CTDs are rare diseases, sample size will usually be small, increasing the risk of nonsignificant result, and potentially the risk of time‐lag publication bias.
Given the importance of this issue, we performed this study to evaluate the time needed for a CTDs‐RCTs to be published after completion and to investigate the relationship between publication delay and main features of trials, including the statistical significance of study results.
PATIENTS AND METHODS
Search strategy
On March 19, 2021, we searched for CTDs‐RCTs registered before this date on ClinicalTrials.gov and having started after January 1, 2000 (ClinicalTrials.gov began registering trials in 2000). We used the advanced search function on ClinicalTrials.gov to identify the diseases of interest within the “Immune system disease” and “Skin and CTDs” categories. We included studies on CTDs encompassing undifferentiated CTD, systemic sclerosis, (systemic) lupus erythematosus, Sjögren syndrome, dermatomyositis/polymyositis/inclusion body myositis, mixed CTD, and antiphospholipid syndrome. The complete list of keywords used for the search is in the online Supplementary Text 1 file. Trial entries provide details on the study population, intervention type, start and completion dates, funding source, design characteristics, and current recruitment status. The retrieved results were then downloaded and uploaded in R v4.1 statistical software (R Development Core Team) for analysis.
Protocol study eligibility criteria
We included only interventional randomized trials (phase 2/3, 3 or 4) with a completed, suspended, terminated, withdrawn, or unknown status, as defined in Clinicaltrials.gov glossary whose results had been published in a peer‐review journal. To avoid immortal time bias (8), we excluded articles published more than 5 years after their completion, and only RCTs having a completion date before the January 1, 2017, were considered (in order to allow included trials to be published from study completion to Janvier 3, 2022, the starting date of our evaluation of publication status). Therefore, all considered RCTs had the possibility to be published within the range of time to publication considered. We also excluded nonrandomized studies, RCTs with another status (ie, recruiting, not yet recruiting, etc), or not targeting a CTD, as well as duplicated studies.
Definitions, data extraction, and characterization
The following study characteristics were retrieved and analyzed by two authors (AB, BR): study design (ie, parallel‐arms, cross‐over), planned sample size, type of intervention (pharmacologic, nonpharmacologic), type of comparator (placebo, active intervention, usual care, or no intervention), and funding sources (industry, nonindustry). A study was considered being industry funded if industry was the sponsor or one of the collaborators (an organization other than the sponsor providing support for a clinical study), as reported in the glossary of Clincaltrials.gov. We classified trials as international if conducted in more than one country. The same two reviewers independently extracted from each published paper (see the following section “Evaluation of publication status and data extraction” for details about the ascertainment of publication status) and/or from the electronic page of journals where papers have been published the following data: e‐publication date; primary outcomes. Primary outcome was the outcome explicitly reported by the authors as primary in the original study or clinical trial registrations, as defined by Boutron et al; if no primary outcomes were explicitly reported, we searched in the calculation of the required sample size for it or, if lacking, we considered primary the outcome consistent with the main objectives for the RCT. A primary outcome was considered statistically significant when the P value was less than 0.05 or the level of significance specified, or a confidence interval for the effect estimate did not include no difference.
The ensemble of the studies included is listed in Supplementary Table 1. All the data collected, the code used for the analysis, and the tables and the figures, are openly available at the following repository:
https://gitlab.com/dmongin/scientific_articles/-/tree/main/time_to_publication.
Evaluation of publication status and data extraction
Two authors reviewed Clinicaltrials.gov to identify the publication status of each included trial (BR, MI). If a link for a publication was available, it was checked for consistency. In case of lack of information about publication status on Clinicaltrials.gov, we searched PubMed, Scopus, and Google Scholar for national clinical trial numbers, title, first and last authors, or intervention. For industry‐funded trials, we also searched company websites. An RCT was defined published if results were delivered in a peer‐reviewed journal in the form of a complete manuscript, even if catalogued in print or in press. We reviewed each full text to ensure a match with the respective Clinicaltrials.gov entry. Disagreement was solved by discussion with a third reviewer (DM).
Outcomes of interest
Time to publication was the interval between the primary completion date and the e‐publication date of the peer‐reviewed manuscript. Completion date was used if primary completion date was lacking. If both primary completion date and completion date were not available in Clinicaltrials.gov, we searched the information on the published manuscript. In this case, completion date was considered when the last patient finished the scheduled follow‐up visits.
Statistical analysis
For each study, we calculated the time from completion to publication (in months). Trials’ characteristics are reported as the number (percentage) for categorical variables or median (interquartile range [IQR]) for continuous variables. Continuous variables were compared with Student's t‐test, Mann–Whitney test, or Kruskal‐Wallis test, and categorical variables with chi‐square test or Fisher's exact test, as appropriate. Multiple linear regression was used to assess the association between the time to publication and the significance of study results, adjusted for variables previously shown to impact this outcome (eg the funder, the size of the trial, the international status, and the year of completion) (4, 9). The impact factor of the journal was also considered as a confounder, as it may be a proxy for the number of submission attempts previous to the actual accepted publication. Multiple imputation was performed to impute missing covariates, considering the ensemble of the covariates in the imputation model. All statistical analyses were computed with R v4.1. P less than or equal to 0.05 was considered significant. Ethical approval was not required (study not involving human participants). The data and the code used to perform the analysis and create the tables have been released openly on the following Gitlab repository: https://gitlab.com/dmongin/scientific_articles/-/tree/main/time_to_publication.
RESULTS
General features of included studies
Among the 1191 RCTs identified (flow‐chart shown in Supplementary Material), 62 studies (5 discontinued) fulfilled our eligibility criteria. Studies were mostly phase 3 (n = 38; 61%), placebo‐controlled trials testing pharmacologic treatments, most of which were conducted in North America (n = 27; 43%), and investigating systemic lupus erythematosus (n = 34; 55%). About half of RCTs were industry funded. Twenty‐two (35%) reported statistically significant results of at least one primary outcome (a trial could not be classified because of the lack of between‐group comparison). The median impact factor of the journals was 8 (IQR: 4‐12) and similar between RCTs reporting significant results and those not reporting significant results. Table 1 provides further details regarding trial characteristics. Studies with negative results were more frequently discontinued, although this result did not reach significance.
Table 1.
Characteristics of included studies
| Overall, N = 62 | Positive, n = 22 | Negative, n = 39 | P | Missing, (%) | |
|---|---|---|---|---|---|
| Time to publication (mo) | 28 (17‐36) | 25 (17‐32) | 29 (17‐37) | 0.61 | 0 |
| Year of study completion, n (%) | 0.63 | 0 | |||
| 2000‐2009 | 23 (37) | 9 (41) | 14 (36) | ||
| 2010‐2013 | 17 (27) | 4 (18) | 12 (31) | ||
| 2014‐2016 | 22 (35) | 9 (41) | 13 (33) | ||
| Disease, n (%) | 0.26 | 0 | |||
| Systemic lupus erythematosus | 34 (55) | 11 (50) | 23 (59) | ||
| Systemic sclerosis | 14 (23) | 4 (18) | 10 (26) | ||
| Sjögren syndrome | 7 (11) | 2 (9) | 4 (10) | ||
| Other | 7 (11) | 5 (23) | 2 (5) | ||
| Continent of the PI, n (%) | 0.68 | 9.7 | |||
| Asia | 13 (21) | 4 (18) | 9 (23) | ||
| Europe | 11 (18) | 5 (23) | 6 (15) | ||
| North America | 27 (43) | 8 (36) | 19 (49) | ||
| South America | 5 (8) | 2 (9) | 2 (5) | ||
| Impact factor of journal, median (IQR) | 8 (4‐12) | 8 (4‐18) | 7 (4‐11) | 0.50 | 0 |
| Industry funded, n (%) | 35 (56) | 12 (55) | 23 (59) | 0.79 | 0 |
| International, n (%) | 24 (39) | 8 (36) | 16 (41) | 1.00 | 8.1 |
| Discontinued, n (%) | 5 (8) | 0 (0) | 5 (13) | 0.15 | 0 |
| Pharmacologic treatment, n (%) | 58 (94) | 20 (91) | 37 (95) | 0.61 | 0 |
| Comparator, n (%) | 0.32 | 0 | |||
| Active | 12 (19) | 6 (27) | 6 (15) | ||
| Placebo | 50 (81) | 16 (73) | 33 (85) | ||
| Phase, n (%) | 0.28 | 0 | |||
| 2/3 | 12 (19) | 2 (9) | 10 (26) | ||
| 3 | 38 (61) | 15 (68) | 23 (59) | ||
| 4 | 12 (19) | 5 (23) | 6 (15) | ||
| Sample size, n, median (IQR) | 131 (61‐288) | 205 (49‐340) | 122 (78‐288) | 0.80 | 0 |
Abbreviations: IQR, interquartile range; PI, principal investigator.
Time to publication
Publication occurred after a median of 28 months (IQR: 17‐36) from study completion. The proportion of studies published after at least 24 months and at least 36 months from completion was of 55% (n = 34), and 27% (n = 17) in the overall sample, 74% (n = 17) and 30% (n = 7) in 2000 to 2009, respectively; 65% (n = 11) and 41% (n = 7) in 2010 to 2013, respectively; and 27% (n = 6) and 14% (n = 3) in 2014 to 2016 (P < 0.01), respectively. In a linear univariable analysis testing the evolution of the time to publication over the years, we find a significant decrease of the time to publication of 1.5 months every year (95% CI: 0.6‐2.4) (Figure 1). This effect has an R‐squared of 0.15, thus explaining a rather low portion of the variability between the RCTs. In the linear multivariable analysis (Table 2), the effect of the completion year remained similar (1.3 months/year; 95% CI: 0.3‐2.4). The significance of study results, funding, international status, type of comparator, or the impact factor of the journal did not have a significant impact on time to publication.
Figure 1.
Evolution of the time to publication for the 62 RCTs considered in the present study. Area of the points are proportional to RCTs enrolment size. RCT, randomized controlled trials; WHO, World Health Organization.
Table 2.
Factors associated with time to publication multivariate analysis
| Factor | Coefficients, mo (95% CI) | P value |
|---|---|---|
| Positive results (ref negative results) | −1.5 (−8.6 to 5.7) | 0.69 |
| Industry funded (ref no) | −0.2 (−9.5 to 9.1) | 0.97 |
| International (ref no) | 3.3 (−8.0 to 14.5) | 0.57 |
| Impact factor | 0.1 (−0.1 to 0.3) | 0.47 |
| Number of patients (per 100 patients) | −1.2 (−2.6 to 0.2) | 0.09 |
| Completion year (per year counting from 2000) | −1.3 (−2.3 to −0.3) | 0.01 |
Abbreviations: CI, confidence interval; ref, reference.
DISCUSSION
Herein, we have shown that CTDs‐RCTs get published faster in more recent years, but still, among published trials, four of ten needed more than 2 years to have their results delivered in a peer‐review journal. Our findings show no evidence of time‐lag publication bias because the time from study end to publication was not influenced by the significance of primary endpoint results. Among the factors investigated, neither the disease, the funding source, nor international status or size, but only the year of study completion, was associated with a different pace of study publication.
On average, CTDs‐RCTs findings were made publicly available in a peer‐review journal more than 2 years after the end of the trial. This means that most studies fail to fulfil the WHO statement asking for trial findings to be submitted within 12 months of study completion, with a further 12 months allowed from first submission to publication (2). This issue, together with our recent observation that one of three CTDs‐RCTs remains unpublished (7), is a barrier to the translation of the knowledge acquired into the management of daily‐life patients. In line with findings in other fields (10, 11), there was an improvement in publication time in recent years, which could be explained by both a faster publication practice (12) (easiness of paper submission, improved tools for potential reviewer selection, faster editorial decision, typesetting, and proofing) and better compliance with recommended research practice. Nevertheless, delay in publication is still observed for a nonnegligible number of trials.
We did not find any evidence of time‐lag publication bias in CTDs‐RCTs. Few studies, and with inconsistent results, have investigated this aspect to date (8). Our results contrast with reports indicating the extensive occurrence of this issue in rheumatology for rheumatoid arthritis trials (13), or in other fields (14). A study published in 2014 showed that RCTs on rheumatoid arthritis published up to 2009 took on average about 38 months to be published, with a 19‐month difference between studies with positive and negative results (13). Song et al pointed out that randomized trials on breast cancer with positive results were published after a median of 2.7 years versus 6.4 years for trials with nonstatistically significant results (15). Ioannidis reported that for RCTs concerned with HIV, the difference between positive and negative results was 3.0 versus 1.7 years. Conversely, our results indicate that if existing, the time‐lag publication is inferior to 9 months, in line with other reports that did not find any influence of the significance of study results on publication time (16) or smaller effects (11). The lack of time‐lag publication bias in studies of CTD could be due to the fact that most CTDs are rare diseases, in which nonsignificant results are more expected, and number of trials are lower than for more frequent diseases, such as rheumatoid arthritis.
The role of funders on timely publication has been the object of various studies achieving conflicting results (9, 17). The observed lack of influence of funding source on time to publication, and on the publication rate as recently shown by us in a sample of CTDs‐RCTs (7), further strengthens the hypothesis that the publication path of academic‐led studies on CTDs does not significantly differ from those supported by the industry.
Our study has several limitations. We should first acknowledge that the study completion date could have been incorrectly reported in Clinicaltrials.gov for a number of studies. Discrepancies in the end trial date between freely available protocol platforms and published manuscript can occur (16). Moreover, our study conclusions are limited to studies registered on Clinicaltrials.gov. However, studies from Europe and Americas recorded in this repository have been found to be representative of the whole sample of registered trials from these continents (18). Additionally, we did not capture other rate‐limiting steps of publication (delayed submission by the authors, peer‐review process and/or multiple resubmissions, quality of the study, etc) for which we did not have data neither in Clinicaltrials.gov nor in published papers. A study of these steps could have explained the mechanisms by which publication was delayed, but its analysis was beyond the objective of the present study. Finally, the relatively low power of our study does not allow to detect time‐lag publication bias less than 9 months.
The strength of this study is to put the accent and analyze for the first time the important aspect of timely publication in a group of rare diseases, by using data gathered over more than two decades.
In conclusion, CTDs are a group of diseases in which management is still complicated by the paucity of efficacious treatment, for which we found evidence of publication bias (7), and delayed publication in a still considerable proportion of studies. Fostering transparent and timely publication practices of study findings in the field is a mandatory step to improve the outcomes of patients suffering from these diseases and avoid waste of time and resources.
AUTHOR CONTRIBUTORS
All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be published. Drs. Mongin and Iudici had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study conception and design
Mongin, Russo, Brigante, Capderou, Courvoisier, Iudici.
Acquisition of data
Mongin, Russo, Brigante, Capderou, Courvoisier, Iudici.
Analysis and interpretation of data
Mongin, Russo, Brigante, Capderou, Courvoisier, Iudici.
Supporting information
Disclosure Form:
Supplementary Figure 1: Flow‐chart of study selection. Values are the number of trials.
Table S1: list of all included published trials in the analysis, with their respective trial registration number, pubmed index and doi if available.
ACKNOWLEDGMENTS
None.
This research was funded by the Swiss National Science Foundation grant 320030_212393.
Data collected for this study, as well as the code used for the analysis, the tables, and the figures, are openly available at the following repository: https://gitlab.com/dmongin/scientific_articles/-/tree/main/time_to_publication.
Author disclosures are available at https://onlinelibrary.wiley.com/doi/10.1002/acr2.11582.
REFERENCES
- 1. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet 2014;383:257–66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Moorthy VS, Karam G, Vannice KS, et al. Rationale for WHO's new position calling for prompt reporting and public disclosure of interventional clinical trial results. PLOS Med 2015;12:e1001819. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Van Lent M, Overbeke J, Out HJ. Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals. PLoS One 2014;9:e104846. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Hopewell S, Loudon K, Clarke MJ, et al. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009;2009:MR000006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Pedersen C, Tai S, Valley E, et al. Unpublished clinical trials of common rheumatic diseases. Rheumatology (Oxford) 2023;kead141. [DOI] [PubMed] [Google Scholar]
- 6. Sterne JA, Egger M, Smith GD. Systematic reviews in health care: investigating and dealing with publication and other biases in meta‐analysis. BMJ 2001;323:101–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Brigante A, Russo B, Mongin D, et al. Extent of and reasons for discontinuation and nonpublication of interventional trials on connective tissue diseases: an observational study. Arthritis Care Res (Hoboken) 2023;75:921–9. [DOI] [PubMed] [Google Scholar]
- 8. Iudici M, Porcher R, Riveros C. Time‐dependent biases in observational studies of comparative effectiveness research in rheumatology. A methodological review. Ann Rheum Dis 2019;78:562–9. [DOI] [PubMed] [Google Scholar]
- 9. Ross JS, Mocanu M, Lampropulos JF, et al. Time to publication among completed clinical trials. JAMA Intern Med 2013;173:825–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Meyburg J, Ries M. Publication bias in pediatric emergence delirium: a cross‐sectional analysis of ClinicalTrials.gov and ClinicalTrialsRegister.eu. BMJ Open 2020;10:e037346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Chen YP, Liu X, Lv JW, et al. Publication status of contemporary oncology randomised controlled trials worldwide. Eur J Cancer 2016;66:17–25. [DOI] [PubMed] [Google Scholar]
- 12. Sebo P. Acceptance and publication times in high‐impact general medical journals. Intern Emerg Med 2022;17:2441–6. [DOI] [PubMed] [Google Scholar]
- 13. Khan NA, Singh M, Spencer HJ, et al. Randomized controlled trials of rheumatoid arthritis registered at ClinicalTrials.gov: what gets published and when. Arthritis Rheumatol 2014;66:2664–74. [DOI] [PubMed] [Google Scholar]
- 14. Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281–6. [DOI] [PubMed] [Google Scholar]
- 15. Song SY, Koo DH, Jung SY, et al. The significance of the trial outcome was associated with publication rate and time to publication. J Clin Epidemiol 2017;84:78–84. [DOI] [PubMed] [Google Scholar]
- 16. Jefferson L, Fairhurst C, Cooper E, et al. No difference found in time to publication by statistical significance of trial results: a methodological review. JRSM Open 2016;7:2054270416649283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Welsh J, Lu Y, Dhruva SS, et al. Age of Data at the Time of Publication of Contemporary Clinical Trials. JAMA Network Open 2018;1:e181065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Viergever RF, Li K. Trends in global clinical trial registration: an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open 2015;5:e008932. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Disclosure Form:
Supplementary Figure 1: Flow‐chart of study selection. Values are the number of trials.
Table S1: list of all included published trials in the analysis, with their respective trial registration number, pubmed index and doi if available.