Abstract
Objective To estimate the frequency with which results of large randomized clinical trials registered with ClinicalTrials.gov are not available to the public.
Design Cross sectional analysis
Setting Trials with at least 500 participants that were prospectively registered with ClinicalTrials.gov and completed prior to January 2009.
Data sources PubMed, Google Scholar, and Embase were searched to identify published manuscripts containing trial results. The final literature search occurred in November 2012. Registry entries for unpublished trials were reviewed to determine whether results for these studies were available in the ClinicalTrials.gov results database.
Main outcome measures The frequency of non-publication of trial results and, among unpublished studies, the frequency with which results are unavailable in the ClinicalTrials.gov database.
Results Of 585 registered trials, 171 (29%) remained unpublished. These 171 unpublished trials had an estimated total enrollment of 299 763 study participants. The median time between study completion and the final literature search was 60 months for unpublished trials. Non-publication was more common among trials that received industry funding (150/468, 32%) than those that did not (21/117, 18%), P=0.003. Of the 171 unpublished trials, 133 (78%) had no results available in ClinicalTrials.gov.
Conclusions Among this group of large clinical trials, non-publication of results was common and the availability of results in the ClinicalTrials.gov database was limited. A substantial number of study participants were exposed to the risks of trial participation without the societal benefits that accompany the dissemination of trial results.
Introduction
Randomized clinical trials are a critical means of advancing medical knowledge. Clinical trials depend on the willingness of participants to expose themselves to the risks of randomization, blinding, and unproven interventions. The ethical justification for these risks is that society will eventually benefit from the knowledge gained from the trial.1 Because the risks involved in trial participation may be significant, and because individual trial participants often do not benefit directly from trial participation, substantial safeguards have been implemented to protect the interests of study participants both prior to and during the trial.2 These safeguards take multiple forms, including oversight by institutional review boards, the informed consent process, and data and safety monitoring boards. Until recently, the protection of the interests of study participants after trial completion has received significantly less emphasis. This began to change in 1997 with the signing of the Food and Drug Administration Modernization Act in the United States, which mandated that the US Department of Health and Human Services establish a registry of clinical trials, thereby providing permanent, public access to information on the conduct of both publicly and privately funded clinical trials.3
In 2005 the International Committee of Medical Journal Editors (ICMJE) required that prospective trials involving human participants be registered prior to the beginning of study enrollment in order to be considered for publication in member journals.4 This requirement was later incorporated into the ICMJE’s “uniform requirements for manuscripts submitted to biomedical journals,” along with the updated CONSORT 2010 statement for the reporting of randomized controlled trials.5 The prospective registration of phase II-IV clinical trials subsequently became federal law in the United States in 2007 with the passage of the Food and Drug Administration Amendments Act. This legislation also expanded the scope of ClinicalTrials.gov to include a database of trial results.6 Results from all registered studies may be posted to ClinicalTrials.gov, including studies completed prior to enactment of the Food and Drug Administration Amendments Act. In addition, reporting results is now mandatory for many trials (box).7 Failure to comply with this mandate can result in substantial penalties, including civil fines of up to $10 000 (£6200; €7400) per day and withholding of funds from investigators sponsored by the National Institutes of Health.6
Trials for which registration and results reporting are required under the Food and Drug Administration Amendments Act, United States
Applicable clinical trials
Prospective clinical study of health outcomes
Controlled interventional study in humans
Involves a drug, biological product, or device regulated by the Food and Drug Administration
Not a phase I drug trial or small feasibility device study*
Trial has at least one US site; or trial is conducted under an FDA investigational new drug application or investigational device exemption; or trial involves a drug, biological product, or device that is manufactured in the United States
Initiated after 27 September 2007 or initiated before 27 September 2007 but completed after 26 December 2007
*All pediatric post-marketing surveillance of device studies are included regardless of study size
The registration of clinical trials serves an important role in protecting the interests of study participants after trial completion. In addition to discouraging investigators from preferentially choosing to report statistically significant positive outcomes, trial registration can increase awareness of possible publication bias within the medical literature by allowing the public to compare the subset of trials with published results to the total number of trials that were registered and conducted. Publication bias can distort the apparent efficacy of interventions, which complicates the interpretation of the medical literature.8 9 10 11 12 13 The non-publication of trial data also violates an ethical obligation that investigators have towards study participants.14 15 When trial data remain unpublished, the societal benefit that may have motivated someone to enroll in a study remains unrealized. Systematic trial registration provides a tool that can help to assess both the magnitude and the causes of these problems. We estimated the frequency of non-publication of large randomized clinical trials and, for unpublished trials, determined the frequency with which trial results are unavailable in ClinicalTrials.gov.
Methods
ClinicalTrials.gov query
ClinicalTrials.gov is the largest existing clinical trial registry, with over 150 000 total registered studies from 185 countries as of August 2013. The regulatory background and logistical processes underlying trial registration in ClinicalTrials.gov have been previously described in detail.16 17 18 Individual trial entries in the ClinicalTrials.gov database include information about the trial registration process, study design, and participants’ enrollment. Information about the trial is entered into the database by study investigators or sponsors and may be updated as the trial progresses. We used the “advanced search” function of ClinicalTrials.gov to search for interventional trials with more than one arm that had been registered prior to 1 January 2009 and were closed to ongoing subject enrollment. ClinicalTrials.gov defines closed studies as “clinical studies that are no longer recruiting participants because they have enough participants already, because they are completed, or because they have been stopped for some reason.”19 We included trials only if the ClinicalTrials.gov recruitment status was listed as one of the following: “active, not recruiting,” indicating that participants were no longer being enrolled but that the study was considered to be ongoing; “completed,” indicating that the trial had “ended normally” (that is, per ClinicalTrials.gov, the “last subject, last visit” had occurred); or “terminated,” indicating that the trial had started to enroll participants but was stopped early for any reason. Each of these designations indicates that participants had been enrolled, that enrollment was no longer taking place, and that there were no plans to reopen enrollment in the future.19 We included trials listed as “active, not recruiting” in our analysis because investigators do not consistently update recruitment status after completing enrollment, and this listing persists indefinitely for many trials. Because of the possibility that publication of these trials was delayed owing to ongoing outcome assessments, we performed a planned subgroup analysis involving trials in which the recruitment status was listed as “completed.” We excluded trial protocols that were withdrawn prior to the beginning of study enrolment; non-randomized trials; duplicate registry entries; and studies where the trial registry listed a study completion date after 31 December 2008. Finally, we excluded trials that were registered with ClinicalTrials.gov more than one month after beginning enrolment, in order to ensure that the decision to register a study occurred separately from the decision to publish the study results. This decision was made because it has been shown that in spite of guidelines endorsing prospective trial registration, trials are often not registered for many months after the initiation of participant enrollment.20 In some of these cases, the registration of the trial is performed in preparation for publication and after the decision is made to publish the results. To the extent that this occurs, the inclusion of trials with delayed registration would bias our results. We limited this source of bias by focusing on trials for which registration was not delayed.
No consensus exists as to what constitutes a large randomized trial. Prior publications examining large trials have used cut-off points based on more than 200 participants or more than 1000 participants.21 22 23 We chose a cut-off point of 500 participants because we believe this defined a group of studies that were unlikely to be pilot studies, that trials of this size have a significant potential to inform medical decision making, and that randomized trials in this group are likely to appeal to journal editors as candidates for publication. Consequently, our search only included trials with an anticipated or actual enrollment of at least 500 participants. Additionally, we conducted a sensitivity analysis in which we calculated publication rates using only those trials with 1000 or more participants. At the time of trial registration the enrollment field lists the planned number of study participants; this field may be updated at the end of study recruitment to reflect the actual number of participants enrolled. Because investigators do not consistently update the enrollment field, we use the term “planned or actual enrolment” to refer to the number of participants listed in the enrollment field at the time of data collection for this study.
At the time of the initial registry search from ClinicalTrials.gov we downloaded a dataset containing the condition studied, funding source, study participants, and trial phase, along with other information.
Publication search
The search for published manuscripts occurred between April and November 2012. We considered a trial to be published when a journal had published a peer reviewed manuscript, either online or in print, which included any outcome data from the study in question. We classified a trial as unpublished if a manuscript containing trial results was not identified after different investigators performed three independent searches. One of these searches was conducted by a health sciences research librarian (LH or KC). The last search for each trial occurred in November 2012, thereby allowing an elapsed time of at least 46 months between trial completion and the final literature search.24
The search for manuscripts containing trial results began with a review of the ClinicalTrials.gov entry. ClinicalTrials.gov encourages trial investigators to update each registry profile with a link to PubMed indexed manuscripts containing results from the registered trials. Additionally, ClinicalTrials.gov uses the unique trial identification number (NCT number) assigned to each entry to automatically identify and link to relevant PubMed entries. For registry entries that did not include a link to a publication containing trial results, we searched Medline via PubMed by trial registration number, keywords, trial title, and investigator’s name for matching manuscripts. For those trials for which no published results were identified, we repeated the search strategy using Google Scholar and then Embase. We assessed matches between registry entries and publications identified by this search strategy by consensus based on the following trial characteristics: study title, trial design, interventions, primary and secondary outcomes, number of participants, recruitment dates, location, and funding sources. We excluded non-peer reviewed publications, including conference proceedings and publications that described study methods without reporting trial results. We considered trials that were only reported as part of a pooled analysis to be published if the manuscript included any unpooled results from the trial in question.
ClinicalTrials.gov provides an optional feature allowing investigators to include their name and contact information as part of the trial registry. When a contact person was listed in ClinicalTrials.gov and no publication was identified using the described search strategy, a study author (CWJ) attempted to contact that investigator once by email to inquire if results had been published. If the investigator indicated that non-publication was due to failed recruitment then we removed the trial from the analysis because our intent was to estimate the frequency of non-publication among trials for which enrollment was successfully completed.
Trial results
For those studies classified as unpublished at the conclusion of our literature search, we reviewed the ClinicalTrials.gov entry for results. This review was performed by a research assistant (LGK) with extensive prior research experience using ClinicalTrials.gov, who underwent training specific to the results database at the start of the present study. We considered only results uploaded prior to 1 November 2012 (corresponding to our final PubMed literature search), because only these results were known to be available in the absence of results in a peer reviewed publication.
When a trial is registered on ClinicalTrials.gov, investigators may list both primary and secondary outcome measures. We recorded whether results were provided for each of the primary outcomes detailed in the original registry. When investigators made the results of statistical analyses available for primary outcomes, we recorded whether results were statistically significant in favor of the interventions (P<0.05 was considered significant unless otherwise specified by study investigators). We assessed analyses identified as non-inferiority or equivalence tests against the non-inferiority margin specified by study investigators. Finally, for each study we determined whether results were provided for safety endpoints or adverse events.
Statistical analyses
We used χ2 tests to compare publication rates between trials grouped according to prespecified characteristics, including funding source, recruitment status, and completion date. P values of <0.05 were considered statistically significant. We used Kaplan-Meier methods to estimate cumulative publication percentages over time since trial completion, with all unpublished studies censored on the date of the final manuscript search. Statistical analyses were performed using SPSS version 20.0 (IBM, Armonk, NY).
Results
Overall, 585 randomized trials met the inclusion criteria (fig 1, see supplementary file). According to available registry data, these trials included 990 136 planned or actual study participants. Median planned or actual enrollment was 800 participants per trial (interquartile range 601-1239). The earliest registry entry was registered with ClinicalTrials.gov on 1 November 1999, and the median registration date was 25 April 2006. At least one publication containing trial results was identified for 410 of these studies during our literature search. An additional four published manuscripts were identified following email correspondence with study investigators, so published manuscripts were identified for 414 of the 585 included trials (71%). The 171 unpublished trials (29% of total) had a planned or actual enrollment of 299 763 participants. Table 1 shows the characteristics of both published and unpublished trials. Phase III studies accounted for 67% of included trials (394/585). Three studies exclusively enrolled patients aged over 65, all of which were published trials. Sixty six studies exclusively enrolled patients under 18; 16 of these trials (24%) were not published. Eleven per cent (n=64) of studies received funding from the National Institutes of Health or other federal sources; 80% (n=468) of studies were supported by industry. Non-publication was less common among trials supported by the National Institutes of Health or other federal sources compared with those without federal funding (17% v 31%, P=0.025). Non-publication was more common among trials that did than did not receive industry funding (32% v 18%, P=0.003).
Fig 1 Flowchart of included trials
Table 1 Characteristics of large randomized trials registered with ClinicalTrials.gov prior to 2009. Values are numbers (percentages) unless stated otherwise
| Trial characteristics | All trials (n=585) | Published trials (n=414) | Unpublished trials (n=171) |
|---|---|---|---|
| Mean (SD) No of participants | 1693 (5580) | 1668 (3217) | 1753 (9045) |
| Trial phase*: | |||
| I/II | 84 (14) | 43 (10) | 41 (24) |
| III | 394 (67) | 290 (70) | 104 (61) |
| IV | 58 (10) | 40 (10) | 18 (11) |
| Other | 49 (8) | 41 (10) | 8 (5) |
| Funding source†: | |||
| Industry | 468 (80) | 318 (77) | 150 (88) |
| NIH/US government | 64 (11) | 53 (13) | 11 (6) |
| Other | 104 (18) | 84 (20) | 20 (12) |
| Trial completion date: | |||
| Unlisted | 43 (7) | 25 (6) | 18 (11) |
| Prior to 2005 | 11 (2) | 8 (2) | 3 (2) |
| 2005-06 | 93 (16) | 71 (17) | 22 (13) |
| 2007 | 192 (33) | 137 (33) | 55 (32) |
| 2008 | 246 (42) | 173 (42) | 73 (43) |
NIH=National Institutes of Health.
*Trials described as “phase II/III” in their registry entry are categorized as phase III.
†Trials with multiple funding sources are listed within all relevant categories; totals therefore add to more than 100%.
For trials in which the ClinicalTrials.gov recruitment status was listed as “completed,” 26% (132/513) remained unpublished. Twenty nine trials were described as “active, not recruiting”; 10 of these (34%) had not been published. Finally, 67% (29/43) of trials that had been terminated remained unpublished. Among the subset of 225 trials with at least 1000 planned or actual participants, 53 studies (24%) were unpublished.
Among trials with a completion date listed in ClinicalTrials.gov (n=542) and that had been published (389/542, 72%), the median time from study completion to publication was 27 months (interquartile range 20-37 months). The observed rate at which these trials were published was highest between approximately 24 and 48 months after study completion (fig 2). The mean time between trial completion and the final manuscript search was similar for published trials (61 months, SD 13 months) and unpublished trials (60 months, SD 12 months).
Fig 2 Kaplan-Meier estimate of cumulative publication percentage by time elapsed from trial completion to publication
Of the 171 trials that remained unpublished as of 1 November 2012, 38 (22%) had results available on ClinicalTrials.gov (table 2). From the initial group of 585 trials, 414 (71%) had published results, 38 (6%) were unpublished but had results available on ClinicalTrials.gov, and 133 (23%) had no results available either in published form or in ClinicalTrials.gov. Fifteen (39%) of the 38 trials with results reported also provided statistical analyses for at least one primary outcome; the remaining 23 trials reported results without including statistical analyses. Nine of these 15 (60%) reported statistically significant results clearly favoring the intervention, whereas another four (27%) reported meeting a standard of non-inferiority. Of 38 trials with results provided, 37 (97%) included information regarding safety endpoints or adverse events.
Table 2 Characteristics of unpublished trials with and without results available in the ClinicalTrials.gov results database. Values are numbers (percentages) unless stated otherwise
| Trial characteristics | Results in ClinicalTrials.gov | |
|---|---|---|
| Available (n=38) | Not available (n=133) | |
| Mean (SD) No of participants | 1199 (1049) | 1911 (10 244) |
| Trial phase*: | ||
| I/II | 5 (13) | 36 (27) |
| III | 24 (63) | 80 (60) |
| IV | 8 (21) | 10 (8) |
| Other | 1 (3) | 7 (5) |
| Funding source†: | ||
| Industry | 38 (100) | 112 (84) |
| NIH/US government | 0 (0) | 11 (8) |
| Other | 0 (0) | 20 (15) |
| Trial completion date: | ||
| Unlisted | 0 (0) | 18 (11) |
| Prior to 2005 | 0 (0) | 3 (2) |
| 2005-06 | 3 (8) | 19 (14) |
| 2007 | 5 (13) | 50 (38) |
| 2008 | 30 (79) | 43 (32) |
NIH=National Institutes of Health.
*Trials described as “phase II/III” in their registry entry are categorized as phase III.
†Trials with multiple funding sources are listed within all relevant categories; totals therefore add to more than 100%.
Of the 150 unpublished industry funded studies, 38 (25%) had results available on ClinicalTrials.gov. None of the 11 unpublished trials funded by the National Institutes of Health or the 20 unpublished trials with other funding sources provided results in the results database. Of 23 unpublished trials completed during or after September 2008 (when the ClinicalTrials.gov results database became functional), 10 (43%) provided results. In total, 237 trials were listed as either being started after 27 September 2007 or completed after 26 December 2007, dates that correspond to the mandatory results reporting period established by the Food and Drug Administration Amendments Act. Of these, 67 (28%) were not published and 38 (16%) were neither published nor had results available in the ClinicalTrials.gov results database. Among all unpublished trials with posted results, the median delay between trial completion and results becoming available was 22 months (interquartile range 13-37 months).
Discussion
Among large randomized trials registered with ClinicalTrials.gov and closed prior to 2009, non-publication was common. We identified an estimated 250 000 trial participants for whom we were unable to find results either in the published literature or in the result database of the registry, which was approximately 26% of the total participants in the included trials. Our findings are consistent with previously observed rates of non-publication among trials representing a broad spectrum of clinical topics and funding sources.9 21 24 25 26 27 28 29 30 31 32 Our results add to existing work by showing that non-publication is an important problem even among large randomized trials, and by providing an estimate of the number of trial participants who accepted the risks of participation in these trials but for whom no results are publicly available. The present study also shows that among large unpublished trials completed prior to 2009, utilization of the results database remains limited. Even among studies completed after the results database became available in 2008, ClinicalTrials.gov contained results less than half of the time.
Prior investigations have shown evidence of publication bias against negative studies as one factor associated with non-publication.9 11 32 33 34 However, it has previously been unclear whether this bias is primarily due to limited motivation on the part of study investigators and sponsors to publish negative data or because journals are less interested in publishing the results of negative trials. By focusing our investigation on studies with at least 500 participants, we greatly limited the possibility that non-publication of trials in this cohort was due to rejection of manuscripts by journals or a lack of time or interest on the part of investigators or sponsors.35 Because of the considerable time and expense required to run these trials and the appeal of such studies to editors, non-publication of the results of large randomized trials is likely in most cases to be a conscious decision made by study investigators and sponsors. Reasons why investigators and sponsors might choose not to publish results include a discrepancy between observed and desired results or the protection of intellectual property rights.36 37 The considerable rate of non-publication even among this group of large trials suggests that decisions by investigators or sponsors are a significant cause of non-publication of clinical trials.
ClinicalTrials.gov has had the capacity to display trial results since September of 2008.38 While this is an important development, there is evidence that compliance with even basic requirements for trial registration still remains poor.20 39 40 41 42 43 44 Furthermore, use of the ClinicalTrials.gov results database is limited, even among trials for which results reporting is required by the Food and Drug Administration Amendments Act.45 Rates of compliance with the ClinicalTrials.gov results database have not been previously described among unpublished trials; our findings suggest that among these studies, utilization of the results database also remains limited. These findings have important implications for those who perform comprehensive literature searches. In over 6% of the large clinical trials we studied, results were available within the ClinicalTrials.gov results database but not within the published literature. Consequently, for authors of systematic reviews, meta-analyses, and clinical guidelines, the ClinicalTrials.gov results database is an important source of data that may otherwise be unavailable. Additionally, for another 23% of the studies in this analysis, results were not available in either the ClinicalTrials.gov database or in published form. For these trials, searching ClinicalTrials.gov may be the only way to determine that an unpublished study was performed. Even without access to results from these trials, information on the number and size of unpublished trials provides valuable information about the possibility of publication bias. Finally, all 38 of the unpublished trials we studied with results available on ClinicalTrials.gov were sponsored by industry. While the sponsors and investigators of these trials deserve credit for utilizing the results database, this represents a limited percentage of the unpublished trials in this group, meaning that currently the availability of results on ClinicalTrials.gov may also be affected by biases similar in nature to the publication and outcome reporting biases that affect the published literature.
Even when study investigators and sponsors make their results available in ClinicalTrials.gov, this should not absolve them from their obligation to publish data in a peer reviewed format. The peer review process plays a critical role in improving the quality of trial reporting. For example, peer review serves to identify potential sources of study bias, helps to ensure statistical rigor, and assists authors and editors with the task of placing study results into context.46 47 While the ability of ClinicalTrials.gov to display results is an important step forward, this capability is not a substitute for the peer review process. Further, publication in the peer reviewed literature also greatly facilitates access to the results by the scientific community and provides readers with the details needed to appropriately incorporate study results into systematic reviews and meta-analyses.48 49
Trial investigators and sponsors have an ethical obligation to study participants to publish trial results.50 This principle is implicit in the US Federal Policy for the Protection of Human Subjects, also known as the “Common Rule,” which outlines the scope and responsibilities of institutional review boards for overseeing research using human participants. The Common Rule states that institutional review board approval requires demonstration that “risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result.”51 Similarly, the Declaration of Helsinki, which was instrumental in developing the modern system of oversight by institutional review boards, also acknowledges the importance of disseminating research results, stating “Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.”52 By directing institutional review boards to assess the societal importance of resulting knowledge in addition to the possible risks and harms to individual research participants, the Common Rule provides justification for institutional review board oversight of results reporting, including trial registration and publication.53 Because the involvement of institutional review boards with clinical trial oversight begins prior to participant enrollment, these institutions are uniquely positioned to protect the rights of study participants throughout all stages of trial conduct, from study planning to reporting results. Given the persistent problem of unpublished trial results despite continued emphasis on trial registration from governmental agencies, funding organizations, and the editorial community, increased institutional review board attention toward this issue may be needed.
Limitations of this study
Several study limitations should be considered when interpreting these results. Firstly, it is possible that we failed to identify published results for some of the trials that were classified as unpublished. We attempted to limit this risk by performing multiple searches for each trial, utilizing multiple databases for each search, and by attempting to contact study investigators before classifying any trial as unpublished. Given the considerable efforts taken to identify publications, any that we missed are sufficiently well hidden that they are unlikely to be identified during routine or even systematic searches. It is also likely that results from some of the currently missing trials will be published in the future, though our data show that publication more than 46 months after trial completion is rare.
An additional limitation of the present study is that registry information contained within ClinicalTrials.gov is inconsistently updated by trial investigators. Trial recruitment status, for example, is not consistently updated by investigators as trials transition from active participant enrollment to closed enrollment with ongoing data analysis to study completion. For this reason, we included in our analysis trials listed by ClinicalTrials.gov as “active, not recruiting.” The publication of these trials may be delayed pending the completion of data gathering and analysis. However, the non-publication rate of these trials (34%) was only slightly higher than that for trials known to have been completed (26%), indicating that outcome data were available for most of these studies, and that studies listed as “active, not recruiting” often proceed to publication without a corresponding change in the registered enrollment status. Study enrollment figures are also inconsistently updated by investigators. Consequently, some registry entries still contain pretrial values for the number of planned study participants rather than updated enrollment information, despite the completion of trial enrollment and follow-up. It is often not possible to tell whether trial enrollment refers to planned or actual enrollment numbers. For this reason, our estimate of the number of people who participated in studies for which results are unavailable should be considered an approximation.
To allow adequate time for study investigators to publish their results, we report on trials that were closed prior to 1 January 2009. It is possible that publication rates are different among more recently conducted trials. Finally, this analysis addresses publication rates among large registered trials. Publication rates may be different for smaller trials and those not registered in ClinicalTrials.gov.
Conclusions
We observed that non-publication is common among large randomized clinical trials. Furthermore, the sponsors and investigators of these unpublished trials infrequently utilize the ClinicalTrials.gov results database. The lack of availability of results from these trials contributes to publication bias and also constitutes a failure to honor the ethical contract that is the basis for exposing study participants to the risks inherent in trial participation. Additional safeguards are needed to ensure timely public dissemination of trial data.
What is already known on this topic
Non-publication of clinical trial results is common among studies across a wide spectrum of both clinical topics and funding characteristics
Utilization of the ClinicalTrials.gov results database by investigators is also limited, even among trials for which results reporting is required by US law
Little is known about publication or availability of results in the ClinicalTrials.gov database for large clinical trials, the subset of trials with the greatest potential impact on clinical care
What this study adds
Among trials with at least 500 participants that were prospectively registered in ClinicalTrials.gov and completed before January 2009, 23% were neither published nor had results available in the ClinicalTrials.gov database a median of 60 months after trial completion
Contributors: CWJ and TPM conceived and designed this study. CWJ, LH, KEC, and LGK acquired the data. CWJ, MAW, and TPM analysed and interpreted the data. MAW provided statistical expertise. The initial manuscript was drafted by CWJ and LGK; all authors contributed to subsequent revisions. CWJ takes responsibility for the overall integrity of the data and the accuracy of the data analysis. CWJ is the guarantor.
Funding: MAW is supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences through Grant UL1TR000083. TPM is supported by the National Center for Research Resources through Grant KL2 TR00084. These sponsors had no role in the study design, data collection, data analysis, data interpretation, or manuscript preparation. The corresponding author had full access to all the study data and had final responsibility for the decision to submit the manuscript for publication.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: The dataset is available from the corresponding author at cjones.unc@gmail.com.
Transparency: The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Cite this as: BMJ 2013;347:f6104
Web Extra. Extra material supplied by the author
Registration and publication information for included trials
References
- 1.Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical? JAMA 2000;283:2701-11. [DOI] [PubMed] [Google Scholar]
- 2.Moreno J, Caplan AL, Wolpe PR. Updating protections for human subjects involved in research. Project on Informed Consent, Human Research Ethics Group. JAMA 1998;280:1951-8. [DOI] [PubMed] [Google Scholar]
- 3.Food and Drug Administration Modernization Act of 1997. US Public Law 105-15. (1997, Nov 21); 21 USC 301.
- 4.De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med 2004;351:1250-1. [DOI] [PubMed] [Google Scholar]
- 5.Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Food and Drug Administration Amendments Act of 2007. US Public Law 110-85. (2007, Sept 27); 21 USC 301.
- 7.ClinicalTrials.gov. FDAAA 801 requirements. National Institutes of Health.
- 8.Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 2004;363:1341-5. [DOI] [PubMed] [Google Scholar]
- 9.Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60. [DOI] [PubMed] [Google Scholar]
- 10.Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012;344:d7202. [DOI] [PubMed] [Google Scholar]
- 11.Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med 2008;5:e217; discussion e217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev 2012;(1):CD008965. [DOI] [PubMed] [Google Scholar]
- 13.Gibson LM, Brazzelli M, Thomas BM, Sandercock PA. A systematic review of clinical trials of pharmacological interventions for acute ischaemic stroke (1955-2008) that were completed, but not published in full. Trials 2010;11:43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Munro AJ. Publishing the findings of clinical research. BMJ 1993;307:1340-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Pearn J. Publication: an ethical imperative. BMJ 1995;310:1313-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Zarin DA, Ide NC, Tse T, Harlan WR, West JC, Lindberg DA. Issues in the registration of clinical trials. JAMA 2007;297:2112-20. [DOI] [PubMed] [Google Scholar]
- 17.Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database—update and key issues. N Engl J Med 2011;364:852-60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Gillen JE, Tse T, Ide NC, McCray AT. Design, implementation and management of a web-based data entry system for ClinicalTrials.gov. Stud Health Technol Inform 2004;107(Pt 2):1466-70. [PubMed] [Google Scholar]
- 19.ClinicalTrials.gov. Glossary of common site terms. National Institutes of Health.
- 20.Viergever RF, Ghersi D. The quality of registration of clinical trials. PLoS One 2011;6:e14701. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Krzyzanowska MK, Pintilie M, Tannock IF. Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA 2003;290:495-501. [DOI] [PubMed] [Google Scholar]
- 22.Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982-9. [DOI] [PubMed] [Google Scholar]
- 23.LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997;337:536-42. [DOI] [PubMed] [Google Scholar]
- 24.Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281-6. [DOI] [PubMed] [Google Scholar]
- 25.Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med 2010;153:158-66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM. Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ 2011;344:d7292. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med 2009;6:e1000144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171:735-40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Toma M, McAlister FA, Bialy L, Adams D, Vandermeer B, Armstrong PW. Transition from meeting abstract to full-length journal article for randomized controlled trials. JAMA 2006;295:1281-7. [DOI] [PubMed] [Google Scholar]
- 30.Von Elm E, Rollin A, Blumle A, Huwiler K, Witschi M, Egger M. Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss Med Wkly 2008;138:197-203. [DOI] [PubMed] [Google Scholar]
- 31.Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med 2012;9:e1001189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Med 2008;5:e191. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009:MR000006. [DOI] [PMC free article] [PubMed]
- 34.Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:iii, ix-xi, 1-193. [DOI] [PubMed] [Google Scholar]
- 35.Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867-72. [DOI] [PubMed] [Google Scholar]
- 36.Blumenthal D, Campbell EG, Anderson MS, Causino N, Louis KS. Withholding research results in academic life science. Evidence from a national survey of faculty. JAMA 1997;277:1224-8. [PubMed] [Google Scholar]
- 37.Campbell EG, Clarridge BR, Gokhale M, Birenbaum L, Hilgartner S, Holtzman NA, et al. Data withholding in academic genetics: evidence from a national survey. JAMA 2002;287:473-80. [DOI] [PubMed] [Google Scholar]
- 38.Clinicaltrials.gov. About the results database. National Institutes of Health.
- 39.Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 2009;302:977-84. [DOI] [PubMed] [Google Scholar]
- 40.Jones CW, Platts-Mills TF. Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med 2012;60:458-64 e1. [DOI] [PubMed] [Google Scholar]
- 41.Dekkers OM, Soonawala D, Vandenbroucke JP, Egger M. Reporting of noninferiority trials was incomplete in trial registries. J Clin Epidemiol 2011;64:1034-8. [DOI] [PubMed] [Google Scholar]
- 42.Sekeres M, Gold JL, Chan AW, Lexchin J, Moher D, Van Laethem ML, et al. Poor reporting of scientific leadership information in clinical trial registers. PLoS One 2008;3:e1610. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.McGee RG, Su M, Kelly PJ, Higgins GY, Craig JC, Webster AC. Trial registration and declaration of registration by authors of randomized controlled trials. Transplantation 2011;92:1094-100. [DOI] [PubMed] [Google Scholar]
- 44.Califf RM, Zarin DA, Kramer JM, Sherman RE, Aberle LH, Tasneem A. Characteristics of clinical trials registered in ClinicalTrials.gov, 2007-2010. JAMA 2012;307:1838-47. [DOI] [PubMed] [Google Scholar]
- 45.Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ 2012;344:d7373. [DOI] [PubMed] [Google Scholar]
- 46.Cobo E, Selva-O’Callagham A, Ribera JM, Cardellach F, Dominguez R, Vilardell M. Statistical reviewers improve reporting in biomedical articles: a randomized trial. PLoS One 2007;2:e332. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 1994;121:11-21. [DOI] [PubMed] [Google Scholar]
- 48.Nieminen P, Carpenter J, Rucker G, Schumacher M. The relationship between quality of research and citation frequency. BMC Med Res Methodol 2006;6:42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 2002;287:2847-50. [DOI] [PubMed] [Google Scholar]
- 50.Strech D. Normative arguments and new solutions for the unbiased registration and publication of clinical trials. J Clin Epidemiol 2012;65:276-81. [DOI] [PubMed] [Google Scholar]
- 51.Federal Policy for the Protection of Human Subjects. 45 CFR §46 (2009).
- 52.World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects. WMA General Assembly:1-5. [DOI] [PubMed]
- 53.Levin LA, Palmer JG. Institutional review boards should require clinical trial registration. Arch Intern Med 2007;167:1576-80. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Registration and publication information for included trials