RESEARCH LETTER
The 2007 Food and Drug Administration (FDA) Amendments Act expanded requirements for ClinicalTrials.gov, a public clinical trial registry maintained by the National Library of Medicine, mandating results reporting within 12 months of trial completion for all FDA-regulated medical products. Despite concerns about its specificity, reporting of mandatory trial registration information on ClinicalTrials.gov is fairly complete, optional registration information less so;1–4 no studies have examined reporting and accuracy of trial results information. Accordingly, we compared trial information and results reported on ClinicalTrials.gov with corresponding peer-reviewed publications.
Methods
We conducted a cross-sectional analysis of clinical trials whose primary results were published between July 1, 2010 and June 30, 2011 in Medline-indexed, high-impact journals (Impact Factor ≥10; Web of Knowledge, Thomson Reuters) that were registered on ClinicalTrials.gov and reported results. For each trial, we assessed reporting of the following results information on ClinicalTrials.gov and corresponding publications and compared reported information in both sources: cohort characteristics (enrollment and completion, age/sex demographics), trial intervention, and primary and secondary efficacy endpoint(s) and results. Sources were concordant if the described endpoint, time of ascertainment, and measurement scale matched. Results were concordant (i.e., numerically equal), discordant (i.e., not numerically equal), or could not be compared (i.e., reported numerically in one, graphically in the other). For discordant primary efficacy endpoints, we determined whether the discrepancy altered study interpretation. Descriptive analyses were performed using Excel® (v14.3.1, Microsoft Corporation).
Results
We identified 96 trials reporting results on ClinicalTrials.gov that were published in 20 high-impact journals. For 70 (73%) trials, industry was the lead funder; the most common conditions studied were cardiovascular disease, diabetes, and hyperlipidemia (n=21; 23%), cancer (n=20; 21%), and infectious disease (n=19; 20%); and trials were most frequently published by New England Journal of Medicine (n=23; 24%), Lancet (n=18; 19%), and JAMA (n=11; 12%). Cohort, intervention, and efficacy endpoint information was reported for 93%–100% of trials in both sources (Table 1). However, 93 of 96 trials had at least one discordance among reported trial information or reported results.
Table 1.
Reporting and comparison of results information on ClinicalTrials.gov and in publications among trials published in a biomedical journal with impact factor ≥ 10 between July 1, 2010 and June 30, 2011 that were registered and reported results on ClinicalTrials.gov (n=96).
| Results Information | Trials Reporting, No. (%) | Comparison of Reported Information among Trials Reporting in Both Sources, No. (%) | |||
|---|---|---|---|---|---|
| ClinicalTrials.gov | Publication | Concordant | Discordant | Could Not Be Compared | |
| Cohort Characteristics | |||||
| Enrollment No. | 96 (100) | 96 (100) | 94 (98) | 2 (2) | 0 (0) |
| Completion Rate | 90 (94) | 90 (94) | 70 (78) | 20 (22) | 0 (0) |
| Sample Age Distribution | 96 (100) | 96 (100) | 56 (58) | 6 (6) | 34 (35) |
| Sample Sex Distribution | 96 (100) | 96 (100) | 85 (89) | 9 (9) | 2 (2) |
| Trial Intervention | 96 (100) | 96 (100) | 65 (68) | 15 (16) | 16 (17) |
| Efficacy Endpoints | |||||
| Primary* | 91 (95) | 91 (95) | 81 (61) | 21 (16) | 30 (23) |
| Secondary† | 89 (93) | 94 (98) | 338 (55) | 53 (9) | 228 (37) |
For primary efficacy endpoints, same 91 trials defined a total of 156 primary efficacy endpoints in either ClinicalTrials.gov or the corresponding publication, 132 (85%) of which were described in both sources.
For secondary efficacy endpoints, 96 trials defined a total of 2089 secondary efficacy endpoints in either ClinicalTrials.gov or the corresponding publication, 619 (30%) of which were described in both sources.
Among trials reporting cohort and intervention information, discordance ranged from 2–22% and was highest for completion rate and trial intervention, where different descriptions of dosages, frequencies or duration of intervention were common.
There were 91 trials defining 156 primary efficacy endpoints (5 trials defined only primary safety endpoints), 132 (85%) of which were described in both sources, 14 (9%) only on ClinicalTrials.gov, and 10 (6%) only in publications. Among 132 endpoints described in both sources, results for 30 (23%) could not be compared and 21 (16%) were discordant. The majority (n=15) of discordant results did not alter trial interpretation, although for 6 the discordance did (Table 2). Overall, 81 of 156 (52%) primary efficacy endpoints were described in both sources and reported concordant results.
Table 2.
Discordant primary efficacy endpoint results reported on ClinicalTrials.gov and in corresponding publication that altered trial interpretation (n=6).
| Trial ID | Primary Efficacy Outcome | Explanation for Why Trial Interpretation was Altered | |
|---|---|---|---|
| ClinicalTrials.gov Reported Results | Publication Reported Results | ||
| NCT00094887 | Median Hours to Resolution of Vaso-occlusive Pain Crisis (95% CI) | Time to resolution in both groups was substantially shorter on ClinicalTrials.gov than in publication. | |
| Inhaled Nitric Oxide: 61.83, 95% CI: (41.75, 78.00); Placebo: 55.16, 95% CI: (46.00, 72.00); No statistical analysis provided. | Inhaled Nitric Oxide: 73.0, 95% CI: (46.0, 91.0); Placebo: 65.5, 95% CI: (48.1, 84.0); P = 0.87 | ||
| NCT00108953 | Median Time to Progression (95% CI) | Median time to progression in both groups was substantially longer on ClinicalTrials.gov than in publication. | |
| Sorafenib + Doxorubicin: 263 days, 95% CI: (146, 384); Placebo + Doxorubicin: 147 days, 95% CI: (66, 244); P = 0.016 | Sorafenib + Doxorubicin: 6.4 months, 95% CI: (4.8, 9.2); Placebo + Doxorubicin: 2.8 months, 95% CI: (1.6, 5); P = 0.02 | ||
| NCT00177671 | Number of Participants With Recurrence of Major Depression | Percentage of participants with major depression recurrence was lower on ClinicalTrials.gov than in publication and hazard ratio on ClinicalTrials.gov is 2-fold greater. | |
| Donepezil: 19/67, 95% CI: (16, 31); Placebo: 11/63, 95% CI: (6, 18); HR=3.97, SD=2.09, 95% CI: (1.00, 4.41); P=0.05 | Donepezil: 35%; 95% CI: (24%, 46%); Placebo: 19%, 95% CI: (9%, 29%); HR=2.09, 95% CI: (1.00, 4.41), λ2=3.97; P=0.05 | ||
| NCT00281918 | Progression-Free Survival, median | Progression-Free Survival was substantially shorter in the rituximab substantially shorter in the rituximab arm reported on ClinicalTrials.gov than in publication. | |
| Fludarabine/Cyclophosphamide: 981.0 days, Range: (1, 1343); Fludarabine/Cyclophosphamide/Rituximab: 1212.0 days, Range: (1, 1372); P<0.0001 | Fludarabine/Cyclophosphamide: 32.8 months, 95% CI: (29.6, 36.0); Fludarabine/Cyclophosphamide/ Rituximab: 51.8 months, 95% CI: (46.2, 57.6); P<0.0001 | ||
| NCT00404079 | Roland Morris Disability Questionnaire, 1 year | ClinicalTrials.gov score was higher for both trial arms than in publication and statistical testing results were different in two sources. | |
| Glucosamine Sulphate: 9, SD: 4; Placebo: 9, SD: 4; Odds Ratio: 4.5 ± 4; P=0.05 | Glucosamine Sulphate: 4.8, 95% CI: (3.9, 5.6); Placebo: 5.5, 95% CI: (4.7, 6.4); P=0.50 | ||
| NCT00426751 | Number of Participants With Complete Sum ST Resolution 60 Min After Percutaneous Coronary Intervention (Intent-to-Treat Population) | Confidence interval of the adjusted difference between arms crossed zero on ClinicalTrials.gov and did not in publication, suggesting a difference in statistical testing of results between the two sources. | |
| Eptifibatide: 124/214; Abciximab: 103/196; Adjusted Difference: 6.8%, 95% CI: (−3.0%, 16.6%) | Eptifibatide: 62.6%; Abciximab: 56.3%; Adjusted Difference: 7.1%, 95% CI: (2.7%, 17.0%) | ||
Note: NCT is term used by ClinicalTrials.gov when assigning a unique clinical trial identifier; CI=Confidence Interval
There were 96 trials defining 2089 secondary efficacy endpoints, 619 (30%) of which were described in both sources, 421 (20%) only on ClinicalTrials.gov, and 1049 (50%) only in publications. Among 619 endpoints described in both sources, results for 228 (37%) could not be compared, whereas 53 (9%) were discordant. Overall, 338 of 2089 (16%) secondary efficacy endpoints were described in both sources and reported concordant results.
Discussion
Among clinical trials published in high-impact journals that reported results on ClinicalTrials.gov, nearly all had at least one discrepancy in the cohort, intervention, or results reported between the two sources, including many discordances in reported primary endpoints. Possible explanations include reporting and typographical errors, journal space limitations, and intentional dissemination of more favorable endpoints/results in publications.5
Our study was limited to a small number of trials that were not only registered and reported results, but also published in high-impact journals. However, these trials likely represent best case scenarios with respect to results reporting. Our findings raise questions about accuracy of both ClinicalTrials.gov and publications, as each source’s reported results at times disagreed with the other. Further efforts are needed to ensure accuracy of public clinical trial result reporting efforts.
Acknowledgments
Funding/support and role of the sponsor: This project was not supported by any external grants or funds. Dr. Krumholz is supported by a National Heart Lung Blood Institute Cardiovascular Outcomes Center Award (1U01HL105270-02). Dr. Ross is supported by the National Institute on Aging (K08 AG032886) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program.
Footnotes
Data access and responsibility: Ms. Becker and Dr. Ross had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Author contributions: Mss. Becker and Ben-Josef and Dr. Ross were responsible for the conception and design of this work and were responsible for acquisition of data. Ms. Becker and Dr. Ross drafted the manuscript and conducted the statistical analysis. Dr. Ross provided supervision. All authors participated in the analysis and interpretation of the data and critically revised the manuscript for important intellectual content.
Previous Presentation of Work: This study was presented at the Seventh International Congress on Peer Review and Biomedical Publication in Chicago, IL on September 9, 2013.
Conflicts of interest: Drs. Krumholz and Ross receive support from Medtronic, Inc. to develop methods of clinical trial data sharing, from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures that are used for public reporting, and from the Food and Drug Administration (FDA) to develop methods for post-market surveillance of medical devices. Dr. Krumholz reports that he chairs a scientific advisory board for UnitedHealthcare. Dr. Ross reports that he is a member of a scientific advisory board for FAIR Health, Inc.
References
- 1.Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS medicine. 2009;6(9):e1000144. doi: 10.1371/journal.pmed.1000144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Zarin DA, Tse T, Ide NC. Trial Registration at ClinicalTrials.gov between May and October 2005. The New England journal of medicine. 2005;353(26):2779–2787. doi: 10.1056/NEJMsa053234. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA: the journal of the American Medical Association. 2009;302(9):977–984. doi: 10.1001/jama.2009.1242. [DOI] [PubMed] [Google Scholar]
- 4.Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database--update and key issues. The New England journal of medicine. 2011;364(9):852–860. doi: 10.1056/NEJMsa1012065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. The New England journal of medicine. 2008;358(3):252–260. doi: 10.1056/NEJMsa065779. [DOI] [PubMed] [Google Scholar]