Ten-Year Update on ClinicalTrials.gov Results Database

In September 2008 ClinicalTrials.gov implemented a results database in response to the Food and Drug Administration Amendments Act (FDAAA).¹ The results database consists of structured tables of summary results information without discussion or conclusions. FDAAA, its implementing regulations (42 CFR Part 11),^2,3 and several policies require results reporting to ClinicalTrials.gov to address non-publication of clinical trial results, and incomplete outcome and adverse event reporting.⁴ These have necessitated process changes for sponsors and investigators in both industry and at academic medical centers (AMCs).⁵ We previously estimated that the regulations and the National Institutes of Health (NIH) trial reporting policy⁶ would affect over half of registered trials conducted at U.S. AMCs.⁷ The scope and significance of these requirements demand that we monitor and evaluate the impact of this evolving results reporting mechanism on the clinical trials enterprise. In 2011 we characterized early experiences with nearly 2,200 posted results.⁴ A decade after launch, the results database currently contains over 36,000 results (as of May 2019). In this paper we describe the current requirements, the state of results reporting on ClinicalTrials.gov, and challenges and opportunities for further advancement.

U.S. Clinical Trial Results Reporting Requirements

Laws, Regulations, and Policies

FDAAA, its regulations, and several U.S. policies require or encourage results reporting on ClinicalTrials.gov (Table 1). These regulations, which were an important milestone in implementing FDAAA, clarified key definitions and the information to be reported, including the additional requirement to submit full protocol documents with results information for studies completed on or after January 18, 2017³. Under the regulations, the party responsible for reporting is generally the “sponsor,” which is defined as either the FDA investigational product application holder or, if none, the initiator of the trial such as a grantee institution. Sponsors may designate qualified principal investigators for meeting the requirements. We will refer to this entity or individual as the “sponsor or investigator.” Both the regulations and the NIH trial reporting policy, which follows the regulatory reporting framework, require sponsors or investigators to submit study results information within one year of the trials’ primary completion date (generally defined as final collection of data for the primary outcome measure), with delayed results submission permitted in certain situations. Registration and results information may also be submitted to ClinicalTrials.gov on an optional basis for clinical studies to which the law, regulations, or policies do not apply, but must follow the established content and quality control review procedures. Although this paper focuses on the U.S. landscape, we note the international scope of results reporting requirements, including the European Union’s Clinical Trial Regulations.⁸

Table 1.

Summary of Key U.S. Laws, Regulations, and Policies Related to ClinicalTrials.gov Results Submission

Name	General Scope	Information Submission Type and Submission Timeline	Relevant Dates	Possible Penalty for Non-Compliance
Food and Drug Administration Amendments Act (FDAAA)1 (U.S. Federal law) and associated regulations, 42 CFR Part 112 (U.S. Federal regulations)	Clinical trials of Food and Drug Administration (FDA)-regulated drug, biological, or device products (“applicable clinical trials” or ACTs) Excludes phase 1 trials and small feasibility studies	Registration: Within 21 days after enrollment of the first trial participant Summary Results: Within 12 months after the primary completion date; submission may be delayed with certification, if specific conditions are met, for up to 2 additional years or delayed for an approved period with extension request for “good cause”	FDAAA Enacted: September 27, 2007 Registration: ACTs initiated after September 27, 2007, or initiated on or before that date but still ongoing as of December 26, 2007 Summary Results: ACTs of FDA-approved, licensed, or cleared products that reached a “primary completion date” (i.e., data collection complete for the primary outcome) after December 26, 2007 Regulatory Effective Date: January 18, 2017 Registration: ACTs initiated on or after January 18, 2017 Summary Results: ACTs that reached their primary completion date on or after January 18, 2017, whether or not the product is approved, licensed, or cleared by the FDA	Civil monetary penalties of up to $10,000/day Withholding of NIH and other federal grant funds
Dissemination of National institutes of Health (NIH)-funded clinical trial information6 (NIH Policy)	NIH funded clinical trials of any type of intervention Includes phase 1 trials and small feasibility studies	Registration: Same as for FDAAA and 42 CFR 11 Summary Results: Same as for FDAAA and 42 CFR 11	Effective Date: January 18, 2017 Clinical trials supported by grants, other transactions, and contracts that submit applications on or after January 18, 2017 for trials initiated on or after January 18, 2017. NIH intramural program-supported clinical trials initiated on or after January 18, 2017	Suspension or termination of NIH grant or contract funding and consideration in future funding decisions
Submission of results of valid analyses by sex/gender and race/ethnicity9,10 (U.S. Federal law & NIH Policy)	NIH-funded, phase 3 clinical trials subject to FDAAA-associated regulations	Registration: Same as FDAAA; also specify outcomes on sex/gender and race/ethnicity Summary Results: Same as FDAAA; also specify results of “valid analyses” by sex/gender and race/ethnicity”	NIH-defined Phase III ACTs under Federal regulations supported by all new, competing grants and cooperative agreements awarded on or after December 13, 2017	Consideration in future funding decisions
International Committee of Medical Journal Editors (ICMJE) Clinical Trial Registration Policy11	Clinical trials of any type of intervention Includes phase 1 trials and small feasibility studies	Registration: On or before enrollment of the first trial participant Summary Results: If required, at time specified by funding and regulatory agencies; otherwise encouraged	Effective Dates: “Clinically-directive” clinical trials that start recruiting on or after July 1, 2005 Ongoing “clinically-directive” clinical trials before September 13, 2005 All clinical trials (expanded to include phase 1) that begin enrollment on or after July 1, 2008	Refusal by editor to publish manuscript
Department of Veterans Affairs Office of Research and Development (VA ORD)12	VA ORD-funded clinical trials	Registration: Prior to enrollment of first participant Summary Results: Within 12 months after the primary completion date	Effective Dates: Registration: Any VA-sponsored clinical trial that began enrollment on or after July 1, 2005 Summary Results: All registered clinical trials starting in 2013 and retrospectively to include trials initiated and/or actively enrolling since 2007	Condition of funding
Patient-Centered Outcomes Research Institute (PCORI)13	PCORI-funded research studies (including observational studies)	Registration: Prior to enrollment of the first trial participant Summary Results: No less than 30 days prior to the research report due date to PCORI	Policy Adoption: February 24, 2015	Condition of funding

Content of Required Results Information

Each ClinicalTrials.gov study record represents one trial with information submitted by the sponsor or investigator. The registration section, which is generally provided at trial initiation, summarizes key protocol details and other information to support enrollment and tracking the trial’s progress. Following trial completion, results information can be added to the record using required and optional data elements organized into the following scientific modules: Participant Flow, Baseline Characteristics, Outcome Measures and Statistical Analyses, Adverse Events Information, and Study Documents (protocol and statistical analysis plan)³ (Table 2).

Table 2.

ClinicalTrials.gov Results Information by Module for Studies Completed on or After January 18, 2017

Results Module Name and Brief Description	Specific Items to Report (Italics indicate optional items)
Participant flow A tabular summary of participants’ progress through the study by assignment group (similar to a CONSORT flow diagram)	Description of any significant events (e.g., wash out, run-in) after participant enrollment, but prior to assignment Number of participants who started study Number of participants who completed study Number of participants reaching study-specific milestone(s) Additional study periods or phases, with number of participants started and completed Reasons and number of participants who did not complete study
Demographic and baseline characteristics A tabular summary of collected demographic and baseline data by analysis group and overall (similar to “Table 1” in a publication)	Baseline measures: age, sex/gender, race and ethnicity (if collected), other measures assessed at baseline and used in the primary outcome measure(s) analysis For each measure: clear title/description, number of participants (or other units) analyzed and analysis population description, measure type (e.g., mean) and measure of dispersion (e.g., standard deviation), unit of measure (e.g., mg/dL or participants) Other baseline measures important for the study
Outcomes and statistical analyses A tabular summary of aggregate results data for each outcome measure by analysis group, and a tabular summary of statistical tests of significance or other parameters estimated from the outcome measure data	All prespecified primary and secondary outcome measures and the results of any scientifically appropriate tests of statistical significance For each measure: clear title/description, time frame, number of participants (or other units) analyzed and analysis population description, measure type (e.g., median) and measure of dispersion/precision (e.g., inter-quartile range), unit of measure (mg/dL or participants) For each statistical analysis: groups compared, statistical method and/or estimation parameter, calculated value, and related descriptive information Other prespecified or post-hoc outcome measures and statistical analyses
Adverse event information A tabular display of adverse events, independent of attribution and whether anticipated, by analysis group	Time frame, collection approach (systematic or non-systematic), and any relevant definitional descriptions 3 tables: All deaths collected during the study All serious adverse events grouped by organ system Other (not including serious) adverse events that exceed a frequency of 5% in any analysis group, grouped by organ system For each table: number of participants affected/at risk (numerator and denominator) overall and for each event, Source vocabulary Other (not including serious) adverse events may be reported with a threshold less than 5%
Study documents Study documents submitted in Portable Document Format Archival (PDF/A) format	Protocol Statistical analysis plan (if not part of protocol) Informed consent form
Other information Administrative and other relevant information about the study	Whether and what certain agreements exist between sponsors and principal investigators (PIs) imposing restrictions on disclosure of results information by the PI Results point of contact Limitations and caveats related to the study results

Quality Control Review Criteria and Process

Information submitted to ClinicalTrials.gov undergoes quality control review, consisting of automated validation followed by manual review by National Library of Medicine (NLM) staff. The goal of quality control review is to ensure that all required information is complete and meaningful by identifying apparent errors, deficiencies, or inconsistencies.¹⁴ Review criteria were developed based on established scientific reporting principles¹⁵ and informed by our experience. Requirements for each data element are explained and reinforced by the system’s tabular structure (e.g., a measure type of “mean” must have a unit of dispersion).¹⁶ Quality control review criteria are described in review criteria documents,¹⁴ and when possible, automated messages are provided before submission within the system. As part of the quality control review process, NLM staff apply the review criteria and provide data submitters with comments noting “major issues” that must be corrected or addressed and “advisory issues” that are provided as suggestions for improving clarity. The quality control review process ends when all noted major issues have been addressed in a subsequent submission. Common types of major issues include invalid or inconsistent units of measure (e.g., “time to myocardial infarction” as measure, with “number of participants” as unit of measure), listing of a scale without the required information about the domain or the directionality (e.g., the minimum and maximum scores), and inconsistencies within the record (e.g., number analyzed for an outcome measure greater than number started the study) (see Table S1 in the Supplementary Appendix).¹⁷

Description of Results Database Experience

We characterized results posted on ClinicalTrials.gov as of January 1, 2019, at which time over 3,300 sponsors or investigators had posted over 34,000 records with results. As of May 2019 approximately 120 new results are posted to the site each week, with an additional 128 posted records with results updated each week. Most posted study results are clinical trials, while 6% (n=1,973) are for observational studies. Table S2 shows the characteristics of posted trials with results. Over 1,500 of the posted results have protocol and statistical analysis plan documents. The median (IQR) time to posting by NLM from primary completion date was 2.0 years (1.3 to 3.8), which includes time from the final collection of data for the primary outcome measure of the trial until the summary results information were submitted to NLM, time for NLM quality control review, and time for sponsors or investigators to address quality control issues.

After the regulations became effective in January 2017 (Table 1), there was an increase in the rate of results posting for completed U.S. clinical trials from an average of 50 new results posted per week in 2016 to 86 new results posted/week in 2017 (Figure). Although assessing the adherence of results reporting with FDAAA may require non-public data (including information not required or collected until after the Final Rule effective date) and further evaluation of study specific considerations, others have used public ClinicalTrials.gov data to approximate frequency of results reporting with components of FDAAA or other metrics (e.g., any results reported in publications or ClinicalTrials.gov within two years). Reported percentages of completed trials that had results reported in ClinicalTrials.gov from these heterogeneous analyses range from 22% of relevant trials completed in 2009¹⁸ to up to 66% in May 2019.¹⁹ These efforts by various groups to highlight rates of reporting, including the naming of specific sponsors, correspond to improvements in reporting generally, and by named sponsors. For example, an updated 2018 STAT analysis documented the most improvement among sponsors previously named in a 2016 STAT analysis.^20,21

Figure.

Cumulative Numbers of U.S. Clinical Trials Registered on ClinicalTrials.gov by Primary Completion Date (Blue Line) and Registered U.S. Trials with Posted Results by the Results First Posted Date (Orange Line) with Key Events. The blue line shows the cumulative number of clinical trials with at least one facility site in the United States (“U.S. clinical trials”) registered on ClinicalTrials.gov with a primary completion date between 2007 and 2018 as of May 8, 2019. The orange line shows the cumulative number of registered U.S. clinical trials with results first posted on ClinicalTrials.gov between 2008 and 2018.

Key Issues in Meeting Results Reporting Requirements

To explore the degree to which sponsors and investigators are meeting the quality control review criteria, on October 31, 2018 we identified all trial records with required and optional results first submitted on or after May 1, 2017 and that underwent quality control review at least once by September 30, 2018. All results submissions (whether required or optional) are subject to the same review criteria, and were considered to have met quality control review criteria (“success”) for a review cycle if no major issues were identified (see Supplementary Appendix). The success rate during the first review cycle for industry records was 31% (862/2,780) and non-industry records was 17% (582/3,486). Cumulative success rates after the second review cycle increased to 77.2% (1,653/2,140) for industry records and 63.2% (1,492/2,359) for non-industry records.

In our analysis of high volume sponsors (sponsors who submitted ≥20 results during sample period), the cycle 1 success rates were heterogeneous. For example, high-volume industry sponsor cycle 1 success ranged from 16.4% to 77.1%, while non-industry sponsors ranged from 5 to 44.4% (see Figure S1 in the Supplementary Appendix). The fact that some high-volume industry sponsors achieved cycle 1 success rates of over 70%, indicates that the reporting requirements can be understood and followed appropriately. Although the median cycle 1 success rate is relatively low, 62% achieved success after two review cycles. We believe that a goal of achieving success within two cycles is reasonable, and is analogous to the need to make changes in response to editorial comments prior to journal publication (see Figure S2 in the Supplementary Appendix). We have observed that industry sponsors tend to have a centralized, well-staffed process for supporting results submission whereas non-industry sponsors tend to rely on individual investigators with minimal centralized support. A 2017 survey found that AMCs had a median of 0.08 full-time equivalent staff (3.2 hours/week) with varying levels of education dedicated to supporting registration and results submission.⁵ In addition to limited support, some sponsors have described challenges with providing structured information in a system that is unfamiliar in format and terminology.¹² NLM recognizes these challenges and has made system improvements over time; we continue to invest in evaluating and improving the system, including providing more just-in-time automated user support prior to submission. We also continue to conduct training workshops, add and improve resource materials (e.g., templates, checklists, and tutorials), and provide one-on-one assistance when needed.

Effect of Results Database on Evidence Base

The effect of the ClinicalTrials.gov registry was reviewed in a previous publication²² and sample evidence for the effect of the results database is provided in Table 3. Additionally, we conducted two analyses to evaluate the relationship between the results database and the published literature (see Supplementary Appendix).

Table 3:

Potential Benefits and Uses of ClinicalTrials.gov Results Database Entries with Examples of Evidence of Benefit

Contribution to Clinical Trial Enterprise	Sample Evidence
Evaluate consistency of reporting	Some discrepancies identified between results entries and publications23–25 Some discrepancies identified between results entries and FDA review documents26
Augment evidence base with results for unpublished trials	Half of sampled results entries do not have publications4 Results entries are the only source of information for many terminated trials27 A quarter of sampled drug trials by the same industry sponsor for the same drug and condition had results entries without publications28
Provide more complete results for published trials	Evidence that many results entries provide more complete information than publications, especially regarding serious adverse events29 Publications refer readers to results entries containing information on prespecified secondary outcome measures and adverse events information: e.g., FLAME (NCT01782326)30 and KIA (NCT01097694)31 trials
Augment evidence base to mitigate publication bias	Results entries uniquely available in ClinicalTrials.gov sometimes provide unique evidence relevant to reviews even though no study has shown that such results entries impact conclusions of systematic reviews32,33
Monitor other effects of results reporting requirements	Implementation of FDAAA has been associated with the following: Concordance between results entries and FDA review documents for sampled trials supporting new cardiovascular and diabetes drug approvals34 Higher proportion of results reporting to ClinicalTrials.gov and lower publication bias among sampled trials supporting FDA new neuropsychiatric drug approvals35
Other uses	Compared discontinuation rates and reasons across trials for pain conditions using results entries36

Relationship to Published Literature

To investigate the broad impact of ClinicalTrials.gov on public availability of trial results, we compared the timing of initial results availability between the results database and corresponding journal publications (when available). On March 1, 2018, we identified 1,902 registered trials with required or optional results first posted on ClinicalTrials.gov between April 1, 2017 and June 30, 2017. We then extracted a 20% random set of 380 records with results, manually identified corresponding publications using previously described methods,³⁷ and compared the date that results were first posted on ClinicalTrials.gov with the publication date. Results posting and publication dates within a one-month period were categorized as “simultaneous” while others were designated as published “before” or “after” posting. Relative to the time of posting on ClinicalTrials.gov, 31% (117/380) of records had a results publication before, 2% (7/380) were published simultaneously, 9% (36/380) were published after posting, and 58% (220/380) did not have any publications by the end of the follow-up period on July 15, 2018. Twenty-four months after the trial primary completion date, 41% (156/380) of trials had posted results available on ClinicalTrials.gov, and 27% (101/380) of trials had publications (see Table S2 in the Supplementary Appendix). These findings are consistent with prior analyses which showed that substantial numbers of trials have no publications 2–4 years after completion,³⁷ with ClinicalTrials.gov providing the only public representation of results for many trials.^4,28

Completeness of Results Reporting

Researchers have previously shown inadequacies in the reporting of clinical trial information in ClinicalTrials.gov and corresponding publications, including for all-cause mortality, which is critical, unambiguous information.^23,38 To improve reporting in ClinicalTrials.gov, an all-cause mortality table is now required for trials completed on or after January 18, 2017. Of 160 trials in our sample with a results publication, we identified 47 trials that included the all-cause mortality table in ClinicalTrials.gov. Of these, 26 had zero deaths reported in ClinicalTrials.gov and 21 had at least one death reported, for a total of 995 reported deaths. The associated publications reported 964 deaths (see Table S3 in the Supplementary Appendix). For those trials with zero deaths in ClinicalTrials.gov, 3.8% (1/26) of publications stated that there were no deaths and 96.2% (25/26) did not specifically mention deaths. For those trials with at least one death in ClinicalTrials.gov, 61.9% (13/21) were concordant with the publication, 14.3% (3/21) reported fewer deaths in the publication, 9.5% (2/21) reported the same overall number of deaths, but in different numbers of arms that were discordantly described, and in 14.3% (3/21) the total number of deaths were ambiguous in the publication. No trials had more deaths reported in a publication than in ClinicalTrials.gov.

Although discrepancies between two or more sources generally raises questions about which is accurate, it is unlikely that sponsors or investigators would report more deaths than actually occurred, especially because the focus on “all-cause mortality” should remove any subjectivity. While not specifically evaluated in this analysis, differences in timing of disclosing trial results may lead to some discrepancies. For example, publication prior to trial completion would only include deaths occurring up to that time while the ClinicalTrials.gov results entry would include all additional deaths observed until trial completion and thereby serve as a key source of final results for such trials.

Discussion

We previously described the mandates to report results to ClinicalTrials.gov as an experiment for addressing non-publication and incomplete reporting of clinical trial results.⁴ A decade after launch, the results database is the only publicly accessible source of results information for thousands of trials, supports complete reporting, and serves as a tool for timely dissemination of trial results that complement existing publications. ClinicalTrials.gov study records provide an informational scaffold by which information about a trial can be discovered, including access to individual participant data (IPD) sharing statements and, in some cases, links to where IPD are deposited.^39,40 This scaffolding function is facilitated when documents about a clinical trial (e.g., publications, IPD repositories, press releases, and news articles) reference the ClinicalTrials.gov unique identifier (NCT number) assigned to each registered study. The recent addition of protocol documents, statistical analysis plans, and informed consent forms further informs users about a study’s design and we encourage their use in meta-research and quality improvement efforts.⁴¹

Efforts to improve the quality of reporting need to consider the full study life cycle. For example, the presumption of both trial registration and summary results reporting had been that required information would flow directly from the trial protocol, statistical analysis plan, and the data analysis itself. Based on experience operating ClinicalTrials.gov, however, we have seen heterogeneity in the degree to which the necessary information is specified or available. Thus, we support recent efforts aimed at strengthening this early stage of the clinical research lifecycle with structured, electronic protocol development tools^42–44 as well as the use of standardized, well-specified outcome measures⁴⁵ that are consistent with scientific principles and harmonized with ClinicalTrials.gov reporting. For such broad efforts and more targeted quality efforts to take root, leadership in the clinical research community is needed to champion their value and provide resources and incentives. In parallel, as the database operators, we continue to evaluate user’s needs, ensure reporting requirements are known and understood by those involved throughout the clinical research lifecycle, to improve the submission process and reporting quality.

We also believe that the full value of the trial reporting system will emerge when various parties recognize and leverage the significant effort invested in the use of this curated, structured system for summary results reporting. For example, providing appropriate academic credit for results posting on ClinicalTrials.gov (as a complement to credit for publications) would incentivize more timely and careful entries by investigators. Additionally, results database tables can be reused in manuscripts and during the publication editorial or peer review process to ensure consistency across sources. Publications can also refer to the full set of results in ClinicalTrials.gov, while focusing on a subset of interest (e.g.,^30,31). Just as the results database supports systematic reviews, we see opportunities for those who oversee research, including funders, ethics committees, and sponsoring organizations to conduct landscape analyses prior to approving the initiation of new clinical trials as well as to monitor a field of research over time. In support of this, we aim to develop tools to optimize search strategies further and enhance the viewing and visualization of search results to support such activities. For instance, NLM recently updated the ClinicalTrials.gov application programming interface to provide support to third-parties with better targeted queries and more expansive content availability as well as changes to the main website search features.

While the results database has evolved considerably in the past decade, efforts to strengthen the culture and practice of systematic reporting must continue. We previously outlined steps that various stakeholders could take to enhance the trial reporting system.²² These actions can be described in two broad themes: (1) facilitating high quality reporting while reducing the burden and (2) modifying incentives to encourage reporting and embracing its value as part of the scientific process. As such, we endeavor to support researchers and institutions in maximizing the value of their efforts and those of the research participants as well as the overall value of the ClinicalTrials.gov results database to the scientific enterprise.