The ClinicalTrials.gov Results Database — Update and Key Issues

Abstract

BACKGROUND

The ClinicalTrials.gov trial registry was expanded in 2008 to include a database for reporting summary results. We summarize the structure and contents of the results database, provide an update of relevant policies, and show how the data can be used to gain insight into the state of clinical research.

METHODS

We analyzed ClinicalTrials.gov data that were publicly available between September 2009 and September 2010.

RESULTS

As of September 27, 2010, ClinicalTrials.gov received approximately 330 new and 2000 revised registrations each week, along with 30 new and 80 revised results submissions. We characterized the 79,413 registry and 2178 results of trial records available as of September 2010. From a sample cohort of results records, 78 of 150 (52%) had associated publications within 2 years after posting. Of results records available publicly, 20% reported more than two primary outcome measures and 5% reported more than five. Of a sample of 100 registry record outcome measures, 61% lacked specificity in describing the metric used in the planned analysis. In a sample of 700 results records, the mean number of different analysis populations per study group was 2.5 (median, 1; range, 1 to 25). Of these trials, 24% reported results for 90% or less of their participants.

CONCLUSIONS

ClinicalTrials.gov provides access to study results not otherwise available to the public. Although the database allows examination of various aspects of ongoing and completed clinical trials, its ultimate usefulness depends on the research community to submit accurate, informative data.

The clinicaltrials.gov trial registry was launched more than a decade ago. Since that time, it has been evolving in response to various policy initiatives. The registry now contains information on more than 100,000 clinical studies and has emerged as a key element of many public health policy initiatives aimed at improving the clinical research enterprise. In 2008, a database for reporting summary results was added to the registry. In this article, we present an update on relevant policies, summarize the structure and contents of the results database, and show how ClinicalTrials.gov data can be used to gain insight into the state of clinical research.

KEY TRIAL-REPORTING POLICIES

Section 801 of the Food and Drug Administration Amendments Act (FDAAA)¹ expanded the legal requirements for trial reporting at ClinicalTrials.gov. It was passed into law amid concerns about ethical and scientific issues affecting the design, conduct, and reporting of clinical trials,² including the suppression and selective reporting of results based on the interests of sponsors,³ unacknowledged alterations of prespecified outcome measures,⁴ “offshoring” of human-subjects research,⁵ and failure to report relevant adverse events.⁶ Among other things, the FDAAA mandates the submission of summary results data for certain trials of drugs, biologics, and devices to ClinicalTrials.gov, whether the results are published or not,⁷ and imposes substantial penalties for noncompliance. The law’s scope is not limited to industry-sponsored trials intended to support marketing applications but includes studies not intended to inform FDA action (e.g., comparative-effectiveness trials of approved drugs or devices), regardless of sponsorship. Table 1 summarizes the scope of key reporting requirements of the FDAAA and two other policies: the registration policy of the International Committee of Medical Journal Editors⁸ and regulations being implemented by the European Medicines Agency for registration and results reporting of clinical drug trials conducted in the European Union.^9,10

Table 1.

Scope of Interventional Studies Covered by Major Reporting Policies.^*

Policy Requirements†	Registration	Results Reporting
FDAAA1	Interventional studies of drugs, biologics, or devices (whether or not approved for marketing); phases 2 through 4; at least one U.S. site or IND or IDE	Same as registration scope, but interventional studies of drugs, biologics, or devices only after FDA-approved for any use
ICMJE8	Interventional studies of any intervention type, phase, or geographic location	Not applicable
EMA9,10	Interventional studies of drugs and biologics (whether or not approved for marketing); phase 1 (pediatrics only); phases 2 through 4; at least one European Union site	Same as registration scope

^*For complete descriptions of policy requirements, see the references cited. EMA denotes European Medicines Agency, FDAAA Food and Drug Administration Amendments Act, ICMJE International Committee of Medical Journal Editors, IDE investigational device exemption, and IND investigational new drug application.

^†ClinicalTrials.gov allows the reporting of interventional and observational studies that are in conformance with any applicable human subject or ethics review regulations (or equivalent) and any applicable regulations of the national (or regional) health authority (or equivalent).

DESCRIPTION OF CLINICALTRIALS.GOV

Data in ClinicalTrials.gov are self-reported by trial sponsors or investigators by means of a Web-based system.⁷ Registration information is generally reported at trial inception. Each record contains a set of mandatory data elements that describe the study’s purpose, recruitment status, design, eligibility criteria, and locations, as well as other key protocol details.¹¹ Additional information may be provided with the use of optional data elements. Before public posting, ClinicalTrials.gov conducts a quality review of the submitted information. Each trial (regardless of the number of study sites) is represented by a single record, which is assigned a unique identifier (i.e., NCT number). Each record is expected to be corrected or updated throughout the trial’s life cycle, and all changes are tracked on a public archive site that is accessible from each record (through a “History of Changes” link). Summary results data are entered in the results database after a trial is completed or terminated (Table 2). Once posted, results records are displayed with corresponding registry (summary protocol) information for each study. Resources and links to additional information are inserted by the National Library of Medicine to enhance the overall usefulness of the database. ClinicalTrials.gov is designed to benefit the general public by expanding access to trial information, but different parts of the database are likely to be of more or less direct use to different audiences.

Table 2.

Summary Objectives and Description of Requirements for the ClinicalTrials.gov Results Database.

Objectives

Satisfy legal requirements

Promote objective, standardized reporting by capturing key trial features in the form of tabular data while minimizing potentially subjective narrative text

Facilitate “good reporting practices,” including accommodation of publishing12 and regulatory13 guidelines

Provide structured data entry to ensure complete reporting, efficient quality review, and consistent display of both required and voluntary data elements

Support detailed searches with the use of the database structure and other National Library of Medicine functions14

Description of scientific modules (in tabular format)

Participant flow: Progress of research participants through each stage of a trial according to group, including the number of participants who dropped out of the clinical trial

Baseline characteristics: Demographic and baseline data for the entire trial population and for each group

Outcome measures and statistical analyses: Aggregate results data for each primary and secondary outcome measure according to group; statistical analyses as appropriate

Adverse events: List of all serious adverse events; list of other (not including serious) adverse events in each group that exceed a frequency threshold of 5% within any group; both lists include adverse events, whether anticipated or unanticipated, and grouped by organ system

Administrative information

Key dates and contact information

Description of agreements, if any, between the sponsor and the principal investigator that would restrict dissemination of results by the principal investigator

QUALITY ASSURANCE

ClinicalTrials.gov uses automated business rules to alert data providers when required information is missing or when certain data elements are internally inconsistent. After passing automated validation, all submissions are individually reviewed before public posting to assess whether entries are complete, informative, internally consistent, and not obviously invalid; specific criteria for this assessment are described on the Web site.¹⁵ Although the review of summary protocol information is generally straightforward, that of results submissions is more complex. The goal, at a minimum, is to determine whether entries provide an accurate depiction of the study design and whether the results can be understood by an educated reader of the medical literature. Some invalid data can be detected by ClinicalTrials.gov staff; however, other data cannot be verified because ClinicalTrials.gov does not have an independent source of study data (e.g., “624 years” is clearly an invalid results entry for mean age, whereas “62.4 years” may or may not be the true mean age). Submissions are not posted on the public site until quality requirements are met; if any important problems are detected (Table 3), results records are returned to the data providers for revision. However, individual record review has inherent limitations, and posting does not guarantee that the record is fully compliant with either ClinicalTrials.gov or legal requirements.

Table 3.

ClinicalTrials.gov Quality Review Criteria.

Quality Review Criterion	Description	Example	Comment
Lack of apparent validity	Data are not plausible on the basis of information provided	Outcome measure data: mean value of 263 hours of sleep per day	Measure of mean hours per day can have values only in the range of 0 to 24, so value of 263 is not valid
Meaningless entry	Information is too vague to permit interpretation of data	Outcome measure: description states “clinical evaluation of adverse events, laboratory parameters, and imaging”; data reported as 100 and 96 participants in each group	Data are uninformative; unclear what counts of 100 and 96 participants refer to; description of outcome measure not sufficient for an understanding of the specific outcome
Data mismatch	Data are not consistent with descriptive information	Outcome measure is described as “time to disease progression”; data reported as 42 and 21 participants in each group	A time-to-event measure requires a unit of time (e.g., days or months)
Internal inconsistency	Information in one section of record conflicts with or appears to be inconsistent with information in another section	Study type is “observational,” but study title includes the word “randomized”	Randomized studies are interventional, not observational
Trial design unclear	Structure of tables and relevant group names and descriptions do not permit a reader to understand the overall trial design or do not accurately reflect the design	Results modules: participant flow and baseline characteristics entered as a two-group study with a total of 400 participants; outcomes entered for three comparison groups with 600 participants	If there is a third group, this should be reflected in the description of participant flow and baseline characteristics

RELATION TO PUBLICATION

ClinicalTrials.gov is designed to complement, not replace, journal publication. The results database provides public access to a complete set of summary results in a structured system that supports search and analysis. These data are primarily tabular in format and lack significant narrative portions. The database facilitates identification of acts of omission (e.g., incomplete reporting of outcome measures) and acts of commission (e.g., unacknowledged changes to prespecified outcome measures). Journals select research articles for publication on the basis of their target audiences, and the articles supplement reported data with peer-reviewed discussions of background, rationale, context, and implications of findings. Journal editors who abide by the standards set by the International Committee of Medical Journal Editors recognize these complementary roles and consider manuscripts for publication even when the results of a trial have already been posted on ClinicalTrials.gov.⁸

DESCRIPTIVE DATA ABOUT TRIALS IN CLINICALTRIALS.GOV

Table 4 provides summary data on registry and results records for interventional studies that were publicly available on September 27, 2010. As of this date, approximately 330 new registrations and 2000 revised registrations had been submitted each week.

Table 4.

Characteristics of Interventional Study Records Posted at ClinicalTrials.gov as of September 27, 2010.^*

Variable	Registry Records (N = 79,413)	Results Records (N = 2178)
Lead sponsor class — no. of records (no. of sponsors)
Industry	28,264 (2880)	1802 (200)
Nonindustry	51,149 (4372)	376 (196)
Recruitment status — no. of records
Recruiting	22,696	0
Active, not recruiting	12,343	74
Completed	34,549	1883
Terminated†	3,551	221
Other‡	6,274	0
Intervention type — no. of records§
Drug or biologic	56,580	1935
Medical procedure	9,636	69
Device	6,012	127
Other¶	16,771	185
Study phase — no. of records∥
0 or 1	9,359	271
1–2 or 2	20,023	393
2–3 or 3	13,822	844
4	7,890	375
Intervention model — no. of records
Parallel assignment	38,813	1321
Single group assignment	21,765	497
Crossover assignment	6,543	331
Factorial assignment	1,524	18
Missing data	10,768	11
Data monitoring committee — no. of records/total no. with responses (%)		359/1509 (24)
By study phase
0 or 1		12/221 (5)
1–2 or 2		113/314 (36)
2–3 or 3		142/520 (27)
4		45/298 (15)
Phase not available		47/156 (30)
By enrollment**
≤500 participants		291/1299 (22)
>500 participants		67/209 (32)
By lead sponsor
Industry		195/1164 (17)
Nonindustry		164/345 (48)
Outcome measures reported per trial — no.
Primary
Median		1
Interquartile range		1–2
Full range		1–71
Secondary
Median		3
Interquartile range		1–7
Full range		0–122

^*The posted registry records totaled 96,026 (which include 16,506 observational studies and 107 expanded-access records). The posted results records totaled 2324, of which 146 were observational studies. In addition, there were 320 registry records for trials of devices not previously cleared or approved by the FDA.

^†Terminated means that “recruiting or enrolling participants has halted prematurely and will not resume; participants are no longer being examined or treated” (http://prsinfo.clinicaltrials.gov/definitions.html).

^‡Other recruiting status categories are as follows: enrolling by invitation (937 records), not yet recruiting (4181), suspended (508), and withdrawn (648).

^§A study record may include more than one type of intervention.

^¶Other interventions might be a behavioral intervention or a dietary supplement.

^∥These data are limited to posted interventional study records with at least one drug or biologic intervention. Information about the study phase was missing in 5486 posted registry records and 52 posted results records.

^**One study record did not provide enrollment information.

RESULTS DATABASE

All studies registered at ClinicalTrials.gov are eligible for results submission; however, submission of results is required only for trials covered by the FDAAA (Table 2). Approximately 30 new and 80 revised records had been received each week; we estimate that full compliance with the FDAAA would lead to results submission for approximately 40% of newly registered studies, or over 100 new records per week.

The results of 3284 registered trials had been submitted by 666 data providers. Of these trials, 2324 had been posted publicly; the remainder either were undergoing quality-assurance review by ClinicalTrials.gov staff or were returned for correction.

Of 2178 clinical trials with posted results records, 20% had more than two reported primary outcome measures and 5% had more than five. For some studies, posted results include more than 100 primary and secondary outcome measures. The FDAAA requires the reporting of all primary and secondary outcome measures, and ClinicalTrials.gov does not limit the number of primary and secondary outcome measures that can be listed. Other prespecified and post hoc outcome measures may also be listed.

Of the 2324 posted results entries, 14% were linked to a PubMed citation through an indexed NCT number¹⁶; other publications that may exist could be found only through focused PubMed searches. We randomly selected a sample of 150 posted results records in September 2009 and conducted manual searches in an attempt to identify all associated publications. Using all available data, we found that 38 of these studies (25%) had an associated citation in September 2009, and 78 (52%) by November 2010. Although this percentage may continue to increase, it is unlikely that all outcomes from these studies will be published.¹⁷

SECONDARY FINDINGS FROM CLINICALTRIALS.GOV DATA

A growing number of researchers are using ClinicalTrials.gov data to examine various aspects of the clinical research enterprise. For example, recent studies evaluated registration records to analyze trends in the globalization of the clinical research enterprise,^5,18 the level of selective publication of study results,^19,20 and the degree of correspondence between registered and published outcome measures.^19-21 Scoggins and Patrick reviewed registration records to identify the types of trials for which patient-reported outcomes were most likely to be reported and the specific instruments used.²² Some authors of systematic reviews have also integrated ClinicalTrials.gov into their search strategies.²³ The integrity of trial reporting is a common theme among these studies, which generally focus on whether prespecified procedures in the study protocol (and any subsequent amendments) are appropriate and were followed. This interest has been fueled by highly publicized cases in which trial protocols were not followed, and the subsequent publication of partial results was considered to be misleading.²⁴ The requirement for registering outcome measures at trial inception is designed to address two problems: publication of only some measures and unacknowledged changes in prespecified measures.^17,21,25

We used ClinicalTrials.gov data to examine two data fields that are integral to the interpretation of study results: outcome measure and analysis population. These can be considered the “numerator” (outcome measure) and “denominator” (analysis population), respectively, of a study result (e.g., the number of events per number of participants in each group studied). The accuracy and specificity of the information within these fields partly determine their usefulness to a reader as well as their usefulness for assessing the fidelity of published reports to prespecified protocols. (Summaries of the methods used in these analyses are available in the Supplementary Appendix, available with the full text of this article at NEJM.org.)

SPECIFICATION OF OUTCOME MEASURES

ClinicalTrials.gov instructs data providers to report the specific measure and time frame for each primary and secondary outcome measure at registration, reflecting the current international standard for trial reporting.^26,27 Experience with reporting outcomes in a tabular format in the results database has emphasized the need for the description of a measure to be specific in order to sufficiently form the rows for the results table (with comparison groups as columns). In addition to time frame, a fully specified outcome measure includes information about the following: domain (e.g., anxiety), specific measurement (e.g., Hamilton Anxiety Rating Scale), specific metric used to characterize each participant’s results (e.g., change from baseline at specified time), and method of aggregating data within each group (e.g., a categorical measure such as proportion of participants with a decrease greater than or equal to 50%) (Fig. 1).

Figure 1.

An Example of the Four Levels of Specification in Reporting Outcome Measures.

We reviewed the first primary outcome measure, as initially registered, from 100 randomly selected non-phase 1 clinical trials in August 2010. Entries were assessed for whether a specific time frame was provided and were categorized according to level of specification (Fig. 1). We categorized 36% as level 1 (i.e., domain only), 25% as level 2, 26% as level 3, and 13% as level 4; of these, 72% included a specific time frame. When only a specific measurement or domain is registered, as occurred in 61% of the entries in our sample, post hoc choices of the specific metric or method of aggregation could mask the fact that multiple comparisons were conducted, potentially invalidating the reported statistical analyses and allowing for cherry-picking of results. Some argue that the method of aggregation (level 4) is part of the statistical analysis plan and may properly be specified later — after data accrual but before unblinding. The archive feature of ClinicalTrials.gov enables those viewing such records to see the originally registered outcome measure and the full timeline of changes (if relevant).

REPORTING OF RESULTS IN ANALYSIS POPULATION

The analysis population is another source of potential bias in results reporting. Substantial distortion of results can occur if all data are not accounted for or if missing data are not handled appropriately. The use of different analysis populations for different outcomes may not be noticed by many readers, but it can exert a strong effect on reported results. In a sample of 700 records (representing 1749 study groups and 5160 outcome measures), the mean number of different analysis populations per study group with at least one participant was 2.5 (median, 1; range, 1 to 25). The magnitude of the difference across groups and outcomes varied. To further explore the magnitude of these differences, we evaluated the percentage of participants who started the study and were analyzed for the first primary outcome measure in a sample of 684 eligible studies (representing 1706 groups). Approximately 31% of trials included 100% of participants in the analysis, and 24% of trials reported results for 90% or less of their participants (see the Supplementary Appendix). Determination of the appropriateness of the analysis population for any specific outcome analysis would require a detailed methodologic review of each study and would potentially involve subjective judgments.

DISCUSSION

In the past 5 years, prospective registration of clinical trials has become standard practice. Public reporting of summary results, independent of the interests of the trial sponsor, represents the next step in this international experiment in systematic disclosure of clinical trial information. It has been 2 years since the launch of the ClinicalTrials.gov results database, and people can now access summary trial data that were not previously available publicly. Researchers, policymakers, and others can now examine features and trends of the clinical research enterprise that were previously difficult to study. For example, methodologists may evaluate the appropriateness of designing trials with many primary outcome measures. Policymakers may consider how current patterns in the use of data monitoring committees might affect the quality and safety of the resulting research. Others may use the data to monitor trends in the clinical research enterprise and raise questions about the portfolio of trials relative to public health needs. The ultimate usefulness of the registry and results database will become apparent as more trial information and results are posted and as persons with different interests and needs incorporate these data into their analyses. ClinicalTrials.gov is continually adding features and linkages to facilitate the use of the data by different audiences, and other groups repackage these data for more specific audiences. For example, BreastCancerTrials.org (www.breastcancertrials.org) augments registry data to serve the breast-cancer community. The Clinical Trials Transformation Initiative is leading an effort to develop a publicly accessible research-quality data set in order to facilitate examination of the clinical research enterprise.²⁸

When one is using the data in ClinicalTrials.gov, however, certain limitations should be kept in mind. First, there are undoubtedly trials that are not registered in ClinicalTrials.gov or any other publicly accessible registry. Coverage in ClinicalTrials.gov is likely to be most complete for trials of drugs or devices that are sponsored by U.S.-based or multinational organizations (e.g., major pharmaceutical companies). Second, some records are missing information (e.g., optional data elements) or contain imprecise entries. We are not able to impose requirements beyond those of the prevailing federal law; trials registered since the passage of the FDAAA have to meet more requirements than do older trials, but investigators who use all ClinicalTrials.gov data will encounter many records with missing fields. In addition, given the demands of individual record review, some problematic entries will find their way onto the public site. Third, new registry and registration policies are being implemented in specific regions and countries around the world. The World Health Organization has established a search portal that includes data from ClinicalTrials.gov and 11 other registries, totaling more than 123,500 records as of November 23, 2010. However, overlapping scope and inadequate coordination internationally have contributed to the difficulty in determining the precise number of unique trials being conducted.

Disclosure requirements for clinical trials continue to evolve. In the United States, the FDAAA calls for the expansion of the basic results database through rulemaking “to provide more complete results information” and mandates the consideration of issues such as requiring results reporting for trials of drugs and devices that have not been approved by the FDA, the inclusion of narrative summaries, and the submission of full study protocols. In general, a guiding principle is that expansion of the registry and results database should only improve on, not reduce, the functionality and usefulness of ClinicalTrials.gov. Information about the status of the rulemaking process, including notification of the opportunity to provide comments, can be found at http://prsinfo.clinicaltrials.gov/fdaaa.html. Internationally, the European Medicines Agency is planning to make summary protocol and results information publicly available for clinical trials of approved and unapproved drugs conducted in the European Union. Efforts are under way to ensure the compatibility of the European Medicines Agency database with ClinicalTrials.gov, thus potentially minimizing reporting burdens for those conducting multinational trials and supporting seamless access to results from many parts of the world.²⁹

Despite the change in cultural expectations regarding trial disclosure and the fact that many trial sponsors and investigators are successfully meeting the requirement to submit summary results, our experience to date indicates that others are still struggling. In addition, the poor quality of some submitted entries is troubling. As Beecher observed in 1966, a “truly responsible investigator [emphasis in the original]” is essential if the rules governing clinical research are to have the intended effect.³⁰ Similarly, the usefulness of ClinicalTrials.gov ultimately depends on whether responsible investigators and sponsors make diligent efforts to submit complete, timely, accurate, and informative data about their studies.